Input Pipeline: capture, learn, distill, consolidate¶

kompend's input pipeline turns free-form notes, facts, and conversation snippets into source material that the next kompend compile folds into the wiki. Four commands work together:

Command	Writes to	What it does	LLM?
`kompend capture "<text>"`	`sources/_captured/`	LLM-extracts knowledge items (decisions, discoveries, corrections, references, notes) from free text — one file per item	✅
`kompend learn "<fact>"`	`sources/_learned/`	Stores a fact verbatim, no LLM — one file	❌
`kompend distill`	`sources/_distilled/`	Groups captured/learned notes by tag and LLM-merges each group into one coherent page	✅
`kompend consolidate`	`sources/_archive/<kind>/`	Finds near-duplicate notes and archives the redundant ones	⚙️ (embeddings optional)

All four operate only on the tool-managed input zones under sources/. Real source documents (PDFs, DOCX, standards) are never touched.

Where files land¶

sources/
├── _captured/   ← capture output (LLM-extracted items)
├── _learned/    ← learn output (verbatim facts)
├── _distilled/  ← distill output (synthesized pages)
└── _archive/    ← notes archived by distill/consolidate
    ├── capture/
    ├── learn/
    └── distilled/

Everything under _captured/, _learned/ and _distilled/ is a source: the next kompend compile parses, chunks, and resolves it like any other document. _archive/ is retired content — excluded from compile and from query retrieval, so archived notes stop contributing without being deleted. kompend organize never flattens or relocates any of these zones.

capture vs learn¶

capture runs an LLM extraction over the text. It is for substantive input — a discussion, a decision log, a paragraph worth mining. The CaptureAgent splits it into one file per knowledge item, each with a type, title, tags, and source: capture frontmatter.
learn stores the text exactly as given, with no LLM call. Use it for short facts you already know you want recorded.

Both accept --tag (repeatable) and --attr KEY=VALUE (repeatable) — tags for hierarchy, attributes for filterable key=value facts. Both are written to the file's frontmatter and flow into the parse cache and compile pipeline.

Nonsense-input refusal¶

capture refuses input that contains nothing extractable. The CaptureAgent is instructed to set refused: true (with a short reason) for gibberish, pure pleasantries, or single trivial fragments — it never invents items or echoes the input back as a "note". When the agent refuses (or returns no items), capture writes nothing and warns:

Nothing captured — the input contained no extractable knowledge items.

Only a truly empty input is rejected up front; the agent is the judge of "nonsense".

distill¶

distill reads all notes from _captured/ and _learned/, groups them by their first tag (untagged → general), and LLM-merges each group into a single page in _distilled/ (deterministic concatenation fallback when no LLM is available).

Two behaviors keep repeated runs from piling up duplicates:

Update in place — a group's page is updated (same file) on re-run instead of accumulating a new timestamped file per run.
Archive originals — by default the distilled source notes are moved to sources/_archive/<kind>/ after a successful write. Pass --no-archive to keep them in place.

kompend distill            # synthesize + archive originals
kompend distill --no-archive

consolidate¶

consolidate finds and archives redundant notes across _captured/, _learned/, and _distilled/. Near-duplicates are detected with rapidfuzz token similarity, plus an embedding tier when an embedding model is configured. For each duplicate group the newest or longest note is kept; the rest are archived to sources/_archive/<kind>/.

kompend consolidate --dry-run   # preview what would be archived
kompend consolidate             # archive duplicates (default)
kompend consolidate --no-archive  # report only, keep everything

The full flow¶

graph LR
    T[Free text / facts] --> C[kompend capture]
    T --> L[kompend learn]
    C --> CAP[_captured/]
    L --> LEARN[_learned/]
    CAP --> D[kompend distill]
    LEARN --> D
    D --> DIS[_distilled/]
    CAP --> CON[kompend consolidate]
    LEARN --> CON
    DIS --> CON
    CON --> ARC[_archive/]
    DIS --> CO[kompend compile]
    CAP --> CO
    LEARN --> CO
    CO --> W[Wiki]

Run kompend compile after distill/consolidate to fold the results into the compiled wiki. kompend compile --distill runs the distill step (with its default archiving) immediately before compiling, and prints a one-line reminder whenever _captured/ or _learned/ still hold undistilled notes.