Cache and State Directory¶

kompend stores derived data, pipeline state, and indexes inside a .kompend/ directory. This document describes what goes where, why, and the design rules for new caches.

Directory layout¶

The .kompend/ directory lives at base_dir/.kompend/, where base_dir defaults to output_dir.parent. For the default layout (output_dir = wiki/, base_dir = .), this means .kompend/ is at the project root — not inside wiki/.

.  (base_dir)
├── .kompend/
│   ├── kompend.db                 # Single non-vector store (SQLite) — see below
│   ├── config.toml             # Project-level config (installed by kompend init)
│   ├── rumdl.toml              # Markdown formatter config
│   ├── log.md                  # Rendered run/lint report (on demand, `kompend log`)
│   ├── cache/                  # Delete-safe parse artifacts
│   │   ├── images/             # Extracted images referenced by the wiki
│   │   ├── encoding/           # Normalized UTF-8 copies
│   │   ├── downloaded.json     # URL download manifest
│   │   └── gen_checkpoint/     # Per-batch generation checkpoints (resumable)
│   ├── vectors/                # Zvec vector index (RAG) — regenerable
│   ├── kompend.db                 # Single SQLite store; `vector_manifest` row = zvec id sidecar
│   ├── memory.db               # Zvec collection dir for semantic query memory (NOT SQLite)
│   ├── dedup/                  # Content-level dedup fingerprints
│   │   └── fingerprints.json   # Per-document fingerprints (embedding + minhash)
│   ├── review/                 # Review queue entries (markdown + _state.json)
│   ├── agents/                 # User-defined conversational agent TOMLs
│   └── last_organize.json      # Last `kompend organize` report
├── wiki/                       # Compiled wiki output
│   ├── index.md                # Auto-generated wiki index
│   ├── log.md                  # Compilation log (regenerated by kompend log)
│   └── ...                     # Compiled wiki pages
├── sources/
│   ├── _captured/              # Captured notes (kompend capture / kompend learn)
│   └── _distilled/             # Synthesized wiki pages (kompend distill)

`kompend.db` — the single non-vector store¶

All non-vector relational/JSON state lives in one SQLite file, .kompend/kompend.db (opened through kompend.core.store.open_store). It holds:

tier/compile state, parsed-document cache, and the LLM chunk/resolve/extract/generate/summary caches;
the compiled-wiki snapshot, extracted assets, bibliographic references, entity-kind registry, claim-verification report, and private-page index;
per-node provenance, memory lifecycle state, the query-memory log (with an FTS5 index), and the attachment manifest (with tombstones);
the lint-run history and the CLI command audit trail.

The table inventory is the single source of truth in kompend.core.store.schema — never inline a table name elsewhere. The store is a clean cut: the flat .kompend/*.json / *.jsonl artifacts from earlier layouts are no longer read or written. kompend doctor (via kompend.core.store.legacy.obsolete_artifacts) flags any leftovers so they can be deleted after a compile --force.

What is stored and why¶

Path	Contents	Typical size	Scope	Created by
`kompend.db`	Single non-vector store: tier state, caches, compiled wiki, assets, references, provenance, memory, attachments, lint/run history (see above)	varies	Project-local	every pipeline stage / command
`config.toml`	Project-level kompend config	< 10 KB	Project-local	`kompend init` (installed from config file)
`rumdl.toml`	Markdown formatter config	< 10 KB	Project-local	materialized on first use
`log.md`	Rendered run + lint trend report	< 100 KB	Project-local	`kompend log --write`
`cache/images/`	Extracted images referenced by the wiki	varies	Project-local	Parser extraction tier
`cache/encoding/`	Normalized UTF-8 copies for parsers needing a file path	varies	Project-local	`normalize_to_utf8_path`
`cache/downloaded.json`	URL download manifest (source URLs + content hashes)	< 10 KB	Project-local	`HTTPDownloader` / `WebCrawler`
`cache/gen_checkpoint/`	Per-batch generation checkpoints (resumable on crash)	< 10 MB	Project-local	`TwoPhaseCompiler`
`vectors/`	Zvec embedded vector database for semantic search	1 – 100 MB	Project-local	`ZvecWikiStore`
`kompend.db` (`vector_manifest` row)	Indexed-node id set + embed-text version for staleness checks	< 10 KB	Project-local	`ZvecWikiStore`
`memory.db`	Zvec collection dir for semantic query memory (rebuildable from the `memory_log` table)	< 100 KB	Project-local	`kompend memory` commands
`dedup/fingerprints.json`	Per-document dedup fingerprints (embedding + minhash)	< 1 MB	Project-local	`DedupOrchestrator` during ingest
`review/*.md`	Wiki nodes pending human review (`.approved.md` / `.rejected.md` / `.locked`)	< 100 KB	Project-local	`ReviewQueue`
`review/_state.json`	Review-queue decisions (approve/reject, reasons, locks)	< 10 KB	Project-local	`ReviewQueue`
`agents/*.toml`	User-defined conversational agent definitions	< 10 KB	Project-local	`kompend agents install`
`last_organize.json`	Last `kompend organize` report (categories, moves, link updates)	< 100 KB	Project-local	`kompend organize`

Project-local vs. global: design rule¶

All kompend caches are project-local and should stay that way.

A cache belongs in .kompend/ (at the project root, alongside sources/ and wiki/) when:

It contains derived data specific to a particular source corpus
It is not reusable across different projects
Rebuilding it requires the original source files

Global caches (e.g. ~/.cache/kompend/) would only make sense for:

Downloaded ML models — but HuggingFace, sentence-transformers, and Docling already manage their own caches under ~/.cache/huggingface/ and ~/.cache/docling/
Parser binaries — pandoc, external tools — managed by the OS package manager or pipx

kompend should not re-invent model or tool caching. The .kompend/ directory is for kompend-derived data only.

Cache lifecycle¶

Creation¶

.kompend/ is created on demand during kompend compile. It is not versioned in git — the directory should be added to .gitignore (it is by default in the kompend project template).

Invalidation¶

Cache	When it becomes stale	What to do
parsed-document + LLM caches (`kompend.db`)	Source file modified or `--no-cache` flag	Delete `.kompend/kompend.db` or re-run compile
`cache/gen_checkpoint/`	After successful generation completes	Auto-deleted on success; persists on crash for resume
tier state (`kompend.db`)	Any source file changes	`--force` flag re-processes everything
run audit (`kompend.db`)	Never (append-only audit trail)	Soft-capped to 1000 records / keep 500
lint history (`kompend.db`)	Never (append-only trend data)	Manual cleanup if needed
compiled-wiki snapshot (`kompend.db`)	After every successful compile	Overwritten automatically
memory state + log (`kompend.db`)	After every successful compile (when `memory_enabled`)	Overwritten/appended automatically
`vectors/`	After compile changes node content	Re-run `--index`
`dedup/fingerprints.json`	New documents added or model changed	Re-runs fingerprinting for new docs only
`review/`	After review actions	Entries removed as they are approved/rejected

Cleanup¶

# Nuke all cached state
rm -rf .kompend/

# Re-parse everything from scratch
kompend compile --no-cache --force

Adding a new cache¶

When adding a new subsystem that needs persistent state:

Non-vector relational state goes in kompend.db — add a table in kompend.core.store.schema (never inline a table name elsewhere). Bulk, delete-safe artifacts (binaries, checkpoints, human-editable files) go under <base_dir>/.kompend/<subsystem>/.
Document it in the tables above (this file).
Make it resilient to missing data — if the store or cache directory is deleted, kompend should rebuild from source files, not crash.
Never store source files or user data — only derived/compiled data.
Use PipelineConfig.paths (a VaultPaths instance) to resolve the path — e.g. config.paths.kompend_db_file or config.paths.cache_dir / "<subsystem>". For modules that only have base_dir, use VaultPaths(base_dir). Never construct .kompend/ paths by hand from output_dir.

External caches (not managed by kompend)¶

These are managed by upstream libraries and listed here for reference:

Path	What	Managed by
`~/.cache/huggingface/`	sentence-transformers models (≈1 GB each)	HuggingFace Hub
`~/.cache/docling/`	Docling model files (≈500 MB)	Docling
`~/.cache/pip/`	Python packages	pip

kompend does not control, configure, or clean these. Users manage them through their respective tools (e.g. huggingface-cli delete-model, pip cache purge).

Config file location¶

Config files are separate from the cache directory:

File	Purpose	Tracked in git
`kompend.toml`	Project config	No (gitignored)
`kompend.toml.example`	Reference config	Yes
`~/.config/kompend/config.toml`	User global config	No

See kompend.core.config AGENTS.md for the full config precedence system.