KB retrieval and token budget — tiered fetch, labels, edge search

decisiondraft owner: philip · updated 2026-11-14

relates: edge-via-seam · agent-skill-packaging · build-process

KB retrieval and token budget

As the KB grows past dozens of entries, agents must fetch only what a task needs, and token cost per task must stay bounded. Proposed contract (draft — awaiting confirmation): tiered retrieval, labels for cross-cutting filters, and edge hybrid search behind the seam.

Problem it solves

A large info base without retrieval discipline makes every task pay for the whole KB; agents drown in irrelevant context and costs scale with repo size, not task size.

The tiered contract (progressive disclosure)

tier	artifact	cost	when
T0	`machine/index.dsl` — one line per entry (`id	type	title	path	status`)	~30 tokens/entry	always first, every task
T1	`machine/<type>.dsl` + one-line "use when" per entry	small	once the type is known
T2	full markdown of selected entries only	per-entry	before instantiating or citing
T3	`relates`/`uses` neighbours, `machine/code/<repo>.dsl`	on demand	only when the task touches them

Rules: cite kb:<id> instead of re-reading; never load the whole repo; index first is already AGENTS.md law — this formalizes the tiers.

Filtering mechanisms

Type projections (exists) — directory = type; task→type mapping via SPEC §9 seams (design reads patterns/frameworks/decisions; copy reads copy/; code context reads machine/code/).
Labels/tags (proposed SPEC v1.12) — optional tags: [a, b] inline list in frontmatter; compile emits machine/by-tag/<tag>.dsl. Tags are cross-cutting (domain:aml, stage:design, surface:ui), never a substitute for type.
Relates graph (exists) — expand to neighbours only when the task needs them.
Edge hybrid search — Cloudflare AI Search (GA, MCP endpoint per instance) as the production retrieval adapter: keyword+semantic, top-k with relevance thresholds, returns chunks not whole entries. Behind the seam: local runner = keyword scan over machine/index.dsl; production = AI Search. Git stays canonical; search is a projection, never truth.

Token reduction (published methods, applied)

Just-in-time retrieval / progressive disclosure (Anthropic, "Effective context engineering for AI agents", 2025-09): lightweight identifiers first, load data at runtime via tools — exactly the T0→T3 contract.
Prompt/prefix caching: stable ordering — system → project context (KB index) → conversation; cached re-reads bill at the cached rate. Citing kb:<id> keeps prefixes stable across turns.
Sub-agent fan-out (Anthropic): broad questions go to an explorer sub-agent that returns summaries + kb: ids; the main agent cites without loading everything.
Retrieval budgeting (MLflow context-window guidance): hybrid search (vector + BM25), re-rank, start top-k low, raise only when retention tests show missing evidence; relevance thresholds drop weak chunks instead of filling the budget.
Extractive compression / recursive summarization: entry summaries live in machine/<type>.dsl so T1 answers "is this relevant" without T2 cost.
Phase-splitting / context resets between delivery stages: each stage of the factory pipeline starts fresh, carrying knowledge forward as kb:<id> citations, not raw transcripts.

Decision (proposed)

Adopt the T0→T3 tiered contract as the agent consumption contract (SPEC §9 update).
Add optional tags to frontmatter + machine/by-tag/ projections in SPEC v1.12, with a governance rule: tags are few, controlled, and reviewed like types.
Wire AI Search as the edge retrieval adapter behind the seam when the KB site gains search; local runner stays keyword-over-index.
Measure tokens per task in learner-retro runs — retrieval efficiency becomes a retro metric.

Anti-patterns

Loading whole entries "to be safe" — T2 only for entries you will cite or instantiate.
Tags as a junk drawer — ungoverned labels recreate the bloat the tiers prevent.
Retrieval without thresholds — filling the context budget with weak matches.
Summaries that drop citations — compression must preserve kb:<id> traceability.
Edge search becoming canonical — git + compile gates remain truth; search is a cache/projection.

Sources

Anthropic, "Effective context engineering for AI agents" (2025-09-29): JIT retrieval, progressive disclosure, sub-agent architectures
Claude Code docs, "How Claude Code uses prompt caching" (prefix caching, cached token rates)
MLflow, "Context Window Management: Why Retrieval Beats Bigger Context" (2026-10-14): hybrid search, re-ranking, top-k, thresholds, compression
SPEC.md §9 (integration seams), AGENTS.md (consuming knowledge)