Ingestion & Retrieval Pipeline — Sub-system Design Doc#

The pipeline that turns raw documents into retrievable knowledge, plus the retrieval operators that serve it. This doc adds one new capability — structured-attribute retrieval — and the coupled pipeline changes it requires, positioned against the current (deployed + local) state. It is the architecture layer above ingestion-foundation (roadmap item 1 of brehob-launch) and the generic substrate consumed by QuoteAI (spec-match) and dev-memory (recency). → North Star B1 / B3.

Status: DRAFT — authored 2026-06-15 out of the Gate-0 red-team, which converged on "Autri is missing exactly one retrieval capability." OD1/OD2/OD4 locked 2026-06-15; review folded in same day (lookup-vs-attribute, tables, versioning split, graph-RAG deferral, OD9–OD12). The remaining Open Decisions are the triage targets.

Risks & Constraints#

Overview#

Where this sub-system stands today, how it got here, and what it is.

Current Status#

The Story#

The Gate-0 red-team set out to measure whether Autri's generic chunk-embed pipeline could ingest the Brehob quote corpus at retrievable fidelity. Grounding the doc's claims against the real repos flipped the premise. Three findings: (1) QuoteAI already extracts quotes into typed records (products with hp_range/cfm_range/psi_range NUMRANGE + lubrication) and retrieves with a hybrid filter-then-rank — the "numeric-filter primitive" the roadmap parked as a maybe-build already exists in the vertical. (2) The spreadsheet→formal-quote transformation is already built (Template-C parser + drafter); its per-line-item retrieval is 3/4 native to Autri (verbatim-description and analog lookups are pure vector/lookup), and the one gap is the numeric spec-match — a validation step, not the spine. (3) Mapping QuoteAI's six retrieval tools onto Autri's three operators, five are already covered; only search_equipment's structured filter is missing.

So the question stopped being "can the generic pipeline do this" and became "Autri is missing exactly one retrieval capability — structured-attribute filtering — and it is generic." A second consumer confirmed it isn't Brehob-specific: dev-memory dogfooding needs the same primitive to weight session decisions by recency (most-recent supersedes; history stays visible). The capability is therefore core substrate, and Gate-0 collapses from "spike before build" into "the eval-acceptance gate on this build."

What Is This Sub-system?#

The ingestion + retrieval pipeline owns the path from a raw document to retrievable knowledge: convert → route → chunk → extract attributes → embed → index, and the operators that query the result. It exists as its own layer because every consumer (QuoteAI, dev-memory, every future vertical) plugs into the same retrieval contract; a change here ripples to all of them. This doc's addition — structured-attribute retrieval — is the fourth retrieval mode alongside vector / FTS / lookup, and the ingestion step that feeds it.

The Gap (verified against code)#

Autri exposes exactly three retrieval operators and no structured-attribute filter:

• vectorSearch — cosine over embeddings; filters only on knowledge_base_id, documentIds, chunkTypes.
• ftsSearch — Postgres FTS; same filter surface.
• lookupSection — exact section_id match (e.g. C7.6.2); not an attribute query.

Chunks carry text + chunk_type + section_id + embedding + bbox — zero typed numeric/categorical attributes, and ingest extracts none. The concrete test "find equipment where 10 ≤ hp ≤ 20 AND lubrication = oilless AND cfm ≥ 90, ranked by relevance" cannot be answered today — no typed columns, no filter operator. (hybrid_search exists as a type name only; unimplemented.)

Target: Structured-Attribute Retrieval (two halves)#

Half 1 — Generate typed attributes during ingest. At chunk time, attach typed attributes to each chunk: numeric (hp, cfm, psi), categorical (lubrication, manufacturer), temporal (date). This is the "LLM does semantics, code does mechanics" principle made concrete: read the value out of messy text, store it typed and queryable.

Half 2 — A filter-then-rank operator. A fourth retrieval mode: WHERE on typed attributes (range / equality / set / date), composed with ORDER BY embedding distance (or FTS rank). Hard constraints prune first; semantic similarity ranks the survivors — exactly QuoteAI's search_equipment shape, generalized.

Both are required: extraction without the operator is inert data; the operator without extraction has nothing to filter.

Deterministic-First, LLM-Fallback Extraction#

The split is not "structured docs → regex, prose → LLM." It's how machine-regular the source is — mirroring the existing chunking philosophy (code is the default; the LLM fires only where the document doesn't declare its own structure):

• Machine-regular sources → deterministic/code extraction (exact, free, no LLM). A small family, by how the value is addressed:
  • Cell-coordinate — the value sits at a known (row, col). Template-C already does this: CFM = col 9, HP = col 11; map columns → declared attributes. Strongest case; needs the template layout.
  • Table-grid parse — a (converted) markdown pipe-table's header row gives column names, cells give values; parse the grid, map columns → attributes. (See Tables below.)
  • Labeled-pattern (regex) — a value following a regular label ("SYSTEM CAPACITY: 31.2 SCFM" → CAPACITY:\s*([\d.]+)). The brittle one; only "deterministic" where the label format is truly regular, else it's the LLM's job.
• Variable prose → LLM extraction. The 8,600+ .doc quotes span 20 years and many templates ("PLEX HORSEPOWER" vs "HP" vs "Horsepower"; drifting units/layout). The LLM's job is "get the HP however it's labeled," piggybacked on the existing Haiku grouping call (no second pass).

Note: "structured doc → no LLM" was about chunking. For attributes, even a structured doc can need the LLM if its values live in free prose — the split is value-regularity, not doc-type. We don't guess it; the harness measures deterministic coverage and the LLM fills the remainder, scored on extraction accuracy.

Tables — the densest attribute source#

A table is where conversion, chunking, and attribute-extraction meet — and it's the richest attribute source in the corpus (a pricing table's CFM/PSI/HP/LIST columns are the typed attributes). The path:

1. Convert to markdown (the conversion stage, ingestion-foundation S1) → the table in a parseable pipe-table form. Necessary, not sufficient.
2. Grid-parse (the deterministic "table-grid" path above) → columns become typed attributes.
3. Row-level granularity → each row becomes its own chunk carrying the column-attributes, so "the 45hp Powerex line item and its price" is retrievable — displayed as the whole table (retrieval ≠ display granularity).

Today the chunker leaves converted tables as chunk_type:'text' (no row granularity, no cell-attributes); closing that is the table-handling work in ingestion-foundation S2, and it's the precondition for line-item-level structured retrieval.

The Autri ↔ QuoteAI Boundary#

Attributes live on chunks (generic, filterable) — this is Autri substrate. Entity rollups (QuoteAI's products / quotes typed tables, used for cross-corpus aggregation like "the cheapest oilless 15hp system we've quoted") stay in the vertical. The trade: per-chunk attributes answer "find content matching these specs" cleanly; entity-level aggregation/dedup is weaker per-chunk and belongs to the vertical that needs it. This line keeps the substrate generic and the vertical thin.

Recency & Supersession (the dev-memory consumer)#

The dev-memory dogfood needs more than date > X: when a decision flips across sessions, the current session must see both the standing decision and the history, with the most-recent treated as authoritative. Resolved 2026-06-15 (ingestion-foundation S5 red-team): this is served by the chunk-level date attribute (a first-class typed attribute) + recency as a rank-boost — not by supersession. History stays retrievable, just ranked lower.

Hard supersession is a separate, document-level concept: superseded_at lives on documents, not chunks (an earlier draft of this section said "on chunks" — corrected), and retrieval already honors it via includeSuperseded?. It's for a replaced document version, not a down-ranked older decision. So the operator needs no new chunk-level temporal model — recency-rank for "newest wins, history visible", document-level supersession for replaced versions. Designing it with Brehob's spec-match and dev-memory's recency in view is what keeps it generic rather than quote-shaped.

Schema Lifecycle#

Defining the schema and evolving it are the same flow — the system proposes, the user curates — at different times. JSONB storage (OD1) is what makes this cheap.

1. Bootstrap (KB creation). The user doesn't hand-author a typed schema cold. On the first batch of docs, the extractor proposes a candidate schema (a townhouse-purchase KB → purchase_price, closing_date, address, loan_amount, interest_rate); the user curates — accept / rename / prune — in the UI. System-controlled KBs (dev-memory) may declare directly. The user curates; they don't author from scratch.
2. Steady state. Docs with values for already-declared attributes are just extracted. Because extraction targets the declared schema, synonyms collapse ("sq ft" → declared square_footage); only genuinely new concepts escape.
3. Expansion. A doc introducing a new attribute is flagged as a candidate (rides the existing call — "also saw property_tax_rate = 1.2%"); the schema does not silently expand. The system surfaces "seen property_tax_rate in N docs — promote to a filterable field?" Noise stays out until the user opts in.
4. Promote-then-backfill. On promote: (a) the field is added to the KB schema; (b) a typed partial-index is created (online, no table change); (c) a targeted backfill re-extracts just that attribute across existing docs (bounded, single-attribute, rides Batch + incremental re-ingestion) and fills the JSONB. No full re-ingestion — and the docs were fully retrievable by vector/FTS throughout; promotion only adds filterability.
5. Versioning (two kinds, don't conflate). This work needs only a light KB-schema version stamp — so "which docs predate this attribute" is answerable for promote-then-backfill. Full document-content versioning (a doc re-uploaded/edited) is a separate, larger capability — the Incremental Re-Ingestion epic — that this work does not block on (it comes after, and composes: re-ingesting a changed doc only re-extracts changed units' attributes).

The JSONB payoff: schema evolution is a metadata + index + bounded-backfill operation, not a table migration + full re-extract.

Architecture#

The internal shape of the pipeline and where it interfaces outward.

Architecture Diagram#

 raw doc ─▶ convert ─▶ route(docClass probe) ─┬─ STRUCTURED ─▶ deterministic chunk ─┐
                                              │                                      │
                                              └─ PROSE ──────▶ Haiku grouping ───────┤
                                                                                     ▼
                                              ┌──────── attribute extraction ────────┐
                                              │  deterministic (cell / grid / pattern) ─┐
                                              │  LLM fallback (rides Haiku call) ───────┴─▶ typed attrs
                                              └──────────────────────────────────────────┘
                                                                                     ▼
                                                  chunk { text, type, section_id, attrs, embedding }

 ─ retrieval ─────────────────────────────────────────────────────────────────────────
   CONTENT SEARCH      query ─▶ rank by ┬─ vector (semantic)
                                        └─ fts (keyword)

   STRUCTURED ACCESS
     • attribute filter (NEW):  WHERE attrs (range/eq/in/date) ─prunes─▶ rank by [ vector | fts | recency ]
                                (the filter is an optional pre-stage; omit it = today's behavior — see OD12)
     • lookup:  by section_id — hierarchy traversal + document order, no ranking

System Boundary#

Inside: conversion, routing, chunking, attribute extraction, embedding, the chunk schema (incl. typed attributes), and the four retrieval operators. Outside: the eval harness (grades this layer, doesn't own it — pipeline-eval-harness); the vertical's entity rollups + drafter (QuoteAI); deploy/tenancy (enterprise-deploy). The one requirement this layer imposes outward: consumers declare the typed attributes they care about (the per-KB attribute schema).

Key Interfaces#

Retrieval Operators — Lookup vs Structured-Attribute#

The four operators answer four different questions:

The boundary that matters: lookup keys on the document's intrinsic identity (the address the author assigned; exact, ordered, hierarchy-aware — it walks the sections tree), while structured-attribute keys on facts we derived (ranges, sets, comparisons + ranking). Given structure vs derived facts; retrieve by address vs retrieve by property. They even shine on different doc types — lookup on highly-structured docs with real addresses (regs, contracts), attribute-filter on fact-laden docs you query by value (quotes, pricing sheets).

Could the attribute operator absorb lookup? Partly — section_id is itself a categorical attribute, so exact-match lookup is expressible as WHERE section_id = X. But lookup uniquely adds (1) hierarchy traversal (ask for C7, get its whole subtree — needs the section tree, not flat equality) and (2) document-order contiguous return (the section as written, no relevance ranking). So lookup isn't redundant — it's the structural-address pattern.

The section tree is also the graph-shaped thing: a containment tree (one-relationship graph), and QuoteAI's entity FKs are a second small graph. A full knowledge graph / graph RAG is deferred (see Decisions Log) — it earns its keep only on multi-hop relationship queries we don't have yet; when one appears, it belongs in the entity-rollup layer, not the chunk substrate.

→ Open: OD11 (make section_id a built-in hierarchical attribute, unifying lookup under the filter?) and OD12 (is structured-attribute a distinct operator or a composable pre-filter prepended to vector/fts/recency?).

Eval Integration (the harness is the gate)#

This capability is a first-class harness citizen, which is why we can build it before fully measuring it. Two new axes on the existing per-index scorecard:

1. Extraction accuracy — precision/recall of extracted typed values vs a hand-labeled attribute gold (did we get hp=20, cfm=31.2, lubrication=oilless, date=2019-07-19?). Cheap to score; no retrieval needed.
2. Filter-then-rank recall — add a structured-filter index to the per-index gold (alongside vector/FTS/lookup), with attribute-filter queries.

Discipline carries over from the harness: baseline-first (delta vs current), per-type floors, significance over point estimates (small-n noise floor stated). An Eval run gates the merge.

Open Decisions (red-team targets)#

Status 2026-06-15: all decisions closed. OD1/OD2/OD4 locked; OD3, OD5–OD8 settled; OD9–OD12 locked on review. Ready for the ingestion-foundation epic refinement + red/blue-team.

Related Epics#

Cross-Cutting Concerns#

Decisions Log#

Known Issues / Tech Debt#

Sub-system docs define architectural boundaries. The test: remove this layer and multiple unrelated features break. Structured-attribute retrieval is consumed by QuoteAI, dev-memory, and every future vertical — remove it and all three lose hard-constraint + recency retrieval. Update this doc when the retrieval contract or the chunk schema changes.