Autri Extraction Pipeline — Design Doc & Refinement#

Status: DRAFT, red-teamed once (session 30, 2026-06-30). Owner: Dan. Leading direction = markdown-as-universal-IR (Option E), to be validated on the eval harness before build.

0. Why this doc exists#

We have hit the ingestion/extraction pipeline from five different angles in a short span — docx clause-lumping, fiction-prompt-on-contracts, scanned-PDF silent failures, the route-blind cost estimate, the prose-grain versioning gap. Each was a real bug, and each got (or nearly got) its own point-fix. The pattern is the concern: document structure is open-ended, so a heuristic-per-style approach is permanently reactive. This doc pulls the whole pipeline and its known problems into one place, states a proposed direction (markdown as a universal intermediate representation), and defines how we pick the final shape on eval-harness evidence, not by argument.

Scope: the path from upload to embedded, queryable chunks — structure detection, routing, chunking, and the chunk object's source provenance. Out of scope (noted where they intersect): retrieval ranking, the versioning flip, the Foundry surface.

Reconcile on adoption: the existing projects/autri/sub-systems/ingestion-pipeline.md (current pipeline) is superseded in part by §1 here (freshly code-grounded). The archived projects/autri/archive/sub-systems/unified-chunking-markdown.md is the prior incarnation of this exact thesis — revive it and fold it in rather than duplicate.

1. Current state — the pipeline as built#

(File-grounded; paths relative to autri/, infra in autri-infra/. main = prod.)

Stage flow#

upload (browser presign → S3 uploads bucket, key = <documentId>.<ext>)
  │  S3 ObjectCreated → SQS ingest-jobs
  ▼
PREP   (handlers/prep.ts)     one document/message
  localize → parse/render/STRUCTURE (per format) → detectUnits
  → cost-ceiling pre-flight → version-link → fan out one extract-job per unit
  ▼  SQS extract-jobs
EXTRACT (handlers/extract.ts) one UNIT/message, parallel
  prepareUnit → routeUnit → { code: deterministic chunk | llm: Haiku grouping }
  → figure-vision pass → writeExtraction → fan-in CAS → finalize claim
  ▼  SQS finalize-jobs
FINALIZE (handlers/finalize.ts) once/document
  linkSections → embed (OpenAI, content-hash skip) → confidence tier → status=ready → version flip
  ▼
embedded, queryable chunks

Three Lambda roles (one image, CMD-selected), three SQS queues each with a DLQ (maxReceiveCount=3), plus a 10-min stuck-doc janitor (autri-infra/lib/autri-ingestion-stack.ts).

Parse layer — only PDF runs structure detection#

Key fact: docx never reaches structure detection — so docClass defaults to PROSE and every docx unit routes to the LLM. (prep.ts:167-169.)

Structure detection (PDF only), routing, chunking, chunk-repair#

• structure.ts + segment.ts: probeAndSegment classifies structurability from word geometry (no LLM); recognizers (PATTERNS) = L-prefix ("C7.6.1"), bare-num ("7.6.1"), article, bible. docClass ∈ {STRUCTURED, FLAT-VERSE, WEAK, PROSE}. STRUCTURED → per-section section_id tags via structureSectioned — family-locked (one numbering scheme per doc).
• routeUnit: !sectionAware → llm ; docClass≠STRUCTURED → llm ; coverage<0.8 → llm ; else code. CODE = deterministic (deterministic-chunk.ts, no LLM). LLM = Haiku grouping, two "group adjacent paragraphs" prompts: chunk-grouping-prose-v1.md (novel scenes/beats, ~600 words — docx + prose PDF) and chunk-grouping-v3.md (thematic + cross-page + figures — PDF non-STRUCTURED).
• chunk-repair.ts + boundary-detect.ts: an LLM header-detector that re-splits oversized chunks — but code-route + appendix-pages only, AC{n} ids, size-triggered. Narrow by construction.
• Embed/finalize: OpenAI text-embedding-3-small; avgConfidence→confidence_tier; status='ready'. Cost ceiling $5/doc (route-aware as of #88).
• Where the LLM is: llm-route grouping, figure-vision, chunk-repair (appendix only). Never on clean STRUCTURED / markdown / xlsx text chunking. OpenAI = embeddings only.

2. Problem catalog#

The through-line is P9: P1–P5 are all "a document style the heuristics didn't anticipate."

3. Proposed direction — markdown as a universal intermediate representation#

Root-cause reframe (from the red-team)#

The NDA failure was not "LLM boundary detection doesn't work." Boundary detection = heading detection, and it already works two ways: deterministically for clean structured docs (the code route), and via an LLM for messy ones (the boundary-detect.ts chunk-repair detector recovered FIA appendix headers, recall 0→~100%). The NDA is structurally a contract, but it got misclassified as PROSE (docx skips structure detection) and handed to the fiction grouper. So the bug is classification + the wrong task, not boundary-detection capability. The residual unknown is generalizing the heading detector's vocabulary beyond numeric headers to legal lettering ("a.", "b.") — which the eval validates.

The thesis (Dan)#

Markdown is the only representation we chunk reliably (explicit, unambiguous heading syntax) — given good markdown; producing good markdown is the hard part. So make it the universal intermediate representation: convert every format → canonical markdown (with real #/## headings), then run the single deterministic markdown chunker. One path, every style. This collapses P1–P5 and is the strongest answer to P9. It revives the archived unified-chunking-markdown.md thesis.

The LLM owns the semantic judgment (what's a heading / what coheres); deterministic code owns the mechanics (split, tag, embed). The unified LLM step runs per-unit in extract (like today's grouping call — fan-out unchanged), with the doc-level classification computed once in prep and passed down as context.

Candidate options (to score on the harness — E is the leading bet)#

C and D are cheap baselines that A/B/E must beat to justify their cost.

4. Implementation approach for Option E (research-backed)#

4.1 Conversion — a format split, not one universal converter#

The best converters (Docling, Marker, MarkItDown, unstructured) are Python and can't run natively in the Node Lambda (PyTorch + model downloads blow the 250 MB / cold-start budget). So:

• docx → mammoth (JS) → html→md (turndown). In-Lambda, tiny, BSD. Word heading styles map straight to h1/h2 — heading fidelity for free.
• xlsx → SheetJS (already pinned) → markdown tables.
• markdown → passthrough to the existing chunker.
• PDF (incl. scanned) → a hosted OCR/layout API. This is the only leg that needs an external service. Mistral OCR 4 is the front-runner: markdown + bounding boxes + block types + confidence, $2–4/1k pages (batch/standard). Docling-as-a-sidecar-container (its docling-serve FastAPI, called over HTTP) is the all-in-one OSS alternative with the richest provenance. LlamaParse is the agentic-quality fallback.

Conversion-stack tradeoffs — and the no-LLM upside#

The pivotal question (Spike 0 in §5): how good is converter output alone? If a converter produces clean markdown with correct headings and low section bleed, the deterministic markdown chunker handles it and the LLM drops out of the chunking path entirely — a near-zero-marginal-cost, linearly-scalable pipeline. The §4.2 LLM normalization then becomes a fallback for the docs converters can't structure (likely legal numbering), not a per-doc tax. Best case: no LLM in chunking at all.

Data-confidentiality is a real axis, not just cost: Brehob's corpus includes signed NDAs and contracts, so "send every page to a third-party OCR API" is a privacy decision — the self-hosted Docling path trades ops cost for keeping confidential documents in-house.

4.2 The thin LLM heading-normalization pass#

No converter reliably promotes legal/contract numbering ("3. Indemnification.", "a.") into a heading hierarchy — they emit it as ordered-list text. That promotion is a semantic call, so it's a small LLM pass after the mechanical converter: feed it the converter's markdown + provenance (cheaper + deterministic substrate than raw pages), and have it lift numbering into #/## and repair heading levels. This is the per-unit LLM step of §3.

4.3 Source provenance — resolving the bbox/source-legibility tension (was H3)#

Re-rendering to markdown does not have to forfeit source-legibility. Production systems (Azure Document Intelligence is the reference) converge on a chunk object with two provenance primitives:

• Logical position — a char span into the canonical content (spans: [{offset, length}]).
• Visual position — a list of {page_no, bbox} regions (always plural, to survive line/column/page spanning).

The provenance-emitting converters already produce this per element: Docling prov = {page_no, bbox, charspan}; Marker chunks = bbox + polygon + page + section_hierarchy; Mistral OCR 4 = bbox + block types. Carry those onto the chunk object and the markdown chunk maps back to the original PDF.

For any LLM step (the §4.2 normalization), the universal rule: never trust an LLM to emit char offsets (architecturally unreliable). Have it emit the verbatim quote, then deterministically fuzzy-align the quote back to source (the LangExtract / Anthropic-Citations / Hypothesis pattern), graded EXACT / FUZZY / NOT_FOUND, dropping/flagging anything below threshold. Anchor with surrounding context (prefix+suffix), not the bare quote, to avoid the first-occurrence trap.

Proposed chunk-object shape:

{
  "chunk_id": "...", "doc_id": "...", "text": "...",
  "headings": ["3. Indemnification"],            // section trail
  "spans":   [{ "offset": 806, "length": 7 }],   // logical: char range into canonical content
  "regions": [{ "page_no": 3, "bbox": {"l":108,"t":405,"r":504,"b":331,"coord_origin":"BOTTOMLEFT"} }],
  "source_elements": ["#/texts/50"], "prev_chunk_id": "...", "next_chunk_id": "..."
}

This is a real refactor (the chunk object becomes markdown-centric with a region list), but it's a known, shippable pattern — not research.

4.4 How the LLM step fits the architecture (was H4)#

Per-unit, in extract — same fan-out as today's grouping call. Doc-level classification ("is this a contract / what scheme") is computed once in prep and passed down. No change to prep→extract→finalize.

5. Eval plan — how we decide on evidence#

We pick the final shape on harness evidence. The first experiment is the one that could collapse the whole design.

Spike 0 — converter-quality eval (the pivotal experiment, run first)#

Before committing the LLM-normalization design, measure the converters directly against the harness. Method: take representative corpus docs across every category (structured / prose / verse / the new legal), run each through the candidate converters (Docling, Marker, Mistral OCR 4, mammoth-for-docx) → markdown → the existing deterministic markdown chunker → score. Compare three pipelines per doc: converter-only (no LLM) vs converter + thin LLM normalization vs the current pipeline (baseline).

Measure: boundary F1 (did the converter's headings yield the right section starts?), section bleed (chunks lumping multiple sections — the NDA failure mode), retrieval recall, and cost/doc. The decision it unlocks: if converter-only clears the bar on most categories, the LLM exits the chunking path (the big cost/scale win); if it clears it only on some, the LLM becomes a targeted fallback. Run it first — its outcome reshapes everything downstream. (Depends on Spike P11-a below.)

Prerequisite stories#

• P11-a — verify the harness can score non-PDF fixtures (was H2). The boundary eval gates on a words layer (fixtures/cache); docx/markdown have a paragraph layer and may be skipped. Confirm/extend the harness to score paragraph-layer (docx/md) fixtures before authoring legal gold — otherwise we author gold the gate ignores.
• P11-b — add ≥2 legal/contract fixtures (n=1 discipline): the Mutual NDA + the QuoteAI services agreement, each with *.boundaries.json (clause-anchored) + *.queries.json (clause-level retrieval). New legal/contract category in manifest.json.

Metrics per option#

• Boundary F1 (eval/boundary.ts, gated, LLM-free) — clause/section starts found?
• Retrieval recall@k / MRR (eval/retrieval.ts, section-anchored) — "what's the Term?" returns the Term clause alone?
• Chunk-coherence LLM-judge — but calibrate it first (was M3): validate against ~20–30 human-labeled chunk examples before trusting it as a metric. Then it's the automated P10 proxy.
• Cost/doc + latency (was M1) — measure cost/doc under markdown-IR, comparing sync vs. Anthropic batch, and re-derive the $5 ceiling (and what #88's route-aware estimator becomes) for an LLM-default world.
• Significance (significance.ts, McNemar) — guard the n=1 / noise-floor trap on every delta.

Regression guard#

Every option must hold (within noise) on the existing FIA / novel / verse corpora — a single unified prompt must not regress the working chunk-grouping-v3.md pericope handling. We do not trade the regs corpus for the contracts corpus.

6. Migration (was M2)#

Adopting markdown-IR means re-converting + re-chunking + re-embedding every existing corpus (FIA, Bible, novels, contracts) — chunks change, so embeddings change. Open: big-bang vs. rolling re-ingest; the compute/time/cost budget; and the versioning implications (a full re-chunk = new chunk rows = a new version per doc). Scope this before adoption — it's a real cost line in the decision, not an afterthought.

7. Open decisions / red-team targets (remaining)#

1. How much LLM does chunking actually need? Converter-only (no LLM) vs. converter + thin LLM normalization vs. an LLM-heavy approach (A/B) — decided by Spike 0. The best case (converter-only) removes the LLM from the chunking path entirely; markdown-IR (E) is the leading substrate either way.
2. Conversion stack: hosted Mistral OCR 4 (cheap, bbox-native, vendor) vs. Docling sidecar (OSS, richest provenance, an extra container to operate) vs. format-split (mammoth/SheetJS in-Lambda + hosted PDF). Decide on cost + provenance + ops evidence.
3. Quality gate (R3 / P10): always-on coherence re-check vs. triggered — and can it emit a per-doc "chunking confidence" surfaced like confidence_tier?
4. Provider portability: "Anthropic locked" is pending the held Bedrock spike — design the normalization prompt to port to Haiku-on-Bedrock.
5. How much of chunk-repair / the geometry segmenter gets retired once everything routes through markdown-IR.

Appendix — research shortlist & sources (session 30 fan-out)#

• Converters: Mistral OCR 4 (hosted, markdown+bbox+blocks, $2–4/1k) · Docling (OSS, best provenance: prov={page_no,bbox,charspan}, Python → sidecar) · Marker (bbox+polygon+section_hierarchy; GPL/OpenRAIL revenue gate) · mammoth (JS, docx→html, in-Lambda) · SheetJS (xlsx) · MarkItDown (no provenance — weaker fit) · LlamaParse (agentic, pricier).
• Source-mapping: never LLM-emit offsets → emit verbatim quote + deterministic fuzzy-align (LangExtract difflib/RapidFuzz, graded EXACT/FUZZY/NOT_FOUND; Hypothesis prefix+suffix anchoring; Anthropic Citations char_location). PDF char boxes available via PyMuPDF/pdfplumber if needed. Chunk-object two-primitive design = Azure Document Intelligence (spans + boundingRegions[]).
• Anthropic locked (current) → batch ingestion is the cost lever. Prior chunking A/B lesson: evaluate the real path. chunk-repair + #88 + scanned-PDF loud-fail are in flight this session (all held, not deployed).