Docling Integration — Extraction Pipeline Redesign, Build & Deploy#

Status: DRAFT, RED-TEAMED session 32 (2026-07-01) via a 3-lens grounding fan-out (provenance, infra, migration/routing/cost). Owner: Dan. The integration design following the sub-systems/extraction-pipeline-redesign decision (Docling chosen, Spike 0). Red-team resolutions folded in; ready for blue-team scope-cut.

Red-team resolutions (session 32)#

Three fresh agents grounded every claim against the real repo + live prod (not the doc's own assertions). Locked decisions:

• RT-1 — Promotion = STRUCTURE-NATIVE. Promote + chunk on Docling's element structure (each element keeps page_no / bbox / charspan), NOT on a markdown string. Forced by provenance: the string path (docling-convert.py → export_to_markdown(), then string-in/out numbering-promote.ts) discards every ProvenanceItem before a chunk exists — so the session-32 chunker verdict was measured on a provenance-free substrate and proves nothing about citation highlighting. This reverses the earlier "convert-only in Python" split — accepted as the cost of citations.
• RT-2 — Routing = ALL DOCS THROUGH DOCLING, then detect + fall back. Doc-class (STRUCTURED / FLAT-VERSE / PROSE) is an output of parsing (probeAndSegment, segment.ts:270), so it cannot be known before Docling parses; the only pre-parse signal is file extension. Route everything through Docling, then use the already-built structural-sanity invariants (evaluateStructuralSanityKb) + a fail-closed reading-order check to detect verse / prose / mis-order and fall back (verse geometry, prose LLM grouper) or flag. Do NOT pre-classify.
• RT-3 — Hosting = SPIKE the numbers. A prepackaged binary can't beat cold start — torch + model weights load into RAM per cold start regardless of packaging. Live prod proves the risk: the current torch-FREE prep worker already pins at Lambda's hard 10s init cap (Init Duration: 9999ms); a torch+OCR container inits in tens of seconds → always blows the cap → re-inits in-handler → 15-min DLQ risk on scans. Spike real init-time / peak-memory / per-doc-time + the lighter-Docling (ONNX + skip-DL-for-structured-formats) delta, then choose: on-demand Lambda (true scale-to-zero, ~$0 idle — viable ONLY if lighter-Docling inits fast) vs Fargate (warm; SQS-polling service; Spot for the migration burst). Steady-state cost is a wash ($35/mo warm, either); the differentiators are the migration burst (Fargate Spot wins) and whether scale-to-zero is achievable.

Factual corrections the red-team forced (in the sections below): the two-locator provenance model already ships; the cost thesis is backwards (coverage/quality play, not COGS win); the migration surface is tiny + pre-Brehob; cold-start + ENOSPC (multi-GB image) are the real infra risks.

0. Why this doc exists#

sub-systems/extraction-pipeline-redesign chose, on eval evidence, to convert every format to markdown with Docling, promote implied numbering to headings deterministically, then chunk with the existing deterministic chunker (no LLM in the common path). This doc takes that from "chosen" to "deployed": hosting, the language split, source provenance / citation highlighting, and corpus migration. Scope: upload → embedded, retrievable, source-linked chunks. Out of scope: retrieval-router tuning, the Foundry surface, the QuoteAI vertical.

1. What the eval settled (session 32)#

Built + ran a self-contained retrieval A/B harness (ingestion/eval/docling-retrieval.ts + ingestion/numbering-promote.ts + spike/docling-eval/docling-{convert,chunk}.py).

• Chunker — deterministic, confirmed. Head-to-head vs HybridChunker on every corpus: legal (tie, 94-100% recall@10), regs (Technical tie 85%, Competition deterministic +4pts 92%), scanned (tie). No corpus favored HybridChunker → take the deterministic chunker (free, no runtime Python-chunker dep, already wired to our chunk object).
• Numbering-promotion is the load-bearing step, not the chunker — 39-79% → 93-100% boundary recall on legal/regs (numbering-promote.ts, unit-pinned to the Spike-0 prototype).
• Scanned/OCR works (Docling + OCR extracted a signed contract the current pipeline drops → P6; retrieval matched the clean docx, 94%) — but via ocrmac (macOS-only); prod needs a Linux engine (§4). Mac-only preview.
• Numbers are vector-only — the prod retriever is a 3-index router; the current pipeline hits 100% on Competition with it. 85-92% is vector-alone, understated.
• Two corpora Docling does NOT win (keep specialized paths): verse (Docling mis-orders dense two-column layout — Genesis came out 1,3,2,4,6,5… — no verse recognizer) and prose/novels (LLM grouper).
• ⚠️ The chunker verdict rode a provenance-FREE pipeline. export_to_markdown() + string-in/out promotion discard all Docling provenance before chunking → §1 validates chunking + retrieval but gives ZERO evidence citation highlighting survives. Re-opened as RT-1 / §3; a precondition, not a follow-on.

2. Target architecture#

The shape: Docling converts as a new first stage; the existing Node pipeline consumes its output. All three sub-decisions (hosting, language split, routing) were red-teamed — see RT-1..RT-3.

2.1 Where Docling runs — hosting (RT-3: spike)#

• Container-on-Lambda is ALREADY the house pattern — the three workers are DockerImageFunction today (correcting the earlier "new pattern" framing). The genuinely new thing is a Python image in a Node monorepo (a build/CI boundary, per [[feedback_monorepo_docker_partial_copy]]) at 4-8 GB (vs the current ~226 MB).
• Cold start is the gating risk, not a footnote. Live prod: the current torch-free prep worker pins at Init Duration: 9999ms — Lambda's hard 10s init cap. Lambda kills any init > 10s and re-runs it inside the handler timeout ([[feedback_worker_tsx_coldstart_10s_edge]]). A torch+OCR container inits in tens of seconds → always blows the cap → on a large scan (which nears the 15-min ceiling) risks a hard-timeout DLQ. "Async" gives no relief from the init cap.
• ENOSPC (multi-GB image) — a real deploy risk the earlier draft missed. The Docker-prune fix is manual + NOT in deploy.sh ([[feedback_docker_vm_disk_enospc_autri_deploys]]); a 4-8 GB image (20-35× today) makes ENOSPC near-certain on the M1 build, plus the M1 arm64-torch trap ([[reference_m1_rosetta_python]]). → build the image in CI/remote, or resize the VM + automate prune.
• Decision RT-3: spike init/memory/per-doc + lighter-Docling; on-demand Lambda only if init is fast, else Fargate (an SQS-polling service; Spot for migration). /tmp is 512 MB → raise ephemeralStorageSize for Docling rasters / OCR intermediates.
• The docling→ingest handoff re-wires a live S3→SQS notification (a documented CloudFormation cycle trap on that edge) + adds a 4th queue/DLQ (more retry/idempotency surface + a monitoring alarm).

2.2 The language split — STRUCTURE-NATIVE (RT-1)#

Docling → element structure WITH provenance. Promotion + chunking operate on that structure (each element keeps page_no/bbox/charspan), then map onto the shipped bbox + source_ranges locators. This means part of promotion is structure-aware (Python-side, or a structured-JSON handoff to Node), reversing the earlier "convert-only in Python" split. Non-negotiable: the markdown-string path is provenance-fatal (§3).

2.3 Routing — ALL through Docling, then detect (RT-2)#

Doc-class is computed by probeAndSegment FROM parsed geometry (the parser Docling replaces), so it's unknowable pre-conversion; only file-extension is pre-parse (can't separate reg / bible / novel PDFs). Route everything through Docling, then use evaluateStructuralSanityKb invariants + a fail-closed reading-order check to detect verse/prose/mis-order → fall back to a specialized path or flag. Do NOT pre-classify.

3. Source provenance & citation highlighting — the PRECONDITION#

Dan's flagged risk, confirmed as the load-bearing gap. The red-team reframed it from "a downstream feature to bolt on" to "a precondition the built pipeline violates at step one."

What already ships (the model to MAP ONTO, not reinvent):

• chunks.bbox (JSONB, normalized 0-1, always-plural regions) — the visual locator (db/migrations/001).
• chunks.source_ranges (JSONB, half-open char ranges into the page's canonical source) — the logical locator (db/migrations/017).
• The inspector's real PDF highlight = the pdftotext per-word layer (WordOverlay + page-NN-words.json), finer than chunk boxes.
• Source-Legibility & Bbox-Trust epic: D1-D12 locked; evaluateStructuralSanityKb (gold-free structural invariants, D4 not-corpus-calibrated) BUILT; D10 = one canonical string built+measured in one pass, offsets never recomputed at read time.

What Docling must do (the real design owed):

1. Keep each element's ProvenanceItem (page_no, bbox, charspan — confirmed in docling 2.107 / docling-core 2.85; TextItem + TableItem carry prov) and map onto the shipped locators.
2. PDFs: export Docling's word/cell boxes into the page-NN-words.json format WordOverlay renders — else the highlight regresses from per-word to coarse union boxes (a visible trust regression). BoundingBox is absolute l/t/r/b + coord_origin → needs a normalize + origin-flip adapter to the 0-1 {x,y,w,h} schema.
3. docx/xlsx: no pixels → use the char-range (source_ranges) locator, ignore bbox. Two substrates, two locators — the earlier one-bbox model conflated them.
4. Structure-native promotion (RT-1) so offsets/prov never drift — D10's discipline extended to Docling, NOT the offset-remapping the earlier §3.1 proposed (which D10 forbids).
5. Fail-closed reading-order gate: fuzzy quote→source alignment returns confident-WRONG matches on reordered text (Genesis). A structural-sanity / reading-order check must fail closed (drop/flag), not emit a wrong box.
6. Table cells: Docling exposes per-cell boxes (TableCell.bbox) — a chance to make table highlight cell-granular (current model = one union box; ties to #68 oversized-table-chunk).

4. OCR on Linux (the scanned path)#

• ocrmac is macOS-only (Apple Vision); the session-32 win used it. Prod Lambda/Fargate is Linux → EasyOCR (torch, bakes in), Tesseract, or a fixed RapidOCR (its default PP-OCRv6 model config failed in the pinned build). This compounds §2.1: EasyOCR pulls its own detection+recognition models → bigger image → worse cold start.
• Re-validate scanned retrieval on the chosen Linux engine — the 94% is a Mac preview; OCR text + bbox quality is engine-specific (and the bbox feeds §3's highlight).
• OCR is slow + memory-heavy → per-doc cost/timeout budget (compounds the 15-min ceiling on large scans).

5. Markdown → structure parser — RESOLVED by RT-1#

Superseded. Structure-native promotion (RT-1) means we never serialize to a markdown string, so there is no markdown to re-parse and markdownToPages's H1/H2 coarseness is moot for the Docling path. (markdownToPages stays for the existing native-.md upload path.)

6. Migration — SMALLER than the earlier draft claimed#

• The 17.7k Brehob corpus is not in prod — a future curated load into brehob-prod (DB-2, post-vend). If Docling ships first, those files are born on Docling → zero migration. Adopt before the Brehob load.
• Current prod ≈ a dozen dev/test docs (sole user, pre-beta). The migration surface is small, not a corpus-wide re-embed.
• Version auto-detection is parked + gated off (AUTRI_VERSION_DETECT off; detectVersionForLoad early-returns) → no live versioning system to entangle. Leave it off — a boundary-changing re-chunk is exactly the anti-pattern that parked it.
• The real migration concern = annotation-anchor breakage: re-chunking changes the section_id labels Foundry annotations anchor to → orphaned citations (the failure class the team already hit). Ride the shipped supersede/flip substrate for an atomic stage-then-flip (old chunks live until the new set finalizes = rollback + no retrieval-gap window).

7. Feedback loops & acceptance#

Reuse the SHIPPED evals; add the provenance-survival gate.

• 🤖 Retrieval recall per corpus — extend the session-32 harness with FTS + lookup (router-level, not vector-only); hold within noise vs the current-pipeline baseline on structured corpora.
• 🤖 Boundary recall — the numbering-promote test (green).
• 🤖 Provenance survival — the GATING check (re-does what §1 skipped): prove structure-native promotion carries prov end-to-end to a CORRECT highlighted region on the legal pair + the scanned contract + Genesis. Reuse the BUILT evaluateStructuralSanityKb invariants + a fail-closed reading-order check — not a new loop.
• 🧑 Scanned path on the chosen Linux OCR engine (re-validate; ocrmac was Mac-only).
• 🧑 Prose/verse routing — specialized paths still win; no regression traded for the Docling win.

Pass bars (draft): structured corpora hold ≥ current within noise (paired significance); scanned validated on Linux OCR ≥ clean-doc parity; provenance/citation accuracy ≥ TBD (structural-sanity invariants pass + curated IoU).

8. Cost — a coverage play, NOT a COGS win#

The earlier "no-LLM = cheaper" framing is backwards:

• The Haiku grouping cost it compared against is already $0 for structured docs — the deterministic code route is default-on (route.ts), so there is no LLM grouping call to save on the common path.
• Embeddings dominate the per-doc floor and Docling doesn't remove them (they aren't even in the current cost model — cost.ts prices only the Haiku step).
• Docling compute is not free (torch/OCR GB-s + cold start) and plausibly exceeds the $0 deterministic path per structured doc.
• So Docling is justified by coverage + quality (scanned docs, legal/reg structure, killing the per-style-heuristic treadmill P1-P9), NOT COGS. Measure Docling GB-s/doc against the deterministic-code baseline, not the legacy Haiku baseline.

9. Open decisions (post-red-team)#

RESOLVED: promotion (RT-1 structure-native), routing (RT-2 all-through-Docling + detect), markdown parser (moot, §5).

STILL OPEN — blue-team / spike targets:

1. Hosting (RT-3 spike) — Lambda-on-demand-if-lighter-Docling vs Fargate; measure init / memory / per-doc + ONNX / skip-DL delta.
2. Linux OCR engine (EasyOCR vs Tesseract vs fixed RapidOCR) + image-size/cold-start impact + scanned re-validation.
3. Provenance mechanism detail (highest-risk build) — the word-layer export format, the BoundingBox coord adapter, the fail-closed reading-order gate, the docx/xlsx char-range path (§3).
4. Migration — ride supersede/flip for atomic re-ingest; annotation re-anchor policy.
5. Build/deploy — Python image in the Node monorepo; ENOSPC (build in CI/remote or resize + automate prune); the S3-notification re-wire cycle trap.
6. Cost — measure Docling GB-s/doc vs the deterministic-code baseline.