Epic: Document Versioning + Version Detection
Status: DRAFT — Step 0. Created 2026-06-27 (Dan & Claude), scoped through a design session + red-team. Makes a re-uploaded / re-published document a detected, version-historied object instead of a silent overwrite: hash-detect that an upload is a new version of an existing doc, keep the prior version as history (soft-supersede, not hard-delete), serve the current one. Editing documents is explicitly OUT — that's the Foundry epic. Incremental reuse (carry-forward of unchanged chunks) is deferred — this epic detects + versions + full-reprocesses; reuse is the later cost optimization. Ladders to North Star B1 / B7 (stop destroying history; dependability) and is compounding infra (dedup, lifecycle, eventual Foundry). Runs in parallel with the Bedrock model spike.
How to read: Current State + the Version-Detection design + the Stories are load-bearing. The detection threshold is eval-gated (V6); the two-quote Jaccard measurement (V1) de-risks it before any classifier is built.
Purpose / Problem Statement
- We silently destroy history today. Re-publishing via MCP
publish_document(replace-by-sessionId) and re-uploading via the app both hard-delete the prior version's entire tree (deleteDocumentTree). The dev-memory loop re-publishes episodes by sessionId, so re-running a session erases the prior episode. No undo, no history. - We have no version detection. Today the only "this replaces that" signal is the MCP sessionId substring; the app path has none (it rejects duplicates). There's no way to know an upload is a new version of an existing doc — so we can't version, dedup, or carry continuity.
- Version detection is compounding infrastructure. Dedup, version history, the document lifecycle, and eventually Foundry all lean on "do we already have a version of this?" Building it now (pre-Brehob) sets up the future.
What This Epic Is
Make a re-uploaded / re-published document a detected, version-historied object. Hash the document's coarse structural units, detect (via best-candidate overlap) whether an upload is a new version of an existing doc / a genuine new doc / a duplicate, keep prior versions as soft-superseded history, and serve the current version. The new version fully reprocesses (its own chunks) — no carry-forward yet. Editing is out (Foundry); incremental reuse is deferred.
Goals & Non-Goals
Goals:
- Version detection — per-unit content hashing +
source_hash; best-candidate Jaccard scoring with a confidence band (auto-link / confirm / new-doc); duplicate detection. Eval-gated threshold. - Soft-supersede, not hard-delete — on a detected new version, keep the prior version (set
superseded_at+logical_documentslineage), flip the current pointer at finalize, atomically. Stop destroying history. - Stable addressing + version surfacing + retention — logical-doc slug → current version; inspector shows history; a prune policy (vector-index-bloat-aware).
- A version-detection eval set — labeled link / not-link pairs to set the threshold safely (key hard-negative: two different template quotes).
Non-Goals:
- Editing documents / editable source of truth — the Foundry epic (where the git-vs-Postgres source-of-truth decision lives). This epic is versioning, not authoring.
- Incremental reuse (carry-forward unchanged units' chunks to skip re-LLM/re-embed) — deferred. It forces the chunk-ownership-across-versions data-model decision and pays off only at re-upload scale. This epic detects + versions + full-reprocesses the new version.
- Chunk-level versioning — versioning stays document-level.
- The Bedrock model spike (parallel, independent).
Current State (grounded)
From a code audit of the document data model + write path:
- A versioning skeleton is already migrated but dormant.
logical_documents(UNIQUE(kb_id, slug),current_version_document_id) exists, empty.documents.superseded_at/logical_document_id/version_labelexist, never written. - The retrieval side of supersession is fully wired + indexed —
includeSuperseded=false→superseded_at IS NULLacross vector / FTS / lookup / v1, plus a partial index. It filters nothing only because the write side never setssuperseded_at. → Activating versioning is mostly a write-side change. - Supersession today = hard delete (
publishDocumentInScope: snapshot stale-by-sessionId → create new → upload →deleteDocumentTree(old)); the app path rejects dupes ("delete first"). - Units are coarse + structural.
units.tsassigns pages to a structural unit (chapter/section); the extractor bundles every paragraph in the unit and fans out one LLM call per unit. A 26-chapter novel = 26 units; Genesis = 45. No fixed size cap — unit size follows printed structure. A short quote ≈ a handful of units. This matters: coarse units fold boilerplate in with unique content, so pure-boilerplate units are the minority case. documents.source_hashexists but is unused — the free byte-identical-dedup signal.- The child graph (CASCADE off documents): pages → figures; sections; chunks (inline
embedding VECTOR(1536)+attributes+unit_index); figures; document_units. Embeddings are inline on chunks → re-embed = row rewrite, and superseded chunks stay in the ivfflat vector index (retention is a perf concern, not just storage).
Design
Version detection (the centerpiece) — ⚠️ PARKED (see re-scope note at top of doc)
For each upload, after the parse step produces units:
- Byte-identical short-circuit. Compute
source_hash; match against the KB → duplicate: skip ingestion, alert the user, point at the existing doc. (This part KEPT — it's the boundary-independent dedup, the high-value piece.) - Hash the coarse units. Per-unit content hash over the deterministic parsed paragraphs (not chunks — LLM chunk boundaries are non-deterministic and won't match).
- Candidate-then-confirm. Narrow to candidate doc(s) by cheap signal (filename / slug / display_name), then aggregate unit-hash matches per candidate and score overlap with Jaccard (shared units / union). Best candidate only — incidental boilerplate matches across many docs are not evidence.
- Threshold + confidence band:
- High Jaccard → auto-link as a new version of the candidate.
- ~1.0 → duplicate (also caught at step 1 for byte-identical files; this catches near-identical).
- Low → new doc (process normally).
- Ambiguous middle band → ask the user "new version of
?" (one click — the edge-case handler; no perfect algorithm needed).
- Threshold is eval-gated (V6): start with a simple Jaccard cut; escalate to boilerplate-discounting (IDF weighting) only if the eval shows template docs false-positive.
⚠️ Steps 2–5 are PARKED as of 2026-06-29. Verified gap: a single-H1 prose doc collapses to ~1-2 coarse units (detectAnchor anchors a page on the top-level section component; markdown = one page per H1), so an organic revision rewrites the whole unit-hash → Jaccard 0 → new-doc. Confirmed via pnpm ingest units (2 units for a 10-section HR policy). The classifier (B-M2) + eval (B-M6) were tuned on multi-unit rule-book/synthetic docs — the wrong doc shape. The fix (detection-specific finer hash grain + prose re-eval) is NOT built, because the design review concluded auto-detection is (a) fragile (depends on a stable segmenter we're still fixing — the FIA section-size bug) and (b) narrow-value (real workflows declare versions explicitly — file id / sessionId / title). See the re-scope banner. Step 1 (source_hash dedup) is KEPT and shipped.
Soft-supersede + flip-at-finalize — ✅ SHIPPED + KEPT (B-M5)
On a detected new version: create the new doc linked to the candidate's logical_document (find-or-create per KB+slug) with a version_label; ingest it normally; at finalize (new version ready), in one transaction, set the prior version's superseded_at + flip logical_documents.current_version_document_id. The flip happens at finalize, NOT at publish — the new version is empty/pending until ingestion completes, so flipping early would serve an empty current version. Retrieval already excludes superseded. Replace the hard-delete in publishDocumentInScope and route the app re-upload through the same path. The new version fully reprocesses (its own chunks); carry-forward is the deferred reuse layer. (Shipped fee1ce2; verified atomic on real local Postgres incl. forced-rollback + dev-memory preservation. KEPT — this substrate is version-declaration-agnostic: it works whether a version is declared explicitly, by title, or detected.)
Deferred — incremental reuse
The same unit hashes that drive detection also identify which units changed — so carry-forward of unchanged units' chunks (skip re-LLM/re-embed) is a natural later optimization. Deferred because it forces the chunk-ownership-across-versions decision (copy / re-point / share-with-range) and its storage trade-off, and pays off only at re-upload scale. Detection produces the hashes now; reuse consumes them later.
Stories
⚠️ RE-SCOPED 2026-06-29 (session 25). All stories SHIPPED (merged to autri main fee1ce2, deploy held), but content auto-detection is PARKED (V2 classifier + V6 eval — dormant, do not invest further). The lifecycle substrate (V3/V4/V5 + the B-M4 undo) and source_hash dedup (V1) are KEPT. Statuses below reflect what shipped. Full rationale: the Decisions Log (2026-06-29) + program decisions.md (session 25).
| Story | Summary | Status |
|---|---|---|
| V1 | Unit hashing + dedup substrate + the de-risk measurement — per-unit content hash (coarse units) + activate source_hash dedup. Measure the two-quote unit-Jaccard (Slate Trucks + a 2nd real quote) to size the template false-positive risk BEFORE building the classifier | ✅ Shipped (B-M1). source_hash dedup KEPT; unit-hashing now feeds only the PARKED detector. (Two-quote measure = structural lower-bound; real body-text Jaccard never landed — see the verified gap.) |
| V2 | Version-detection classifier — candidate scoping (filename/slug) → per-candidate Jaccard → threshold + confidence band (auto-link / confirm / new-doc); the new / version / duplicate / ambiguous decision | ⚠️ Shipped (B-M2) but PARKED — auto-detection deprioritized |
| V3 | Soft-supersede write core + flip-at-finalize — on detected version, soft-supersede prior (superseded_at + lineage) at finalize, atomic current-pointer flip; replace hard-delete in publishDocumentInScope + route the app path through it; new version full-reprocesses | ✅ Shipped (B-M5) — KEPT |
| V4 | Stable addressing — find-or-create logical doc per (KB, slug); slug→current-version resolution; resolve the non-unique name / LIMIT 1 ambiguity | ✅ Shipped (B-S1) — KEPT |
| V5 | Version surfacing + retention — inspector/list shows current vs superseded history + the confirm-version UX; prune/retention policy (superseded chunks bloat the ivfflat index, not just storage) | ✅ Shipped (B-M3 confirm UI + B-S2 preserve-all/partial index) — KEPT; the confirm UI is repurposable for explicit/title-declared versioning |
| V6 | Version-detection eval set — labeled link / not-link pairs (key hard-negative: two different template quotes); precision/recall swept across the threshold; sets V2's threshold on data | ⚠️ Shipped (B-M6) but PARKED — tuned on the wrong doc shape (multi-unit rule-book/synthetic, not single-H1 prose) |
| B-M4 | Undo a version link (corrective) — un-supersede the prior version + split the two back into separate docs; every link reversible (atomic, chunk-flag-aware) | ✅ Shipped (B-M4) — KEPT; the reversibility primitive + the basis for a user-declared "not a version" action |
Edge Cases & Gotchas
| Scenario | Expected Behavior | Why It's Tricky |
|---|---|---|
| Two different template quotes share boilerplate units | Low Jaccard → new doc, not a version | Coarse units make boilerplate a minority fraction; the threshold rejects it; the gray-zone confirm catches mostly-boilerplate quotes |
| Ambiguous overlap (gray band) | Ask the user "new version of X?" | No perfect auto-classifier; human-in-the-loop for the middle |
| Re-publish same sessionId (dev-memory) | Old version → superseded (kept) | MCP sessionId is an explicit version key — keep it alongside hash-detection |
| New version still pending at publish | Flip the current pointer only at finalize | The new version is empty until ingest completes; an early flip serves an empty doc |
| Reordered sections | Match by content hash, not position | unit_index shifts; positional matching is wrong |
| Split / merged units | Read as changed/new; overlap drops | Heavy restructuring can fall below threshold → the confirm band catches it |
| Superseded versions accumulate | Retention prune | Superseded chunks stay in the ivfflat vector index → search bloat, not just storage |
Decisions Log
| Date | Decision | Rationale | Alternatives Considered |
|---|---|---|---|
| 2026-06-27 | Editing is OUT — this epic is versioning, not authoring | Editing a doc in place IS the Foundry integration (where the source-of-truth decision lives); versioning is independent and shippable now | Bundle editing here (conflates two concerns; pre-empts the parked Foundry design) |
| 2026-06-27 | Build version detection now; defer incremental reuse | Detection is compounding capability (dedup / lifecycle / Foundry), not just a cost optimization — and we have pre-Brehob runway. Reuse is the cost optimization that forces the chunk-ownership decision; defer to re-upload scale | Defer all of it (misses the compounding capability); build reuse now (premature + hard data-model decision) |
| 2026-06-27 | Hash coarse units, not chunks; source_hash first for dupes | Units are deterministic + diffable; LLM chunks are non-deterministic. Coarse units fold boilerplate with unique content → fewer false positives | Hash chunks (non-deterministic, unreliable) |
| 2026-06-27 | Best-candidate Jaccard + confidence band, eval-gated; IDF only if needed | "Some units match" false-positives on template docs (Brehob quotes); best-candidate Jaccard + a tuned threshold + a gray-zone user-confirm handles it; boilerplate-discounting is the escalation if the eval demands it | Binary "some match → version" (template false positives); pre-build IDF (over-engineering) |
| 2026-06-27 | Soft-supersede; flip at finalize, not publish | The retrieval filter + skeleton already exist; the new version is empty until ingest finishes, so the supersede/flip must wait for ready | Hard-delete (data loss); flip at publish (serves an empty current version) |
| 2026-06-29 (session 25) | REVERSAL: content auto-detection PARKED; lifecycle substrate + source_hash dedup KEPT | After shipping all stories + verifying on a real HR-policy corpus: a single-H1 prose doc collapses to ~1-2 coarse units, so an organic revision scores Jaccard 0 → new-doc (confirmed via pnpm ingest units; the classifier/eval were tuned on the wrong doc shape). Auto-detection is also fragile (needs a stable deterministic segmenter — blocked on the FIA section-size bug) and narrow-value (real workflows declare versions explicitly — file id / sessionId / title; most DMS don't content-detect at all). The high-value, robust pieces (dedup + the supersede lifecycle) don't depend on detection. | Build the prose-grain fix (over-invests in the fragile, narrow piece); finer units globally (re-tunes the chunking unit model + its eval); keep coarse units as-is (silently misses every prose revision) |
Risks
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Template quotes false-positive as versions | Medium | Lineage corruption | V1 two-quote Jaccard measurement sizes it; best-candidate threshold + gray-zone confirm; IDF if the eval demands |
| Non-atomic supersede → zero or two "current" versions | Medium | Retrieval empty / duplicated | Single-transaction supersede + flip at finalize |
| Superseded versions bloat the vector index | Medium | Search perf + cost | V5 retention prune; consider a partial vector index / hard-prune-after-N |
| Detection threshold mis-set | Medium | Mis-versioned docs | V6 eval (precision/recall) sets it on data; confirm band absorbs gray cases |
name non-uniqueness corrupts resolution | Low | Wrong version served | V4 resolves via the logical-doc slug + a uniqueness story |
Open Questions (red-team targets)
- Candidate signal: is filename/slug enough as the candidate gate, or add title/structural signals?
- Threshold + band boundaries: the auto-link / confirm / new-doc cutoffs — set entirely by V6's eval, or seed with a default?
- Confirm UX: where does the "new version of X?" prompt live in the app upload flow?
- MCP / headless confirm: the MCP path has no interactive user — fall back to the explicit sessionId key (no hash-confirm needed), or auto-link only on high confidence?
- Retention default: keep N / TTL / manual; per-KB configurable? And the vector-index-bloat mitigation (partial index vs hard-prune-after-N).
- Chunk-ownership (for the deferred reuse): copy / re-point / share-with-version-range — flagged now so the deferred reuse epic inherits it.
Relationship to other docs: ladders to North Star (B1 / B7); editing + editable-source-of-truth live in the foundry-integration epic (this is its versioning prerequisite). Builds on the dormant D15 skeleton (logical_documents + superseded_at) and the Wave-2 write core (retrieval/src/documents.ts). Incremental-reuse design: archive/sub-systems/incremental-re-ingestion (deferred). Runs in parallel with epics/bedrock-model-spike.