Spike: Bedrock Model Comparison vs Haiku
Status: SCOPED — information-gathering spike. Created 2026-06-26, scoped 2026-06-27 after red-team (Dan & Claude). Answers one question: is there a Bedrock-hosted model that does Autri's semantic chunking at materially lower cost than Claude Haiku 4.5, at equal-or-better retrieval quality? Pure measurement — bank the scorecard, no implementation, no production change. A future conversion is a separate decision this spike informs; it does not perform it. Ladders to North Star B2 (know cost cold) with B1 (quality floor) as the gate. Runs in parallel with the document-versioning epic (independent — no shared code).
How to read: the Method + Tasks are the spike; Acceptance defines what "banked" means.
Purpose
Cost is the north-star denominator, and the chunking LLM call is the largest variable cost in LLM-routed ingestion. Bedrock offers models (Amazon Nova Micro/Lite) roughly an order of magnitude cheaper per token than Haiku 4.5. If one of them holds Autri's retrieval quality, that is the single biggest ingestion-cost lever available — far bigger than batch (~10× vs ~2×). But we will not touch production inference or embeddings without data-driven proof. This spike produces that proof (or disproves it) and banks the result. Nothing ships off the back of it without a follow-up decision.
Goals & Non-Goals
Goals:
- Build the missing cost-vs-quality scorecard in the eval harness (the cost↔recall join doesn't exist today — see Background). Reusable instrument, kept regardless of outcome.
- Run a model frontier-search: Haiku 4.5 (baseline) vs Nova Lite / Nova Micro (+ Haiku 3.5) on the semantic-grouping call, scored on retrieval recall + $/doc with paired significance.
- Bank a quality-vs-price frontier and a go/no-go recommendation for a future conversion.
Non-Goals (this spike does NOT):
- Implement any conversion. No swapping production inference, no embedding swap, no chat/QuoteAI provider change, no cutover. The decision to convert is a separate future epic this spike feeds.
- Move embeddings. (An optional secondary embedding-quality arm — T3 — banks a data point, but changes nothing.)
- Batch ingestion (deferred separately).
- Document versioning (its own epic, runs in parallel).
- Resolve Brehob residency — Dan's read is Brehob won't require strict in-AWS residency (one-person IT, reasonable counterpart); the all-AWS question is therefore a uniformity/cost choice, not a contract mandate. (Light follow-up: confirm the signed contract's residency wording — a one-line attorney tweak if needed, not engineering.)
Background — what the red-team established (so this doc stands alone)
- The chunking call uses forced tool-use (
sdk-client.ts:tool_choice:{type:"tool"}+ a Zod schema). Bedrock supports forced tool-use for both Claude AND Nova (the Nova tool-choice doc uses "structured output" as its worked example) — so candidate models drop into the existing path without a prose-JSON rewrite. - Nova / Titan are single-region on-demand (bare model ID) — no
us.*cross-region profile, no ~10% geo premium. (Only newer Claude models are forced onto the cross-region profile + premium.) Strengthens the Nova cost case. - The significance layer is already built —
ingestion/eval/significance.tshas an exact paired McNemar test, a noise-floor "n=1" guard, and a paired bootstrap CI. Use it. - The per-index query gold exists (vector / FTS / attribute-filter forms) across a real multi-category corpus (19 docs + the Slate Trucks pair + 5 distractors). The experiment runner already takes pluggable chunker arms with a pinned model ID.
- What's missing (the spike's first task): no runner co-emits $/doc alongside recall —
run-experiment.tshas zero cost wiring. The frontier is a (quality, cost) plane; that join must be built. The baseline snapshot is also stale (frozen 2026-06-03,prompt-grouping-v3) — re-freeze before any A/B. - Quality yardstick = retrieval bars, not extraction bars. A grouping-model swap moves chunk boundaries → retrieval recall; it does not move typed-attribute extraction (and the S4 extraction stage isn't built / its gold is "not yet scoreable"). So the floor is retrieval recall + $/doc, not the ≥98% / ≥90% extraction bars.
Method — frontier-search (one variable at a time)
- Instrument (T0). Wire per-arm
$/doccapture intorun-experiment.tsalongsiderecall@k/ MRR; re-freeze the baseline against the current pipeline. Output: a{recall, mrr, $/doc}per-arm scorecard from one run. - Arms (T1). Add Bedrock model arms via the Converse API, forced tool-use preserved: Haiku 4.5 (
us.anthropic.claude-haiku-4-5-…, baseline) · Nova Lite (amazon.nova-lite-v1:0) · Nova Micro (amazon.nova-micro-v1:0) · (optional) Claude Haiku 3.5. Same grouping prompt + schema for all; add a</tool>-style stop sequence + a trailing-text post-parse guard for Nova. - Score (T2). Run all arms against the existing per-index gold; compute retrieval recall + $/doc with paired significance (McNemar + bootstrap + noise floor) vs the Haiku-4.5 baseline.
- (Optional) Embedding arm (T3). Titan v2 / Cohere Embed vs OpenAI
text-embedding-3-smallon retrieval recall only (no cost lever — embeddings are already cheap; this banks the quality-parity half of a future all-AWS option). Clearly secondary. - Bank (T4). Write up the quality-vs-price frontier + a go/no-go recommendation for a future conversion epic. No implementation.
(Confirm exact Bedrock model IDs + current pricing when the arms are built — IDs and rates drift.)
Tasks
| Task | Summary | Status |
|---|---|---|
| T0 | Cost-vs-quality scorecard instrument — per-arm $/doc co-emitted with recall/MRR in run-experiment.ts; baseline re-freeze. (Keeper — reusable for all future model decisions.) | Not started |
| T1 | Bedrock model arms — Nova Lite/Micro (+ Haiku 3.5) via Converse, forced tool-use, Nova trailing-text guard; Haiku 4.5 baseline | Not started |
| T2 | Run the frontier — recall + $/doc per arm with paired significance vs baseline; the scorecard | Not started |
| T3 | (Optional) Embedding-quality arm — Titan v2 / Cohere vs OpenAI on retrieval recall; banks the all-AWS-parity data point | Not started (optional) |
| T4 | Bank + recommend — frontier write-up + go/no-go for a future conversion. No implementation | Not started |
Acceptance — what "banked" means
A committed scorecard plus a recommendation:
- Per-arm
{vector recall@10, FTS recall@5, hybrid lift, $/doc}, each delta vs the Haiku-4.5 baseline carrying paired significance + noise-floor verdicts (no point estimates). - Quality floor (the gate): a candidate is "viable" only if it holds vector recall@10 ≥ 0.90, FTS recall@5 ≥ 0.85, and positive hybrid lift past the noise floor. The winner is the cheapest viable arm — never the cheapest arm.
- A one-paragraph recommendation: go (a Bedrock model is materially cheaper at equal-or-better retrieval quality → a future conversion is data-justified) or no-go (keep Anthropic + OpenAI; re-check when models change). Either way the result is banked, not acted on.
Decisions Log
| Date | Decision | Rationale | Alternatives Considered |
|---|---|---|---|
| 2026-06-27 | Spike, don't convert | Don't churn working, secure prod inference/embeddings without data-driven proof of cost-savings at equal-or-better quality. Bank the result; convert later only if the data justifies it | Build the conversion now (churn without proof); do nothing (leaves the ~10× cost lever unmeasured) |
| 2026-06-27 | Residency is not a forcing function | Dan's customer read: Brehob (one-person IT, reasonable counterpart) won't require strict in-AWS residency; an issue, if any, is a conversation with Andy. So all-AWS is a uniformity/cost choice, not a contract mandate | Treat residency as a hard go-live blocker (drives a painful, maybe-needless re-index) |
| 2026-06-27 | Quality floor = retrieval bars, not extraction bars | A grouping-model swap moves chunk boundaries → retrieval recall; extraction accuracy is a different axis (and S4 isn't built / not scoreable). Judge the model on recall + $/doc | Import the full ingestion-foundation floor (wrong yardstick + unmeasurable) |
| 2026-06-27 | Cost-scorecard instrument is a keeper | The cost↔recall join is missing today; building it serves every future model/price decision, not just this spike | Hand-merge two disjoint runs (error-prone, not reusable) |
Risks
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Nova schema adherence / trailing text under forced tool-use | Medium | Noisy or invalid grouping | Stop sequence + post-parse guard; the eval surfaces it as a quality miss anyway |
| Per-arm cost capture inaccurate (current cost model uses a calibrated multiplier on cached artifacts) | Medium | Misleading $/doc | T0 captures real per-run tokens per arm, not the cached-artifact multiplier |
| Stale baseline skews deltas | Low | Wrong reference | Re-freeze in T0 before any A/B |
| No Bedrock model clears the floor | Low (acceptable) | Spike returns "no-go" | That IS a valid, valuable result — banks "keep what we have," re-check later |
Open Questions
- Model set: is Nova Micro too weak for semantic grouping to bother including, or start at Nova Lite + Haiku 3.5? (T1 — cheap to include both.)
- Embedding arm (T3): run it in this spike to bank the all-AWS-parity data point, or strictly inference-only? (Cheap to add once the harness is up.)
- Corpus scope: run the frontier on the full eval corpus, or a representative slice + the Slate Trucks (Brehob-shaped) docs for the headline number?
Relationship to other docs: ladders to North Star (B2 / B1); consumes the eval harness (ingestion/eval/ + the eval-pipeline discipline) and cost observability. Supersedes the prior conversion-epic framing of this doc — conversion is now a future, data-gated decision, not committed scope. The document-versioning epic runs in parallel (independent). Batch ingestion remains deferred (archive/epics/batch-ingestion).