What I built17
- A novelty ledger recording eight mechanisms with no published counterpart, each paired with the instrument that would measure it, so a claim of novelty is falsifiable rather than decorative: continuous audit-sample QC, promotion criteria for local-to-global learning, confusion-driven codebook merge and split, the two-call versus combined-call ablation on arbitrary codebooks, the tokens-per-card knee, fitted non-uniform class-definition rendering, cluster-shared versus per-row candidate sets, and fork-versus-fix for shared vocabularies.
- A four-rung candidate-presentation ladder that makes a large codebook affordable: send it whole, split it by its own top level, cluster similar responses behind one shared shortlist, then retrieve per response. The rung that was missing is the one for codebooks with no usable top level, which used to fall straight to the expensive answer.
- Two measurements decide when a project drops a rung, total prompt size and label count, whichever is hit first, because one number alone let oversized prompts through.
- Learning scopes, so a fix travels exactly as far as its evidence: a correction fixes one row, a codebook edit reaches one project's future runs, a shared definition change needs every affected department to sign off, and a prompt change needs evidence from many projects. Nothing alters a run already in flight.
- Fork-with-provenance for shared vocabularies: when two departments pull a definition in opposite directions it forks with its history kept, instead of one team silently winning. Error, ambiguity and perspective are triaged differently.
- Continuous QC as a blind random audit sample per run, powering both the quality score and a corrected estimate published with an honest margin of error beside the raw number.
- Conformal accept and abstain thresholds so the error rate on accepted labels stays under a stated bound, with every label carrying a calibrated 0–100 confidence rather than a vibe.
- Correction history that is never overwritten: the model's original answer and each human change with who and when, plus run versions that reuse unchanged rows so a re-run shows the edit's effect rather than model randomness, at no cost.
- An estimator registry where published numbers come only from registered statistical code, weighted designs carry correct variance, and a model can never produce a published figure.
- One model gateway with a one-way valve stated as model authority never widens, rather than the weaker no-model-call-below-the-boundary it replaced.
- Four agent surfaces with bounded jobs (DataPlan, Discovery, Diagnosis, Findings Assistant), none of which owns control flow.
- Managed agent frameworks ruled out of the control path with a reason: Bedrock Agents, Knowledge Bases and Flows own prompt construction, retrieval and orchestration, three of the four artifacts a published number's reproducibility depends on. Step Functions as run truth is out for the same reason, with run state assigned to a Postgres state machine.
- A technology radar written as scope markers instead of a wish list: in scope, out of scope, or field survey only, after the document was twice misread as a build list and once as a commitment to train models. Training, fine-tuning and self-hosted weights are a permanent product boundary.
- PII redaction ahead of every model call, sensitive-disclosure triage with escalation, sign-in on Cognito while every permission decision stays in code we own, and browser-direct uploads so large files never pass through the services.
- Bedrock Batch designed in as a run mode with its fencing pattern resolved, immutable model versions pinned where the provider offers them, and multi-region warm standby as the target recovery posture.
- Four sequenced layers, core AI system then platform services then product surfaces then operations, each with exit criteria rather than dates, because delivery is one developer working in sequence.
- An interactive diagram set alongside the prose: a system map, a stakeholder view, an AI explorer, and a Structurizr DSL model, so a reader can see the platform rather than read 8,600 lines about it.
Tools
- AWS Bedrock
- PostgreSQL
- pgvector
- TypeScript
- Zod
- DuckDB
- AWS Cognito
- OpenTelemetry
- Structurizr