AI Software Engineer

Benchmarks and checking

• Held the platform against Claude Opus 5.5 working blind on ten survey files and their 28 published charts: judge-free rules cut wrong charts 93% and oversized charts 100%, while Opus still leads on focus.
• Built four benchmarks scored by code, not a judge: published toplines, nvBench 2.0, StatQA and QRData. StatQA test choice went from 40% to 68%, against GPT-4o's best of 64.83%.
• Added 25 statistical tests checked against scipy and statsmodels, causal effect estimators matching statsmodels, and causal discovery that says "unsettled" rather than guess a direction.
• Gave every column one meaning in one place: a census found eight facts each decided in several places, and one semantic model per run took disagreements across 411 runs from 30,405 to 263.
• Made every reading of a file a recorded decision, applied where checked and asked of the reader where a guess would move a number. Code labels are read twice, reversed, and applied only where both agree.
• Moved every component onto one semantic layer after more reasoning and bigger models helped none: accuracy on the hardest questions rose from 60% to 93% with zero wrong.

Highlights in full

• Own the architecture, roadmap and delivery of a platform that takes any structured dataset a team uploads and returns analysis traceable to its source: twelve agents over 76 registered tools, orchestrated by a Step Functions machine generated from a declarative graph, with Distributed Map fan-out, a bounded repair loop and a failure circuit breaker. 347k lines of TypeScript and Vue across 12 workspace packages, 4,845 tests, and no per-dataset pipeline to maintain.
• Shipped the product surface rather than stopping at a pipeline: a reader asks a question in plain English and gets a chart back, then edits it directly - stack, swap, sort, top-N, a second shape, highlight one category, filters on more than one value, and undo with redo - each of which had previously been reachable only by typing exactly the right sentence.
• Found where agency actually pays by moving it from executing analyses to choosing them: an analyst agent that sees one line per survey question and never a row returns typed intents that code compiles into plans, taking headlined claims recovered from a published analysis from 13% to 52-59% on held-out surveys against a rule-built baseline, at about $0.05 per 13 surveys.
• Eliminated a previously unmeasured failure mode with schema-constrained decoding, per-response correlation tokens and citation verification: out-of-codebook labels are rejected, silently overwritten rows are detected, join integrity reached 100% and label churn was zero across 100% of rows on repeat runs. The benchmark pipeline destroyed a label on 1 in 8 rows (12.7%) without flagging one.

The agent system

• Grew the pipeline to twelve agents with their own tool sets: intake, label, qa, curate, adjudicate, analysis, analytics, viz, conclude, coordinator and explore, under an orchestrator whose tools are the other agents.
• Added a curate agent that repairs the codebook itself: cluster confusable codes, find the gaps, draft the distinction that separates two of them, then freeze the profile so labelling runs against a fixed target.
• Introduced an adjudicate agent that locates contested rows and re-decides only those, turning ensemble disagreement into a bounded second pass instead of a full re-label.

Labelling reliability

• Recovered scrambled label batches instead of dropping rows: discard the batch whole, re-label at batch size 1, then quarantine and abstain, capped at 2 iterations. Correlation integrity has never dropped below 100% since.
• Calibrated codebook fit before spending anything on labelling: embed the question scope, sample-label 25 rows. The right codebook separates at 0.598 and a 4% abstain rate against 0.316 and 88%, for about $0.001.

Intake

• Split intake into five gated stages (frame, repair, retype, classify, review) where the orchestrator is ordinary code and the model is consulted only where the rules are visibly unsure.
• Unlocked a time axis on roughly seven in ten real exports, which had been reporting no date column while carrying the timestamp packed inside an identifier. The same intake survives UTF-16 null-byte headers, duplicate headers overwriting a column, and 0/1 flags typed as rating scales.
• Emitted every intake decision as one replayable JSON recipe: across every export shape we could construct, the model is consulted a handful of times, the sweep costs a fraction of a cent, and a run has never once diverged from its replay.

The ask path

• Structured the ask path as eight tools around a single model call: describe the survey, bind the question, choose a rung, execute the plan, compose the answer, compare runs, propose follow-ups.
• Compiled questions into Metabase's MBQL plan shape so one format serves both the agent and a UI query builder. A dashboard panel posts the plan the model would have written, at ~20 ms and $0.
• Designed refusal as the last rung of a ladder: substitute, decompose, sample, extend, request. An unanswerable question returns the missing column by name instead of an invented reason.
• Cached question-to-plan bindings with the plan hashed into the key, turning the one non-deterministic step in the read path into a lookup, and making falling hit similarity a signal that the corpus moved.
• Shipped a question passport on every answer (plan hash, dataset, registry and skill versions) that re-executes byte-identically, with shadow re-execution diffing recent answers whenever a definition changes.
• Wired metric lint to CI, so a declared metric that can no longer be computed against a stored run fails the build rather than rotting silently between releases.

Visualization

• Composed a visualization agent of seven tools (propose, check, draw, repair, restyle, describe, shorten labels) over nine marks, where the request vocabulary is deliberately wider than the draw vocabulary so a refusal can name the word it could not honour.
• Fanned one chart spec into five artifacts through a constrained Vega-Lite compiler: the chart, an accessible data table, alt text, a CSV and an ASCII rendering, with repeatability tested by diffing specs instead of images.
• Made charts conversational: a reader asks about the chart in front of them and the reply lands where they are looking, still resolving to the facts the analytics stage computed.
• Derived mark selection from published visualization research (Cleveland & McGill, Bertin, Mackinlay's APT, Draco, Brehmer & Munzner) so chart choice cites evidence rather than taste, with the agent confined to proposing mark and encoding behind four gates.
• Attacked my own scorer with an oracle over the agent's search space before trusting it. A word cloud sized by a free-text column scored a perfect 1.00, which is where the measure gate came from.

Data model

• Modelled the platform over a deliberately narrow fact table where dimension and key are columns, so a survey of departments and a survey of campuses union without a migration and without new SQL.
• Typed change itself with four lineage relations (relabel, recode, wave, wave_recoded) where only a new wave may claim topic movement, so a second labelling pass cannot present itself as a trend.
• Blocked the cross-survey join that silently double counts: a respondent fingerprint catches runs that share respondents, after a pooled breakdown counted every one of them twice while reporting a denominator that looked perfectly self-consistent.

Application

• Delivered a 14-page Nuxt 4 front end over a NestJS API of ten modules (uploads, surveys, codebooks, analysis, comparisons, exports, governance, templates, jobs), all compiling against one shared TypeScript contract.

AWS deployment

• Compiled one graph file into two deployment targets, Lambda-bound and AgentCore-bound, gated at load time so an edge to a node that does not exist fails before a run starts rather than halfway through one.
• Traced a fully SUCCEEDED execution that had silently dropped part of the corpus to concurrent Distributed Map iterations writing the same slot, then partitioned the writes and added a real fan-in, because green meant nothing until that held.
• Benchmarked a managed agent runtime against a hand-rolled loop, then removed it when the benchmark was repeated: AgentCore took labelling from partial to complete coverage where our own loop stalled, but once that loop was gone the stage made one tool call and stopped, and 49.8s of its 65.4s was the container booting. Both remaining harnesses now call their tool directly.
• Bounded blast radius with a ToleratedFailurePercentage circuit breaker at 5%, halting a run once that share of batches fails instead of grinding through every remaining batch and paying for all of them.

Measurement

• Retired the 0.637 baseline every prior claim in the project rested on, after fresh runs reproduced it at 0.523 and traced the gap to a delivered spreadsheet that had been human-reviewed before it shipped.
• Held every agent to improve-or-discard: across the whole evaluation corpus not one chart proposal has beaten the deterministic rule table, so the rule table stayed. A consensus signal measured on held-out data separated about 4× better than the risk score it replaced.

Platform architecture

• Built the semantic layer that decides what any dataset can be asked: plain English compiles to an MBQL plan, clears seven validation checks, then executes against stored facts with DuckDB on a miss. Metrics bind to column kinds rather than column names, so a survey the system has never seen is fully answerable the moment it lands, and a question nobody predeclared still returns a cited number.
• Authored the target architecture for the production platform that succeeds it: six documents, 80 logged decisions, four sequenced layers gated on exit criteria rather than dates, and eight candidate mechanisms each paired with an experiment that can validate or reject it.
• Staged the candidate-presentation ladder that makes a large codebook affordable: send it whole, split it by its own top level, cluster similar responses behind one shared shortlist, then retrieve per response. Each rung is tried before the next, and two measurements decide the drop.
• Specified learning scopes so a fix travels exactly as far as its evidence: a correction fixes one row, a codebook edit reaches one project's future runs, a shared definition change needs every affected department to sign off, and nothing alters a run already in flight.
• Made shared vocabularies forkable instead of contested: when two departments pull a definition in opposite directions it forks with its history intact, rather than one team silently winning.
• Sampled QC as a blind random audit per run, powering both the quality score and a corrected estimate published with an honest margin of error beside the raw number.
• Set accept and abstain cutoffs from conformal thresholds so the error rate on accepted labels stays under a stated bound, with every label carrying a calibrated 0–100 confidence rather than a vibe.
• Kept every correction instead of overwriting it: the model's original answer and each human change with who and when, plus run versions that reuse unchanged rows so a re-run shows the edit's effect rather than model randomness, at no extra cost.
• Ruled managed agent frameworks out of the control path with a stated reason: Bedrock Agents, Knowledge Bases and Flows own prompt construction, retrieval and orchestration, which are three of the four artifacts a published number's reproducibility depends on.
• Wrote the technology radar as scope markers rather than a wish list: in scope, out of scope, or field survey only, after the document was twice misread as a build list and once as a commitment to train models.
• Put PII redaction ahead of every model call and specified sensitive-disclosure triage with escalation, with sign-in on Cognito while every permission decision stays in code we own.

Platform and governance

• Deployed the governance surface in the POC to match: Bedrock guardrails deployed as their own stack, entitlement-gated respondent-level retrieval, and spend caps enforced inside stages after an audit found the guard doing nothing. The personal-data layer was later removed by decision, with name-shaped columns kept as a descriptive kind that is never analysed, which has to be revisited before a real upload.
• Ran the platform with no database: DuckDB in-process over per-run JSON, Lance loaded as a DuckDB extension for vector and BM25 retrieval straight from S3, facts bounded at ~150 KB per run against 14–178 MB of payload.
• Proved scale with five reproducible checks rather than slides: 200× rows at unchanged ask latency, cross-scope queries over 216 runs in milliseconds, and replays that call no model at all.

The design record

• Grounded the architecture in 12 logged prior-art passes: SSSOM mapping predicates, the DDI survey-metadata model, MBQL, Vega-Lite, Draco, Sato and Sherlock context-aware typing, XLSForm, GPTCache, LinkML, SDMX and SKOS.
• Documented what was deliberately not built and what would reopen it: no warehouse, so Cube and dbt's semantic layer stayed out; no text-to-SQL, since the model never sees a row after labelling; no agentic orchestrator, since the repair it was buying is a Choice state and a counter.
• Drew the line between what may extend and what may not: parsers, kinds, metrics, operators and marks extend without touching existing code, while the validator, the gate, provenance and grain enforcement require human review.
• Authored the decision record the team works from: roughly 200 measured experiments and rejected alternatives, each stating what it establishes and what it does not, including that at equal model the pipeline moves F1 by -0.007 while the model choice moves it by +0.230, and that the codebook-description lever the project had named its largest was measured and lost.