All projects

Coding & Analysis Platform

Survey Agents

Ask any dataset a question in plain English and get an answer you can make a decision on. Every number is computed by code, cited to the rows it came from, and read back and checked before anyone sees it; when unsure, it asks instead of guessing. Owned end to end, architecture through deployment.
When
June 2026 – Present
Type
Featured build
rule-built report13%agent picks the analyses52–59%
mapworkersjoinrepair loop
Distributed Map fan-out, a bounded repair loop and a 5% failure circuit breaker

Highlights8

  • Cut wrong answers 85% on 164 published figures by reading every answer back and checking it before it is shown. 68% on StatQA, above GPT-4o's best reported 64.83%.
  • Quadrupled the headline findings a page recovers, 13% to 52-59% on surveys it had never seen, with an analyst agent that chooses the analyses while code computes every number.
  • Architected and shipped twelve agents over 76 registered tools, orchestrated by a Step Functions machine generated from a declarative graph: Distributed Map fan-out, a bounded repair loop, a 5% failure circuit breaker, 347k lines of TypeScript and Vue, and 4,845 tests.
  • Schema-constrained decoding plus per-response correlation tokens: out-of-codebook labels rejected, silently overwritten rows detected, and 100% join integrity across recorded runs. The benchmark pipeline lost roughly one row in eight without flagging it.
  • Questions compile to MBQL plans, clear seven validation checks, then execute over stored facts with DuckDB on a miss. Metrics bind to column kinds, so an unseen survey is answerable on arrival.
  • Charts are specs, never images: one constrained Vega-Lite spec emits the chart, an accessible data table, alt text, a CSV and an ASCII rendering, with mark selection deterministic.
  • Every number in a drafted report resolves to a fact the analytics stage computed, or the sentence is redrafted naming the offending figure and then removed.
  • Answers statistical and causal questions on any table, and every answer is read back and checked before it is shown: wrong answers on 164 published figures fell 85%, and StatQA test choice reached 68% against GPT-4o's best of 64.83%.

Build notes45

  • Ten agents with their own tool sets: intake, label, qa, curate, adjudicate, analysis, analytics, viz and conclude, under an orchestrator whose tools are the other agents.
  • A curate agent that repairs the codebook itself: cluster confusable codes, find the gaps, draft the distinction that separates two of them, then freeze the profile so labelling runs against a fixed target.
  • An adjudicate agent that locates contested rows and re-decides only those, turning ensemble disagreement into a bounded second pass rather than a full re-label.
  • The ask path as eight tools around a single model call: describe the survey, bind the question, choose a rung, execute the plan, compose the answer, compare runs, propose follow-ups.
  • A visualization agent of seven tools (propose, check, draw, repair, restyle, describe, shorten labels) over nine marks, where the request vocabulary is deliberately wider than the draw vocabulary so a refusal can name the word it could not honour.
  • Charts you can talk to: a reader asks about the chart in front of them and the reply lands where they are looking, still resolving to the facts the analytics stage computed.
  • A 14-page Nuxt 4 front end over a NestJS API of ten modules (uploads, surveys, codebooks, analysis, comparisons, exports, governance, templates, jobs), all compiling against one shared TypeScript contract.
  • Five-stage intake pipeline (frame, repair, retype, classify, review) where the orchestrator is ordinary code and a model is consulted only where the rules are visibly unsure; every decision lands in one JSON recipe that replays without it. Over 30 hand-written export shapes: 6 files need a stage, 11 model calls, $0.0023, and 0 divergences between a run and its replay.
  • Label integrity by construction: a schema enum makes an out-of-codebook code unrepresentable rather than discouraged (the prompt this replaced threatened a $1,000 penalty and got 9 invented tags anyway), and a per-response correlation token catches batch scrambles that count-matching cannot see.
  • Escalation ladder for failed batches: discard the batch whole, re-label at batch size 1, then quarantine and ABSTAIN, capped at 2 iterations. Correlation integrity 100%, every row joined, and zero rows carrying a destroyed label against roughly one in eight on the pipeline it was benchmarked against.
  • verify_citations makes the reporting valve mechanical: every figure in the drafted prose must resolve to a fact the analytics stage computed, or the sentence is redrafted naming the offending number and then removed.
  • Ran the control experiment that the architecture could lose: at a fixed model the two pipelines tie on F1 (0.685 vs 0.692, overlapping ranges), model choice moves F1 by 0.230 and architecture by −0.007, and the pipeline costs ~4–8× and ~6× the wall clock of three plain API calls. Also showed the 0.637 baseline every earlier claim rested on was never reproducible.
  • Batch size measured as a quality parameter rather than a throughput knob: the sweep leaves F1 flat and moves the precision/recall split, so the ladder is a dial with a documented shape instead of a guess.
  • Consensus routing wired in after ensemble voting was tested on runs already on disk: the consensus signal separates ~4× better than the hand-built risk score, which measured as useless. Nested ensemble arms carry no information and nearly shipped.
  • Corrected my own published finding twice on reasoning: non-termination came from an unbounded reasoning budget and a rules block, not from an incompatibility between reasoning and constrained decoding. Reasoning then won on F1 and lost the product.
  • Deployed on AWS with CDK: Step Functions state machine (Distributed Map over label slices, merge, Choice-based repair loop, ToleratedFailurePercentage circuit breaker) reaching complete coverage on consecutive executions, after an early green run silently dropped part of the corpus because concurrent map iterations overwrote a shared slot.
  • Established where a managed agent runtime earns its place and then measured it away: AgentCore took labeling from partial to complete coverage where our own loop stalled, but a state machine beats it at the orchestration level, and on a repeat benchmark both remaining stages turned out to make one tool call and stop, paying 49.8s of a 65.4s stage for a container boot.
  • Semantic layer over the facts: ~10 declared operators, metrics minted by usage and bound to column kinds rather than names, a binding cache so repeat questions are lookups, and a refusal ladder that substitutes, decomposes, samples, extends and requests before it refuses.
  • Question passport on every answer (plan hash, dataset version, registry version, skill version) so an answer re-executes byte-identically and a stale one is detectable; shadow re-execution diffs recent answers when a definition changes, so a moved number is found before a reader finds it.
  • Negative catalog computed at intake: what a dataset cannot answer and why, so the interface grays out the control instead of refusing after the fact.
  • Visualization layer on a constrained Vega-Lite subset: one spec produces the chart, an accessible data table, alt text, a CSV and an ASCII rendering. Mark selection is deterministic, borrowed from Cleveland & McGill, Bertin, Mackinlay's APT, Draco's constraint split and Brehmer & Munzner's task vocabulary.
  • Chart agent held to improve-or-discard behind four gates: across the whole evaluation corpus not one proposal beat the rule table, and the single case with headroom scored five different ways across five identical runs. An oracle over the agent's own search space found the scorer's exploits first: a word cloud sized by a free-text column scored a perfect 1.00, which is where the measure gate came from.
  • Nuxt 4 web app over a NestJS API, both compiling against one shared TypeScript contract, rendering the same specs the CLI prints; DuckDB in-process as the compute engine with no server, and Lance serving vector plus BM25 search straight from S3.
  • Intake hardened against real files: UTF-16 headers full of null bytes, a duplicate header silently overwriting a column, 0/1 flags typed as rating scales, report titles in the header row, and roughly seven in ten exports hiding their timestamp inside the identifier so a header-trusting pipeline reports no date column while holding the dates the whole time.
  • 347k lines of TypeScript and Vue across 12 workspace packages, 4,845 tests and roughly 200 measured experiments, each stating what it establishes and what it does not, including that at equal model the pipeline ties a single prompt on F1, that every accuracy number is self-graded, and that the page answered only 19% of the questions a survey asked while every chart-level score looked healthy.
  • Located where agency pays rather than assuming it: the agent ties or loses at executing a chart, coding a response and sequencing a pipeline, and beats its rule baseline 13% to 52-59% on held-out surveys when it chooses which of a survey's questions deserve a chart at all.
  • Codebook fit is checked before any spend: question scope is embedded and 25 responses are sample-labeled, giving mean best match 0.598 with a 4% abstain rate on the right codebook against 0.316 and 88% on the wrong one, for about $0.001. The thresholds written from a guess before calibration ran would have let the wrong codebook through with a warning.
  • Upload identity is content, not filename: a normalized content hash means two people uploading the same export get one run instead of two conflicting sets of numbers for one survey, and the first sighting wins for the wave's date.
  • Run lineage on two axes with four relations (relabel, recode, wave, wave_recoded), because only a new wave may claim topic movement. A second labeling pass over the same responses cannot present itself as a trend.
  • Cross-run comparison guarded by what the runs actually are: 69% of recorded runs are the same respondents, so a pooled total is refused where a dimension is missing, drift across unequal runs is reported as a share rather than a count, and the axis is never assumed to be time.
  • Cross-codebook mappings earn trust by behaving alike: a proposal is checked against the co-occurrence and sentiment profiles of both codes before confirmation, and carries an SSSOM predicate so a narrowMatch is never aggregated as an exact match. Two real codebooks produced 0 lexical proposals, which is itself the finding.
  • Time buckets follow the academic calendar rather than the Gregorian one, because a month boundary falls mid-semester and term-to-term is the comparison a reader can act on.
  • Deterministic insight sweep says what stands out the moment an upload finishes, corrected for multiple comparisons, every claim resolving to a fact. Building it surfaced two false-discovery traps: correcting after an effect-size filter, and a null model that preserved the sparsity pattern, both of which made noise look like findings.
  • Chart properties are classified rather than opened up: 75 styling properties each carry a class, which is how three accepted-and-inert bugs were found where a restyle answered a field no branch ever read.
  • Capability menus are generated from the registry rather than hand-written, after the same staleness bug appeared three times, and a panel's declared requirement is the same predicate the validator enforces, so the interface cannot offer what the run will refuse.
  • Spend control that actually binds: caps on both tokens and dollars, checked inside a stage rather than only between stages, after an audit found the guard had been doing nothing at all.
  • The graph is data with load-time gates rather than a trusted stage list, and the pipeline emits one replayable recipe; ten agent harnesses were deployed with per-agent tool sets, one withheld at runtime until its precondition exists after telemetry showed it never firing.
  • Progressive transcript compaction borrowed from production harnesses, then measured: batch size rather than compaction is the dominant cost lever, and the 8,192-token output cap is the real constraint on batch size.
  • Five reproducible scaling proofs instead of slides: 200x rows at unchanged ask latency, cross-scope queries over 216 runs in milliseconds, zero-model-call replays, namespace and entitlement checks, and byte-identical re-execution from a passport.
  • Skills authored in the open Agent Skills standard and loaded into the cached system prefix, then held to the same bar as everything else: the first one changed no outcome on the batch and cost slightly less, which is the result rather than the pitch.
  • A review stage that reads the finished profile instead of each decision, because per-decision gates are blind to upstream bugs. Verified by reintroducing a known sampler defect (a stride that aliased against alternating data and silently halved a dimension's values on any file over ~200 rows) and confirming the stage catches what the gates missed.
  • Held against Claude Opus 5.5 working blind on ten survey files: judge-free rules cut wrong charts 93% and oversized charts 100%, while Opus still leads on focus.
  • One semantic model per run: a census found eight column facts each decided in several places, and moving every reader onto one model took disagreements across 411 runs from 30,405 to 263.
  • Every reading of a file is a recorded decision: code labels are read twice, the second time reversed, applied only where both agree, and asked of the reader where a guess would move a number.
  • One semantic layer for every component raised accuracy on the hardest questions from 60% to 93% with zero wrong, after more reasoning and bigger models helped none.

Tools

  • TypeScript
  • AWS Bedrock
  • AgentCore
  • Step Functions
  • AWS CDK
  • Nuxt 4
  • NestJS
  • Vega-Lite
  • DuckDB
  • Lance
  • Vitest
  • llama.cpp
Next projectEdSpace