Case study · Oct 2026 · AI & Machine Learning Systems
Filling missing data at scale
An agentic RAG architecture for a private markets dataset
A private markets client had a field missing for approximately 35,000 records. The answers existed in unstructured public sources, but not in any structured field. We designed an agentic retrieval-augmented system that retrieves, extracts, adjudicates and validates each value, routing every record to the cheapest model that can resolve it.
The problem
Sparse features, unstructured evidence
Private markets datasets are dense in some fields and sparse in others. Every missing value weakens the models and analysis built on top, and manual imputation does not scale to ~35,000 records.
The naive approach, calling a frontier model with web search on every record, works but costs €0.19–0.41 per record, about €6,650–14,350 for 35,000 records, and gives no audit trail. An ungrounded prediction is worse than an abstention.
Requirements
- Coverage across approximately 35,000 records
- Provenance and a confidence score for every output
- Explicit abstention instead of hallucinated values
- Inference cost measured in cents per record
- Incremental re-runs on every new data release
Agentic architecture
A model cascade routed by difficulty
Each stage only receives what the cheaper stage before it could not resolve. Retrieval, a lightweight extraction model and deterministic consensus scoring handle most of the volume; reasoning models are invoked only on contested records.
Retrieval agent no inference
Plans a small set of queries per entity, ranks sources by reliability and caches documents. When a page cannot be fetched, the search snippet is used as a fallback context.
Extraction model lightweight LLM
Converts each document into structured candidates under a strict output schema, each grounded in a verbatim quote. High throughput, low cost per token.
Consensus scorer no inference
A weighted ensemble vote across independent sources, weighted by source quality and evidence strength. Resolves a record when at least two sources agree.
Judge agent reasoning model
Invoked only on conflicts. Reasons over the surviving candidates, their quotes and known context, and must clear a calibrated confidence threshold.
Verifier agent reasoning model
Treats the model's parametric knowledge as a hypothesis, then runs a targeted retrieval to ground it. Ungrounded hypotheses are discarded.
Plausibility check no inference
Constraint validation on every accepted output: valid ranges, consistency and no self-references. A rejected output falls through to the next stage.
System design
Agents, tools and routing
An orchestrator routes each record through the cascade, persists every intermediate output and stops cleanly on budget limits.
Retrieval layerA search tool with query planning per entity, reliability-weighted source ranking and a shared document cache, so a document retrieved once serves every record that cites it.
Extraction modelA lightweight LLM with constrained, schema-validated outputs: value, context and the grounding quote. Malformed outputs are rejected and retried, never repaired.
Decision layerA deterministic ensemble vote, a judge agent for conflicts and a verifier agent for model-generated hypotheses, followed by constraint validation on every output.
OrchestrationRecords are partitioned into fixed shards for distributed inference across machines. Each output is checkpointed atomically, so an interruption loses only in-flight records.
Memory storeA persistent store of the current best value per record, with provenance, stage and confidence, plus a full change history and a log of every refresh.
Cost governanceToken and retrieval costs are tracked per record. Quota or credit errors halt the run instead of burning through retries, and provider outages pause it with back-off.
Results
Lower cost and higher precision, from the same routing decision
The deterministic consensus stage is both the cheapest and the most accurate. Routing records through it before any reasoning model is what drives the cost down.
Inference cost for ~35,000 records
Euros, retrieval and inference included; cost per record in grey. Frontier model: list prices with built-in web search, ~4 searches per record (estimate). Cascade: measured per-record rate × 35,000.
Exact-match accuracy by stage
Held-out evaluation set with known labels, hidden from the system.
What made this work
Four design choices
Route by difficulty. Deterministic consensus scoring resolves the easy majority with no inference at all. It is the cheapest stage and the most accurate, so reasoning models are reserved for records that genuinely need adjudication.
Ground every output. Each value carries a verbatim quote and a URL. A model's parametric knowledge is only a hypothesis: ungrounded, it matched the label 44% of the time, so it never becomes an output without retrieval evidence.
Abstain rather than hallucinate. Records without agreeing evidence stay unresolved. A gap is safer than a confident wrong value, and it is logged so the record can be retried in a later cycle.
Incremental inference. Each new data release processes only the delta: new records, records newly in scope and earlier abstentions after a retry interval. The memory store keeps every output and its history, so coverage compounds over time.
Have gaps in your data?
Fund, deal or company fields: the same agentic architecture applies.
Tell us which fields are missing. We will scope the sources, the cascade and the cost per record before anything runs.