An AI agent for SEC filings that cites every claim and computes every number.
A RAG and function-calling agent over the 10-K filings of 15 public companies. Explanations come from hybrid search over filing text; figures come from SEC XBRL data through typed tools, and every number in an answer is checked against its sources. Every pull request must pass an eval gate in CI, and each answer is traced with its cost.
Skills demonstrated
Data science foundation
AI engineering
Headline result
Pass rate on the held-out test set, 92 questions × 3 runs
97.5%
+4.0 points, 95% CI [+1.1, +7.2] from a paired bootstrap, with 14% fewer prompt tokens.
A real output
› Compare Apple, Microsoft and NVIDIA's gross margins in their latest fiscal years, and what does NVIDIA say about its customer concentration?
Tool calls
get_financial_metric(ticker='AAPL', metric='gross_profit', fiscal_year=2025)
→ [1] Apple Inc. gross_profit FY2025: 195,201,000,000 USD
get_financial_metric · search_filings(revenue and gross profit for all three, plus one filing search)
→ [2]–[7] XBRL facts and an NVIDIA 10-K passage
calculate(operation='ratio', a=153463000000, b=215938000000)
→ ratio = 0.7107
Answer
| Company | FY | Gross Profit | Revenue | Gross Margin |
|---|---|---|---|---|
| Apple | FY2025 | $195.2B [1] | $416.2B [2] | 46.9% |
| Microsoft | FY2026 | $225.5B [3] | $331.8B [4] | 67.9% |
| NVIDIA | FY2026 | $153.5B [5] | $215.9B [6] | 71.1% |
NVIDIA states that its revenue is concentrated among a limited number of direct and indirect customers [7]. In FY2026, one direct customer was 22% of total revenue and another 14% [7].
How it works
01
Question
About any of 15 companies' 10-Ks
02
Agent loop
DeepSeek, async, parallel tool calls
03
Tools
Hybrid search · XBRL facts · calculator · compare_metric
04
Grounding guard
Every number checked against tool results
05
Cited answer
[n] citations, traced with cost and latency
Filings are parsed into Item sections, chunked and embedded locally with bge-small in LanceDB. Every question is an OpenTelemetry trace, and every pull request runs unit tests plus a 48-question eval gate.
Adding 13 semiconductor companies tested whether the pipeline generalizes. A batch ingest prints a health report per filing, and the first run passed 5 of 13: headings vary by filer, and several companies don't report lines like gross profit or total liabilities. After the fixes 11 of 13 pass, and missing metrics are derived and labelled as derived in every answer.
Evidence
Retrieval ablation, held-out test set
92 questions × 3 runs; 95% CI from a paired bootstrap.
| Retrieval | Recall@5 | Pass rate | Note |
|---|---|---|---|
| Vector only (v1) | 0.57 | 93.5% | |
| Hybrid: BM25 + vector (shipped) | 0.80 | 97.5% | +4.0 points, CI [+1.1, +7.2]; 14% fewer tokens |
| Hybrid + cross-encoder reranker | 0.79 | 97.5% | No gain, +1.7 s median latency: left off |
Agent vs plain RAG, original benchmark
The first 58-question set; it saturated at 100%, which is why the harder 198-question set exists.
| Question type | Agent | Plain RAG |
|---|---|---|
| Overall (58) | 100% | 60% |
| Numeric (12) | 100% | 83% |
| Comparison across companies (5) | 100% | 0% |
| Sector rankings, 13 companies (7) | 100% | 14% |
| Unanswerable, refused correctly (8) | 100% | 100% |
The speedup came from concurrency, not async
Wall time for the full 140-question eval run. 94% of answer time is waiting on the model.
What the evals caught
01
A deliberately broken pull request shifted every fiscal year by one. The agent noticed the wrong dates in its tool results and re-queried, so the eval gate passed it. The fix was a unit test: end-to-end evals measure answers, not components.
02
8.7% of answers changed between identical runs, so a 2-point change could be luck. Every comparison uses 3 runs and a paired bootstrap CI, and the CI gate's tolerance comes from the measured run-to-run spread.
03
A cross-encoder reranker improved ranking but not answers: +0.0 points, CI [−1.1, +1.1], for 1.7 s more latency. It ships disabled, with the ablation in the README.
04
The judge spent its token budget reasoning and returned nothing, so correct answers scored zero. Judge failures are now reported separately, never counted as wrong answers.
Limitations
Stack