An AI analyst for A/B tests that never does the math itself.
An MCP server that gives any AI client validated experiment statistics: power analysis, sample-ratio checks, CUPED, delta-method ratio metrics, sequential and Bayesian tests, and segment-level effects. An agent uses those tools to write ship / don't-ship readouts in which every number comes from a tool call.
Skills demonstrated
Data science foundation
AI engineering
Headline result
Decisions that match the pre-registered rule, 40 held-out simulated experiments × 3 runs
98%
DeepSeek V4-Pro in both. Without tools the model spent 2.7× as much and took 3× as long, mostly doing arithmetic.
A real output
Tool calls
describe_experiment(experiment_id='hillstrom')
→ 3 arms, 64,000 units. Primary spend (MDE 15% relative, alpha 0.05); guardrails: none
check_srm(experiment_id='hillstrom')
→ Observed split 33.29% / 33.29% / 33.42%; chi2 = 0.2025, p = 0.9; SRM check passed
analyze_metric(metric='spend', cuped_covariate='history')
→ mens_email vs no_email: +117% (95% CI +64% to +197%), p < 0.0001; CUPED variance reduction 0.0434%
adjust_pvalues(method='holm')
→ Holm over 2 comparisons: 2 significant at alpha 0.05
Answer
Decision: SHIP mens_email (vs no_email).
How it works
01
Question
“Should we ship this experiment?”
02
Readout agent
LLM with a capped tool loop
03
MCP server
12 read-only tools over stdio
04
Statistics engine
Pure Python, validated by simulation
05
Experiment store
DuckDB registry + Parquet units
Tools take experiment ids, never raw data, and every result carries a stable id the readout cites. Ground truth lives in a separate store that only the eval oracle reads.
Hillstrom: the agent ships the men's e-mail (+117% spend), and CUPED barely helps: prior-year spend cuts the variance of two-week spend by 0.04%. The women's e-mail works on new customers, and an uplift model (Qini 0.039) shows it's the only one worth targeting.
Criteo, a seeded 1M-user sample: the SRM check correctly passes the planned 85/15 split, visits rise +30.2%, and the lift is concentrated, with the uplift model's top decile at +5.66 points against +1.21 overall. The simpler T-learner beat the X-learner (Qini 0.052 vs 0.032).
Evidence
Agent evals on the held-out test split
40 simulated experiments × 3 runs; mean across runs. The oracle applies the pre-registered rule; the truth is the simulator's known effect.
| Metric | Agent (V4-Pro) | Agent (V4.1-Flash) | Same model, no tools |
|---|---|---|---|
| Decision matches the pre-registered rule | 98% | 100% | 73% |
| Decision matches the true effect (ceiling 88%) | 86% | 88% | 62% |
| Required checks done (from the trace) | 100% | 100% | n/a |
| Injected SRMs caught | 100% | 100% | 89% |
| Numbers grounded in a tool result | 99% | 96% | 47% |
| Cost / latency per readout | $0.0072 / 33 s | $0.0016 / 18 s | $0.0191 / 95 s |
Peeking inflates false positives
False-positive rate over 14 daily looks when there is no true effect, across simulations.
5% target
CUPED only helps when the covariate predicts the outcome
Confidence-interval width relative to no adjustment (lower is better).
What the evals caught
01
A tool reported a significant result whose relative CI still crossed zero, and three agent runs read it three different ways. A Fieller interval now uses the same statistic as the test, so the two can never disagree: 0 disagreements on 5,000 near-boundary cases, down from 1,248.
02
Given only counts and means, the no-tools model compared an absolute CI bound with a relative 5% margin and turned a failing guardrail into “ship”. With tools, the server returns the verdict, so the agent never handles units.
03
The first O'Brien-Fleming implementation spent exactly α overall but used the one-sided formula. Only comparing against published boundaries caught it: the first look's boundary was 4.38 instead of 4.88.
04
A 5% non-inferiority margin failed about 1 in 5 changes with zero true harm at the planned sample sizes. Guardrails now fail only on statistically significant harm, and the eval includes small real harms to measure what that misses.
Limitations
Use it from Claude Desktop
Add the server to claude_desktop_config.json (absolute paths), then ask Claude to read out an experiment.
{
"mcpServers": {
"experiment-copilot": {
"command": "/path/to/experiment-copilot/.venv/bin/expc",
"args": ["serve", "--data-dir", "/path/to/experiment-copilot/data"]
}
}
}Stack