AskBench — ask your data, get a verdict, not a paragraph.

BCG trials · 13 published RCTs

GitHub · Leaderboard

Reproducible offline: ASKBENCH_STUB_LLM=1 python3 web/server.py — no API key, no credits. Clone and run python3 eval.py to replay the Skeptic benchmark.

200seeds in eval.py

1.58%FPR after BH

FLAGGEDBCG I² 92.1%

IDENTICALacross replays

Try:

Panel verdict

Panel discussion

Publication-style figure of the vetted finding

The panel declined to combine

Worklist click to replay prior questions

Question	Dataset	Verdict	Note

About AskBench

Most tools give you an answer. AskBench tells you whether the statistics support one. The toolkit computes; the Skeptic refuses when checks fail.

Measured over 200 seeds · 1.58% false-positive rate after Benjamini-Hochberg FDR · refuses when the statistics do not hold.

Data

AskBench is deliberately explicit about provenance.

Real published dataset: BCG trials (Colditz 1994) · 13 studies · shown on load
Synthetic datasets (for measurement): Perturb-seq screen · VTE meta-analysis · planted traps
Reproduce: python3 eval.py · python3 real_data.py
MCP server: Call the Skeptic from inside Claude Code · askbench_mcp.py

How we measure this

A single good answer is easy to cherry-pick. So both tracks are re-run over 200 random seeds and scored against planted ground truth: every trap the Skeptic must catch, every real finding it must let through, and the rate at which pure-noise candidates slip past. The figure below is generated from those runs, not typed by hand.

Horizontal bar chart of the Skeptic's correct-verdict rate over 200 seeds. Structural traps KO_3 and IVF_ART are caught on 100% of seeds; the statistical traps immobility heterogeneity and maternal_height near-null are caught on 92.5% and 91.5%; real findings KO_0, KO_1, previous_VTE and thrombophilia pass on 99.5%, 95.5%, 100% and 99.5%. A dashed reference line marks the 1.58% null false-positive rate. Publication-style figure over synthetic illustrative data. — Skeptic operating characteristic over 200 random seeds. Structural traps (fixed cell and study counts) are caught by construction; the statistical traps sit in the low nineties; real findings pass from 95.5% to 100%. The amber line marks the null false-positive rate (1.58%, 82 of 5200 null candidates), reported not hidden. Synthetic illustrative data.

Every number comes from the tested toolkit, never the model. Deterministic analysis over synthetic data with planted ground truth (illustrative Perturb-seq knockouts and a pregnancy-VTE meta-analysis; the readout gene GENE7 is synthetic) and one real published dataset (the 13 BCG-vs-tuberculosis trials, Colditz et al., JAMA 1994); Claude narrates the interpretation on top. Set ASKBENCH_STUB_LLM=1 to run offline and spend no credits.

GitHub

Built with Claude Code · Life Sciences 2026.