AskBench
GitHub Leaderboard

Your table

Paste per-study estimates. The panel pools each factor and refuses when one combined number would mislead.

Study table

Checking…
Column format

factor + HR/RR with lower & upper CI columns is the usual paper format. Also: log_rr + se. Map columns above if your headers differ.

Ask

Panel preview

Loading preview…

Panel verdict

Paste a study table — preview appears on the right as columns map.

Panel verdict

Panel discussion

Publication-style figure of the vetted finding
The panel declined to combine

Worklist click to replay prior questions
Question Dataset Verdict Note
About AskBench

A statistical panel checks the evidence before the model speaks — and refuses when pooling is not safe. Numbers from the toolkit, never the model.

Most tools give you an answer. AskBench tells you whether the statistics support one. The toolkit computes; the Skeptic refuses when checks fail.

Reproducible offline: ASKBENCH_STUB_LLM=1 python3 web/server.py (no API key). Measured over 200 seeds · 1.58% false-positive rate after Benjamini-Hochberg FDR.

Data

AskBench is deliberately explicit about provenance.

Real published dataset
BCG trials (Colditz 1994) · 13 studies · benchmark tab
Synthetic datasets (for measurement)
Perturb-seq screen · VTE meta-analysis · planted traps
Reproduce
python3 eval.py · python3 real_data.py
MCP server
Call the Skeptic from inside Claude Code · askbench_mcp.py

How we measure this

A single good answer is easy to cherry-pick. So both tracks are re-run over 200 random seeds and scored against planted ground truth: every trap the Skeptic must catch, every real finding it must let through, and the rate at which pure-noise candidates slip past. The figure below is generated from those runs, not typed by hand.

Horizontal bar chart of the Skeptic's correct-verdict rate over 200 seeds. Structural traps KO_3 and IVF_ART are caught on 100% of seeds; the statistical traps immobility heterogeneity and maternal_height near-null are caught on 92.5% and 91.5%; real findings KO_0, KO_1, previous_VTE and thrombophilia pass on 99.5%, 95.5%, 100% and 99.5%. A dashed reference line marks the 1.58% null false-positive rate. Publication-style figure over synthetic illustrative data.
Skeptic operating characteristic over 200 random seeds. Structural traps (fixed cell and study counts) are caught by construction; the statistical traps sit in the low nineties; real findings pass from 95.5% to 100%. The amber line marks the null false-positive rate (1.58%, 82 of 5200 null candidates), reported not hidden. Synthetic illustrative data.

Reproducible from a clean clone: python eval.py prints these exact numbers, deterministically, with no model and no credits spent.