Most tools give you an answer. AskBench tells you whether the statistics support one. The toolkit computes every number, the Skeptic refuses what it cannot support.

Refuses on real published data: 13 BCG trials, RR 0.49, no pooled number at I²=92.1%
Measured, not asserted: 200 seeds, 1.58% false positives after Benjamini-Hochberg
Numbers identical with Claude switched off

BCG · 13 trials

Published per-trial data (Colditz 1994). Pooled RR 0.49, but I²=92.1% is too high for one honest pooled number.

Ask

Study table

Checking…

Outcome

Column format

Raw 2x2 counts work directly, no maths needed first: factor + tpos,tneg,cpos,cneg (or events_treat,n_treat,events_ctrl,n_ctrl). Also: RR/OR/HR with lower & upper CI, or log_rr + se. Odds and hazard ratios are pooled and labelled as such, never reported as risk ratios. Map columns above if your headers differ.

Panel verdict

Verdict is computed instantly from the toolkit. Tap a question to rephrase the headline.

Panel verdict

Proof: switch Claude off and the numbers do not move (worked example: the 13 BCG trials)

The toolkit computes the numbers; the model only writes the sentence

Claude narrating

Pooled RR 0.49
95% CI 0.345 to 0.695
I² 92.1%
Verdict FLAGGED

Claude removed (model-free)

Pooled RR 0.49
95% CI 0.345 to 0.695
I² 92.1%
Verdict FLAGGED

Same 13 BCG trials, identical numbers with the model switched off. Reproduce from a clean clone: ASKBENCH_STUB_LLM=1 python3 eval.py

The panel declined to combine

Worklist click to replay prior questions

Question	Dataset	Verdict	Note

About AskBench

A statistical panel checks the evidence before the model speaks, and refuses when pooling is not safe. Numbers from the toolkit, never the model.

Most tools give you an answer. AskBench tells you whether the statistics support one. The toolkit computes; the Skeptic refuses when checks fail.

Reproducible offline: ASKBENCH_STUB_LLM=1 python3 web/server.py (no API key). Measured over 200 seeds · 1.58% false-positive rate after Benjamini-Hochberg FDR.

Data

AskBench is deliberately explicit about provenance.

Real published dataset: BCG trials (Colditz 1994) · 13 studies · benchmark tab
Synthetic datasets (for measurement): Perturb-seq screen · VTE meta-analysis · planted traps
Reproduce: python3 eval.py · python3 real_data.py
MCP server: Call the Skeptic from inside Claude Code · askbench_mcp.py

How we measure this

A single good answer is easy to cherry-pick. So both tracks are re-run over 200 random seeds and scored against planted ground truth: every trap the Skeptic must catch, every real finding it must let through, and the rate at which pure-noise candidates slip past. The figure below is generated from those runs, not typed by hand.

Horizontal bar chart of the Skeptic's correct-verdict rate over 200 seeds. Structural traps KO_3 and IVF_ART are caught on 100% of seeds; the statistical traps immobility heterogeneity and maternal_height near-null are caught on 92.5% and 91.5%; real findings KO_0, KO_1, previous_VTE and thrombophilia pass on 99.5%, 95.5%, 100% and 99.5%. A dashed reference line marks the 1.58% null false-positive rate. Publication-style figure over synthetic illustrative data. — Skeptic operating characteristic over 200 random seeds. Structural traps (fixed cell and study counts) are caught by construction; the statistical traps sit in the low nineties; real findings pass from 95.5% to 100%. The amber line marks the null false-positive rate (1.58%, 82 of 5200 null candidates), reported not hidden. Synthetic illustrative data.