BCG · 13 trials

Published per-trial data (Colditz 1994). RR 0.49 — but I²=92% is too high for one honest pooled number.

Ask

Study table

Checking…

Outcome

Column format

factor + HR/RR with lower & upper CI columns is the usual paper format. Also: log_rr + se. Map columns above if your headers differ.

Panel verdict

Verdict is computed instantly from the toolkit. Tap a question to rephrase the headline.

Panel verdict

The panel declined to combine

Worklist click to replay prior questions

Question	Dataset	Verdict	Note

About AskBench

A statistical panel checks the evidence before the model speaks — and refuses when pooling is not safe. Numbers from the toolkit, never the model.

Most tools give you an answer. AskBench tells you whether the statistics support one. The toolkit computes; the Skeptic refuses when checks fail.

Reproducible offline: ASKBENCH_STUB_LLM=1 python3 web/server.py (no API key). Measured over 200 seeds · 1.58% false-positive rate after Benjamini-Hochberg FDR.

Data

AskBench is deliberately explicit about provenance.

Real published dataset: BCG trials (Colditz 1994) · 13 studies · benchmark tab
Synthetic datasets (for measurement): Perturb-seq screen · VTE meta-analysis · planted traps
Reproduce: python3 eval.py · python3 real_data.py
MCP server: Call the Skeptic from inside Claude Code · askbench_mcp.py

How we measure this

A single good answer is easy to cherry-pick. So both tracks are re-run over 200 random seeds and scored against planted ground truth: every trap the Skeptic must catch, every real finding it must let through, and the rate at which pure-noise candidates slip past. The figure below is generated from those runs, not typed by hand.

Horizontal bar chart of the Skeptic's correct-verdict rate over 200 seeds. Structural traps KO_3 and IVF_ART are caught on 100% of seeds; the statistical traps immobility heterogeneity and maternal_height near-null are caught on 92.5% and 91.5%; real findings KO_0, KO_1, previous_VTE and thrombophilia pass on 99.5%, 95.5%, 100% and 99.5%. A dashed reference line marks the 1.58% null false-positive rate. Publication-style figure over synthetic illustrative data. — Skeptic operating characteristic over 200 random seeds. Structural traps (fixed cell and study counts) are caught by construction; the statistical traps sit in the low nineties; real findings pass from 95.5% to 100%. The amber line marks the null false-positive rate (1.58%, 82 of 5200 null candidates), reported not hidden. Synthetic illustrative data.