Which models ask before answering.
One honest table. AskBench hands a model the same vetted findings the deterministic Skeptic already checked, then measures whether the model's final answer respects those checks: does it defer to a flag, avoid claims the numbers do not support, and still carry real findings through. Every row is a real run, with N disclosed.
The bottom row is the control: the same panel with the Skeptic removed. Strip the Skeptic and flag deference collapses to 0%, it asserts every flagged finding as settled. Every Claude tier holds it at 98 to 100%. That gap is the point: the leaderboard measures deference to the checks, and it tells a panel that respects them apart from one that ignores them.
How to read this
The Skeptic is deterministic: given the data it always produces the same checks, and its accuracy against planted ground truth is fixed. That fixed accuracy is the baseline below, and it measures the deterministic checks, not a model. The table under it measures the opposite thing: given checks that are already correct, how faithfully a given model's final answer honours them.
Skeptic operating characteristic
Straight from eval.py, run deterministically over 200 seeds with
no model and no credits. Reproducible by anyone who clones the repo.
- Structural traps (hold by construction)
- 100.0% / 100.0% caught (underpowered cells; too few studies)
- Statistical traps (honestly not perfect)
- 92.5% / 91.5% caught (too heterogeneous; near-null)
- Real-finding pass rate
- 95.5% to 100%
- Null false-positive rate
- 1.58% (82/5200, after Benjamini-Hochberg FDR, disclosed not hidden)
What the model columns mean
When the Skeptic flags a finding as unsafe to trust, this is the share of those flagged findings the model's final answer does not present as settled. A model with low deference overrides the checks and asserts the finding anyway.
The share of claims in the model's final answer that go beyond what the vetted numbers support. A claim counts as unsupported when no number from the toolkit backs it, so this catches confident language the data did not earn.
When a finding is genuinely real and the Skeptic passes it, this is the share the model still carries through to its answer. It is the honest counterweight to deference: refusing everything would look cautious but would drop real signal, and this column catches that.
Leaderboard
| Model | N | Flag deference | Unsupported-claim rate | Real-finding pass-through | Run date |
|---|
On real-finding pass-through: the answer names the top-ranked real findings by design, so live models sit near 60% and the deterministic floor names all of them. The Skeptic-off control passes 100% through precisely because it defers to nothing and reports everything, which is why deference, not pass-through, is the column that tells respect from override.
Add your model
A row appears here only when a real run has written its result file. One command:
python3 leaderboard.py --model <id>
That writes leaderboard/results/<id>.json; this page reads that
folder on load. No file, no row: the benchmark does not carry a number it did not run.