Measurement, not vibes: the yardstick the whole stack can use

“Trust us, it’s accurate” is a marketing line, not a measurement. Yet that’s how almost every SAST tool is sold: the vendor self-reports, the public benchmarks are frozen and long since memorized, and the toy single-language suites look nothing like a real finding. The result is a market where nobody can actually prove which scanner is good.

BenchProctor provides an open corpus and public scorer for compatible SARIF 2.1.0 output: true-positive rate, false-positive rate, and Youden’s J. The result remains tied to the benchmark version and the tool output being scored.

What the public release provides

It reduces label leakage. Published files omit comments, CWE tags, category names, and label-bearing file names. Versioned releases can rotate cases while retaining the published scoring contract.
The filename leaks nothing. No comments, no CWE tags, no category names, and file names that give away neither category nor label. The public answer key is the ground truth used by the scorer.
It looks like real software. 11 languages, 21 framework targets, 242 categories across 231 CWEs, with broken-sanitizer traps that a pattern-matcher fails. Cross-file and polyglot shapes remain on the public roadmap. The score subtracts false alarms from catches, so “flag everything” nets zero.
It’s free and open. Apache-2.0, and the scorer is a single zero-dependency Python file. Clone it, point it at your SARIF, read your number.

A frozen, single-language, self-reported benchmark isn’t in the same conversation.

Wired into the stack, it’s the receipt for everything else

A stack that makes strong claims needs a measurement surface. That is the job:

TheAuditor is a SAST engine you can hold to the yardstick, so its accuracy is a measured number, not an adjective.
Warden acts on findings you’ve proven trustworthy, not on noise.
Arbiter can orchestrate scoring runs across providers as a standing gate.
Curator carries the standards those runs are held to, across every project.

Every other tool in this stack is willing to be measured, and BenchProctor is the measuring stick. That’s a posture legacy vendors structurally can’t match: you can’t sell “trust me” to a buyer who can score you in three commands.

Read the same case from each of their seats: TheAuditor, Warden, Arbiter, and Curator.

The old way is already over

Buying security tooling on a vendor’s self-reported number is how teams end up with a scanner that’s great at its own benchmark and useless on their code. The era of taking accuracy on faith is closing. Proof is becoming table stakes, and the tools that fear an open yardstick are telling you something.

Get the yardstick

Release 2026.07.22 is live now with all 11 standalone languages and 2,938,418 cases. Start with Introducing BenchProctor, the completed staged-release rationale, or get the scorer on GitHub.

The whole stack, in its own words: TheAuditor · Warden · Arbiter · Curator · BenchProctor (you’re here)