Proof, not vibes: the yardstick the whole stack answers to
SAST vendors grade their own homework. BenchProctor is the open, machine-verifiable benchmark that scores any tool on a real number, and it's the proof layer the entire stack is built to survive.
“Trust us, it’s accurate” is a marketing line, not a measurement. Yet that’s how almost every SAST tool is sold: the vendor self-reports, the public benchmarks are frozen and long since memorized, and the toy single-language suites look nothing like a real finding. The result is a market where nobody can actually prove which scanner is good.
BenchProctor ends that. It’s an open, machine-verifiable benchmark corpus that scores any tool emitting SARIF 2.1.0 on one honest number: true-positive rate, false-positive rate, and the Youden’s J between them. No self-grading. No vibes.
On its own, it’s the only benchmark built to be un-gameable
- It can’t be memorized. The corpus rotates every quarter, so the code changes completely while every scoring-relevant invariant stays put. Last quarter’s score still compares, and last quarter’s answers don’t help.
- The filename leaks nothing. No comments, no CWE tags, no category names, and file names that give away neither category nor label. The answer key is the only ground truth, and it’s locked and verified before a release can ship.
- It looks like real software. 9 languages, 18 real frameworks, 234 categories across 219 CWEs (~85% of the OWASP Top 10 2025), with cross-file chains, polyglot microservice flows, and broken-sanitizer traps that a pattern-matcher fails. A 50/50 balance means “flag everything” scores zero.
- It’s free and open. Apache-2.0, and the scorer is a single zero-dependency Python file. Clone it, point it at your SARIF, read your number.
A frozen, single-language, self-reported benchmark isn’t in the same conversation.
Wired into the stack, it’s the receipt for everything else
A stack that makes strong claims needs something that can prove them. That’s the job:
- TheAuditor is a SAST engine you can hold to the yardstick, so its accuracy is a measured number, not an adjective.
- Warden acts on findings you’ve proven trustworthy, not on noise.
- Arbiter can orchestrate scoring runs across providers as a standing gate.
- Curator carries the standards those runs are held to, across every project.
Every other tool in this stack is willing to be measured, and BenchProctor is the measuring stick. That’s a posture legacy vendors structurally can’t match: you can’t sell “trust me” to a buyer who can score you in three commands.
Read the same case from each of their seats: TheAuditor, Warden, Arbiter, and Curator.
The old way is already over
Buying security tooling on a vendor’s self-reported number is how teams end up with a scanner that’s great at its own benchmark and useless on their code. The era of taking accuracy on faith is closing. Proof is becoming table stakes, and the tools that fear an open yardstick are telling you something.
Get the yardstick
Java ships first, production-ready, full public launch by the end of June 2026, with Python next and the rest following as each clears the same bar. Start with Introducing BenchProctor and why we release one language at a time, or get the scorer on GitHub.
The whole stack, in its own words: TheAuditor · Warden · Arbiter · Curator · BenchProctor (you’re here)