Bring any scanner's SARIF and we'll find the CWE
Scoring against BenchProctor is one standard-library Python file with no dependencies. The catch most benchmarks trip on is that tools don't report CWEs the same way, so the scorer recovers the CWE from wherever your tool actually writes it, with no per-tool adapter, and grades every tool on the same honest footing.
Now that all 11 standalone languages are live, the question we get is refreshingly practical: will it score my tool? The honest answer most benchmarks have to give is “if your tool reports results exactly the way we expect.” That caveat is where fairness quietly dies, because no two scanners report a CWE the same way.
BenchProctor’s scorer is built around that reality instead of against it.
One file, no dependencies
Scoring is a single standard-library Python script. No install, no framework, nothing to pin. You run your tool, hand the scorer its SARIF output and the answer key, and you get a number. SARIF is a standard, so compatible tool output can be scored without a per-tool adapter. Matching still depends on the locations and CWE data a tool provides.
The CWE can hide anywhere, so the scorer looks everywhere
Here’s the part that makes it fair. A detection only counts if it lands on a vulnerable file and carries that file’s CWE, and most tools don’t put a bare CWE number where you’d naively look for it. One writes it into the rule’s metadata. Another tags it. Another references a CWE taxonomy. So the scorer recovers the CWE from wherever your tool actually puts it, rather than demanding a specific field and penalizing every tool that formats things differently.
The result: no per-tool adapter and no special-casing of popular scanners. If a tool emits no CWE, the scorer has an explicit filename-only mode. That mode can reward over-flagging, so the default CWE-aware mode is the comparable result.
One number, and it can go negative
You get a true-positive rate, a false-positive rate, and a single number that subtracts them, reported both with every vulnerability class weighted equally and in flat aggregate. Perfect is +100%, guessing is 0%, and a tool that’s worse than a coin flip goes negative. There’s nowhere for a bad detector to hide behind a flattering headline.
Why fairness is the credibility
A benchmark that only scores tools shaped like its favorite is just marketing for that tool. Making the scorer dialect-agnostic is what lets the number mean the same thing for everyone, and that is the only way a benchmark earns the right to be cited.
Where it stands
The scorer is open (Apache), the corpus is live across all 11 standalone languages, and the answer key remains separate from the files you analyze. BenchProctor is the measurement layer for TheAuditor, Warden, Curator, and Arbiter. Read release 2026.07.22 or follow the RSS feed.