Bring any scanner's SARIF and we'll find the CWE
Scoring against BenchProctor is one standard-library Python file with no dependencies. The catch most benchmarks trip on is that tools don't report CWEs the same way, so the scorer recovers the CWE from wherever your tool actually writes it, with no per-tool adapter, and grades every tool on the same honest footing.
Now that Java and Python are live, the question we get is refreshingly practical: will it score my tool? The honest answer most benchmarks have to give is “if your tool reports results exactly the way we expect.” That caveat is where fairness quietly dies, because no two scanners report a CWE the same way.
BenchProctor’s scorer is built around that reality instead of against it.
One file, no dependencies
Scoring is a single standard-library Python script. No install, no framework, nothing to pin. You run your tool, hand the scorer its SARIF output and the answer key, and you get a number. SARIF is a standard, so any tool that can emit it can be scored. That’s the whole point of meeting tools where they already are.
The CWE can hide anywhere, so the scorer looks everywhere
Here’s the part that makes it fair. A detection only counts if it lands on a vulnerable file and carries that file’s CWE, and most tools don’t put a bare CWE number where you’d naively look for it. One writes it into the rule’s metadata. Another tags it. Another references a CWE taxonomy. So the scorer recovers the CWE from wherever your tool actually puts it, rather than demanding a specific field and penalizing every tool that formats things differently.
The result: no per-tool adapter, no special-casing the popular scanners while everyone else scores artificially low. A tool isn’t punished for its SARIF dialect, only for what it did or didn’t find. And if a tool emits no CWE at all, there’s an explicit, clearly-labeled fallback mode, so you always know which footing you’re grading on.
One number, and it can go negative
You get a true-positive rate, a false-positive rate, and a single number that subtracts them, reported both with every vulnerability class weighted equally and in flat aggregate. Perfect is +100%, guessing is 0%, and a tool that’s worse than a coin flip goes negative. There’s nowhere for a bad detector to hide behind a flattering headline.
Why fairness is the credibility
A benchmark that only scores tools shaped like its favorite is just marketing for that tool. Making the scorer dialect-agnostic is what lets the number mean the same thing for everyone, and that is the only way a benchmark earns the right to be cited.
Where it stands
The scorer is open (Apache), the corpus is live for Java and Python, and the answer key stays unreadable from the files you analyze. BenchProctor is the proof layer under TheAuditor, Warden, Curator, and Arbiter. Read Java and Python are live or follow the RSS feed.