A wrong answer key punishes the tools that get it right
A SAST benchmark with a mislabeled 'safe' file doesn't just measure wrong. It scores a correct finding as a false positive and penalizes the tool that was right. BenchProctor won't let a misplaced label ship.
Every SAST benchmark is, at bottom, an answer key: a pile of files, each labeled vulnerable or safe, and a score that rewards a tool for agreeing with the labels. So a wrong label doesn’t just cost you a data point. It inverts the score.
Picture a file labeled safe that actually contains a live SQL injection. A correct scanner reads it, finds the injection, and reports it. Against that answer key, the correct finding counts as a false positive. The tool that was right just lost points for being right, and a weaker tool that missed the bug entirely scored better. A benchmark that penalizes accuracy is worse than no benchmark at all, because it actively points you toward the wrong tool.
So before BenchProctor measures anyone, it has to be sure of its own answer key.
Every label carries its evidence
You don’t have to take a label on faith. Each file in the corpus ships with the reasoning behind its label: the CWE it tests, and the code that makes the label true. That makes the ground truth auditable without running a scanner. If you think a label is wrong, you read the case and check it against the file, rather than arguing about it. A benchmark whose labels can’t be inspected is asking for trust; one whose labels arrive with the evidence earns it.
A misplaced “safe” can’t ship
Labels are checked, not assumed. Before a release goes out, every safe file has to genuinely differ from its vulnerable counterpart. If one doesn’t, that’s the exact mistake that produces an inverted score, and the release fails instead of shipping it. This holds across all nine languages, and a standard release can’t skip it.
There’s a second backstop on the same principle. A flawless tool would score a perfect 100% detection, 0% false alarms against the answer key, and every release is held to exactly that. If it comes up short, the labels and the scorer disagree, which is a defect in the benchmark, and the release fails before it can ever mislead a user.
Verified means checked, not “looks about right”
Label correctness is one property among several that every release has to clear. Each one is a deterministic check against the generated code, no machine learning and no judgment calls:
| Property | What it holds |
|---|---|
| Real compilation | Generated code is built by the actual language toolchains, not parsed, compiled |
| Structural quality | No synthetic stubs or tells; the code reads like code a person would write |
| Label integrity | Every label’s evidence is present, well-formed, and internally consistent |
| Ground-truth correctness | Safe and vulnerable counterparts are observably distinct |
“Verified” on a BenchProctor release means the artifacts were checked and passed, not that they look plausible.
What stays fixed while the code rotates
A benchmark earns trust by being hard to game, and a balanced answer key is the first defense. BenchProctor holds vulnerable and safe cases to a 50/50 split, within a two-percent tolerance, so the laziest strategy of flagging everything catches every real bug and every safe file alike and scores roughly zero. Difficulty is held too: every category keeps a floor of trivial, realistic, and hard cases rather than collapsing to whichever is easiest to produce. And every quarter the corpus rotates. The code changes completely while a category keeps testing the same weakness, so last quarter’s score stays comparable and none of last quarter’s files help this quarter.
Coverage currently spans nine languages and eighteen frameworks; the public catalog stands at 234 categories across 219 CWEs, mapping roughly 85% of the OWASP Top 10 (2025) CWE set. The corpus launches in phases as each language clears the same bar. Java is first, with a public launch targeted for the end of June 2026, Python next, and the remaining seven to follow.
Where it stands
The scorer that reads a SARIF file and returns your true-positive rate, false-positive rate, and Youden’s J is public and zero-dependency, under Apache 2.0, with a reference scan you can validate your own setup against. The Java corpus isn’t out yet; it’s the first in line.
BenchProctor is the independent yardstick for the rest of the stack: TheAuditor’s findings, the work Warden acts on, the runs Arbiter orchestrates, and the standards Curator keeps. Background on the method: why static SAST benchmarks rot, introducing BenchProctor, and the Java-first release plan.