A wrong answer key punishes the tools that get it right
A SAST benchmark with a mislabeled safe file measures the wrong thing. Here is why public answer keys and transparent scoring matter.
Every SAST benchmark is, at bottom, an answer key: a pile of files, each labeled vulnerable or safe, and a score that rewards a tool for agreeing with the labels. So a wrong label doesn’t just cost you a data point. It inverts the score.
Picture a file labeled safe that actually contains a live SQL injection. A correct scanner reads it, finds the injection, and reports it. Against that answer key, the correct finding counts as a false positive. The tool that was right just lost points for being right, and a weaker tool that missed the bug entirely scored better. A benchmark that penalizes accuracy is worse than no benchmark at all, because it actively points you toward the wrong tool.
So before BenchProctor measures anyone, it has to be sure of its own answer key.
Public labels and answer keys
The public release pairs each test case with an answer key containing its category, label, and CWE. If you think a label is wrong, inspect the released case and answer key and report the specific discrepancy. The internal per-file proof metadata is deliberately not public.
A misplaced “safe” can’t ship
BenchProctor states that labels are checked before a release goes out, including the distinction between safe and vulnerable counterparts. The public release does not include the internal proof objects or oracle output, so users should distinguish that release assertion from independently reproducible public evidence.
What the public release lets you check
Users can verify ZIP hashes, inspect the public scorer, and recompute a score from their own SARIF output and the public answer key. Those checks do not prove the private generation or label-review process, so the public wording should not claim that they do.
What stays fixed while the code rotates
A benchmark earns trust by being hard to game, and a balanced answer key is one piece of that.
Release 2026.07.22 holds vulnerable and safe cases to an exact 50/50 split, so once the scorer
subtracts the false-positive rate from the true-positive rate, flagging everything nets zero. Future
releases are versioned and may rotate cases while retaining their published scoring contract.
Coverage now spans 11 languages and 21 framework targets; the public catalog stands at
242 categories across 231 CWEs, with 2,938,418 balanced cases in release 2026.07.22.
Where it stands
The scorer that reads a SARIF file and returns your true-positive rate, false-positive rate, and Youden’s J is public and zero-dependency, under Apache 2.0. All 11 standalone languages are live.
BenchProctor is a public benchmark for the rest of the stack: TheAuditor’s findings, the work Warden acts on, the runs Arbiter orchestrates, and the standards Curator keeps. Background on the method: why static SAST benchmarks rot, introducing BenchProctor, and the Java-first release plan.