A Benchmark That Resists Pre-Training
A frozen benchmark can reward memorization. Here is how 2,938,418 balanced cases and versioned releases support a more useful test.
Every hand-written SAST benchmark has the same expiration date. It is public, it is small, and it never changes, so the tools scored against it slowly bend to fit. A rule gets tuned to the handful of known cases. A model sees the corpus in training and learns the answers instead of the analysis. The score climbs while the tool gets no smarter. A benchmark you can pre-train against is not measuring detection anymore. It is measuring memory.
BenchProctor addresses that problem with scale, separate answer keys, and cases designed not to expose their category or label through names or identifiers.
Scale defeats hand-tuning
The current release contains 2,938,418 labeled cases across 63 framework-size suites, balanced exactly fifty-fifty between vulnerable and safe. That size is not a vanity number. A corpus this large is pointless to hand-tune, because there is no short list of “the cases” to shape a rule against. Scale alone will not stop a model from training on a public corpus, though, which is what rotation is for. The cases span 11 languages and 21 framework targets, and each language lives in three depth tiers: quicktest, normal, and enterprise. A tool that does well here did well on breadth, not on a cheat sheet it happened to see.
Rotation is what beats pre-training
Scale and clean files still leave one hole: the labels are public, so nothing stops someone training a model on this release’s files and their answers. What closes it is that releases rotate. A model fit to the exact cases in one release learns those cases, not the analysis, and the next release draws fresh ones behind the same published scoring contract. A score you pre-trained for expires the moment the corpus turns over, so the only durable way to score well is to analyze the code.
Anti-leakage by construction
Scale alone is not enough if the files leak their own answers. So the cases carry none of the usual tells. There are no explanatory comments. There are no CWE tags sitting in the source. The identifiers are shuffled, so a filename reveals nothing about whether the code behind it is vulnerable or safe. A scanner cannot read the label off the page, because the label is not on the page. The only place ground truth lives is a separate answer-key CSV that the scanner never sees during the run.
This is the same discipline covered in what makes a SAST test actually hard: if a pattern-matcher can ace it, it was never a test.
Gate-verified labels
A benchmark is only as honest as its answer key, and a wrong label inverts the whole score: a correct tool gets marked wrong, a broken one gets rewarded. BenchProctor states that every case must clear its release gates before shipment: compile or parse, source-to-sink behavior for vulnerable cases, an effective safe counterpart, and a recorded sink line. The public release includes cases and answer keys, not the internal per-file proof data.
The payoff is a number you can trust
All of that construction exists to serve one practical outcome. Point compatible SARIF 2.1.0 output
at the corpus, hand the results and the answer-key CSV to a zero-dependency scorer, and read a true
positive rate, a false positive rate, and Youden’s J. Release 2026.07.22 covers 242 emitted
categories mapped to 231 distinct CWE IDs. The score is reproducible from the public scorer, answer
key, and your SARIF output.
Why it matters
The value of a benchmark is its documented scope and reproducible scoring. A small frozen set can invite overfitting. A large release with separate labels and versioned artifacts offers a more useful measurement, while still requiring users to interpret the score within the release scope.
BenchProctor is open under Apache 2.0. Score compatible SARIF output and inspect the public scorer, answer key, and checksums yourself.