The 2026.2 corpus: versioned and rotated
A historical note on the 2026.2 corpus and the value of versioned benchmark releases.
Historical release note: This post describes the 2026.2 Java and Python release. The current public release is
2026.07.22, with 2,938,418 standalone cases across 11 languages.
The 2026.2 corpus covered Java on Spring and Jakarta, and Python on Flask, Django, and FastAPI, with every case labeled vulnerable or safe and split exactly fifty-fifty, so false alarms weigh against as much safe code as real bugs and, once the scorer subtracts the two rates, flagging everything nets zero. It maps to 213 of the roughly 249 CWEs OWASP designates as the actionable set every serious organization should cover, drawn from MITRE’s full catalog of some 3,000 weaknesses, about 85 percent coverage. Those were the release figures at the time.
A frozen benchmark measures the wrong thing
Publish a fixed set of test files once and leave it there, and two things happen. Tools get tuned against it, and the models behind them read it in training. After a while a high score stops meaning “analyzes code well” and starts meaning “has seen this corpus.” You are measuring memory, not analysis, and you can’t tell the two apart from the score.
What rotation changes
A rotated release can use different test code from its predecessor, while preserving the published scoring contract: vulnerability classes, fifty-fifty balance, difficulty mix, and framework coverage. That makes a score meaningful only when its benchmark version is stated alongside it.
Hard to game what you can’t recognize
The corpus is built to give a scanner nothing to pattern-match on. The files carry no comments, no category names, and no hints in their names, so a tool has to actually analyze the code to label it. The only ground truth is a separate answer key the scanner never sees, and every label is verified before a release ships.
Why it matters to you
If you are choosing a SAST tool, you want a number you can trust over time and against a vendor’s own claims. A versioned benchmark gives you that. Run a named release, record its version with your score, and recompute the result from the public scorer and answer key.
BenchProctor is open under Apache 2.0. Grab the current corpus and score compatible SARIF output.