Why static SAST benchmarks rot, and how versioned rotation helps
A frozen benchmark can measure memorization as much as analysis. Versioned releases help preserve comparability while introducing fresh cases.
Publish a benchmark once and leave it still, and it starts decaying the moment anyone looks at it. The code doesn’t go stale. The answers do, because they leak into the world around it.
The failure mode
A static corpus is a fixed set of files with fixed labels. Over time:
- Tool authors tune against it. Detection rules get shaped to the exact patterns in the suite, which is great for the suite and unremarkable everywhere else.
- The files end up in training data. Once a public benchmark is scraped, a model can recognize a case without analyzing it. The score measures recall of a dataset, not reasoning about code.
- “Improvements” become unfalsifiable. When the same 2,000 files are scored year after year, you can’t tell a genuinely better analyzer from one that has simply seen the answer key more times.
The result is score inflation that looks like progress. A number climbs, and nobody can say whether the tool got smarter or just more familiar.
What we actually want to hold fixed
The instinct is to keep generating new files. But if every release is a fresh random corpus, you lose the thing a benchmark is for: comparability. A score from one release has to mean roughly the same thing as a score from the next, or you cannot track regressions.
So the real requirement is subtle: change the code, keep the contract. The specific test files can be different between releases, so nothing can be memorized, while everything that determines what the score means stays constant.
What rotation holds fixed
For a rotated release, the actual test code can change. What does not change is everything that decides what a score means:
| Invariant | Held constant |
|---|---|
| CWE identity per category | Fixed, so a category always tests the same weakness |
| Difficulty distribution | Published release contract |
| True-positive / true-negative balance | 50 / 50 |
| Language & framework coverage | Unchanged across rotations |
Two properties fall out of that split:
- Deterministic, not a lucky draw. A release is a fixed artifact with a fixed answer key, not a random pile that happens to look right. Rebuild the same release and you get the same corpus.
- Fresh but comparable. A new release can use fresh files while retaining the scoring contract needed to compare results by version.
Why balance still matters
Holding the corpus at 50% vulnerable and 50% safe isn’t cosmetic, but it is not what defeats the laziest gaming strategy. Flag everything and a tool catches every true positive and every false positive at once, so its detection rate and its false-alarm rate both hit 1.0, and Youden’s J, the difference between them, lands at roughly zero. That is true however the classes are split, because J is a difference of rates measured within each class. What the 50/50 split protects is the false-positive rate itself: keep the safe half large and equal, and there is always enough known-good code to catch a tool that flags indiscriminately, while the flat-aggregate view never gets swamped by one class.
The point
A benchmark earns trust by making its scope and version clear. A rotated release can introduce fresh code while retaining the published scoring contract. Record the exact release with every result.
Next: the scoring itself. A confusion matrix, Youden’s J, and why we report a category-averaged number as the headline.