BenchProctor Releases Are Versioned
A score is meaningful only with its scanner and benchmark versions. BenchProctor uses dated, periodic, event-driven releases.
BenchProctor publishes dated releases when there is a public artifact worth shipping. The schedule is periodic and event-driven, not a quarterly promise. A score should always name both the scanner version and the benchmark release used to produce it.
That sounds like a logistics change. It is really a statement about what a score means.
A score is a pair, not a point
Here is the principle the whole cadence rests on. A benchmark result is never just a property of your scanner. It is the pair (scanner version, benchmark version). “Tool X scores J 0.91” is an incomplete sentence until you say which corpus produced it. The same scanner run against a larger, harder corpus will score differently, and it should, because the test got better.
Once you accept that a score is a pair, a new benchmark release becomes another dated measurement point. A larger or harder corpus can yield a different result, so compare like with like and cite the release date.
Dated artifacts and reproducibility
The public release provides dated bundle names, a manifest, per-file hashes, and an aggregate checksum list. Record those details with a result so someone else can retrieve the same release artifact and reproduce the scoring run.
The guarantee that does not change
Release bundles contain test code, an answer-key CSV, the public scorer, a manifest, and a bundle README. How cases are generated and their internal proof metadata stay outside the public repo.
So the promise is narrow and ironclad. Faster releases, more of them, each one frozen the moment it lands, all of them kept. Same two artifacts, same open scoring you can recompute by hand, as covered in how BenchProctor scores a SAST tool.
Why we are doing it now
The benchmark can grow without a calendar promise. The important discipline is to state the release date, validate its checksums, and keep the benchmark version alongside every score.
BenchProctor is open under Apache 2.0. Record a release version, score compatible SARIF output, and inspect the public scorer and integrity data yourself.