BenchProctor blog
Engineering notes
How we build a SAST benchmark you can't game: methodology, scoring, coverage, and the occasional war story from generating millions of labeled test cases.
All 11 languages are live: 2,938,418 cases, one public yardstick
BenchProctor release 2026.07.22 is live with 2,938,418 balanced benchmark cases across 11 languages and 21 framework targets. Score compatible SARIF 2.1.0 output.
Read post →An Honest Score With Public Inputs
BenchProctor provides a versioned corpus, public answer key, and scorer for interpreting compatible SARIF results.
Read post →BenchProctor Releases Are Versioned
A score is meaningful only with its scanner and benchmark versions. BenchProctor uses dated, periodic, event-driven releases.
Read post →A Benchmark You Cannot Pre-Train Against
A frozen benchmark can reward memorization. Here is how 2,938,418 balanced cases and versioned releases support a more useful test.
Read post →How to Actually Measure a SAST Tool's False-Positive Rate
Most quoted SAST accuracy numbers have no ground truth behind them, so they cannot be checked. Here is what a defensible false-positive measurement actually requires.
Read post →Reading a SAST scorecard
A single SAST score can hide as much as it shows. Here is how to read recall, false positives, and the traps that make a good tool look bad.
Read post →The 2026.2 corpus: versioned and rotated
A historical note on the 2026.2 corpus and the value of versioned benchmark releases.
Read post →Bring any scanner's SARIF and we'll find the CWE
Scoring against BenchProctor is one standard-library Python file with no dependencies. The catch most benchmarks trip on is that tools don't report CWEs the same way, so the scorer recovers the CWE from wherever your tool actually writes it, with no per-tool adapter, and grades every tool on the same honest footing.
Read post →What makes a SAST test actually hard
A benchmark only means something if a pattern-matcher can't ace it. Here's what's inside a corpus designed to be hard: real framework idioms instead of toy snippets, sanitizers that are present but broken, taint that travels several steps, and a strict 50/50 split that makes 'flag everything' score zero.
Read post →A wrong answer key punishes the tools that get it right
A SAST benchmark with a mislabeled safe file measures the wrong thing. Here is why public answer keys and transparent scoring matter.
Read post →Measurement, not vibes: the yardstick the whole stack can use
BenchProctor is an open benchmark corpus with a public scorer for compatible SARIF 2.1.0 output.
Read post →Why BenchProctor released one language at a time
A look back at the staged verification plan that led to all 11 standalone languages shipping in release 2026.07.22.
Read post →How BenchProctor scores a SAST tool
The whole scoring model is a confusion matrix and one subtraction. Here's how true-positive and false-positive rates become a single number, why we average per category, and how the benchmark checks itself.
Read post →Why static SAST benchmarks rot, and how versioned rotation helps
A frozen benchmark can measure memorization as much as analysis. Versioned releases help preserve comparability while introducing fresh cases.
Read post →Introducing BenchProctor: a SAST benchmark you can't game
An introduction to BenchProctor's public SAST benchmark corpus and transparent SARIF scoring model.
Read post →