BenchProctor blog
Engineering notes
How we build a SAST benchmark you can't game: methodology, scoring, coverage, and the occasional war story from generating millions of labeled test cases.
An answer key you cannot grep for
In a single file, anti-leakage means no comments, no CWE tags, no telltale names. When the benchmark is a whole application, the label can leak through the shape of the repo instead. Here is what a polyglot corpus has to hide that a single file never had to.
Read post →When the fix lives in another language
In a compound chain, the safe version and the vulnerable version are identical at the sink. The only difference is a control several files and one language upstream. That is the case you cannot pattern-match, and the reason a new browser tier of cases calls no sanitizer at all.
Read post →How BenchProctor scores a cross-language vulnerability chain
A single confusion matrix cannot describe a finding that crosses files, services, and languages. Here is the three-part score BenchProctor reports for a polyglot scenario, and why it never collapses them into one number.
Read post →The vulnerability a single-service scan cannot see
BenchProctor's standalone corpus scores a tool one file at a time. The new polyglot scenario category scores it on the hardest real case: a vulnerability that exists only as a chain crossing several languages and services.
Read post →All 11 languages are live: 2,938,418 cases, one public yardstick
BenchProctor release 2026.07.22 is live with 2,938,418 balanced benchmark cases across 11 languages and 21 framework targets. Score compatible SARIF 2.1.0 output.
Read post →An Honest Score With Public Inputs
BenchProctor provides a versioned corpus, public answer key, and scorer for interpreting compatible SARIF results.
Read post →BenchProctor Releases Are Versioned
A score is meaningful only with its scanner and benchmark versions. BenchProctor uses dated, periodic, event-driven releases.
Read post →A Benchmark That Resists Pre-Training
A frozen benchmark can reward memorization. Here is how 2,938,418 balanced cases and versioned releases support a more useful test.
Read post →How to Actually Measure a SAST Tool's False-Positive Rate
Most quoted SAST accuracy numbers have no ground truth behind them, so they cannot be checked. Here is what a defensible false-positive measurement actually requires.
Read post →Reading a SAST scorecard
A single SAST score can hide as much as it shows. Here is how to read recall, false positives, and the traps that make a good tool look bad.
Read post →The 2026.2 corpus: versioned and rotated
A historical note on the 2026.2 corpus and the value of versioned benchmark releases.
Read post →Bring any scanner's SARIF and we'll find the CWE
Scoring against BenchProctor is one standard-library Python file with no dependencies. The catch most benchmarks trip on is that tools don't report CWEs the same way, so the scorer recovers the CWE from wherever your tool actually writes it, with no per-tool adapter, and grades every tool on the same honest footing.
Read post →What makes a SAST test actually hard
A benchmark only means something if a pattern-matcher can't ace it. Here's what's inside a corpus designed to be hard: real framework idioms instead of toy snippets, sanitizers that are present but broken, taint that travels several steps, and a score that subtracts false alarms from catches so 'flag everything' nets zero.
Read post →A wrong answer key punishes the tools that get it right
A SAST benchmark with a mislabeled safe file measures the wrong thing. Here is why public answer keys and transparent scoring matter.
Read post →Measurement, not vibes: the yardstick the whole stack can use
BenchProctor is an open benchmark corpus with a public scorer for compatible SARIF 2.1.0 output.
Read post →Why BenchProctor released one language at a time
A look back at the staged verification plan that led to all 11 standalone languages shipping in release 2026.07.22.
Read post →How BenchProctor scores a SAST tool
The whole scoring model is a confusion matrix and one subtraction. Here's how true-positive and false-positive rates become a single number, why we average per category, and how the benchmark checks itself.
Read post →Why static SAST benchmarks rot, and how versioned rotation helps
A frozen benchmark can measure memorization as much as analysis. Versioned releases help preserve comparability while introducing fresh cases.
Read post →Introducing BenchProctor: a SAST benchmark built not to be gamed
An introduction to BenchProctor's public SAST benchmark corpus and transparent SARIF scoring model.
Read post →