BenchProctor blog
Engineering notes
How we build a SAST benchmark you can't game: methodology, scoring, coverage, and the occasional war story from generating millions of labeled test cases.
What the First Full Release Run Caught
The next BenchProctor release ships when its full gate run is green. Here is what the first full runs over the expanded surface caught, how each was fixed at the source, and where the release stands.
Read post →Applications: A First Public Release of Whole-Project Benchmarks
Single-file cases measure one thing at a time. Real systems are whole projects. The application shape gets its first public release: deployable, multi-language projects with real weaknesses planted across services.
Read post →A Label Has to Match the Code Under It
The next BenchProctor release adds eight weakness classes the corpus never covered, and generates each one only in the languages where it can honestly exist. A CWE label that names the wrong weakness costs the tool that names the right one.
Read post →Compose the Benchmark to Your Stack
A benchmark locked to a fixed set of language and framework combinations measures a scanner on someone else's stack. BenchProctor's corpus is now composed, not canned: request any lane, pin the exact library, or specify a whole multi-service graph.
Read post →Your Code Is Polyglot, So Is the Benchmark
Real systems mix languages, glue scripts, native modules, and a stack of cloud and infrastructure config. BenchProctor's corpus covers that surface, so you measure a scanner against your real attack surface instead of a sliver.
Read post →The Frameworks You Actually Use Are Coming to Standalone
The current release covers 21 framework targets. The next standalone release widens that to the frozen surface's full set of lanes: more web frameworks per language, GraphQL and serverless lanes, command-line lanes, and C host and kernel cases.
Read post →A Benchmark Surface You Can Audit
BenchProctor's generation surface is now frozen and published: every lane, every substrate, every pinnable library, with counts a build refuses to let drift. You can see exactly what the benchmark can cover before you trust a score from it.
Read post →Every File We Ship, Its Compiler Has Already Accepted
At benchmark scale, millions of generated files is a warning sign, not a boast. Nothing enters a BenchProctor release until the real toolchain for its language accepts it, and a suite of deterministic gates, with no AI judge, holds every label.
Read post →Whole Apps, Not Snippets
BenchProctor's v0.14.0 release ships whole, deployable, multi-language applications with several real weaknesses planted in each one. A benchmark that finally looks like the codebases teams actually run.
Read post →The Benchmark Caught Its Own Mistake
A benchmark is only as good as its answer key, so BenchProctor audits its own key before publishing. In one pre-release audit it found a batch of artifacts that would have punished a correct tool, and fixed them before anyone was scored.
Read post →One Score Hides the Gap
A single F-score folds three different abilities into one number, so a buyer never learns which one the tool is bad at. BenchProctor reports a triple and refuses to average it.
Read post →An answer key you cannot grep for
In a single file, anti-leakage means no comments, no CWE tags, no telltale names. When the benchmark is a whole application, the label can leak through the shape of the repo instead. Here is what a polyglot corpus has to hide that a single file never had to.
Read post →When the fix lives in another language
In a compound chain, the safe version and the vulnerable version are identical at the sink. The only difference is a control several files and one language upstream. That is the case you cannot pattern-match, and the reason a new browser tier of cases calls no sanitizer at all.
Read post →How BenchProctor scores a cross-language vulnerability chain
A single confusion matrix cannot describe a finding that crosses files, services, and languages. Here is the three-part score BenchProctor reports for a polyglot scenario, and why it never collapses them into one number.
Read post →The vulnerability a single-service scan cannot see
BenchProctor's standalone corpus scores a tool one file at a time. The new polyglot scenario category scores it on the hardest real case: a vulnerability that exists only as a chain crossing several languages and services.
Read post →Built to Test What It Can Generate
The benchmark deliberately covers a wider language surface than any single scanner, so a tool is graded beyond its home turf. Because the graded artifacts carry no labels and the answer key is held separately, it still cannot be tuned to the test.
Read post →All 11 languages are live: 2,938,418 cases, one public yardstick
BenchProctor release 2026.07.22 is live with 2,938,418 balanced benchmark cases across 11 languages and 21 framework targets. Score compatible SARIF 2.1.0 output.
Read post →An Honest Score With Public Inputs
BenchProctor provides a versioned corpus, public answer key, and scorer for interpreting compatible SARIF results.
Read post →BenchProctor Releases Are Versioned
A score is meaningful only with its scanner and benchmark versions. BenchProctor uses dated, periodic, event-driven releases.
Read post →A Benchmark That Resists Pre-Training
A frozen benchmark can reward memorization. Here is how 2,938,418 balanced cases and versioned releases support a more useful test.
Read post →How to Actually Measure a SAST Tool's False-Positive Rate
Most quoted SAST accuracy numbers have no ground truth behind them, so they cannot be checked. Here is what a defensible false-positive measurement actually requires.
Read post →Reading a SAST scorecard
A single SAST score can hide as much as it shows. Here is how to read recall, false positives, and the traps that make a good tool look bad.
Read post →The 2026.2 corpus: versioned and rotated
A historical note on the 2026.2 corpus and the value of versioned benchmark releases.
Read post →Bring any scanner's SARIF and we'll find the CWE
Scoring against BenchProctor is one standard-library Python file with no dependencies. The catch most benchmarks trip on is that tools don't report CWEs the same way, so the scorer recovers the CWE from wherever your tool actually writes it, with no per-tool adapter, and grades every tool on the same honest footing.
Read post →What makes a SAST test actually hard
A benchmark only means something if a pattern-matcher can't ace it. Here's what's inside a corpus designed to be hard: real framework idioms instead of toy snippets, sanitizers that are present but broken, taint that travels several steps, and a score that subtracts false alarms from catches so 'flag everything' nets zero.
Read post →A wrong answer key punishes the tools that get it right
A SAST benchmark with a mislabeled safe file measures the wrong thing. Here is why public answer keys and transparent scoring matter.
Read post →Measurement, not vibes: the yardstick the whole stack can use
BenchProctor is an open benchmark corpus with a public scorer for compatible SARIF 2.1.0 output.
Read post →Why BenchProctor released one language at a time
A look back at the staged verification plan that led to all 11 standalone languages shipping in release 2026.07.22.
Read post →How BenchProctor scores a SAST tool
The whole scoring model is a confusion matrix and one subtraction. Here's how true-positive and false-positive rates become a single number, why we average per category, and how the benchmark checks itself.
Read post →Why static SAST benchmarks rot, and how versioned rotation helps
A frozen benchmark can measure memorization as much as analysis. Versioned releases help preserve comparability while introducing fresh cases.
Read post →Introducing BenchProctor: a SAST benchmark built not to be gamed
An introduction to BenchProctor's public SAST benchmark corpus and transparent SARIF scoring model.
Read post →