Introducing BenchProctor: a SAST benchmark built not to be gamed

Update, July 22, 2026: Release 2026.07.22 contains 2,938,418 balanced standalone cases across 11 languages and 21 framework targets. This post describes the original launch rationale.

A SAST tool is only as trustworthy as its accuracy, and accuracy is meaningless without ground truth. You can run a scanner against your own codebase all day, but unless someone has already labeled every line as vulnerable or safe, you have no idea whether a clean report means clean code or a blind tool.

That is what a benchmark is for: a body of code where the answer is known in advance, so you can ask two precise questions of any tool. Does it find the real bug, and does it stay quiet on code that is actually fine? BenchProctor exists to answer both, for any tool, across eleven languages, with a corpus built so the score can’t be inflated by memorization or naming tricks.

The benchmarks we had were measuring the wrong thing

Most public SAST benchmarks share the same three weaknesses:

• They are hand-authored and frozen. A fixed set of human-written cases gets published once and never changes. Tools, and increasingly the models behind them, overfit to it. A high score starts to mean “has seen this corpus,” not “analyzes code well.”
• They leak the answer in the filename. When a test lives at sqli/BenchmarkTest01729_true_positive.java, a scanner can score well by pattern-matching the path. You are no longer measuring analysis.
• They cover one language, one shape. Single-file, Java-only suites with no sanitizers don’t resemble the findings that matter: taint that crosses files, services, and languages, sitting next to defenses that almost work.

What BenchProctor does differently

Real taint flows, not hand-written snippets. Each vulnerability class is a taint flow: untrusted input enters at a source, moves through a propagator, and reaches a dangerous call, the sink, with a sanitizer being the thing that would make it safe. A vulnerable case is missing an effective sanitizer; its safe twin has one. Every case is idiomatic code constrained to a realistic flow, so a scanner never gets a free win from a nonsense pairing.

Anti-leakage by construction. Emitted test files contain no comments, no CWE tags, no category names, and no hints in their identifiers. A file name tells you nothing about its category or its label. The only ground truth is a separate CSV answer key that the scanner never reads.

Polyglot. Eleven languages (Python, Go, Java, JavaScript, TypeScript, PHP, Ruby, Bash, Rust, C, and C++) across twenty-one framework targets.

Versioned rotation. New releases can use fresh variants while preserving the published scoring contract. Release timing is periodic and event-driven, rather than tied to a quarterly promise.

Transparent scoring. The public bundle ships its cases, answer key, scorer, manifest, and checksums. BenchProctor states that releases pass its gate suite before publication; per-file proof metadata and oracle output remain private.

Running it takes three commands

BenchProctor scores compatible SARIF 2.1.0 output.

# 1. run your scanner, export SARIF 2.1.0
your-tool scan ./corpus --format sarif -o results.sarif

# 2. score against the answer key
python score_sarif.py results.sarif expectedresults-2026.07.22.csv

# 3. read TPR, FPR, and your Youden J

The scorer is a single standard-library Python file. No dependencies, nothing to trust but source you can read in a sitting.

Free, and staying that way

The corpus and the scorer are released under Apache 2.0. No signup, no account, no telemetry. Corpora are versioned and published periodically. Grab the current release, score your tool, and raise a specific issue if the released case and answer key appear to disagree.

This blog documents the methodology, scoring, and release scope. Cross-file, polyglot, and adversarial shapes remain on the public roadmap rather than in the current standalone release.