Why BenchProctor released one language at a time
A look back at the staged verification plan that led to all 11 standalone languages shipping in release 2026.07.22.
Update, July 22, 2026: The staged plan is complete. All 11 standalone languages are live in release
2026.07.22, covering 2,938,418 balanced cases across 21 framework targets.
BenchProctor’s first production plan covered nine languages. C and C++ later joined the matrix. We did not release them all at once. Here is why.
”Supports” is not “production-ready”
A generator that emits Rust or Ruby is not the same thing as a corpus we’d stake a tool’s accuracy score on. The whole value of a benchmark is that its answer key is correct. If even a small fraction of the labels are wrong, every number you compute against it is quietly wrong too, and a benchmark that gives false confidence is worse than no benchmark at all.
So before a language goes public, it has to clear a hard bar:
- Compile-clean across the real toolchain for every framework, not just “looks like valid syntax.”
- Release gates. BenchProctor states that labels and scorer consistency are checked before a language ships. The public bundle carries the cases, answer key, scorer, manifest, and hashes, not the internal per-file proof metadata.
- No leakage, per file. No comments, no CWE tags, no naming hints, and file names that give away neither category nor label.
- Idioms a real developer would actually write, framework by framework, not toy snippets that no production codebase resembles.
Until a language clears that bar in public, we hold it back. We’d rather ship one rock-solid language than nine shaky ones.
Where the plan landed
Java and Python shipped first. Go, Rust, PHP, Ruby, JavaScript, TypeScript, Bash, C, and C++ followed after clearing the same standard. The current release covers all 11.
Honesty is the product
It would have been easy to publish a broad benchmark headline and let people discover the rough edges themselves. That’s not the deal. A benchmark earns trust by being right, and being right about Java first is worth more than being approximately right about everything.
Today you can point any SAST tool that emits SARIF 2.1.0 at any of the 11 languages, run the one-file scorer, and get a true-positive rate, a false-positive rate, and a Youden’s J you can recompute by hand.