All 11 languages are live: 2,938,418 cases, one public yardstick

BenchProctor release 2026.07.22 is public now.

It contains 2,938,418 independently labeled benchmark cases: exactly 1,469,209 vulnerable and 1,469,209 safe. All 11 standalone languages are live across 21 framework targets:

• Java, Python, Go, Rust, TypeScript, JavaScript, PHP, Ruby, Bash, C, and C++;
• Spring, Jakarta EE, Flask, Django, FastAPI, Gin, net/http, Actix-web, Axum, Express, Koa, NestJS, Laravel, Symfony, Rails, Sinatra, cpp-httplib, and standalone targets;
• quicktest, normal, and enterprise sizes.

The release represents 242 emitted categories mapped to 231 distinct CWE IDs.

Bring a scanner, not an integration project

For compatible SARIF 2.1.0 output, the bundled dependency-free Python scorer can grade it. The scorer recovers CWE identity from common SARIF fields and reports true-positive rate, false-positive rate, and Youden’s J, both category-averaged and as a flat aggregate.

your-tool scan ./testcode --format sarif -o results.sarif
python score_sarif.py results.sarif expectedresults-2026.07.22.csv

The same public corpus and scorer work for commercial vendors, open-source maintainers, researchers, and buyers. No scanner implementation details need to be shared.

What the number means

Every suite is balanced exactly 50/50, so flagging everything earns a zero rather than a trophy. Files do not expose their category or label through comments, names, or identifiers. BenchProctor states that each published language clears compile-or-parse, source-to-sink, sanitizer, and sink-line gates before shipment. The public bundle does not include the per-file proof metadata.

Cross-file chains, polyglot service scenarios, and adversarial evasion cases remain in active development. They will join the public corpus when their labels clear the same bar, not before.

Get the release from TheAuditorTool/BenchProctor.