Every File We Ship, Its Compiler Has Already Accepted
At benchmark scale, millions of generated files is a warning sign, not a boast. Nothing enters a BenchProctor release until the real toolchain for its language accepts it, and a suite of deterministic gates, with no AI judge, holds every label.
When a benchmark says it has millions of files, the honest reaction is suspicion, not applause. Auto-generated code that nothing checked is slop, and a benchmark built on slop measures nothing. The scale only means something if every file in it earned its place. So none of ours ships until its own toolchain accepts it.
The toolchain is the gate
Every emitted file is put through the real, native toolchain for its language before it can be part of a release, compiled, type-checked, or syntax-checked depending on the language: the Go compiler, cargo and rustc for Rust, javac and Maven on Java 21, node and tsc in strict mode for JavaScript and TypeScript, php -l, ruby -c, gcc and g++, and bash -n. Not a sample of the corpus. Every file. A file its own compiler rejects never reaches a release.
Deterministic gates, no judge
On top of the compile bar sits a suite of deterministic gates, with no AI and no model asked to judge anything. A vulnerable case must carry a real source-to-sink flow. Its safe twin must actually neutralize that flow for that sink, not merely look like it does. The recorded sink line must match the operation being scored. Dozens of further structural and idiom checks must hold. A gate whose toolchain is missing reports that honestly and, in a release run, fails rather than quietly passing.
Applications get run, not just built
For the application shape, building is not enough. A runtime gate stands representative projects up under Docker and runs them. A project that does not run does not ship.
The scale is the reason the bar matters
The current public release is 2,938,418 labeled cases, exactly balanced, every one cleared through that bar. Producing a release is deliberately expensive: at all-combos and depth 10, a full gate run puts tens of millions of files through the toolchains over roughly thirty hours of compute. Those are ballpark figures, and the exact counts are not known until a release is actually produced. But the expanding standalone surface and the arriving application corpus are expected to take the published total past ten million files, and every one of them clears the same bar before it ships. That is the point of the number: a benchmark worth citing is one whose ground truth compiled, ran, and held, at every file, at any scale. More on the corpus at BenchProctor and the wider stack at Code Reality Labs.