A Benchmark Surface You Can Audit
BenchProctor's generation surface is now frozen and published: every lane, every substrate, every pinnable library, with counts a build refuses to let drift. You can see exactly what the benchmark can cover before you trust a score from it.
A benchmark score is only as trustworthy as your knowledge of what the benchmark can and cannot cover. If the surface is a moving target you take on faith, the number is too. So we froze ours and wrote it down.
The whole surface, in one inventory
The complete menu of what the corpus generates is now a single published catalogue: eleven general-purpose languages plus a Solidity emitter, eighty-five requestable lanes (sixty-four code-framework lanes and twenty-one infrastructure substrates), and one hundred eighteen pinnable libraries. Not a marketing round number. A named list, lane by lane and library by library, with the role each lane plays and the dependency each library brings in.
Frozen means it cannot drift
The three sets, the lanes, the substrates, and the libraries, are pinned, and a test holds the published inventory against the live generator in both directions. Add, remove, or rename any lane and the build fails until the inventory and its count are amended together, in one deliberate change. The set a reader sees and the number a reader is given can never quietly disagree, because the system will not let them. It is the same discipline TheAuditor holds its framework registry to: a count you can hold us to because the build holds us to it first.
Absence on the record
A frozen surface also lets us be honest about what is deliberately left out. Test frameworks are not generated: this corpus plants weaknesses in production code, and a test suite is not that. A short list of other lanes is excluded because they would have to be padded to exist here, and a padded lane is a mislabelled one. Those absences are decisions on the record, not gaps we are hoping you do not notice.
Why a buyer should care
You can look at the surface before you run anything and know exactly what a score does and does not speak to. You can pin an evaluation to a version of it and rerun the same measurement later. And when the surface grows, it grows on the record, in one reviewed change, not by quiet accretion. A benchmark you can audit is the only kind worth citing. More on how the corpus is scored and rotated across the BenchProctor blog, and how the stack fits together at Code Reality Labs.