What the First Full Release Run Caught
The next BenchProctor release ships when its full gate run is green. Here is what the first full runs over the expanded surface caught, how each was fixed at the source, and where the release stands.
The next BenchProctor release, the expanded standalone corpus and the first application corpus together, ships when its full gate run is green. The first full runs over the expanded surface were not, which is the gate doing exactly what it is for. Here is what it caught and what changed.
Real compilers, real fixes
The new lanes had passed every in-memory and unit check before they met a real compiler, and the compilers caught what those checks could not. The catches clustered where the surface had grown: the new framework lanes and the multi-file shapes. Some Java code paths needed an explicit return, a TypeScript handler’s declared type had to match what it returned, and multi-file Rust scenarios needed module paths that resolve once the files are assembled into one crate.
Every one was fixed where the code is generated, never by patching the output after the fact, and the affected files compile under their real toolchains.
Stricter gates
A gate that reports a partial answer is worse than one that fails, so the gates themselves were tightened in the same pass:
- Every check scans to the end and writes out its full list, rather than a sample.
- Java’s compiler reports every error in a unit, not only the first one it meets.
- The strict TypeScript check runs in bounded batches, so even the largest suite completes in full.
When a gate failed a file, the fix went into the gate only if the gate was wrong about that file. Every other fix went into the generator.
Applications have to run
For applications, the bar moved up too. A project can build cleanly and still fail at its first query because the table it reads does not exist. Generated applications now bring up the data stores their code talks to, a relational database, a document store, and a directory service, populated with the records the code expects. And the runtime check does not stop at “the input reached the dangerous call.” It is built to confirm the query actually ran against the store.
Where it stands
A targeted re-run over the regenerated lanes passes the compile checks and the check that each case’s sink matches its CWE. Next is the full run at release depth, including standing the applications up under Docker. Until that run is green, the current public release remains 2026.07.22, with 2,938,418 labeled cases.
The label work from the same pass is in a label has to match the code under it. The bar itself is described in every file we ship, its compiler has already accepted.