Compose the Benchmark to Your Stack
A benchmark locked to a fixed set of language and framework combinations measures a scanner on someone else's stack. BenchProctor's corpus is now composed, not canned: request any lane, pin the exact library, or specify a whole multi-service graph.
Most SAST benchmarks ship a fixed menu of combinations. Python with Django and Postgres. A Node service with Express. If that is not what you run, the score tells you how a scanner does on someone else’s system, not yours. Until recently ours had the same limit: a lane was a language welded to a framework and whatever that pairing happened to pull in. You took the combinations we shipped or you went without.
That is gone. The corpus is composed now, not canned.
Three ways to ask for a system
You can request the exact shape you need to measure, from broad to precise:
- A lane by name. Any code-framework lane stands alone as a suite: a Go
ginservice, a Rustaxumservice, a PHPsymfonyapp, a Javaspringapp. Sixty-four code-framework lanes across eleven languages, each requestable on its own. - A specific library pinned onto a lane. “Just Kafka” is not a vague ask anymore. It is the real client for that language:
kafka-pythonorconfluent-kafkain Python,saramaorkafka-goin Go,kafkajsin Node,kafka-clientsin Java. “Just AWS” isboto3,aws-sdk-js,aws-sdk-go-v2,awssdk, or the SDK for whichever language you name. One hundred eighteen libraries, each pinnable by name. - An exact multi-service graph. Name each node’s language, framework, and pinned library, the boundary between the nodes, and the infrastructure they deploy onto. “Three Kafka producers and one consumer that writes to AWS, over an HTTP boundary, on a Terraform substrate” returns exactly that, not the closest thing we had lying around.
Why this is the point, not a convenience
A scanner’s real test is the seam between services: the boundary where a language changes, a message queue sits, or a cloud call happens. A benchmark that only offers a handful of prebuilt combinations can never put a scanner in front of your seam. Composition means the corpus can mirror the system you actually ship and measure a tool on that, rather than on a stand-in.
The honest edges
Two things this does not mean:
- It is a frozen menu, composed freely, not arbitrary code. The surface is a published inventory: eighty-five requestable lanes, one hundred eighteen pinnable libraries, twenty-one infrastructure substrates. You combine those however you like. You cannot ask for something off the menu and have it invented on the spot.
- A pin has to exist for the language you put it on. A Kafka client is there for Python, JavaScript, Go, and Java; an ORM pin only applies where that language has an ORM. Ask for a pin a lane does not carry and the request is refused by name, not silently dropped, so you never get a suite that quietly ignored what you asked for.
We generate all of this with our own generator, which stays proprietary. What is public is the result and the menu it draws from. New lanes clear a compile gate before they count toward a scored release, and the newest additions are working through that gate now. The composition surface itself is frozen and published, which is the subject of a companion post. See how the pieces fit at TheAuditor and Code Reality Labs.