What makes a SAST test actually hard
A benchmark only means something if a pattern-matcher can't ace it. Here's what's inside a corpus designed to be hard: real framework idioms instead of toy snippets, sanitizers that are present but broken, taint that travels several steps, and a strict 50/50 split that makes 'flag everything' score zero.
Most security benchmarks are easy to beat for the wrong reason: the vulnerable cases look
vulnerable. Take a bare eval() on obvious user input, or a string-concatenated query three lines
from the request. A regex can “find” those, post an impressive number, and tell you nothing about
whether the tool understands code. A benchmark is only worth running if a pattern-matcher can’t
ace it.
So before BenchProctor’s first language went out, the design goal was the opposite of easy. Here’s what makes a test genuinely hard.
Real frameworks, not toy snippets
The vulnerable and safe cases are written the way real services are written: through framework request paths, ORM and database-driver calls, the idioms of Spring, Django, Flask, and FastAPI. A scanner can’t lean on a tiny library of textbook sink shapes, because the code looks like code people actually ship, not like a tutorial.
Sanitizers that are present but broken
This is the one that separates analyzers from matchers. Plenty of the safe-looking cases contain a sanitizer, and plenty of the vulnerable ones do too, except the sanitizer is wrong for the context, or bypassable, or applied where it can’t help. A tool whose rule is “I saw a sanitizer, must be safe” walks straight into the trap. Only a tool that reasons about whether the protection actually holds gets these right.
Taint that travels
The dangerous value rarely sits next to the dangerous call. It moves through a collection, a round-trip, a conditional, or a couple of reassignments before it reaches the sink. Tests that make taint travel several steps reward tools that actually follow data flow and quietly punish the ones that only look one line deep.
A 50/50 split that kills the shortcut
Every category is balanced exactly half vulnerable, half safe. That single constraint defeats the oldest trick in benchmarking: a tool that flags everything looks like a perfect detector on a corpus that’s mostly vulnerable. Here, flag-everything scores like a coin flip, because for every real bug there’s a safe twin built to look just like it.
Why we build it this way
The whole point of a benchmark is that the answer is known in advance and can’t be faked. Making the tests hard, with real code, broken sanitizers, traveling taint, and balanced classes, is how the number ends up measuring analysis instead of pattern-matching. An easy benchmark flatters tools. This one grades them.
Where it stands
BenchProctor’s corpus is open (Apache), and the scoring is designed so any tool’s results can be checked against an answer key it never sees. It’s the independent yardstick the rest of the stack answers to: TheAuditor, Warden, Curator, and Arbiter. Read how BenchProctor scores a SAST tool or follow the RSS feed.