Benchmark methodology

The figures live on the gauntlet page. This page is the methodology, and it deliberately restates no number — a third copy of figures that already exist in two places is how they drift, and ours had drifted once before.

What the benchmark is for

A review gate’s failure mode is not missing a bug. It is noise: a tool that comments on everything gets muted, and a muted tool is a cost rather than a control.

So the measurement that matters is not “how many findings” but how many repositories got silence. A high finding count on real, well-maintained open-source infrastructure is evidence against us, not for us.

The corpus

Real open-source Terraform, Terragrunt and Crossplane repositories, pinned to exact commit SHAs, run offline with garboard gate — no AI, no keys, no network. Pinning is what makes the numbers reproducible: an unpinned corpus produces a different answer every week and cannot be a regression test.

CloudFormation is a separate corpus, reported separately and never averaged in. It is template libraries whose purpose is to demonstrate every service, so its silence rate is structurally different and combining them would flatter neither number honestly.

Negative controls

Three YAML-heavy repositories that are definitively not Crossplane — Kyverno policies, OPA Gatekeeper’s library, and Argo CD example apps. The harness asserts zero Crossplane findings on each. Any hit is a classifier regression and fails the run outright.

This exists because one of our real false positives was a classifier that swept in anything with a spec.parameters field. Live ammunition catches that; unit fixtures did not.

What the harness forbids

Two rules, and they are the ones that make the benchmark worth reading:

1. Never edit a corpus repository or a negative control to make Garboard pass. The corpus is the fixed thing. If it fails, the product is wrong.
2. A baseline change needs an explained, reviewed product decision. The numbers cannot move because someone re-ran it and liked the new result better.

We publish our own false positives

The gauntlet page lists the false-positive classes the run found — including one we have not fixed, with the reason. A page about our false positives that omitted the one we know about would disprove its own argument, and would be the exact drift this benchmark exists to catch.

What it does not tell you

It measures signal quality on public infrastructure code. It does not tell you how the gate behaves on your repository, whose conventions are derived rather than assumed, and it is not a security-coverage claim against any framework. For coverage, see the rule catalog and the control tags in SARIF output.