A 403 Is Not Proof: What Building a CVE Toolkit Taught Me About Evidence

Your load balancer returned a 403. Did the request still reach the origin?

I have asked some version of that question for years, usually during an incident, which is the worst possible time to discover the answer. Plenty of edge rules respond with a rejection after forwarding the request anyway. The status code says blocked. The origin logs say otherwise. Both are telling the truth about the layer they can see, and only one of them is telling the truth about your security.

This fall I built a toolkit in MockForge to answer that question in a lab instead of an incident. It started the way my favorite features usually start: a user asking hard, specific questions about whether a mocking tool could help prove a WAF rule actually works. Three releases later it is rule packs, a remediation feed, a monitor, and managed labs that reproduce a real CVE against a real vulnerable application. This is what building it taught me.

🔗Lesson 1: Judge at the layer where the truth lives

Every layer of a system reports its own success. The edge reports a status code. The load balancer reports a routing decision. Neither one knows whether the request arrived at the application, because that is not their job. Security testing that reads the edge's report and calls it a verdict is grading the student's own homework.

So the toolkit's core mechanic is a recorder that sits where the origin would be and logs what actually arrives. A pass is not "the edge said 403." A pass is "the edge said 403, and nothing arrived, and the benign requests did." And the verdict vocabulary includes the uncomfortable middle: inconclusive, for the runs where the evidence cannot establish blocking. A test that cannot prove a negative should say so instead of guessing. I now think that word is the most important output the tool produces.

🔗Lesson 2: Safety belongs in the schema, not in the policy document

A CVE testing tool walks a line. Build it wrong and you have shipped an exploit kit with your logo on it. I did not want that line held by documentation, good intentions, or a terms-of-service page. I wanted it held by physics.

So a rule pack profile can only contain inert markers: header and parameter names, with values restricted to letters, digits, and a few punctuation characters. There is no request body field at all. A profile cannot carry a command, a template expression, or a traversal payload, not because we forbid it, but because the format has nowhere to put it. When we later let an LLM draft candidate signatures from advisories, the same schema became the trust boundary: advisory text is untrusted input, drafts are rebuilt and validated, and anything questionable lands as needs-review, never runnable. A prompt-injected draft cannot produce a payload because the schema admits nothing payload-shaped.

The general principle travels well beyond this tool: if a safety property matters, encode it in the data model. Policies get skipped under deadline pressure. Schemas do not care about your deadline.

🔗Lesson 3: Honest tools say what they cannot prove

The toolkit does not prove an application is patched. It does not establish protection against every variant of a CVE. When a data feed is unreachable, the remediation output says null, not false, because "we could not check" and "it is not exploited" are different claims and merging them is how dashboards lie. Those limits are printed in the guides, in the same size type as the features.

I believe this is a competitive advantage, not a concession. Security tooling runs on trust, and trust compounds when a tool's claims survive contact with a skeptical reader. The teams who will rely on a verdict at 2 AM are the teams who read the limitations section first.

🔗The short version

A 403 is a claim. Claims are cheap at every layer of every system I have ever operated. The work, in security and in shipping generally, is building the recorder: the thing that sits where the truth lives and tells you what actually arrived.

The toolkit is open source, the guides are in the MockForge book, and the managed labs shipped this week in 0.3.240. If you run rules at an edge anywhere, I would genuinely like to know what your recorder says.