I wanted to disagree with 'AI made me a worse reviewer' from Michael Amachree (@dev_michael) . Instead I counted 204 of my own guards — and 89 % of them have never been asked to prove they can fail.
Michael wrote something that I couldn't put down: AI didn't make me a worse coder, it made me a worse reviewer. Here is the number, and it's worse than his thesis: of the 204 automated checks in my repositories that draw a conclusion, only 22 can prove they are able to fail. That's 11 %. The other 89 % have never once been shown a known-bad input. They are green. Whether they are green because everything is fine, or green because they are incapable of finding anything - I could not have told you last week. And I'm the person who wrote them.
What I actually counted
First the definition, so you can reject it or reuse it.
A conclusion-bearing guard is any test that reads source code, config, or system state and asserts a claim about it. Not "does this function return 4" - but "no workflow downloads its cache over the network", "every page passes the same quarter filter", "this feature flag matches the deployed spec". The tests that stand in for a human reviewer.
A negative control is a probe that feeds that guard a known-bad input and asserts it gets rejected for the expected reason. Our convention marks them KONTROLLE: in the test name.
Counting is mechanical: 204 guard files across three repositories, 22 with at least one control probe, 54 probes total. The counter is a proxy - marker-based, so unmarked controls and false-positive guard files put the true number at plus or minus a few points. The shape survives any correction: most of my reviewers have never been reviewed.
Three green-and-blind checks, one ordinary week
This isn't theoretical. All three of these happened to me in the last seven days, in production tooling.
The deploy gate that died of its own medicine. A pipeline step existed specifically to catch a silent failure mode - a missing tool falling back to an empty result. It called node -e to parse a health response. The deploy runner has no Node. Six consecutive deployments failed with exit 127 - the check against missing tools failed on a missing tool, and nothing shipped for six hours. The step had been green in review because nobody had ever run it where it actually runs.
The harvester that threw away its own work. An autonomous job collected data from public repositories and judged each run by exit code. One run wrote seven perfectly good records, then hit a non-fatal warning and exited non-zero. The machine booked its own completed work as "failed, retry later" - because interrupted-with-partial-results had no representation, only success and failure. We caught it because the result file was sitting on disk right next to the exit code that denied its existence.
The pattern that matched the wrong 500. An error classifier looked for server errors with the pattern 50[024] - anywhere in the output. It matched the "500" inside "4258 of 5000 quota points remaining" and classified a successful run as a server failure. Every field it read was real. It was answering a different question than the one asked.
Three different systems. One shape: the check watched a messenger - an exit code, a pattern, a status - while the artifact that mattered told a different story.
What this has to do with AI making you a worse reviewer
Here's where I think Michael's post lands harder than he says.
AI moved my job. I used to spend most of my day producing artifacts and a little of it verifying them. Now an agent produces most of the artifacts, and my job is verification. Which means my real codebase - the one my judgment actually ships through - is those 204 guards.
And that codebase is held to a standard I would reject in application code. No test coverage (11 %). No review of the reviewer. Green as the default state, silence booked as success.
When Michael says AI made him a worse reviewer, I'd sharpen it: AI promoted us all to reviewers, and none of us tested the reviewer. The model isn't the weak link. The unfalsifiable green checkmark is.
The rule that survived the week
Everything above collapses into one sentence we now apply mechanically:
Judge the artifact, not the messenger.
Exit codes are messengers. Summaries are messengers. The agent's own "done" is a messenger. Green badges are messengers. The artifact is the diff, the file on disk, the served response body, the row in the database. When a messenger and an artifact disagree, the artifact is right - and a check that only ever reads messengers should be treated as unverified, however green it is.
The corollary for guards: a green zero is the most dangerous answer a check can give. "Found no violations" and "is incapable of finding violations" produce identical output. Only a negative control separates them.
Count your own ratio (60 seconds)
This is the part you can use without believing me. Drop this in your repo root - it counts test files that read source or state, and how many carry a marked negative control (adjust the marker to your convention):
// count-controls.mjs — node count-controls.mjs
import { readFileSync, readdirSync, statSync } from "node:fs";
import { join } from "node:path";
const files = [];
(function walk(d) {
for (const n of readdirSync(d)) {
if (n === "node_modules" || n === ".git" || n === "dist") continue;
const p = join(d, n);
statSync(p).isDirectory() ? walk(p) : /\.test\.(t|j)sx?$/.test(n) && files.push(p);
}
})(".");
let guards = 0, withControl = 0, probes = 0;
for (const f of files) {
const t = readFileSync(f, "utf8");
if (!/readFileSync|readdirSync|execSync/.test(t)) continue; // "reads state" proxy
guards++;
const n = (t.match(/KONTROLLE|negative.control|can.?not.?find/gi) ?? []).length;
if (n) withControl++;
probes += n;
}
console.log(`${guards} conclusion-bearing guard files · ${withControl} with a negative control (${guards ? Math.round(100 * withControl / guards) : 0} %) · ${probes} probes`);
If your number is above 30 %, I'd genuinely like to know how you got there - that's the discussion I'm hoping for below.
Where I was the punchline, twice, while writing this
Rule 2 of writing these posts is correcting yourself unprompted, so:
While building the feature this article's data comes from, my equivalence test failed by exactly 0.25 - and the bug was in my test, not the code: min-max spreading turns a column of zeros into a column of 0.5s and adds a constant. I had built a probe that answered a different question than the one asked, in the middle of measuring exactly that failure class.
And one push in that same hour went out with a red test - because npm test | grep replaces the test's exit code with grep's. My pipeline read a messenger. The artifact - the failing test - sat right there.
The person telling you to test your reviewers failed to test his reviewer, twice, in one evening. That's not irony. That's the base rate, and it's why conventions beat discipline.
What this does not prove
One developer, three repositories, one week - this is a case series, not a sample. The 11 % is marker-based and approximate. And I have not shown that raising falsifiability coverage improves outcomes downstream; I've shown that at 11 % I couldn't distinguish my working guards from my decorative ones. Whether the number that matters is 30 % or 80 %, I don't know yet - we're raising ours and measuring as we go.
There's also a fair objection: negative controls are themselves tests that can rot. True. But a control that rots fails loudly the next time the guard changes - that's the asymmetry that makes them worth writing.
So: what's your ratio? And more interesting - what's the greenest check in your pipeline that you now suspect has never been able to fail?
I build cachly — memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks — and which earlier decision it contradicts. Every assistant you use reads the same memory, and every lesson carries the name of whoever learned it — so nobody has to learn it twice.
Free tier, hosted in the EU: cachly.dev
Top comments (77)
AI really said: “Don’t worry, I reviewed it.”
Meanwhile the test: I have never seen a bad input in my life. 😂
That 89% is less a test suite and more a very confident collection of green-colored decorations.
"A very confident collection of green-colored decorations" - there goes my CI dashboard. I'll be seeing a Christmas tree every morning now.
And "I have never seen a bad input in my life" is testimony, not a test result. Most of my suite would say exactly that under oath, with the same straight face.
😂 Exactly. At this point CI isn't continuous integration, it's continuous decoration.
And apparently “I swear I've never failed” is now a valid test strategy. 😭
Continuous decoration is better than mine and I'm taking it.
"I have never failed" as a test strategy also has the nice property of being technically true right up until the moment it isn't.
Ai just tries to write test cases which will always pass.
When you tell any edge test case then it says - Smoking gun - This was the real test case I forgot
The pattern where you name an edge case and it says "ah - that's the real test" is the tell, and it's worth saying out loud: that's not the model finding the case. That's you finding it, and the model agreeing enthusiastically.
Which is fine as long as nobody logs it as "AI-generated test coverage."
the negative-control ratio is useful, but i think it still lets one class of decorative control through: a guard that can fail for the wrong reason. feed malformed input, the parser crashes, the test goes red — technically falsifiable, still not judging the artifact.
the 2x2 control i'd add follows ur rule literally: good artifact / good messenger, bad artifact / good messenger, good artifact / bad messenger, bad / bad. the two off-diagonals prove which side the guard trusts. in the 5000 case, a good result plus a bad-looking messenger should stay green; a bad result plus a clean exit code should go red.
did any of the 22 controls hold the messenger constant while mutating the artifact, or are most proving the whole pipeline can fail somewhere? that split may be harsher than 11%.
Sergei - this is pretty sharp, and it's right. The negative-control ratio proves a guard can go red. It does not prove it went red for the right reason. A parser that crashes on malformed input is falsifiable and useless in the same breath - red, but blind to the artifact.
Your 2×2 is the fix, and the diagonal we actually run is the weak one. good/good→green and bad/bad→red is most of what our controls test - and bad/bad is exactly the ambiguous cell: you can't tell whether the red came from judging the artifact or from the pipeline breaking somewhere. The signal lives in the off-diagonals:
That second cell is the one we were missing. Our own controls mutate the artifact three ways - absent, wrong, and naive-but-plausible - and each must go red while a correct one stays green. Not one of them holds the artifact good and degrades only the messenger. You named a hole in our harness, not just in the 22.
Honest answer to your question: most of the 22 prove "the pipeline can fail somewhere," not "the artifact was the variable." The messenger-constant subset - same harness bytes, only the artifact mutated - is a strict subset of the 11%, so it's smaller by construction. I haven't recomputed the split that way yet; I will, and I'll post the number, because you're right that it's the harsher and more honest denominator.
One refinement I'd want nailed before building the off-diagonal - a real design question, not a quibble: "bad messenger" has to mean degraded, not absent. Noisy stderr, a wrong exit code, reordered output, timing jitter - those are messenger-only, and the guard must stay green through them. But a messenger broken enough that you literally can't read the artifact (the parser crash) is a loss of observability, and red is arguably correct there - abstaining beats guessing. So the off-diagonal only discriminates while the bad messenger still lets a competent guard read the artifact. Where do you draw that line - what's your canonical "degraded but still readable" messenger mutation?
degraded but still readable should preserve the normalized artifact and mutate only the transport. my canonical cell would be correct artifact + exit 1 + noisy stderr, while the parser still reconstructs identical artifact bytes. if the artifact cannot be recovered at all, that is observability loss and red is correct, but it should be a separate failure class from artifact rejection.
The "500 inside 4258 of 5000 quota points" example is the same shape of bug I shipped without realizing it. A deterministic gate on my project was matching "carbon" as a brand name inside the ordinary phrase "carbon copy," declining a completely unrelated question because it watched the string, not what the string meant. You caught it, not a test, because I didn't have one.
The regression suite I added afterward has negative controls now, tests confirming genuine brand mentions still get caught, not just that the false ones stop firing. But it only exists because someone found the bug by hand first, and that someone was you. Your 11% number reads like that's the usual order: incident, then negative control, not the other way round.
Genuinely curious whether your data can actually tell the difference between teams that built the negative control proactively and teams that built it the way I just did.
Honest answer to your genuinely curious question: no - our data cannot tell those two teams apart, and I want to be precise about why. The 11% is a snapshot of the guard population (how many have a negative control TODAY), not a time series. It doesn't see when a control was born or what prompted it. To measure your "usual order" hypothesis you'd need git archaeology: for each negative-control test, compare its commit date against the date of the incident/fix it guards - doable, and now I want to run it, but I haven't.
Anecdotally, on our own codebase the order is almost always yours: incident first, control second. Today alone, twice - a trimming filter of ours would have silently eaten readme-generator.go because the pattern matched "readme", and a golden value in a reference solution was wrong because of a float edge (550 × 1.19 = 654.4999...). Both caught by controls that exist only because we'd been burned into requiring them.
Which is the one structural fix I know for the ordering problem: make the negative control an ADMISSION rule instead of a reaction. In our benchmark harness, no checker is allowed into a run until it has proven all three gates - fails on the unsolved state, passes on the reference solution, fails again on a known-bad mutation. The control exists before any incident can, because without it the check simply doesn't run. Your carbon/carbon-copy guard would have needed a "matches brand, ignores idiom" pair on day one - not because anyone was wise, but because the gate refuses decoration.
The 89% number is brutal and probably understated. I'd argue the same blind-trust bug is now repeating with LLM reviewers: most teams wire every call to one frontier model and call it a day, then never feed it a known-bad input either. The fix isn't a bigger model — it's routing by scenario. The mechanical 80% of reviews (format, obvious violations, "does this match the spec") don't need a frontier model at all; a smaller, cheaper one handles them, and you only spend frontier budget on the 20% that needs real judgment. That gets you the negative-control discipline you're describing and a 70%+ cost drop, because the expensive model is finally used where it can actually fail differently. Green checks that never saw a known-bad input are exactly what scenario routing is meant to stress-test.
The extension to LLM reviewers is the right next domino: a model-based check that never saw a known-bad input is my 89 % with a bigger invoice. One friendly disagreement, though: routing and falsifiability are orthogonal. Routing changes who reviews; a negative control tests whether the reviewer can fail - and a cheap model that never sees a planted violation is exactly as blind as the frontier one, just cheaper per blind spot. So I'd flip the order: build the known-bad corpus first, run it through every tier, and let the measured catch rates set the routing thresholds - not the task taxonomy. That would also test your most interesting claim, "used where it can actually fail differently": do you have per-tier catch rates on planted violations? If the small and the frontier model miss known-bads in different places, that disagreement is itself a routing signal - and that's the number I'd genuinely love to see.
Agreed — and that's the part I hadn't fully separated.Routing and falsifiability are orthogonal axes. Routing answers "send the right model to the right task, stop paying frontier prices for mechanical traffic." Falsifiability answers "has this reviewer ever seen a known-bad input." They don't substitute: a router pushing 70% of calls to a cheap model, paired with a cheap reviewer that never saw a known-bad, just trades your 89% for "89% with a smaller invoice." Root cause untouched.
The complement I'd want: treat known-bad regression as its own routed stream. Normal calls go through normal routing; a small persistent stream of known-bad traffic is pinned to a reviewer channel that runs regression checks. Routing saves the money, the regression channel keeps proving the reviewer still recognizes the boundary. Two axes, two jobs.
(Your line "a model-based check that never saw a known-bad input is my 89% with a bigger invoice" is going straight into the next post's thesis — too good to leave buried.)
The pinned known-bad stream is the right complement - one addition, because there's a third reviewer hiding in your design: the router itself. A misrouted hard call is the new silent failure - "hard, but classified mechanical" produces a cheap answer that looks fine and is quietly wrong, and no per-tier regression stream catches it, because each tier only sees the traffic the router sent it. So the known-bad corpus needs a third slice: inputs that are known-hard-disguised-as-mechanical, pinned through the classifier, scoring its confusion rate. Route the models, regression-test the reviewers, and regression-test the thing that decides who reviews.
The "router is the third reviewer" framing is the part most teams miss. The silent failure is real precisely because each tier only ever sees the traffic the router already decided was its's — so a misroute never surfaces as a tier regression, it just becomes a quietly-wrong cheap answer.
The fix you're pointing at is making the router's own confusion rate visible: pin an adversarial slice (known-hard-disguised-as-mechanical) and replay it through the classifier every release, the same way you'd regression-test a model. Route the models, regression-test the reviewers, and regression-test the thing deciding who reviews — exactly. The only addition I'd make: log the router's confidence on that slice over time, so drift shows up before it reaches a call.
Logging confidence on the pinned slice - agreed, with one sharpening: track it as a calibration curve per release, not a raw average. The dangerous quadrant is confidence flat while the slice's error rate moves - confidently-wrong is the only failure mode that reaches production without a symptom. A per-release curve on the same pinned slice gives you that drift almost for free, since you're replaying it anyway.
Solid sharpening. One addition from the trenches: the pinned slice itself drifts. Replaying it per release is nearly free, but if the slice was labeled against last quarter's traffic, a flat confidence line can hide that the slice no longer represents production — you get false calibration-drift alarms, or worse, silent complacency. We now version the slice next to the model and track "slice freshness" as its own signal, not just the curve. The curve tells you if the reviewer regressed; slice freshness tells you whether you can still trust the curve. Skip either and the per-release discipline slowly rots.
"Struggle ≠ genuinely hard" names a bias I hadn't caught in our own labeling - reviewer frustration as a difficulty proxy. The intersection rule plus publishing the disagreement band is the piece my methodology writeup was missing, and I'll adopt it as written: known-hard = the two-labeler core, the band as its own bucket instead of forced labels.
Two questions on the band in practice: how big does it run for you (share of cases), and does it shrink when you tighten the written rubric - or only when you add labelers? If the band is rubric-sensitive, it doubles as a measure of how teachable your definition is, which would be worth publishing on its own.
"Struggle ≠ genuinely hard" is a sharp catch — reviewer frustration as a difficulty proxy is exactly the trap. And yes, adopt the intersection rule as written; the disagreement band as its own bucket (not forced labels) is the piece I'd have insisted on too.
On the band in practice:
If you write that up, cite yourself — the band-as-teachability-measure is yours, and it's the cleaner idea.
The two-component split is the part I'll take as-is. I think the ordering matters more than either component, though.
If you tighten the rubric first and then add cross-team labelers, the band shrinks and you cannot tell which part shrank. Worse: a sharper rubric can make same-team labelers agree with each other more confidently while staying collectively wrong. Definitional ambiguity falls, the blind spots do not, and the aggregate reads like progress. The only attribution I have found is to run cross-team labeling on the same cases before and after the rubric change, which is expensive and I do not have a cheaper answer.
On the residual, I would carve a third bucket out of your "genuinely irreducible hard core" before publishing it as one number. Some disagreements survive both treatments not because the case is hard, but because it is underdetermined by the evidence the labelers have. Same rubric, different teams, still split - and the reason is that the answer is not recoverable from what is in front of them. The remediation is different again: not re-label, not re-staff, but collect more signal or mark the case undecidable.
I am confident that bucket is real because I built it and it stayed empty. We added an explicit "insufficient evidence, decline to answer" verdict to a retrieval system, with tuned thresholds, then measured how often it actually fired on the benchmark. Zero. Not rarely - zero in both directions: it never wrongly declined, and it never correctly declined either. A bucket that never fills is indistinguishable from a bucket that does not exist, and I had shipped it believing it worked, because nothing had gone wrong.
So if you publish the residual band as a teachability measure, the number I would want printed next to it is how often the undecidable bucket gets used. If that is zero, the residual is not measuring irreducible difficulty - it is measuring everything you have not separated yet.
I will write it up, but not as mine. The one-line version was a hunch with nothing under it. What makes it publishable is your split - definitional ambiguity and same-team blind spots as separate populations with separate remediations - and the residual-after-both is your framing, not mine. If it goes out, it goes out with your name on it too.
The third bucket is the part I'd have pushed for too, and "underdetermined by available evidence" is the sharper name than "irreducibly hard." It reframes the residual as an epistemic limit you can measure, not a difficulty you can't. That's a cleaner primitive than a single teachability number.
Your zero-firing decline-to-answer story is the same shape as the empty-world gate on the routing thread, in a different domain: a control that never triggers is not evidence of absence. You shipped it believing it worked because nothing went wrong — and "nothing went wrong" was exactly the signal that was blind. The bucket that never fills is indistinguishable from the bucket that doesn't exist. Stating "print how often the undecidable bucket gets used next to the residual" is the honest version of what most teams skip.
On ordering: your rubric-first trap is why we insist the disagreement band stays its own bucket rather than forcing labels. If you collapse it, you lose the ability to see the band before and after — and then you can't attribute which component moved, which is your exact complaint. The cheaper attribution you wanted might be a single-rubric design where two labeler cohorts are measured separately on the same cases; you still get the band, and the cohort split is observable without a before/after study. Not free, but less than re-labeling twice.
One caveat on the third bucket: marking a case "undecidable" is itself a verdict that can be wrong in two directions — you can give up on a case that more signal would have resolved, or you can keep grinding past the point where more signal changes nothing. The remediation ("collect more signal or mark undecidable") hides a threshold. So the question I'd actually want answered before this goes out:
How do you decide when to stop collecting signal and mark a case undecidable — is there a cost threshold, or is it a per-case judgment call? And when you do mark it, does that mark ever get revisited, or does "undecidable" become permanent the way "resolved" can?
On the byline — it goes out as a conversation, not a claim. The split is mine, the third bucket is yours, and the residual-after-both is ours. That's the honest attribution, and it's the part worth publishing.
Both directions is exactly right, and it's the part I hadn't thought hard enough about. "Undecidable" is a verdict, and a verdict that can only be entered and never revisited turns into a landfill.
My honest answer: not a cost threshold. A threshold implies the cost of more signal is the thing that varies, and usually it isn't - what varies is whether more signal would change the label. So the test I'd want is closer to: if we collected the next obvious piece of evidence, name in advance what verdict it would produce. If you can't name it, more signal isn't going to help and you should mark it. If you can, go get it - you've already done the reasoning, the collection is just bookkeeping.
That has a nice side effect: the mark carries its own reopening condition. "Undecidable given X" is revisitable the moment X exists. "Undecidable" alone isn't, and that's the version that becomes permanent.
Which makes me want to invert your question back: should the undecidable bucket be allowed to hold anything with no named missing evidence at all? I'm starting to think that entry is really "unexamined" wearing a nicer word.
And yes - conversation, not claim. That attribution is right and I'm glad you spelled it out
The 89% figure stuck with me because I keep seeing the same shape with coding agents. The agent says done, the exit code is zero, and the only thing that catches the lie is opening the file it claimed to edit. Your rule about judging the artifact, not the messenger, is the whole job now. I treat any green check that has never been fed a known bad input as unverified, same as an untested function. Curious how many of those 22 controls were added after a production miss versus written up front.
Honest answer: I can't tell you yet, and the reason is itself part of the finding. The marker count is a snapshot - it knows WHICH guards have controls today, not WHEN or WHY each was born. Anecdotally, every one I can date was incident-born, including two this week (a filter that would have silently eaten readme-generator.go, and a golden value that was wrong because of a float edge). Your question - same one Daniel Nwaneri asked an hour before you, independently - just became a measurement on our board: git archaeology, dating each control's introducing commit against the fix it guards, "unclear" reported as unclear. I'll ping this thread with the split when it's run. The one structure we've found that flips the order: admission gates - no check enters our benchmark harness until it has already failed on a known-bad. There, the control exists before any incident can.
You've already granted half of this further down the thread — the
.get()case, where the guard reads absence and there was never a bad input to construct. The other half is the one I keep running into, and I don't think it has a fix.The admission rule needs three things: fails on the unsolved state, passes on the reference, fails on a known-bad mutation. That works when you own the thing you're mutating. A lot of our guards sit over a boundary someone else owns. The known-bad input is something a bank does — a settlement that reverses, a duplicate arriving with a different reference format, a field that quietly changes shape without an announcement. I can't manufacture any of those on demand. So the control gets built from recorded traffic instead of synthesised input, and recorded traffic only contains the failures we've already survived.
Which drops me straight back into your ordering problem with no structural fix. The guard gets admitted on the strength of a mutation I could only construct because the thing had already happened to us once.
The green count is the part I'd put on a wall. Our version of it is a reconciliation job reporting zero mismatches. Zero is also what it reports when the date window is off by one and it compared an empty set against an empty set. Same output, opposite meaning, and the second one turns into a regulatory conversation rather than a ticket.
So the question back: for a guard sitting over an external system you can't mutate, does anything in your scheme still work? Or is "replay recorded bad traffic and accept that you only cover what already hurt you" the honest ceiling?
Your question deserves a straight answer: no, the scheme doesn't fully survive crossing into a system you can't mutate - but the ceiling sits higher than "replay what already hurt you," and the gap between the two is buildable. Three pieces. First: recorded traffic gives you instances; the known-bad you need is a broken invariant. You can plant the break yourself - take a recorded settlement and flip the sign, permute the reference format, delete the field the guard supposedly watches. The bank never sent it, but your guard must still catch it, because what you're testing is your detector, not their generator. That covers whole families of shape-drift you haven't met yet, not just the ones you survived. Second, for the quiet format change: a per-field format histogram with an alarm on first-seen shapes - a cousin of the eligible_seen idea from the other thread - turns "changed without announcement" from a post-incident discovery into a same-day signal. Third, your reconciliation zero: make zero illegal on its own. The job's output should be "0 mismatches across N compared," where N==0 is UNKNOWN, never PASS - and once a week a deliberately shifted record from your own side must show up as a mismatch, through the real comparison path. If the planted mismatch goes quiet, the window broke, and it fails in the direction someone investigates.
What I'll grant completely: for failure classes you cannot imagine, the ordering problem stands. When we audited our own controls' birth dates, most were born after an incident, not before. The honest claim for the admission gate was never "no first blood" - it's "no second blood from the same wound, and no guard admitted on faith." Over an external boundary, that's still the difference between a test suite and a scrapbook.
The 89% number is brutal and I recognize it. I ran a smaller version of this audit after a "green" CI gate let a broken migration through — out of 40-ish repo guards, exactly 3 had ever been fed a deliberately broken input. The rest were tautologies wearing a badge.
The
KONTROLLE:naming convention is the part I'm stealing. Marker-based counting is the only way this stays honest as the suite grows; if the control probe is optional-and-unmarked, it silently stops being written the first time someone is in a hurry.One thing I'd add to the taxonomy: guards that can fail but only on inputs that no longer occur. I had a lint rule rejecting a config format we deprecated 8 months ago — technically testable, practically dead. Did you count those as "able to fail" in your 22, or did you filter for probes tied to a currently-live failure mode?
Curious how you handle the pushback when a negative control itself becomes the brittle part of the suite — probing a guard against a known-bad input that's too known-bad (nobody would ever actually write it) feels like testing the wrong thing. Where did you draw that line?
3 of 40 - thank you for counting before commenting; may I add your 92.5% next to our 89% when I write the follow-up? Your two questions, honestly:
Dead guards: you caught a real gap. Our 22 counted marker PRESENCE - "this guard has been fed a known-bad" - not liveness. A control probing a config format retired 8 months ago would have counted. Your case is now the third dimension in our counting scheme: can it fail / against a failure mode that still occurs / guarding a boundary that still exists. (Marco added the time-axis version in a sibling thread: regenerate the bad state from the CURRENT system, and assert the boundary is still present.)
Where we draw the too-known-bad line: the known-bad must be the mistake a hurried human or agent would actually make, not a constructed absurdity. In practice we take it from incident history or from the most plausible reflex - Math.round instead of merchant rounding, forgetting the second mandatory file, dropping the sort. And we pair it with a near-miss known-GOOD (something that looks like the violation but isn't) - that pair is what keeps the control honest in both directions; ours caught an over-eager filter this week exactly that way.
The "admission gate" idea further up (a checker can't run until it's proven it fails on a known-bad, passes on the reference solution, and fails again after the mistake is re-planted) is the strongest fix in this thread, and it generalizes cleanly to security gates specifically — with one extra trap worth naming. A security-relevant negative control is itself vulnerable to testing the wrong layer: it's easy to write a KONTROLLE probe that feeds a known-bad string into the detection logic (a SQL literal, a header, a synthetic payload) and call that a negative control, when the actual enforcement boundary — the RLS policy, the IAM scope, the API auth check — never gets exercised at all. That control turns green the moment your regex or SAST rule matches, which tells you nothing about whether the underlying permission boundary would actually have stopped the real thing. The fix is the same shape as your admission gate, one layer down: the known-bad input for a security guard has to travel through the real enforcement path, not a mock of it.
The trap you name is real and I can bring a scar as a second specimen. We had exactly this with Postgres row-level security: the negative control fed a forbidden query through the test suite and went red as expected - but the suite connected as the table owner, and Postgres lets the owner walk past RLS. The control was exercising the SQL, not the boundary. The fix wasn't a better payload; it was a worse identity: the known-bad has to run as a role the enforcement actually applies to, through the production connection path, or it proves nothing. Your one-layer-down version of the admission gate is going into our rules verbatim: for security guards, the known-bad must travel the real enforcement path, and the test principal must be one the boundary is supposed to stop.
The open end I don't have a clean answer for: how do you prove the path was the real one? A regex control turns green when the regex matches; a boundary control turns red when the boundary blocks - but from the test's point of view both are just an assertion failing somewhere. The best I've got is a marker that is only observable from behind the boundary (a row you can only see if RLS let you through, a header only the real gateway strips). Curious whether you've standardized something like that, or whether "same connection string as production" is where you draw the line.
A self-inserted marker still has to be trusted, since the test wrote it too. The more structural version: don't mark anything — read a value that only exists because the boundary let you through, not one you put there. For RLS specifically, that's checking current_user against the exact role production traffic uses, plus rolbypassrls, plus table ownership combined with relforcerowsecurity — because any one of those three can silently make "connected, reading rows, policy says allow" true while RLS was never evaluated at all. Assert on the thing that determines whether the boundary applies, not on the output the boundary would have blocked. Same principle probably generalizes past Postgres: find the precondition that has to be true for the enforcement layer to even be in the loop, and gate the negative control on that precondition being real, not just on the request failing.
The 11 percent number would sting less if I did not immediately recognize all three of your failure shapes from my own repos, especially the classifier matching the "500" inside "5000 quota points". The KONTROLLE: marker convention is a nice forcing function, because right now most of my negative controls live as tribal knowledge in whoever last touched the guard. I am stealing the marker idea and running the count on my checks this week.
Stealing the marker is its entire purpose - and "negative controls living as tribal knowledge in whoever last touched the guard" is a better one-line justification for it than anything in my article. One warning before you run your count, from a mistake that cost us: make sure your counter reads code, not comments. When we first counted, 13 guards showed as "has a control" because the promised assertion existed only in a comment - the counter matched the string, exactly the failure shape you just recognized in the classifier. Strip comments first, then count. And please post your number when you have it - we're at 89% (ours) and 92.5% (another reader's 37/40), and I'd love to add yours to what's becoming an accidental community measurement.
This is the gap we're building Mneme to close: giving the reviewer something deterministic to check against instead of just judgment and fatigue. Promoting everyone to reviewer only works if the review has actual teeth.
"Review with actual teeth" is the right target, and deterministic beats judgment-and-fatigue every time it's available. The question that decides whether teeth are real, though, is one layer down: what does Mneme's check refuse, and when did it last refuse something in production? We've started surfacing exactly that as a visible timestamp - "last refusal: N days ago" - because a reviewer that never says no is indistinguishable from a reviewer that stopped looking, and both wear the same green badge. If your deterministic layer can answer that question on a dashboard, you've closed the gap you're describing. Genuinely curious what it refuses today.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.