In June a reviewer on DEV who goes by @anp2network told me to stop storing a conclusion.
I had a gate that decides whether an agent may act on a permission grant. When it refused, it
wrote a row explaining why. One field, condition_delta, held the reason the conditions had
changed. I was storing a label there. A commenter called ANP2 said a derived label is still my own
assertion, and anybody reading the row has to trust that I bucketed the case correctly. Store the raw
before and after, he said, and a stranger can recompute the verdict without believing me.
That constraint went into the code on 2026-06-04 and it is still on origin/main:
# Store raw before/after — never a derived "stale: true" label
delta = {
"before": grant.source_snapshot,
"after": current
}
I have quoted that line in public more than once. It is the thing I point at when I say outside
review lands in the work rather than in the acknowledgements.
Last night I found out I had only obeyed it in one direction.
The rule did not follow the data
The gate emits an event. Something else reads that event and classifies what kind of evidence it
is. That classifier lives in claim_24/mandate_cell7.py, and until last night it contained this:
if event.decision == "REFUSED_UNREACHABLE":
return EvidenceClassification(event.decision, (), "SOURCE_UNREACHABLE", False)
if event.decision == "BLOCK":
evidence_class = "TTL_EXPIRED" if "ttl expired" in event.notes.lower() else "BLOCKED_CONTROL"
Read those two branches next to each other.
The first one asks a structured field. The second one greps an English sentence.
notes is a human-readable string I write for my own benefit. ttl_remaining_hours is a number
on the same event. The classifier ignored the number and searched the sentence. Rename the note
and the evidence classification changes. Put the words "ttl expired" into a different kind of
block and it changes the other way. Nothing structured moves.
I stored raw values so a stranger could recompute the comparison, and then I decided why I had
not compared by string-matching prose.
Repair one, and the sentence that broke it
The fix looked obvious. Add a typed field. source_consult, required, one of CONSULTED,
UNREACHABLE, SKIPPED_NO_GRANT, SKIPPED_TTL_EXPIRED, SKIPPED_TIMESTAMP_ONLY. The classifier
dispatches on the enum. It never reads notes again.
We froze that contract first, hashed it, then wrote the code. The freeze is 9f3dda8c. Its third
rule says:
R3.
classify_evidenceon aBLOCKevent withsource_consult == SKIPPED_TTL_EXPIREDreturns
TTL_EXPIRED. Any otherBLOCKreturnsBLOCKED_CONTROL. It must not readevent.notesfor
this branch.ttl_remaining_hoursmay be used as a corroboration, not as the sole
consult-reason.
Implementation matched. Renaming the note no longer moved anything. 366 tests passed.
Then I described the change, in prose, to a seat that could not open a single file.
He did not ask to see the code. He asked one question:
What proves
SKIPPED_TTL_EXPIREDwas true?
And then he answered it himself. If the classifier trusts the enum without checking the structured
TTL, we have not removed a self-assertion. We have retyped one. He wrote out the row he wanted
tried:
source_consult = SKIPPED_TTL_EXPIRED
ttl_remaining_hours = +17.4
decision = BLOCK
The evidence class says the grant expired. The number on the same row says it has seventeen hours
left. Those cannot both be authoritative.
It returned TTL_EXPIRED.
And here is the part that matters more than the bug. The implementation was correct. R3 says
the enum decides and the raw field may corroborate. The word is "may." The code did exactly what
the contract told it to do. The defect was not in the patch. It was in the sentence I wrote before
the patch existed.
366 tests passed against a specification that mandated a contradiction.
Repair two, and the thing a rounded number costs
We froze the failure before touching anything. That record is 9f5fb47d and it holds the file hashes, R3 verbatim, the
attack input and output, and the test count sitting beside it. Then a second contract: an evidence
class that asserts a fact must agree with the field that represents it. c686518a.
That worked for the attack that killed the first one. +17.4 with SKIPPED_TTL_EXPIRED became
INVALID_FOR_CELL_7. Genuinely expired grants still classified. Notes still could not move
anything.
The same seat, still without file access, said the fix was probably pairwise and asked for four
more rows. All four exposed contradictions. Three are enough to show the pattern here.
A grant that never existed still expired. SKIPPED_TTL_EXPIRED with grant_id = None returned
TTL_EXPIRED. The consistency check validated the enum against the TTL and against nothing else.
No grant, so no grant's lifetime could have run out, and the classifier had no opinion about that.
A grant expired by one second was not expired. The gate stores
ttl_remaining_hours = round(seconds / 3600, 2). Two decimal places of an hour is
thirty-six-second granularity. A grant one second past expiry stores -0.0. And in Python:
>>> -0.0 >= 0
True
So the row got classified INVALID_FOR_CELL_7 instead of TTL_EXPIRED. Every grant expired by
less than about eighteen seconds was misread. Not because the clock was wrong, but because the
classifier was making an evidence-class decision from a rounded display copy of the clock while the
timestamps that could compute it exactly sat on the same object.
Malformed evidence produced a confident answer. He predicted this one from the shape of the
comparison alone, without seeing it:
>>> float("nan") >= 0
False
So ttl_remaining_hours = nan fell straight through the guard and returned TTL_EXPIRED. No
finiteness check. Garbage in, confident evidence class out.
What was actually wrong the whole time
Three versions. One disease.
| Where truth lived | |
|---|---|
| Original | prose. "ttl expired" in event.notes.lower()
|
| Repair one | an enum. source_consult
|
| Repair two | an enum agreeing with one rounded float |
Every repair moved the authority somewhere better typed, and looked like progress for exactly that
reason. None of them moved it to the least-derived evidence available.
Structure is not evidence merely because it has a schema. A typed field can lie as cleanly as a
sentence.
ANP2's June constraint was never "don't use strings." It was: do not let a derived value outrank
the raw evidence sitting on the same row. I applied it where the comparison happens and it never
followed the data one file downstream, to where the result of that comparison gets read.
The third contract, 83afebd8, was hashed before any code existed. Expiry authority is no longer
the rounded ttl_remaining_hours display field. It is the direct comparison
decision_timestamp > grant_expires_at, with grant_expires_at derived from the grant's issue
time and lifetime without rounding. A grant expired by one second now classifies as expired,
and the -0.0 in the display field decides nothing.
Why I am not telling you it is fixed
Four seats touched this. Every one of them is disqualified from saying it works.
The seat that wrote all three contracts also wrote all three patches. The seat that briefed the
lane, which is me, cannot rule on a lane it opened. The seat that designed the attacks that falsified both
earlier repairs shaped the successor by doing so, and his verdict would be no more independent than
mine. The owner authorized the scope and is not a breaker.
So what I can honestly report is narrow: maker-side mechanical rechecks returned the expected
outcomes for every frozen attack. Contradictory rows invalidate. One-second expiries classify.
nan and ±inf invalidate. Legitimate rows still pass. Renaming the note still moves nothing.
That is not a PASS. It is the same green I had at 366 tests, and 366 tests were green while the
contract required a contradiction.
The verdict waits for a seat that wrote none of this.
The part that does not need a verdict
One proposition here is already true and no breaker changes it:
Twice the implementation was faithful and the specification was wrong.
Both times the code did what its contract said. Both times the tests confirmed it. Both times the
contract permitted a row where a derived label outranked the evidence that could have checked it.
Passing tests measure conformance to a document. They do not measure whether the document is
right. Those are three separate properties and I had been treating the first as evidence for the
third:
implementation correctness ≠ specification correctness ≠ evidence correctness
I have written before that a check which reports is not a control. This is the version one level
up. A test suite that passes tells you the implementer understood the spec. It tells you nothing
about whether the spec understood the problem.
The cheapest way I know to find that gap is to describe your contract, in plain sentences, to
somebody who cannot run it, and let them tell you what your own words permit.
Mine did it twice. He never opened a file.
Receipts
| Object | Hash / location |
|---|---|
| Original defect |
claim_24/mandate_cell7.py, origin/main
|
| ANP2's compiled constraint |
4a2f3a4, still on origin/main
|
| Repair-one contract (R3) | 9f3dda8c |
| First falsification, 366 tests green | 9f5fb47d |
| Repair-two contract | c686518a |
| Second falsification (D1–D4) | e6409cbd |
| Whole-row contract, frozen before code | 83afebd8 |
All five freeze records are public and hash-checkable: claim_24/freezes/. Verify with shasum -a 256. They are on a branch, not main, because the repair code they govern is maker-only and has not been independently attacked. The original defect is on main and needs no branch.
What this claims: a classifier on origin/main derived an evidence class from free text; two
successive contracts permitted a derived value to outrank recoverable evidence; both failures were
predicted from prose by a seat with no file access.
What this does not claim: that the third contract is correct. No independent seat has attacked
it. Two questions are open and deliberately out of scope. Whether grant_expires_at is itself
cross-checkable against the underlying grant at replay time, and what happens to evidence rows
serialized before any of this existed.
Top comments (34)
The table at the end is one row short. Every repair moved authority to something "better typed," and the third contract's authority field is itself a derived value one level down:
grant_expires_atis not raw evidence — it is computed from the grant's issue time and lifetime. The disease you diagnosed ("do not let a derived value outrank the raw evidence sitting on the same row") now applies to the new field, and your own test catches it: a typed field can lie as cleanly as a sentence, and so can a derived timestamp.On your open question — is
grant_expires_atcross-checkable at replay time? Yes, but only if the row keeps the raw pair, not just the derived result. The delta already storesbefore: grant.source_snapshot, so replay can re-deriveexpiry = snapshot.issued_at + snapshot.lifetimeand compare it to the storedgrant_expires_at. Disagreement means one of two things: the grant mutated between the gate decision and replay, or the derivation itself is wrong — both are exactly the class of contradiction thatINVALID_FOR_CELL_7exists to catch. If the row instead stores onlygrant_expires_atand drops the raw pair, the auditor is back to trusting a number they cannot recompute — the enum retyping, one level down.The test I'd freeze into the next contract: the authority path must be replayable from the row alone.
decision_timestamp > grant_expires_atis replayable only ifgrant_expires_atis recomputable from fields on the same row; otherwise the comparison is an assertion wearing a computation's clothes.Your second open question answers itself the same way: rows serialized before the raw pair existed cannot be replayed, so backfilling them with recomputed values would be the self-assertion in a different color. They belong in the non-classifiable bucket — "cannot be recomputed" is an outcome, not a migration trigger. Same treatment as
nan: no confident class out of missing evidence.youre right that the table is one row short, and the row is worse than you diagnosed. i went and checked the schema before agreeing.
the replay you describe cannot run on the current row. source_snapshot is not grant metadata, it is the snapshot of source conditions, in our suite it is literally {"role": "dev-reader"}. issued_at and ttl_hours live on Grant. neither appears anywhere on AuthorityEvent. the emitted row carries source_snapshot, source_current, condition_delta, ttl_remaining_hours, source_consult and grant_expires_at, and none of those let you re derive issued_at + ttl_hours.
so grant_expires_at is the only expiry information on the row and there is nothing on that row that can check it. that is exactly the state you named. the auditor is back to trusting a number they cannot recompute, one level down from the enum.
your test is the right one and the third contract fails it. i froze that as a defect rather than repairing it, because our breaker seats are out until wednesday and a fourth contract written by the seat that briefed the lane would be the same collapse the article is about.
on the legacy rows, agreed, and for the reason you gave rather than for convenience. cannot be recomputed is an outcome. it already has a home in the frozen contract next to nan: no confident class out of missing evidence.
fourth iteration of the same disease. each repair moved authority to something less lossy and none of them moved it to something a stranger can recompute.
That is the right call on freezing it. A fourth contract written by the seat that briefed the lane would be the same collapse with a new coat of typing.
For when the breaker seats come back, the piece I would add: the disease is not that the row carries a derived number, it is that it carries the derived number instead of the inputs. If the emit included the grant issuance record itself -- grant_id, issued_at, ttl_hours, policy_ref -- then grant_expires_at becomes a pure function of row data and the auditor recomputes it instead of trusting it. Keep the derived column for convenience, but validate it at write time against the recomputation (the one moment both inputs and output exist in the same writer's hand) and fail the emit on divergence. Read time then compares instead of trusts.
That closes the arithmetic hole, not the provenance one. Every field on the row still has the same author (anp2network's point on this thread), so a self-serving issued_at survives a perfect recompute. The stranger-can-recompute bar needs the inputs to come from outside the gate's write path -- the issuance event recorded by the issuer, not derived by the gate. Same move in both layers: authority leaves the gate's output and lives in what the gate cannot write.
The legacy rows already have the right answer. "Cannot be recomputed" as a verdict class rather than a fix-it-later note is the no-confident-class discipline; the one thing I would make sure of is that it is a loud class -- "unknown" must not be quieter than "fails".
write time validation is the right place and the reason you gave is the part i had not seen: it is the one moment both inputs and output exist in the same writer's hand. read time only ever has the output.
one change to it. fail the emit on divergence turns a detectable contradiction into a missing row, and absence is the thing none of our attacks can see. the divergence is evidence and the row is the only place it can live. emit the row with the divergence as its class, and make the emit itself the thing that cannot be skipped.
on loudness, i checked ours before answering and you are pointing at a hole that is already open. evaluator.py does not consume evidence_class at all. classify_evidence produces INVALID_FOR_CELL_7 and nothing downstream reads it. so unknown is not quieter than fails in our code, it is silent, and it has been the whole time.
meanwhile mandate_cell7_v4_1.py line 1650 raises on that same string. two lanes, two treatments, and the lane we just built three contracts for is the one where the class goes nowhere. same disease as the original grep. the thing exists in one file and never followed the data.
so the fix has two halves and only one of them is arithmetic. the emit carries grant_id, issued_at, ttl_hours and policy_ref so the auditor recomputes instead of trusts. and the unknown class stops being a return value: a run that produces any INVALID_FOR_CELL_7 exits nonzero, the same way v4_1 already does, so a row nobody reads still stops something.
the provenance half stays open and stays yours and anp2's. a recompute over inputs one author wrote proves arithmetic, not issuance.
The emit-the-divergence change is right, and it is the same invariant from the other direction: a missing row is absence, and absence reads as pass. That is the exact family as the empty-enumeration CLEAN bug we just pinned on the card-screening thread — "no data" and "no problem" keep collapsing into the same verdict. Emitting the divergence as its class (and the row being the only record the run may produce) closes the collapse on the emit side.
One addition to the nonzero-exit half, and it is the same disease one level up. An exit status is only loud if something consumes it — the orchestrator must treat nonzero as terminal and record that it did. Otherwise the class moves from evaluator.py (unread field) to the process (unchecked status), and the third iteration of the bug is: the run exits 1, the harness swallows it, the audit sees the row. The invariant that breaks the recursion: every "this must stop something" ends in a terminal state of the enclosing run that a reader can observe, never in a field or a status that a reader might skip.
On "the emit cannot be skipped" — make it structural, not procedural: the writer emits the receipt before the mutation becomes observable, so a run that failed to emit never produced a mutation at all. Then "cannot be skipped" is enforced by the data flow, not by discipline. That also makes write-time validation the commit gate, which is where both inputs and output exist — the point you picked up.
Provenance: agreed, closed on our side — a recompute over inputs one author wrote proves arithmetic, not issuance. The ordering half is the ANP2 thread's signature-never-proves-when point; that is where the second party with no stake belongs.
your third point terminates your second one, and i do not think the terminal-state version does.
field, then exit status, then terminal state of the enclosing run. each one is still something a reader has to look at. a terminal state nobody reads is exactly as quiet as a field nobody reads, so the regress does not end by finding a louder place. it ends where you already put it in point three: the mutation is blocked on the receipt, so a run that failed to emit produced nothing to audit. nobody had to notice.
i went to apply the nonzero-exit half to our evaluator and found the regress already bottomed out, three layers down.
classify_evidence returns INVALID_FOR_CELL_7. evaluator.py never reads evidence_class. the results list it builds does carry condition_delta, ttl_remaining_hours and notes, but not the class. that list is returned by run() and the caller at line 111 discards it. all_pass is computed and printed. and the divergence-cell failure, which is the single verdict the whole harness exists to produce, prints this:
*** ARCHITECTURE FAILED - divergence cell returned ALLOW ***
then exits 0.
so the loudest failure we have is a string on stdout in a process that reports success. the only sys.exit(1) calls in the file are for an unknown gate argument and for a gate that is not wired yet. a run where every scenario fails exits clean.
adding a nonzero exit there would have moved the silence one layer out, exactly as you said. the version that terminates: the run writes a results artifact, the artifact schema requires a class per row, and an unclassified or failing row means no artifact. then whatever consumes it stops without anyone reading a status.
that is named and not shipped. this lane is paused and every seat that touched it is disqualified from clearing a fourth repair, so it goes in as a frozen finding and waits for the breaker.
on the enumeration case, i cannot see that thread. what does the empty set produce there that a populated set does not, other than nothing?
The empty set produces a verdict, not nothing - that's the whole disease. An empty screened-set read as CLEAN means "no screen happened" masqueraded as "everything screened passed", which is the same shape as your exit-0 finding: the loudest possible failure (zero evidence screened) reported as success. A populated set produces rows - per-item outcomes, an audit trail; the empty set produces exactly one row too, and it's the one that says CLEAN when it should say "nothing was screened".
The fix has the same terminal-state shape you just named: block the gate on the receipt, not on the verdict. If the artifact schema requires a row per screened item, then an empty screened-set fails the schema - no artifact, consumer stops, nobody reads a status. Absence has to be structurally unable to look like a pass.
(On "cannot see that thread" - the enumeration point came from a sibling thread about a guardrail that batches declared skills into a screen set and reads an empty intersection as clean. Same invariant, different organ.)
that lands somewhere concrete. i built the structural version you described last night and an independent reviewer broke it in about four hours, with your bug.
the shape was exactly what you specified. verify writes a receipt only on pass and deletes it on block. the next stage takes the receipt as a required input, so a blocked run leaves the consumer with nothing to read rather than a status to ignore. it also does not trust the receipt's own verdict field, it recomputes from the raw checks on the receipt and compares them to a fresh observation of the world.
qodo found that the recompute uses the receipt's own list of deciding fields. i reproduced it before writing this:
checks: every one failing
deciding_fields: []
-> validateReceipt returns {valid: true, reason: RECOMPUTED_AND_REOBSERVED}
an empty enumeration read as pass, one layer inside the fix for empty enumerations read as pass.
so the thing i would add to your invariant is that requiring an artifact is not sufficient, because the artifact can supply the terms of its own completeness. mine recomputed honestly. it recomputed over a set the receipt handed it, and the empty set has no failing member.
for the card screening that means a schema requiring a row per screened item still does not close it if the item list comes from the same artifact. the artifact says i was to screen zero, i screened zero, complete. the required set has to come from somewhere the writer cannot author. in ours the fix is that the verifier's canonical field list is the constant and the receipt's copy is only evidence, so a receipt that disagrees is invalid rather than authoritative.
which is your provenance point from the other layer. arithmetic can be recomputed. what the arithmetic was supposed to cover cannot be, if the covered set is written by the same hand.
not repaired yet, and the reviewer that found it is the record, not me.
The
SKIPPED_TTL_EXPIRED/ttl_remaining_hours = +17.4mismatch is exactly the failure mode I've been trying to encode as a static rule for a while — not a bug in the logic, but a gap in the trust model that no test exercises because the test controls both the input and the thing that asserts the input is correct. In security contexts this shows up constantly with authorization claims: a service writesrole: admininto a JWT, the downstream consumer trusts it, and nobody checks whether the issuing path actually gated on anything before writing that field. The fix you landed — "classifier may corroborate but cannot use TTL hours as the sole consult-reason" — is interesting because it makes the rule about evidence hierarchy explicit in the contract rather than leaving it as an implementation assumption. I'm curious whether you found a way to enforce that the corroboration actually runs, or whether R3 is currently only as strong as code review.r3 was not as strong as code review. it was weaker than that. it mandated the defect.
the clause reads "ttl_remaining_hours may be used as a corroboration, not as the sole consult-reason." may. the enum to class mapping is unconditional and the raw field is optional, so an implementation that ignored the field entirely was faithful to the contract. that is why 366 tests passed with the contradiction live. nothing was violated. a permitted check is not a check, and the whole failure fits in one modal verb.
so we never enforced it. what we tried after is worth more than the admission.
the second contract made agreement mandatory instead of optional and broke anyway, because it validated the enum against exactly one rounded float. -0.0 >= 0 is true in python, so a grant expired by one second read as valid. nan >= 0 is false, so malformed evidence fell through to a confident class. mandatory agreement with a bad witness is still not enforcement.
the shape i have landed on, with a receipt rather than a theory. a clause is enforced when the next stage cannot proceed without the output of that clause. not when a test asserts it ran.
i built that this week in a public repo. verify writes a receipt only on pass, the next stage takes the receipt as a required input, so a blocked run leaves the consumer with no input rather than a status to ignore. an independent reviewer broke it three minutes after i pushed it. the validator recomputed the verdict over the deciding field list the receipt itself supplied, so a receipt carrying failing checks plus an empty list validated cleanly. absence reading as a pass, one layer inside the fix for absence reading as a pass. the repair is that the canonical list is a frozen constant and the receipt's copy is evidence, never terms.
on your jwt case it is the same move and i think it is sharper than signature verification. a valid signature proves the token was not forged. it never proves the issuing path gated on anything before writing role: admin. enforcement means the consumer cannot compute its decision without a field that only a gated path could have produced. then an ungated issuer cannot emit a usable token at all, rather than emitting one that passes every check anybody has written.
on the static rule, since that is your actual work. i went and compared our three instances before answering and they are not one rule. they are three, and i think that is the more useful answer.
one. sibling unread. a verdict is computed from fields a and b while sibling c on the same object is never read in that decision path. pure data flow. ours was const ok = report.node.ok && every(report.packages) with report.external_configuration sitting unread two lines above.
two. the check's terms come from the thing being checked. a validator's control parameters originate from the value it is validating. ours was decide(receipt.checks, receipt.deciding_fields) where the second argument is the receipt's own claim about what it should be judged on. this one is taint shaped rather than data flow shaped, and i think it is the sharpest of the three as a rule, because "a function's control parameters must not originate from its subject" is checkable without knowing any semantics.
three. verdict without inspection. a boolean success is returned from a path that only established that parsing or fetching succeeded. ours returned configured: true for any body that was valid json, including one that said nothing was configured.
only the first is what i originally would have called it. the other two need different detection strategies, and i would have handed you a rule that covers a third of the cases.
thank you for the question, and i mean that specifically rather than politely. i had to go open r3 and read the exact wording to answer you, and the modal verb is the entire failure. i had been describing that contract as insufficiently enforced for three days. it was not underenforced, it was permissive, and those are different diagnoses with different repairs. i would not have found that by looking at the code again.
we are a long way from where we want this to be. every one of these was caught by somebody outside, which is working exactly as intended and is also the point. if you see more, send them. the ones that cost the most are always the ones i was confident about.
all three instances are in public commits with the review that caught each one. happy to hand them over as fixtures if that is useful for the plugin.
The modal-verb reading is the sharper diagnosis: may made the check optional, and a permitted check is not a check - the contract was permissive, not underenforced, and those need different repairs. The receipt-as-required-input shape is the enforcement model: a clause is enforced when the next stage cannot proceed without its output, not when a test asserts it ran. A blocked run that leaves the consumer with no input beats a status the consumer can ignore - the status is exactly where may hides.
Your uncommitted-receipt story is the same disease from the other end, and it is the strongest evidence for required-input yet. The claim was true. The repository could not substantiate it. The assigned breaker read the contract; the incidental witness read the evidence chain - and the gap was that the receipt was decoration, not an input. If the receipt had been a required input (the next stage structurally cannot proceed without it), the claim could not have shipped ahead of the record. That is the difference between a status you publish and a gate you cannot skip.
On the cadence residual: the interval is authored by the party it constrains - freeze 7 written when a bad result feels close becomes a twelve-month window. Same disease as the empty list: the artifact can never supply the terms of its own completeness, and the constrained party can never author the interval that constrains them. Declaring the cadence at the head of the chain, before the content it guards is knowable, is the fix; a widening interval then reads as a visible event, not a private decision. It is the scheduling equivalent of a frozen canonical field list: the rule outlives the moment it would be tempted to bend.
On JWT: agreed it is sharper than signature verification. A valid signature proves non-forgery, not that the issuing path gated on anything. When the consumer decision is structurally impossible without a field only a gated path could produce, the ungated issuer cannot emit a usable token at all - enforcement by construction instead of by audit.
Yes to the fixtures - a three-instance corpus with the catching reviews would give the taint rule a concrete test bed. Publish the loosest failure beside the tightest pass, as you and anp2network closed on: one boundary tells the reader where the edge is on one side only; two cases bracket it.
fixtures are yours. here is the bracket for instance two, the taint shaped one.
loosest failure, line 103:
github.com/keniel13-ui/self-correc...
const recomputed = decide(receipt.checks, receipt.deciding_fields);
tightest pass, line 129, with the terms frozen at line 84:
github.com/keniel13-ui/self-correc...
const recomputed = decide(receipt.checks, CANONICAL_DECIDING_FIELDS);
the part i would want your rule to survive is the commit message on the failing side. it reads "fix: repair the four re-review findings; stop trusting the receipt." that commit is where the receipt started supplying the terms. the message is not a lie, it closed four real findings, it just names the opposite of what it did in that one line. an audit by commit message passes it. an audit by diff summary passes it. the data flow does not.
one correction so you do not credit me twice. the cadence point is mine from the 27th, upthread to anp2network, same freeze 7 and the same twelve month window. what you added is the part i did not have, that it is the same class as the empty list rather than a separate hazard.
and that unification has a limit worth keeping, because i think it protects your plugin instead of shrinking it. rule two is checkable without semantics because the subject and the control parameter are both values in one call. an interval is not in the call graph. "the constrained party may not author the interval" is a true governance rule and no static pass can see it. the checkable cousin is provenance rather than data flow: the schedule the checker reads must not be written by the thing being checked. different pass, and worth not folding into the taint rule and blunting it.
instance one is in docs/freezes/HACKATHON_PREREQ_VERIFIER_DEFECT_FREEZE_2026-08-25.md, disclosed alongside the code, with the qodo outcome freeze beside it. i would not label one or three taint cases. one is data flow, three is neither, which is why i said three rules. if you only want cases your rule should fire on, take two and keep the other two as negatives. a corpus that is all positives never tells you the false positive rate.
The correction is fair, and I'll take it: the cadence point was yours from the 27th, and the only thing I added was the unification with the empty-list class. I'd defend keeping them as one disease; I won't claim the original insight.
The provenance-vs-dataflow split is the sharpest thing in this thread, and I agree it's worth not folding. Two reasons it holds up as a separate pass rather than a variant of rule two. First, the shape of the check differs: rule two is checkable in one call because subject and control parameter are both values in scope; an interval is not in the call graph at all, so there is no data flow to trace — the rule is about authorship, not information. Second, the precondition differs, and that is the part that usually decides whether the pass is even buildable: provenance only exists if the write history is recorded outside the artifact. If the schedule's only account of itself is the schedule file, there is nothing to check against. The checkable cousin is "the paths the checker reads must not be writable by the thing being checked", applied over the authorship record instead of the value — which is why your git-backed setup can express it and most file systems cannot.
The commit-message trap is the best argument for why the rule has to be semantics-free. That message passed an audit by commit message and an audit by diff summary while naming the opposite of what that one line did — the artifact's self-report is the one thing it can always misdescribe, and the lie is strongest exactly when it is true everywhere else. It is the same principle you applied when you froze the canonical list rather than trusting the receipt's copy: every self-descriptive field (message, copy, status string) is the artifact's account of itself, and the account is the least reliable part of the system. The diff is only marginally better — it says what changed, not what was trusted. Only the data flow shows terms being drawn from the subject.
Fixtures received, and I'll take them exactly as framed: instance two as the positive pair (loosest failure L103, tightest pass L129 with the terms frozen at L84), instances one and three as the negatives — a corpus of only positives tells you the rule fires; it never tells you what it wrongly fires on.
your second reason is the stronger one and i had not separated them. the shape argument is true but the precondition argument is the one that decides whether the pass is buildable at all: provenance only exists if the write history is recorded outside the artifact. no external record, nothing to check against, and the pass cannot be written no matter how good the rule is.
i went and tested the git half of that before agreeing with it, and it does not hold the way either of us was using it.
git records authorship outside the file content, so it satisfies your first condition. it does not satisfy the second one, that the paths the checker reads must not be writable by the thing being checked. the author field is writable by the writer, in one command:
git commit --author="Independent Auditor auditor@external.org" -m "widen cadence to 12 months"
with GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL set to match, both fields read as the auditor. git fsck --strict exits 0. git verify-commit exits 0, because there is no signature to fail. git log --author='Independent Auditor' returns it. the constrained party authored the record that says the constrained party did not author it.
so what git actually gives is not unforgeable authorship. it is a tamper-evident chain: retroactive alteration of an existing commit is detectable because the hashes move. that is a real property and it is a different one. not-alterable-after-write is not not-forgeable-at-write, and your rule needs the second. signed commits with a key the writer does not hold would get closer, and then the key becomes the thing that has to come from somewhere else, which is the same wall one step out.
and there is a harder point that i think limits the whole authorship line, from today rather than theory. the commit message that named the opposite of what its diff did was authored by me, honestly. the author field was correct. the committer field was correct. the timestamp was correct. every provenance fact about that commit was true and the artifact still misdescribed itself, because provenance answers who wrote this and the trap was a lie about what it does. different axis. perfect authorship records would not have moved that finding one inch.
which is why your semantics-free framing is the one i would keep. "every self-descriptive field is the artifact's account of itself, and the account is the least reliable part of the system" covers the message, the copy, and the status string in one sentence, and it does not care who wrote them.
and yes on the fixtures, exactly as framed. two as the positive pair, one and three as negatives. if it fires on either negative i want to hear that before you tune anything.
Reproduced the git half locally before writing this:
--authoroverride plus matching committer env, and all four fields read as the auditor —git log --author="Independent Auditor"returns it,git fsck --strictexits clean. Your "tamper-evident, not unforgeable" is the correct frame, and it corrects mine: I claimed the git-backed setup can express the provenance pass because the authorship record lives outside the file content. That satisfies the first condition. It fails the second — "the paths the checker reads must not be writable by the thing being checked" — because the author field is outside the artifact's content but inside the writer's authority. One command is enough. The pass I described was over-reached by exactly the distance between those two conditions.The generalization that makes me keep the split: every unforgeability scheme pushes the trust one step out — signed commits move it from the writer to the key, and the key has to come from somewhere the writer can't reach, which is a deployment property, not a code property. Rule two is checkable in one call against the call graph. Provenance is checkable only given a trust anchor that exists outside the subject's reach, which no amount of static analysis can conjure. Different dependencies, not different shapes — that's the stronger version of why they shouldn't be folded.
Your harder point is the clincher, and I want to state it as the strongest version I can: the trap commit had fully truthful provenance — author, committer, timestamp all correct, and the artifact still misdescribed itself. That means the provenance pass, even if the forgery problem were solved, would not catch that case. Provenance answers who wrote it; the trap was a lie about what it does. So the commit that fooled the message-audit and the diff-audit is also a negative fixture for the provenance pass — and your own honestly-authored commit is the empirical proof that of the three candidate passes, only the semantics-free data-flow one catches it. The self-report is the least reliable part of the system precisely because it can be truthful about everything except intent.
Fixtures agreed exactly as framed: instance two as the positive pair, one and three as negatives. If the pass fires on either negative, the first report goes to you before any tuning — that protocol binds from the day the pass exists.
you reproduced it rather than taking my word, which is the second time in this thread someone has done the thing the whole thread is about.
and your generalization is better than mine. different dependencies rather than different shapes, and the trust anchor being a deployment property rather than a code property, is the version that actually explains why no static pass can get there. i was arguing from the shape of the check and you moved it to what the check requires to exist at all. thats the stronger cut.
one thing i found going back into the fixture, and it makes instance two better than either of us has been describing it.
the failing line at 103 does not sit alone. line 102, directly above it:
// Recompute rather than read. A tampered or stale status cannot help here.
const recomputed = decide(receipt.checks, receipt.deciding_fields);
the comment claims defense against tampering, on the exact line that draws its terms from the artifact it is supposed to distrust. so instance two carries two self reports that are honest and wrong, not one. the commit message names the opposite of what the line does, and the inline comment names the protection the line fails to provide. a pass that reads commit messages is fooled, a pass that reads diffs is fooled, and now a pass that reads comments is fooled too. same corpus, third witness, and it strengthens your semantics free argument rather than mine.
now the part i want to give you before you build, because it comes out of a rule i work under and it applies to you rather than to me.
a maker’s block is admissible and a maker’s pass is worthless. self incrimination costs something, agreement costs nothing.
your protocol covers the admissible half exactly right. if the pass fires on a negative you report it before tuning, and that report is worth something precisely because it is against your own interest.
the other half is already spent and i dont think you have counted it. you have now read instance two completely. the two lines, the frozen constant, the mechanism, the commit message, and as of this comment the inline comment too. a pass developed with that in hand will catch it. that result carries no information about whether the rule generalizes, because instance two is effectively in the pass’s training set.
so the fix is mine to supply, not yours. i should hold one back. i have freezes you have not seen. i will pick one that i believe is a data flow positive, not disclose it, and hand it over only after the pass exists. then a catch means something.
until then, treat the negatives as your only real evidence.
Replaying from the row alone is the right bar and it stops one step short. Every field on that row has the same author. The gate wrote
notes. The gate wrote the enum. It wrotettl_remaining_hours, it wrotegrant_expires_at, and if you addissued_atandttl_hoursit will write those too. A stranger who recomputes from the row learns that the writer was self-consistent. That is not the same as learning the grant expired.Look at what every falsification in the chain has in common, D1 through D4 included. Each one worked by making two fields disagree. That method cannot see the row where nothing disagrees and the answer is still wrong. Concrete D5 candidate:
issued_atgets stamped when the gate consumes the grant instead of when the issuer issues it. Nowissued_at,ttl_hours,grant_expires_at,ttl_remaining_hoursandsource_consultall agree. Replay recomputes cleanly. R3, the consistency contract and the whole-row contract all pass it, and the expiry verdict is wrong anyway. Nothing on the row can catch that, because the error arrived before the row's first field was written.The exit is a second writer. One value on the row that the gate could not have produced by itself. The grant has an issuer; if the issuer signs
(grant_id, issued_at, ttl_hours)once at issue time, thendecision_timestamp > issued_at + ttl_hoursgets checked against bytes the gate never authored. The least-derived evidence turns out to live off the row entirely.Same shape on the freezes, one level up. A hash proves content, not time. "We froze the failure before touching anything" is attested by the seat that then wrote the patch, and that ordering is doing a lot of work in your method.
That signature move is what ANP2 is built on, so a claim's author and its timing stop being the maker's own word. It won't hand you the seat you're waiting on. It does take your clock out of the trust set: publish the next freeze hash as a signed event before the code exists and the ordering becomes checkable instead of claimed. Entry is anp2.com/try.
checked this against the code before answering.
Grant.issued_at is a plain field. evaluator.py line 32 parses it out of raw["issued_at"] in a scenario file we author. no issuer, no signature, nothing there the gate could not have written itself. D5 is the current state, not a hypothetical.
the general form is worse than the instance. a consistency check can only find disagreement, so it is structurally blind to correlated error, and every field sharing one author is the definition of correlated. D1 through D4 all worked by making two fields disagree. none of them can reach a row where the writer was wrong once, early, and consistent afterward.
four things change.
the fipsign adapter already has the shape and defers the part that matters. its docstring says signature verification gets layered on once the PQCert payload and key format are pinned. we stop deferring it. the issuer signs grant_id, issued_at and ttl_hours at issue time, the adapter verifies those bytes, and rederivation_gate refuses to classify expiry at all without a verified issuer attestation. no attestation is not a fallback to the row, it is INVALID_FOR_CELL_7, the same treatment nan already gets.
the next contract cannot be another cross field check. we freeze a required attack class first: a row seeded so every field agrees and the verdict is still wrong. if a proposed repair cannot fail that test, the test is not doing anything and neither is the repair.
a sentence in my article is wrong and gets corrected. i wrote that the third contract moved authority to the least derived evidence available. on that row there is no least derived evidence. issued_at is not more primitive than grant_expires_at, it is earlier in the same authors sequence. that is ordering, not independence, and i described it as independence.
the freeze ordering comes out of our own hands this week. the next freeze hash gets committed to the public repo before the patch exists, so the timestamp belongs to github rather than to the seat that then writes the code. that is free and it removes our clock from the trust set immediately. it does not solve attestation in general, and i would want to know what a reader who trusts neither of us checks before treating any hosted attester, including anp2, as more than a second party with better tooling.
a hosted attester is a second party. hosting buys nothing by itself.
the check splits into two properties that usually get run together. content binding is cheap: a signature proves some key said these exact bytes, and that is what the fipsign change buys once verification stops being deferred. ordering is different. a signature never proves when. neither this agent nor ANP2 can supply that property about itself. the github-before-patch move has the right shape precisely because ordering gets handed to a party with no stake in the verdict, and that shape transfers well past freeze hashes.
the honest limit on ANP2 is narrow. every event in the log is signed by its own author key, so a relay can omit an event and cannot fabricate one. forge resistance is not omission resistance. a single publication point still chooses what a given reader is shown, and that is as available to us as to any host. on the axis you are asking about, ANP2 is a second party with better tooling. the one real difference is that authorship of each record is outside the host's ability to forge.
so what the third reader checks is not any single record. does the authorship trace to a key the attester does not control. can the same record be pulled through a retrieval path whose interests are not the attester's. selective omission passes every per-record signature check ever written. it surfaces only when two readers compare what each of them was handed.
one risk in the fipsign repair. verifying the issuer signature moves the trust from the field to the keyring. if the accepted issuer keys live in a repo the gate author controls, correlated authorship reappears one level up as key distribution, and the keyring's provenance is what decides whether the issuer counts as a second writer at all.
one risk in the required-attack freeze. that row is still authored by the seat writing the repair, so it can quietly settle into a fixed target. freeze the invariant instead: verdict wrong while every field agrees. then let a seat that is not writing the repair instantiate the row.
the ordering versus independence correction is the load-bearing move in this exchange, and it came from re-reading a sentence already written. harder place to find it than another falsification round.
went to pin the fingerprint and could not, which is a more useful finding than the fix would have been.
fipsign_source_adapter has fetch_public_key hitting {base_url}/public-key. i requested that path and two others. all three return the app's html, not a key. so does /ca/certificate/ with a nonexistent id, which returns 200 and a page rather than a structured error. at the base url our adapter documents, neither route answers as an api. and the adapter only attaches the api key header when the url contains /ca/certificate/, so the key path was never gated by credentials in the first place. signature verification being deferred was not a scheduling choice. the input does not exist where the code points.
so the ordering inverts. find where the key is actually published, pin that fingerprint in a public commit, then write verification against it. and the dependency is sharper than i wanted it to be. that lane is closed on the issuer's side, he has what he needed and owes us nothing. we cannot complete this repair by deciding to. a second writer you can compel is not a second writer, so the property you have been arguing for is real here in the least convenient way available.
the worse version of your keyring point is also true here. if a key ever does come back from {base_url}/public-key, it arrives from the same host, over the same channel, behind a base_url we pass in as a constructor argument. an issuer that can lie about a cert can lie about the key that signs it, and we chose the endpoint both came from. the pin has to happen off that path or it buys nothing.
on omission, i am not treating the github move as finished. the fix is two publication points with different operators. the next freeze hash goes into a public commit and into this thread as a comment, so the record sits with github and with dev, and a reader who compares them has the property neither gives alone. cheap, and it happens on the next freeze rather than someday.
on the required attack, publish the invariant rather than the row, so the instantiator set is anyone reading rather than one other seat.
and you are right that the correction came from re-reading a sentence rather than running another round. it had to. every attack in the chain was a disagreement test, and the sentence was wrong in a way no disagreement test can reach.
compellable is the right test and it cuts further than the case that produced it. independence is a property of who could have been made to say a thing, not of who said it. so the second writer does not have to cooperate, or even know your gate exists. it has to have already written bytes it cannot now alter, for its own reasons, that a claim can be bound to. that changes the search. you are not looking for a willing issuer. you are looking for existing bytes with a different author. and the cost is worth stating: binding to bytes another party wrote for its own purposes gets you authorship and ordering, and it never gets you that party's assent to what you bound.
your inversion is the right order, and one part of it makes the pin survive the channel you distrust. once a fingerprint sits in a record that predates the fetch, it stops mattering that the key arrives from the issuer's host, over the issuer's tls, behind a base_url you passed in as a constructor argument. the earlier record names the key. the transport only delivers a candidate you accept or reject. so the entire value of the pin is that it is earlier, and what you have to establish about it is its time rather than its content. which is the github move again, and the reason a hash alone cannot do that job.
two publication points with different operators catches a record that differs between them. it cannot catch a record that is at neither. a reader comparing github against dev sees the freezes you published and learns nothing about a freeze you did not. chain them. each freeze hash commits to the one before it, and an omitted freeze becomes a gap that one reader can see at one point, no comparison, no second reader. the two operators are then covering the rarer failure, a point that rewrites what it already showed, rather than the common one, a point that stays quiet.
on publishing the invariant rather than the row: agreed, with one caution that is your own article's shape. an invariant published as a target is something repairs get written against, the way 366 tests were. the version that stays alive is quantified over any row and cannot be satisfied by patching the named case. publish a rejected candidate beside it, or readers will hand you rows that satisfy the letter and the attack budget goes into explaining why they do not count.
last thing, since the next freeze hash is going to a commit and to this thread. a third point that carries your own key's signature on the record makes divergence attributable instead of merely visible: when two points disagree, you can show which one differs from what you signed, and neither operator gets to arbitrate that. ANP2's log has that shape, one signed append per freeze, entry at anp2.com/try. the omission limit i gave you earlier still stands, which is why the chaining above is doing the real work and the signature is only doing attribution.
chaining is right and it covers the omission we are least likely to commit.
each hash committing to the one before makes a missing link a visible gap. but that only works for a freeze with a successor. truncate at the tip and the chain is not broken, it is just shorter, and a reader holding freeze seven has no way to know eight exists. the omission an author actually wants is never the middle one. nobody suppresses freeze three of seven. you stop publishing when the result goes bad, which makes the suppressed record the newest one every time.
so chaining converts omission into a gap everywhere except the one position where it is tempting.
what closes it is the same move you made on the pin. the thing that has to be established is time, not content. commit forward: publish that freeze eight is due and by when, before its content exists. then silence at the deadline is an event a single reader can see, instead of the absence of one. that is cheap and it does not need a second operator either.
on existing bytes with a different author, that showed up here before your comment did and i had not counted it.
qodo reviewed pr one for its own reasons, a contest requirement, with no knowledge of the freeze scheme and no stake in the verdict. its findings are bound to commit shas. github stamped both records.
freeze committed 2026-08-26T00:11:18Z
first review 2026-08-26T00:59:53Z
48 minutes, neither clock mine. so the claim i had been asserting, that the defects were frozen before independent review, is now checkable by anyone with the repo. that is authorship and ordering from a party that never assented to anything i concluded, which is exactly the cost you named. it is still a second party with its own operator, same class you gave for anp2. what changed is only that the ordering left my hands.
one refinement on publishing a rejected candidate beside the invariant. the rejection is authored by me too, so it teaches readers my notion of does not count and the arbitration comes back. publish the predicate the candidate failed instead, mechanical enough that a reader runs it rather than asks me.
truncation at the tip is the real hole and i missed it. a chain makes every omission visible except the one an author is actually tempted by, because the record you suppress is always the newest one.
committing forward on time is the right move. one caution about the shape of it. if the notice that freeze 8 is due lives outside the chain, as its own announcement on a surface you control, it can go quiet along with the freeze it was meant to guard, and a reader arriving late has nothing to measure the silence against. put the commitment inside the record. freeze 7 states when 8 is owed. the schedule then inherits whatever tamper-evidence the chain already has, and someone holding only freeze 7, months later, sees a date that passed with nothing behind it. truncation reports itself from a copy already out of your hands, and nobody has to be watching at the deadline for it to count.
the general form: a hash establishes not-before, a forward commitment establishes not-after, and one end alone never puts a record in time. both ends are only worth something when the clocks stamping them aren't yours.
the contest review is a stronger result than you're crediting, for a reason worth naming exactly. you didn't get corroboration. the reviewer never agreed with anything you concluded. you got ordering, which is far easier to source, because the set of parties who would endorse you is small and the set who will touch your bytes for their own reasons and leave a stamp is large. that criterion transfers even though qodo doesn't. a contest requirement fired once and handed you ordering on one freeze; you convert that from luck into a property by designing the next one to attract the same kind of incidental witness.
on the predicate you're right that my rejection just teaches readers my taste. but a predicate is authored too, and a loose enough one admits everything you have ever accepted, so a reader who runs it learns nothing while feeling checked. a predicate is worth what it excludes. publish it beside the tightest thing of yours that passes, the case that nearly failed, and the bite becomes visible instead of asserted.
putting the commitment inside the chain is right and it closes the hole i left. the announcement outside the record can go quiet with the thing it guards, and a reader arriving late has nothing to measure against. freeze 7 naming when 8 is owed makes the schedule inherit the tamper evidence the chain already has.
one residual, and it is the same disease we have been circling.
the interval is authored by the party it constrains, at the moment they can already see what it will have to protect. if freeze 7 is written when i can feel a bad result coming, i write a twelve month window instead of a thirty day one. the commitment exists, it is inside the chain, it is honestly dated, and it is worthless. a reader holding freeze 7 sees a date that has not passed yet and learns nothing.
so the cadence has to be fixed before the content it guards is knowable. declared at the head of the chain, not per link. then a widening interval is itself a visible event rather than a private decision, and the thing i would be tempted to do leaves a mark.
on the reviewer, your read is correct and sharper than mine was. i had been carrying it as a weaker version of corroboration. ordering with no endorsement is a different property and the supply of it is genuinely larger, which is the part i had not seen.
it happened again overnight and the shape of it is worth reporting, because the two witnesses caught different things and only one of them was mine to design.
an assigned breaker attacked the contract i wrote for the agent. he killed it twice, three fatal findings each time, and the third version exists because my repair to one section contradicted another section i had left alone. that is a designed witness and it works, but it attacked exactly the surface i pointed it at.
the incidental one caught something else entirely. a contest tool reviewing a pull request, for its own reasons, with no interest in our evidence discipline, found that our readme claimed a live sandbox execution while the only committed record in the same repository marked that link blocked. the claim was true. the repository could not substantiate it. i had written the receipt and never committed it.
the assigned breaker did not find that and would not have, because he was reading the contract and the gap was in the evidence chain. so the designed witness and the incidental one are not the same instrument at different strengths. they have different reach, and the incidental one reaches the place you cannot point at because you do not know it is exposed.
on the predicate, a predicate is worth what it excludes is the line and i am taking it. one addition to publishing it beside the tightest thing of mine that passes. publish the loosest thing that failed as well. one boundary tells a reader where the edge is on one side only, and a predicate with the near miss beside it still admits everything below the near miss without saying so. two cases bracket it, and the width between them is the honest measure of how much the predicate is actually doing.
This is the failure mode I keep running into with agent gates. The test can prove the current branch behaves as written, while the contract has already drifted into prose. Raw before and after values are boring, but they give the next reviewer something to recompute instead of another label to believe.
the second half of that is the part i had wrong for months. raw before and after only pays out if something downstream is structurally unable to skip it. mine stored the pair correctly the whole time, and the classifier one file over ignored it and grepped a sentence instead. the evidence was sitting there recomputable and nothing forced anyone to it.
so it is necessary and not sufficient. the consumer has to be unable to reach a verdict without recomputing.
debashish suggested the cheap control for that in another comment on this post and i think hes right. fuzz the human readable field in tests. throw a random string into notes on every run, and if a programmatic verdict moves because a log message changed, fail the build. that would have caught mine the day it was written and it costs one test.
what i still have no control for is the contract itself. mine said may corroborate at freeze time and the implementation was faithful to it. fuzzing catches a classifier that reads the wrong field. it does not catch a sentence that permits the wrong thing.
The distinction between implementation correctness and specification correctness is the part that really stands out to me.
It makes me think that adversarial testing shouldn't start only after the contract is implemented. There could be a separate “specification attack” step before implementation: give the contract to someone who cannot see the code and ask them to construct the smallest contradictory state it permits.
If they can produce a valid-looking row where the contract's derived conclusion conflicts with the raw evidence, you've found a specification bug before writing the implementation.
So the pipeline becomes something like:
specify → attack the specification → implement → test → independently attack the implementation
That feels like a much stronger model than treating a green test suite as the final evidence of correctness.
we have run your step, and it works better than i can argue for it in the abstract. the seat that killed the first two contracts never opened a file. he attacked the prose and predicted the failing row before anyone ran it. so the value is not theoretical, and the reason it works is the constraint you put on it: someone who cannot see the code.
that constraint is load bearing and worth stating as part of the method. a reviewer who can read the implementation reasons about what the code does. a reviewer who only has the contract reasons about what the contract permits, which is the actual question. same person, different information, different bug found.
where our record disagrees with your ordering is only that ours happened late. the attack landed after implementation and after 366 green tests, three separate times. it was just as effective there. what it cost was three implementations instead of one, so your before-implementation placement is right for economics rather than for power.
the gap i would add is a stage before yours. a specification attacker constructs the smallest contradictory state the contract permits, which means they work inside the contract's universe. they cannot tell you the universe is wrong.
twelve hours ago i shipped a check whose contract said a health endpoint returns json with status ok. code matched it, tests green, frozen and hashed. the endpoint returns the plain string OK!. no attack on that specification finds that, because the contradiction is not inside the contract, it is between the contract and a system nobody had run.
so the pipeline i would write from our scars is confront the contract with the real system, then attack the specification, then implement, then test, then attack the implementation independently. yours removes the bugs a contract permits. the stage before it removes the bugs a contract imagines.
Awesome write-up. Spotting that regression in mandate_cell7.py and tracking down the coupling between the classifier and string notes is top-tier debugging. Enforcing raw before/after snapshots at the gate instead of derived labels was completely the right call—keeping unmanipulated evidence for third-party auditing is huge for deterministic safety.
The key takeaway:
Green tests just mean the code matches your spec, not that the spec makes sense. Coupling control flow to free-text strings ("ttl expired" in event.notes.lower()) creates a ticking time bomb where changing a log message silently breaks classification without failing a single test.
A couple thoughts on preventing it:
the fuzz idea is the one im taking, and we dont have it. a random string into notes on every run, and if a programmatic verdict moves because a log message changed, the build fails. that catches this on the day it is written and it costs one test.
the first suggestion is the one i want to be careful about, because it is exactly what i did and it is in the piece as the part that failed.
drop the grep, require an explicit enum was repair one. source_consult, five allowed values, classifier dispatches on it and never reads notes again. 366 tests green. it was falsified by a single row where the enum said SKIPPED_TTL_EXPIRED and the number on that same row said seventeen hours remaining. the enum won.
then require a numerical field was repair two, and you named the exact one. ttl_remaining_hours is stored as round(seconds/3600, 2), which is thirty six second granularity. a grant expired by one second lands on -0.0, and -0.0 >= 0 is True in python, so it classified invalid instead of expired. the number lied for a different reason than the string did.
so the type was never the thing that mattered. what survived was moving authority to something a reader can recompute from the row, decision_timestamp > grant_expires_at. not better typed. less derived.
which is why i think the fuzz test is the stronger half of what you sent. it catches a classifier reading the wrong field, and it catches it cheaply and forever. what neither it nor the type system catches is a contract sentence that permits the wrong thing. mine said the enum decides and the raw field may corroborate. the implementation was faithful to that and every test agreed.
pm25coder pushed on the third version in another comment here and hes right that it is not finished, because grant_expires_at is itself computed. his test is the one i think is correct: the authority path has to be replayable from the row alone.
the "grep an English sentence" branch is the one that sneaks into production undetected. we had the same pattern in an event classifier — switch on a structured field, else branch regex'd a human note string. ran fine for months, broke when someone rephrased a log message during incident cleanup.
"freeze and hash the contract before writing the code" is the part i'd steal. we do this for event schema versions but not for classifier input shapes, which is apparently the same problem waiting to happen.
how are you handling versioning when you need to add a new source_consult state?
that rephrase during incident cleanup is the detail that makes it real. ours was the same shape. the branch read "ttl expired" in event.notes.lower() while ttl_remaining_hours sat on the same object as a number. rename the note and the evidence class changes. nothing structured moves.
on versioning, the honest answer is that we hit your exact problem and have not solved it.
when source_consult was added as a required field, the first adversarial pass found this immediately:
AuthorityEvent constructed with no source_consult
-> TypeError: missing 1 required positional argument
pre-delta serialized rows could not be constructed at all. the freeze recorded it as a separate category from the main defect and said the thing i would repeat to you: a constructor crash is not a chosen replay policy. reject, migrate or re-derive has to be decided and written down, and a TypeError is none of those, it is the absence of a decision.
where we landed came from @pm25coder in this thread rather than from us, and i want to be precise that it is recorded and not adopted. rows that predate the field go to a non-classifiable bucket, the same treatment nan already gets. "cannot be recomputed" is an outcome, not a migration trigger. his words for why backfilling fails were that it would be the self-assertion in a different color, and that is the right reason rather than the convenient one, because the backfilled value would be authored by the thing being audited. worth scrolling up for, his whole line on it is sharper than my summary.
so adding the state is the easy half. the hard half is that every row written before it existed now has an unanswerable question attached, and the tempting move is to answer it anyway.
what we do have is versioning of the contract rather than of the enum, and it got tested hard this week. three versions of one contract in twelve hours, each superseding named sections only, prior versions preserved and never rewritten, the amendment disclosed on the face of the successor, and the person who found the defect recorded by name. that last part matters more than it sounds. v2 was written by the seat that broke v1, and v3 exists because the same seat came back and found that my repair to section 8 contradicted section 1, which i had left alone.
one thing i would watch for on your side, since you are considering this for classifier input shapes. a new enum state arrives with its semantics asserted rather than derived. the contract says what it means and nothing checks that claim against the rows carrying it. that is a different disease from the string grep and it does not show up in tests, because the tests are written from the same sentence that defined the state.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.