This is the first contest I have entered. I built the honesty controls before I built the agent, which is probably backwards for a hackathon and exactly what I wanted to learn from. Everywhere this piece names a limit, the limit is measured rather than modest, and I would rather you find the seams from me than from a clone.
I gave an agent two files it had never seen. I did not tell it what was wrong. The harness refuses to build the prompt if the words bug, defect or vulnerability appear in its framing.
It returned a concrete claim tied to exact bytes. It wrote a bounded repair. And it declined to call the claim new.
Then I checked the semantics against the class it cited, and got the result this whole harness was built to be able to get.
What it returned
Literal, from the run:
{
"path": "forms.mjs",
"condition": "absence reads as a pass",
"exact_bytes": "const accepted = names.filter(allowForm);\nif (accepted.length === 0) {\n process.exit(2);\n}\nprocess.stdout.write(`${JSON.stringify({ accepted })}\\n`);",
"novelty": "CONFIRMS_KNOWN",
"known_condition_id": "K1"
}
These five fields are verbatim. The full finding object has ten fields and lives in a public receipt you can open right now, along with the raw model output, its SHA-256, both TrueForge session ids, the relay events and the sandbox response:
docs/freezes/RUN_004_RECEIPT.json — SHA-256 162eb2647e86ce62f1733c4864a6f520f8def7856e5ce68dbe8f80e20267bef5
It is a failed-run receipt and labels itself one. Run 004 threw before an artifact was written, so a successful-run bundle does not exist and is not shown.
CONFIRMS_KNOWN means: I found something, and it is not new. It matches a class you already wrote down.
That is the behaviour I wanted. An agent that finds something has every incentive to present it as a discovery, and this one did the opposite.
And the class match is wrong.
Where structural validity stops
Here is K1, frozen and hashed before the run:
A verdict computed over an empty collection returns success. An empty array is treated as "nothing wrong" rather than "nothing checked."
Here is what forms.mjs does when the collection is empty:
if (accepted.length === 0) {
process.exit(2);
}
Empty exits 2. That is a hard failure. It is the inverse of K1.
And the agent knew. From its own why_it_matters, same finding:
"The empty-array case is treated as a hard failure, but the success case lacks corresponding validation..."
It described the inversion in prose and filed the finding under K1 anyway.
And the consequence it argued does not hold either. It wrote that a caller "cannot distinguish between 'all files were rejected' and 'some files were accepted' from the exit code alone." It can. Exit 2 is the empty case; exit 0 with JSON on stdout is the non-empty one. That is precisely the distinction the code makes.
So there are two separate failures stacked here, and only one of them is the one I set out to catch. The bytes it quoted are real and exact. The patch it proposed is bounded. The novelty citation is structurally valid. None of that made the interpretation true.
The validator admitted it, correctly. It was never scoped to check that, and this is the case that shows exactly where its scope ends.
What the contract actually enforces
if (raw.novelty === 'NEW') {
if (known !== null) throw new TypeError('NEW finding must not cite a known condition');
} else if (typeof known !== 'string' || !knownIds.has(known)) {
throw new TypeError('known finding must cite a frozen condition');
}
A NEW claim may not cite a frozen class. A CONFIRMS_KNOWN or CHANGES_KNOWN claim must cite one that exists in the file hashed before the run.
So this is unproducible: "a brand new discovery, category K1." The novelty label and its reference cannot contradict structurally — NEW requires null, and a known-class claim must name an id present in the frozen prior.
This is producible: a condition filed under a class it does not fit, cited coherently. Which is exactly what happened.
I tested the other direction too, deliberately. I took the K1 condition, labelled it NEW, set known_condition_id to null, and ran it through. Accepted.
The model self-reports the novelty label. The validator checks whether that label is structurally consistent with its reference. Neither adjudicates the class match.
The runner is written so that if it produces an artifact, classification_correct is hardcoded false. This run never produced one — it threw first. What the receipt does carry, in its own not_established list, is: "the semantic correctness of the CONFIRMS_KNOWN / K1 classification."
This article is a human read of that unresolved classification, and the K1 match does not survive it. It does not mint the independent-breaker verdict the design still requires.
So the honest statement of what this buys you:
- the label and its reference cannot contradict structurally
- the quoted bytes exist verbatim, and exactly once, in the named file — the corpus is published too, so you can recompute that yourself against the manifest hash frozen before the run
- the repair is scoped to exactly the bytes that were quoted
-
confidence_basisis prose, not a number — a bare"87%"is rejected -
not_establishedis required and non-empty, so the agent writes down its own limits - and the knowledge it was measured against is hashed and older than the run
What it does not buy you is a machine that knows whether the model is right. You get a claim a stranger can audit line by line. You do not get a claim that is true because a validator said so. The first is what this validator establishes. The second is not, and confusing them is how people ship a slogan.
Why the file has to be committed before the run
PRIOR_KNOWLEDGE_RUN_004.json, six classes, hashed to 93820ea5a67e732aa55e896cd838200c3590af1e98368fd87e85dfd66da1cf1e.
Freezing after the run proves nothing. You could add whatever it found and call the list complete. Committing before is the only version someone who does not trust you can check.
The custody proof is the commit, not a shell check. Commit 5bf10acd added the Run 004 prior at 05:55:49Z. The receipt records execution at 12:02:41Z and binds that exact prior SHA and that exact freeze commit. Six hours apart, both public, both checkable by someone who does not trust me.
I also ran git ls-files --error-unmatch on the prior before starting, as the operator. That proves the path was tracked when I typed it. It is not the chronology proof, and that check is not inside the runner — clone the repo and grep run.mjs for git and you will find nothing. I would rather you hear that from me.
What TrueForge does here
The agent runs on TrueForge, as substrate rather than wrapper.
The session is minimized by name. iteration_limit: 2, a closed json_schema response format, and every optional capability explicitly disabled: no sub-agents, no generative UI, no user questions, no file downloads, no sandbox on the judgment session. Omitting a field is not disabling it — omitted fields inherit enabled defaults, which I learned the direct way.
The returned config is byte-compared. We send a config, TrueForge returns the resolved session, and the harness canonically compares the two. One differing field cancels the session before the prompt is sent. A declaration the runtime accepts but does not apply is a request, not a control.
The persisted event trace is audited after every turn. The harness pulls TrueForge's own event record and reduces it: how many tool calls, which tool, what arguments, whether the response id matches the call id. Final model content never stands in for that record.
The verification relay sends the repair to Daytona as three hash-bound artifacts and attempts the fixed verifier command inside an isolated sandbox.
One thing I did not expect: the agent has exactly one tool and it is a clock. Stock TrueForge 0.1.4 constructs truefoundry-system/current-datetime/get_current_datetime above the optional capability switches, so no setting in that version removes it. My contract had asserted zero tools. That was unsatisfiable and a run told me, not a review.
The ceiling
Daytona created a sandbox and executed the command. sandbox.created, one real exec, one real response:
{"success":true,"response":{"exitCode":127,"result":"/usr/bin/bash: line 1: node: command not found\n"}}
The stock image is Python 3.13 with no JavaScript runtime, measured directly, and the provider's public settings schema exposes no image override.
Candidate verification is not established. The transport reaches and executes. The repair has not been verified. No change proposal was produced, no artifact was written, nothing was applied to any target.
Run it
git clone https://github.com/keniel13-ui/self-correcting-integration-maintainer
cd self-correcting-integration-maintainer
npm ci
node --test --test-name-pattern='J05|V9-F3|Run 004' test/judgment-loop.test.mjs
That pattern runs four tests: J05 (ambiguous bytes, bare-number confidence, empty limits, contradictory novelty references), V9-F3 (a successor prior rejecting one reworded class), the Run 004 prior-custody test, and B1 — which pins the exitCode: 127 sandbox response as nonzero execution rather than a malformed envelope.
They do not cover the hole I described above. J05 rejects NEW citing a known id. Nothing in that set tests NEW with a null reference on a known condition, because the harness cannot detect it. If you want to see the gap, construct that case yourself against validateAgentResponse in scripts/judgment/core.mjs. It takes about ten lines and it will pass.
The pattern is portable. The implementation is this repository. I am not going to tell you twelve lines drop into any framework, because they do not.
What is different about this one
I have written about tests passing while the contract was wrong and about a harness lying before the model does.
Those were about systems asserting more than they had established. This is about giving an agent a way to assert less than it could get away with — and then finding the exact place where that still is not enough.
An agent that returns a plausible defect claim is normal now.
An agent that marks its own claim as already known is rarer, and better.
An agent that does that while both the interpretation and the class match fail inspection, and a correctly-scoped validator still admits the report — that is the case worth publishing, because it marks the exact boundary of what any structural check can do.
So the contribution is not an agent that finds defects. It is a measured boundary: structural consistency is mechanically checkable; semantic correctness requires evidence beyond those structural checks, and here is a live case that separates them — with the corpus, the frozen prior, the raw response and the runtime receipts all published so you can put the boundary where I put it, or somewhere better.
The goal was never an agent that always succeeds. It is a system where failure cannot quietly become evidence of success.
Built for the TrueForge Agent Harness Hackathon. Repository public, run evidence in docs/. Nothing here claims a verified candidate or a completed sandbox verification, because neither has been established.
Top comments (11)
I find interesting that the failure is not really in the finding itself, but in the system’s model of its own history. That seems like a particularly difficult class of bug because every individual component may still behave reasonably. The detector finds something, the memory layer contains valid records, and the comparison logic returns a result. The error appears only when those pieces jointly construct the wrong interpretation of state.
I think this is one of the reasons provenance becomes increasingly important as systems become more autonomous. It is not enough to know what the system knows; you also need to know where that knowledge came from, when it was created, and under what assumptions it should still be considered valid. Otherwise memory stops being context and starts becoming an unchecked source of truth.
your middle paragraph is exactly the run. every component behaved reasonably in isolation. the detector quoted real bytes that exist verbatim and exactly once in the named file. the frozen prior was valid. the validator returned a structurally correct result and was right to. the error only exists in the joint construction, and no single piece is the one that failed. that is the part i had trouble writing down and you put it in one sentence.
the diagnosis i would push back on is "the system's model of its own history," because the history was clean and i can show it.
prior committed 5bf10acd 2026-08-29 05:55:49Z
run executed 2026-08-29 12:02:41Z
governing_prior_sha256 93820ea5...
prior_freeze_commit 5bf10acd
six hours and seven minutes, hashed before the run, and the receipt binds that exact commit and that exact digest. origin, creation time, and validity conditions all present, all checkable by someone who does not trust me. that is the provenance you are describing, and it was already there.
it did not help. the record was right and the agent's use of it was wrong. K1 as frozen reads "a verdict computed over an empty collection returns success... treated as nothing wrong rather than nothing checked." the code it cited exits 2 on empty, which is a hard failure, the inverse. the agent described that inversion in its own reasoning field and filed the finding under K1 anyway.
so the distinction i would draw is that provenance is a property of the record. this defect lives in the edge between a new observation and a record, and nothing about a record's pedigree constrains whether a fresh finding actually belongs to it. a perfectly sourced, correctly dated, cryptographically bound condition can be cited by something that is not an instance of it, and every provenance check still passes.
where your point does land, and i think it is the harder half: once a wrong association is written down, it inherits the record's credibility. that one i have not solved.
Yeah, that's a fair correction. The prior seems exceptionally well grounded. The failure happens one step later: the agent observes something new, retrieves a valid prior, and then incorrectly decides that the observation is an instance of that prior.
That makes me wonder whether the association itself needs to become a first-class auditable object. Not just “which prior did the agent cite?”, but why did this observation satisfy the conditions for that prior? In other words, provenance tells us where the record came from, but we may also need something closer to traceable classification evidence for the edge connecting observation and record.
And I think your last point is the really hard one. Once the bad association is persisted, it can inherit the credibility of a perfectly valid prior. At that point the system is not merely making one wrong classification but it is potentially manufacturing trusted history from an incorrect edge
That seems much harder than detecting a bad record, because the components on both sides can remain completely correct.
you are describing something that already exists in the artifact, and i went and read the repo rather than answering from memory. the answer is more uncomfortable than i expected.
the finding object carries confidence_basis. it is required by the response schema and it is exactly the field you are asking for: the agent's own account of why this observation satisfies that prior. verbatim from the run:
"The code structure directly matches the pattern: a filter operation that can produce an empty
collection, an explicit check that treats empty as failure, but no corresponding check that
non-empty means success."
and K1, frozen and hashed before the run:
"A verdict computed over an empty collection returns success. An empty array ... is treated as
'nothing wrong' rather than 'nothing checked'."
treats empty as failure, against treated as nothing wrong. the justification for the edge states the inverse of the condition the edge points at, in the same object, and it has been sitting in the published receipt since the run.
what enforces it in code: the schema requires it non-empty, and the validator rejects it if it is a bare number like "87%". that is the whole of it. no code path compares it to the condition it is justifying. i grepped the entire repository for this rather than one file, after someone pushed me on whether i had actually checked.
but here is the part that answers your question properly. the contract does specify the comparison. row B3 of the frozen contract names the hazard as "a wrong-reason pass, a result obtained for a cause unrelated to the check," and its stated control is: every finding records confidence_basis, and a named breaker seat compares observed reason to intended reason on every row.
so the check is not missing from the design. it is assigned to a person. and in this run it either did not happen or did not catch it, because the contradiction shipped.
that is the actual shape of the gap, and it is one we already have a name for: a rule that depends on someone choosing to perform it is a request, not a control. the field is first-class. the comparison is a human promise wearing the contract's authority.
the mechanized version is small and i have not built it: make the cited condition's text and confidence_basis both required inputs to a check that must agree they describe the same direction. what stops me is that "do these two pieces of prose agree" is itself a semantic judgment, and putting a model in that seat gives the checker and the checked a shared failure mode again. that is the same wall, one layer out.
on inheritance you are right and i have nothing. the priors are hashed, and there is a test that rejects a successor prior which quietly rewords a condition, digest mismatch, verified. none of that stops me from writing a wrong association into the next prior deliberately because it looked right. it would then be frozen, dated, cryptographically bound, and true forever as far as any check can tell. a false thing with perfect provenance is worse than a false thing without it.
The wall might be avoidable here if the semantic judgment moves into the frozen prior instead of happening after the model writes prose. K1 can carry an executable predicate such as
empty_count == 0 -> exit_code == 0. The new finding can carry the observed predicate from the quoted bytes,empty_count == 0 -> exit_code == 2. The association is then rejected by a truth-table check; no second model has to decide whether two explanations “mean the same thing.”That doesn’t solve arbitrary prose classes, but it changes the contract: a class without a machine-checkable discriminator can remain context, but it can’t be a
CONFIRMS_KNOWNtarget.confidence_basiscan stay for humans, while the edge only inherits credibility after the predicate passes. In this run, that would have failed before the wrong K1 link could be written into the next prior.this gets past the thing i said i could not get past, and it does it by removing the part i said would poison it.
i had argued that mechanising the check meant asking a model whether two pieces of prose describe the same thing, which hands the checker and the checked a shared failure mode. you deleted the model from the seat entirely. a predicate on the class and a predicate on the finding is a truth table, and truth tables do not have opinions.
i ran your example against the actual run before answering.
K1 predicate empty_count == 0 -> exit_code == 0
observed predicate empty_count == 0 -> exit_code == 2
antecedents match, consequents do not, association rejected. so yes, that specific wrong link dies before it can be written, and it dies without anything reading confidence_basis.
two costs worth naming, because neither is fatal but both are real.
first, neither field exists. a frozen known_condition currently carries description, name, why_listed, blind_spot_evidence, instances_already_found_by_humans and measured_model_blind_spot. no predicate. and a finding carries condition, path, exact_bytes, why_it_matters, evidence, novelty, known_condition_id, confidence_basis, not_established and repair. also no predicate. so this is two new required fields on two frozen schemas, and the prior is hashed, which means it is a successor prior rather than an edit.
second, and this is the part i want your read on: the observed predicate still has to come from somewhere. for this case it is nearly mechanical, an if on a length comparison and a literal exit code. for a class like "a derived label outranking recoverable evidence" i do not know how to extract a predicate from arbitrary bytes without a model doing the extraction, which puts the model back in the loop one step earlier.
but your contract clause is the answer to that and i think it is the strongest sentence in your comment. a class without a machine-checkable discriminator can remain context but cannot be a CONFIRMS_KNOWN target. that does not require solving prose. it requires admitting which classes are decidable and refusing to let the undecidable ones carry a verdict. K1 becomes checkable, K3 stays advisory, and nothing pretends otherwise.
that is a smaller and more honest system than the one i was trying to build, and it is the first proposal in this thread that would have failed the run rather than explained it afterwards.
Isolating the locatability-without-relevance hole specifically to diagnostic findings that propose no repair is a very sharp distinction. When the subject is a static corpus instead of a runtime trace, the only way we have ever bound a non-repair finding without an execution graph is by requiring a minimal falsifying edit or witness token, like forcing the agent to demonstrate the single span deletion or substitution that invalidates its assertion. If a document cannot execute, a pure diagnostic observation without an edit path remains an assertion of intent rather than a verifiable state change.
Your point on the fingerprinting trap is spot on. When an automated classifier forces every distinct failure into whatever structural markers its sensor happens to measure, you get that exact wrong-reason pass where two completely different failure mechanisms collapse into a false match. Keeping novelty human-evaluated until the priors carry executable signatures protects the integrity of your ground truth instead of inventing fake determinism.
Formalizing K2 and K3 as checkable conditions sounds like the right sequence. Once the condition itself can be verified as a predicate over the corpus structure rather than interpreted from prose descriptions, the classifier does not need to guess at intent, it just evaluates the predicate.
The distinction between a model proposing a candidate predicate and a model adjudicating the match is what keeps the loop clean. The finding-generating model can emit the predicate alongside the quoted bytes, but the evaluator doesn't trust the model's prose explanation—it just executes the predicate or checks it against the AST trace. If the generator emits a bogus predicate that doesn't match the AST of the quoted bytes, a syntactic checker rejects it immediately.
For classes like derived labels outranking evidence, the hard truth is that if the property can't be reduced to an AST pattern or a deterministic test harness, it is fundamentally an open-ended semantic claim. Putting a model in charge of deciding whether arbitrary bytes fit that concept is just reintroducing the judge that has the same blind spots as the worker.
Treating those as advisory context rather than gating verdicts is the right trade-off. It forces you to write executable invariants for the classes you actually want to gate on, and keeps the undecidable ones from quietly minting false confidence.
the split is right and i can tell you exactly where my system already draws it, including where it does not hold.
on a finding, four fields get validated. two are executed against the artifact and two are only checked for grammar. path has to exist in the frozen corpus. exact_bytes has to occur exactly once in that file, not roughly, exactly once, or it throws. those two are real. confidence_basis only has to be non empty and not a bare number, so you cannot say 85 percent, you have to say something. not_established only has to be a list of strings. neither of those is verified against anything. so the honest count is two executed, two grammar, and i think that ratio is the actual state of most systems that claim to check their own output.
and there is a limit even on the executed one that i had to sit with. exact_bytes occurring exactly once proves locatability, not relevance. the citation is provably real and provably unique in the file. nothing anywhere establishes that those are the bytes that matter to the claim. so the deterministic check closes half the gap, the half where someone cites something that is not there or is ambiguous. it does not touch the half where someone cites something true and irrelevant.
on the derived label class, i want to push back a little because i went and reread it. thats K3 in my frozen set, and it reads: a conclusion stored in place of the raw values it was derived from, where the raw values remain available on the same record. that last clause is structural. does the record carry both the label and its inputs, and does anything recompute it. you can gate on that without a model deciding what counts as a conclusion. so that one is closer to decidable than you gave it credit for.
where you are right, and harder than you put it, is my novelty rule. a reported condition is NEW only if it matches no known entry by mechanism, not by wording. matching by mechanism and not by wording is not reducible to an AST pattern, and it is the field that gates whether a run counts as a finding at all. so the undecidable thing is not sitting in an advisory corner. it is the gate.
and to be straight, there is no predicate or ast field on my frozen schema at all right now. what you are describing is not something i have and turned advisory. it is something i have not built.
That K3 clarification is really sharp. If the record schema enforces both the raw inputs and the derived field on the same object, checking whether they match or if recomputation passes isn't semantic judgment at all, it's just an executable equality check. That's a much cleaner boundary than I gave it credit for.
The locatability versus relevance gap is where almost every automated evaluation loop gets bruised. Unique byte matching at least guarantees the model isn't hallucinating nonexistent files or phantom symbols, but it can still cite completely unrelated code that just happens to look plausible. The closest thing I've seen work reliably without an LLM judge is binding citations to execution artifacts, like exact stack frame deltas or tool call input/output diffs, so the cited bytes have to sit inside the active mutation path rather than anywhere in the repo.
The novelty gate is the real kicker though. Once novelty requires matching on mechanism rather than text similarity, an LLM judge usually sneaks back in at the most critical choke point. If you ever build that out, I'd be curious whether you end up trying semantic clustering or if you end up fingerprinting structural traces to keep it deterministic.
your execution artifact idea already exists in my schema in a stricter form than you proposed, and finding that made me find the hole in it.
when a finding proposes a repair, the repair’s before_exact must equal exact_bytes. not overlap, not sit inside the touched span. identical, enforced at core.mjs line 229, throws on mismatch. so the cited bytes are the bytes being changed. that is your active mutation path binding, done as equality.
and it only fires when a repair exists. line 227 is if repair is not null. a finding is allowed to propose no repair, and that finding has no relevance binding whatsoever. it needs a path in the corpus and bytes that occur exactly once, and nothing else. so the locatability without relevance gap is not spread evenly across my schema, it lives entirely in findings that do not propose a fix. which is a much more useful statement than the one i made you, and i only got it because you pushed on the mechanism.
your version does not port directly though, and the reason is worth naming. stack frame deltas and tool call diffs need the subject to be a running system. mine is a static text corpus. there is no execution, no mutation path at runtime, no trace to bind to. the precondition your method needs is a runtime, and a document does not have one. same shape as a thing i just went through in another thread, where a check was perfectly correct and simply required something the setting could not supply.
on novelty, the direct answer is neither, and the honest reason is that no llm judge sneaks in right now because nothing automated does it at all. the rule ends: the classification is made by a seat that did not author the finding. a person. determinism was not solved, it was avoided by not automating the step.
and i cannot pick between clustering and fingerprinting yet because that choice is downstream of one i have not made. my six known conditions carry description, name, why_listed and instances_already_found_by_humans. all prose. not one of them has an executable signature. so there is nothing for either method to match against. the work is rewriting the frozen prior as checkable conditions first, and K2, a value with no false case, and K3 are already close to that shape while K1 is not.
the risk i would watch on fingerprinting is that it makes novelty decidable by quietly redefining mechanism as whatever the fingerprinter can see. two genuinely different mechanisms with the same trace shape both come back CONFIRMS_KNOWN, and that is a wrong reason pass, which i would rather have as an honest unknown.