I built ExceptionZero for Google's All Things Agentic Hackathon. This post covers how it was built, and specifically the failure mode that cost me the most time — I wrote it for the purposes of entering this hackathon.
Five times in five days, I shipped a component that returned a clean, well-formed, entirely plausible response while doing nothing at all.
Not one of them threw an error. Not one showed up in a log as a failure. Each was caught by noticing that a number was wrong.
That turned out to be the most interesting thing I learned building a multi-agent system, and it's not what I expected to write about.
What I was building
Every business has someone who opens a spreadsheet of failed payments every Monday and works it by hand. Pull the invoice, check the customer record, look at what happened last time, work out what went wrong, fix it, confirm it worked.
An exception is, by definition, the case automation couldn't handle. If a rule could resolve it, it wouldn't be an exception. That's why this has never been automated — it needs judgment on incomplete information, with money at stake.
ExceptionZero is ten agents that do that investigation autonomously, and — the part I care about — stop when they shouldn't touch something.
Failure one: the guardrail that quarantined everything
I wired Model Armor in as an ADK before_model_callback, screening the assembled prompt before it reaches Gemini. Then I deployed and ran twenty cases.
All twenty came back quarantined.
The service returned 200s. The JSON was well-formed. Every case had a sensible-looking reason attached. The system had stopped doing its job entirely and looked completely healthy.
The cause was a substring check:
findings = [k for k, v in filter_results.items() if "MATCH_FOUND" in str(v)]
NO_MATCH_FOUND contains MATCH_FOUND. Every clean filter result registered as a hit.
What makes this worth writing down isn't the bug — it's that the only signal was the tally. A fleet that quarantines 100% of its input produces exactly as many HTTP 200s as one that works.
Failure two: the fallback that never fell back
Before that, the same guardrail had the opposite problem. My screening function was:
return _armor_managed(text) or _armor_local(text)
Managed Model Armor first, local detector as fallback. Except a verdict object saying "clean" is truthy — so whenever the managed API answered at all, the local detector never ran.
I'd described this in my README as defence in depth. It was defence in depth until the first layer answered.
Failure three: the registry that was never written
The agent registry publishes each agent's capability, version, service account and tool scope to Firestore, so another team can discover approved agents. GET /registry returned a perfect catalog.
It was reading from memory. The Firestore client library wasn't installed in the container, the write failed, and my graceful fallback returned the in-memory copy — correct behaviour, and completely indistinguishable from success.
The tell was two missing fields. domain and published_at only exist on the Firestore write path. The response looked right; it just had less in it than it should have.
Failure four: the scheduler pointing at nothing
I wrote a docstring saying "Cloud Scheduler hits /sweep" and then repeated it as though it were true. There was no Cloud Scheduler job and no /sweep endpoint. It took someone asking "are you sure?" for me to check.
Failure five: the fan-out that was invisible
The context coordinator dispatches four specialists concurrently, each under its own service account, each scoped to one table. I instrumented each with an OpenTelemetry span, deployed, and opened Cloud Trace.
The spans were there. The attributes were right. And the trace said Spans: 1.
OpenTelemetry context is thread-local. The fan-out spawns threads. Each specialist became its own root trace, so the thing the spans existed to demonstrate — four agents running in parallel under one case — was the one thing you couldn't see.
The fix is three lines: capture the parent context before submitting, attach it inside each worker, detach after. But the failure is instructive. The delegation was real. The instrumentation was real. The evidence was absent, and nothing anywhere reported a problem.
The pattern
In ordinary software, a broken component usually announces itself. It throws, it times out, it returns a 500.
In an agent system, the components most likely to fail silently are the ones you added for safety: guardrails, registries, schedulers, tracing. They're written to degrade gracefully — a guardrail that crashes the fleet is worse than one that lets a case through — and graceful degradation is, by construction, indistinguishable from working.
Three things I'd do differently from the start:
Check numbers, not status codes. Every one of these was caught by a count being wrong: 20 quarantines instead of 1, spans: 1 instead of 6, a missing field, an empty store. None by an exception.
Make fallbacks loud. My identity module now prints when impersonation fails rather than quietly using ambient credentials. It's noisier and it's correct — a silent fallback means the security model stops being enforced and nobody finds out.
Distinguish "couldn't" from "wouldn't". My first identity self-test reported both impersonation failure and IAM denial as "denied". Those are opposite outcomes: one is broken, one is the control working. Separating them turned an unusable diagnostic into a proof.
The thing the project is actually about
None of that is what I set out to build. What I set out to build is an agent that knows when to stop.
The demo's best moment isn't a resolution. It's a case where an €81,000 invoice was paid at €44,000, and the fleet says:
The payment amount is significantly less than the invoice amount, and there is no evidence explaining this large partial payment.
Confidence lands below the floor. The value lands above the ceiling. Two independent conditions, either sufficient. The agent wanted to resolve it, and the system stopped it.
Three design decisions make that possible, and all three are about taking authority away from the model.
The diagnosis agent has no tools and no data credentials. It reasons only over evidence another agent retrieved. Its service account holds aiplatform.user and cloudtrace.agent and no data role at all — so it cannot query the estate even if it decided to. That's IAM refusing the call, not a prompt asking it not to.
The risk gate is deterministic code, not a model call. A probabilistic gate isn't a control. Confidence floor, value ceiling, reversibility, screening status, counterparty history — five conditions, evaluated identically every time, auditable afterwards.
Reversibility isn't self-reported. Early on, the diagnosis agent decided that voiding a duplicate submission was irreversible, and blocked its own valid resolutions. The fix wasn't a better prompt. It was removing the question from the model: an action is reversible if and only if a compensating action is defined for it. Letting an agent describe the consequences of its own proposal is the same failure the gate exists to prevent.
One more finding
I built stub agents first, to exercise the pipeline without model calls. They resolved 40% of cases.
Then I connected real Gemini and it resolved zero.
It was right to. My stubs had a hardcoded type-to-action mapping, which papered over the fact that my synthetic data didn't contain the evidence needed to reach a conclusion. A duplicate submission with no original payment on file is genuinely unverifiable, and escalating it is correct.
Every jump in resolution rate after that — 0%, then 30%, then 40% — came from fixing the data, never from loosening the agents. A model that refuses to assert what the evidence doesn't support turns out to be an excellent test of whether your evidence is any good.
ExceptionZero runs on Gemini 3.5 Flash via Vertex AI, with Google ADK, Cloud Run, Pub/Sub, Firestore, BigQuery, Cloud Trace, Model Armor and Cloud Scheduler. All data is synthetic and reproducible from a seeded script.
Live: https://exceptionzero-vkh44dcwiq-uc.a.run.app
Code: https://github.com/sgade123/exceptionzero
Demo: https://www.youtube.com/watch?v=koK85iqdSQc
Built for the All Things Agentic Hackathon. #AllThingsAgenticHackathon
Top comments (0)