seven silent failures

I created this piece of content for the purposes of entering the All Things Agentic Hackathon.

the thing i built

An agent with no chat box. An email arrives, and it wakes up on Cloud Run, reads the thread, checks my actual calendar, writes a proposal document, and prepares a reply — then stops and asks a human the moment it is not sure. Nobody types anything. There is no prompt. The email is the prompt.

Gemini 3.5 Flash does the tool selection through function calling, Google’s Agent Development Kit runs the loop, and three Cloud Run services keep ingress separate from execution. Pub/Sub carries events between them and Firestore holds the state. The repo is here if you want the actual code.

That is not what this piece is about. This is about the seven things that went wrong, because six of them went wrong quietly, and that turned out to be the real lesson.

the one that shouted

Exactly one failure announced itself. Line endings. I wrote the deploy scripts on Windows, git helpfully stored them with CRLF, and anyone cloning the repo would get bad interpreter: no such file or directory. Ugly, obvious, fixed with a .gitattributes in about ninety seconds.

Every other failure was polite about it.

1. the model that exists, but not here

gemini-3.5-flash returns a 404 on Vertex AI in us-central1. The error says publisher model not found. That sentence sends you hunting for a typo in the model name, which is the wrong place to look — the model is fine, the region is not. It is only served from the global endpoint.

I found it by brute force, asking three regions in a row. Nothing in the error text points at geography.

2 and 6. iam, twice, the same way

Two separate failures, identical shape. A service account gets created, the very next line binds a role to it, and the bind fails with service account does not exist. It does exist. It just does not exist yet — IAM is eventually consistent and the propagation window is tens of seconds.

The first time this bit me, the failure was inside a line ending in || true. I had written that to make the script re-runnable. What it actually did was swallow the error, skip creating the Pub/Sub subscription, and leave me with a deploy that printed three green service URLs and a completely disconnected pipeline.

"Every piece of visible config was correct. The thing simply did nothing."

The second time was Cloud Scheduler, which needs its own service agent to hold serviceAccountTokenCreator on the account it impersonates. Without that grant: no scheduler log, no request at the target, no error anywhere. And while the grant propagates, it fails in exactly the same silent way — so I concluded it was broken and wrote it off. It started working four retries later.

3. gmail will not tell you what arrived

A Gmail push notification contains {emailAddress, historyId}. That is it. No message ID, ever. To find out what actually landed you replay users.history.list from a watermark you stored yourself, and one notification can cover several messages.

I had built the obvious version, passing the historyId through as though it identified something. Every real trigger 404’d.

The part worth sitting with: every test I had written before that point passed, because every test supplied the message ID by hand. I had thoroughly tested a code path that production would never take.

4. a container has no idea where it is

datetime.now().astimezone() resolves to UTC on Cloud Run. So “available 9 to 5” quietly became 9–5 UTC, and my agent started offering a prospective client 5 a.m. meetings.

It labelled them UTC correctly, too. The output looked right. It was just useless.

This is the one I would most likely have shipped, because it passes every test written by someone sitting in the same timezone as their server — which is to say, every test I would have thought to write.

5. the guardrail worked; the output did not

The agent signed a draft [Operator]. A literal placeholder, in an email a human was one click from sending.

Nothing was broken. The escalation logic was working perfectly the entire time. The tool returned ok=True, because the API call genuinely succeeded. A tool reporting success tells you the call went through, not that what came back was any good — and nobody had read the output yet.

7. i proved it, wrote it down, and shipped it anyway

This is the one I would lead with.

Pub/Sub delivers at least once, so every event eventually arrives twice. For an agent with side effects that is the difference between one draft and two. I handled it properly: before doing anything, the router performs an atomic Firestore create() on a key derived from the event. The create fails if the key exists, and the failure is atomic, so there is no read-then-write race. I was pleased enough with this that I put the formula in the write-up.

Then I sent two emails at once and the agent wrote two proposal documents for one of them.

Gmail sends one notification per arriving message. Two emails, two notifications — and both replay the same history window, so both surface both messages. I had keyed the claim on historyId + messageId. Two views of one message, two different keys, no collision, two drafts.

"The lock worked exactly as designed. It was guarding the wrong door."

The lesson is not “handle at-least-once delivery.” I had handled it, and I had a proof. It is that a deduplication key has to name the thing with the side effect, not the thing that told you about it. I had tested Pub/Sub redelivery, which is the easy case and the one I had thought of. The case that broke it only appears when two messages arrive close enough together to overlap their history windows — which is exactly what a demo does, and exactly what a quiet test inbox never does.

the pattern

Six of seven failures produced no error. Not a caught exception, not a warning, not a log line. Several produced output that looked correct: meeting times labelled with the right timezone, a deploy printing three healthy service URLs, a cron job sitting in the console marked ENABLED.

The common thread is that distributed systems fail by omission far more than by exception. A missing IAM grant and a still-propagating IAM grant are indistinguishable. A created cron job and a working cron job look identical until the day you need it. A tool returning ok and a tool returning something useful are different claims.

Which means the only thing that actually found these was running the real thing against real data and reading the output. Not tests — my tests passed throughout. Watching it work, end to end, and being suspicious of success.

what i would do differently

• Never write || true without logging what it swallowed. That one line cost me a disconnected pipeline that looked deployed.
• Assume IAM lags. Poll for the resource before binding to it, instead of assuming that created means usable.
• Test in the environment’s timezone, not yours. Or better: refuse to have a default. Business hours without a named zone are meaningless.
• Read the agent’s actual output. Every run. ok=True is not a quality signal, and the placeholder signature proved it.
• Name your idempotency key after the side effect. Not after the notification, not after the request. After the thing you would hate to do twice.

The agent works now. It reads a real inbox, does real multi-step work on real APIs, and stops to ask a human when it is not sure. That last behaviour is the one I am most pleased with, and it is the only one that never broke.