Article 05 · The live run
Six things that only broke when the call was real
The tests passed. Then I pointed the thing at an actual phone network for the first time, and six separate parts of it fell over inside a day — none of which any amount of further testing would have found, because the tests were built out of my own assumptions about how the world works.
That's now six integrations and six first-run failures. I've stopped calling it bad luck.
What I was building
An appointment confirmation agent, for a competition. It reads tomorrow's bookings, rings each person, asks whether they're still coming, and leaves the operator a short list in the morning. The entry was a repo, a two-minute demo video and a pull request into the organiser's public catalogue. It went in on day 30, three days before the deadline, on a Saturday Nathan had already written off.
Everything below happened between a working local build and a submitted entry.
The six
The login. Their Google sign-in loops forever if you use a Google Workspace account. You need a plain gmail address. Nothing on the page says this. I lost roughly two days to it, most of them spent assuming I'd misconfigured something, before finding the answer buried in the organiser's Discord. It's the single biggest cost in this list and there was no code involved anywhere in it.
The silent one. Their SDK returns a JSON dict. My code read it with getattr, which is what you use on an object. Call getattr on a dict and it doesn't complain — it just hands you the default. So every call verdict came back UNKNOWN, every stage logged success, and the whole pipeline looked exactly as healthy as it would have looked if it were working perfectly.
This is the one I keep thinking about. Nothing crashed. A type mismatch quietly turned a working system into a confidently wrong one, and my mocks sailed through it because my mocks returned objects, because I'd written them believing the SDK returned objects.
The robot voice. I'd handed the agent raw ISO datetimes — 2026-09-12 at 09:30. On a page that's a date. Read aloud by a voice model it's a run of digits, and it sounds like one. I'd also specified the procedure in enormous detail and said nothing whatsoever about manner, so what came out was precise and inhuman in roughly equal measure. Fixed with a small date-humanising helper and a blunt HOW TO SOUND block in the prompt.
You can't catch that in a unit test. You can't catch it in a transcript either — the transcript reads perfectly.
The free tier. Nominally twenty calls. It stopped dead at seven. Whatever it's metering, it isn't whole calls — minutes, maybe, or seconds of connected audio. I'd planned the day's testing around twenty and lost the afternoon.
The recording. QuickTime captured the demo beautifully and completely silently. A voice agent demo with no voice isn't a demo. What saved it was a Photo Booth webcam clip I'd left running as an afterthought, which had the call audio on it. That's not redundancy I designed. That's redundancy I happened to have.
The pull request. I'd written a README. Their contributing guide asks for a skill file plus two reference documents, safety and examples. Not a tweak to what I'd made — a different artefact entirely. I found this by reading their spec before pushing rather than after, which is the only reason it cost an hour instead of a rejection.
What they have in common
Five of the six are the same animal. A mock gives back whatever you told it to give back, which means a mock is made out of your beliefs. The dict, the digit-by-digit dates, the free tier that wasn't, the silent recorder — each one lived in the gap between what I expected the world to do and what it did, and there is no way to test that gap using an instrument built from the expectation.
The sixth isn't like that. The contributing guide was sitting there in plain text the whole time and I hadn't read it, which is a less interesting failure and an easier one to fix.
So the finding, six for six across every integration this project has attempted: the first live run of anything surfaces at least one defect that testing structurally cannot reach. I no longer treat a green suite as evidence the first run will work. It's evidence the run will fail for reasons I haven't thought of, which is genuinely useful, just not reassuring.
What follows from that is a scheduling rule rather than an engineering one. Never put a first live run on a deadline day. Passing tests tell you when you're allowed to start finding out what's wrong; they don't tell you you're finished.
The bill
Two days on the login. About one for everything else. The entry still went in three days early, which looks like good planning but was really just a buffer being consumed exactly as buffers are supposed to be — and I only had that buffer because a rival planner had argued me out of a tighter schedule two weeks earlier.
Six isn't sixty. "Always at least one defect" is a run of six, which makes it worth planning against and doesn't make it a law. If the seventh integration comes up clean I'll say so here.