Disconnected Contains Connected: How a Substring Check Verified a Failed Deploy as Good

On 25 August 2026 our deploy script restarted a service, read its health endpoint, and printed "bridge is connected". The endpoint had just returned a status of disconnected. The gate tested for the substring connected, and disconnected contains connected, so the deploy exited zero on a component that was down.

The fix is three changed patterns in one file. The reason it is worth writing up is that it was the second time that morning our own verification lied to us about a system that was behaving exactly as built.

What did the deploy verification actually check?

Whether the health response contained the letters c-o-n-n-e-c-t-e-d anywhere in it. The script polled the bridge's health endpoint up to 36 times, five seconds apart, and broke out of the loop the moment the response matched the shell glob *connected*. It then ran the same match again to decide whether the deploy had succeeded.

In POSIX shell, case "$h" in *connected*) is a wildcard pattern match, not a word match. The Open Group's Shell Command Language specification defines the asterisk as matching any string, including the empty one. So the pattern is satisfied by connected, by "status":"connected", and just as happily by "status":"disconnected".

Every negation English builds by prefix does this. disconnected contains connected. unhealthy contains healthy. unauthorized contains authorized. invalid contains valid. unavailable contains available. The failure token is usually the success token with something bolted onto the front.

Which means a substring health check does not fail randomly. It fails in one direction only: it turns failures into passes. It will never produce a false alarm on a working system, so it will never annoy anyone into fixing it.

How did we notice, if the deploy reported success?

Because the script printed the raw health payload one line above its own verdict. The transcript read health: {"status":"disconnected"...} and then, on the very next line, bridge is connected. Two adjacent lines contradicting each other are hard to read past.

That was luck wearing the costume of a habit. The line that printed the payload exists so a human can see the evidence and not just the conclusion. Strip it out and the run becomes indistinguishable from a good one.

Rob Ewaschuk, author of the monitoring chapter in Google's Site Reliability Engineering book, sets the bar as: "Your monitoring system should address two questions: what's broken, and why?" Ours answered neither, and answered them with confidence.

Why did 49 passing tests not catch it?

Because not one of them tested the deploy script. The suite covered the feature being shipped and was green throughout: 49 tests at the moment of the apply, 56 by the end of that day. The gate deciding whether the shipped code was actually running lived outside the thing under test.

That is not really an argument for testing deploy scripts harder. It is an argument that a deploy script is software with a return value, and this one's return value was wrong. The same script had already produced a second verification defect that morning. It set four environment keys inside a loop reading from a pipe, and ssh inside that loop consumed the loop's stdin, so only the first key was ever written. The dry-run printed one key where four were expected, which is the only reason it surfaced before the live apply.

Neither failure was the system breaking. Both were our instruments reporting on a system that was doing precisely what it had been told. We have written up the expensive version of this before, in the six days our agent mesh was dead while every check stayed green.

What does the corrected check look like?

It matches the whole field rather than the word. The pattern became *'"status":"connected"'* in all three places the old glob appeared, with a two-line comment above the last one naming the trap so nobody reintroduces it.

This is still a substring match. It is a far narrower one, because "status":"disconnected" no longer satisfies it, but it is pattern-matching over JSON and it will break the day the payload gains a field called last_status or the serialiser changes its spacing. The genuinely correct fix is to parse the response and compare the value. We did not do that here, because the check runs through a shell inside a container and adding a JSON dependency to that path costs more than the narrower glob is worth today. That is a trade with an expiry date on it, not a solution.

We build and run agent infrastructure for other companies using the same deploy scripts we point at our own boxes. If you want that work done by people who publish their false passes, our AI agent development services page covers the engagement, and the live mesh running our agency covers what it holds up.

Where else does this pattern hide?

Anywhere one valid status value is a substring of another and the matcher is not anchored. A short list of shapes worth grepping your own scripts for:

• grep 200 on a status check, which also matches a content-length of 1200
• *ok* matching broken, blocked, and token expired
• *success* matching the literal string success: false
• docker ps | grep -q Up, which matches Up 2 seconds (unhealthy) exactly as well as it matches a healthy container
• *connected* matching disconnected, reconnecting, and connection refused

The asymmetry is what makes it rare enough to feel safe. disabled does not contain enabled, so half the status vocabularies you check are fine, and the half that are not look identical from the outside.

What rule did we take out of it?

Grep for success and you will find it. If a check searches output for the token that means good, and the output that means bad contains that token, the check reports good. The defect is not in the string. It is in asking a question with only one available answer.

The operational form, which is the part that costs something: before a gate ships, feed it one output you know is bad and watch it fail. A gate that has never returned non-zero has not been tested. It has been observed agreeing with you. We made the same point from a different angle in stated controls are not controls: a control described but never triggered is a description.

Frequently asked questions

Should deploy scripts use jq instead of shell globs?

Where jq is present on the machine running the check, yes. Compare the parsed value directly, along the lines of printf '%s' "$h" | jq -r .status, and test that against connected. It fails loudly on malformed JSON rather than quietly matching nothing, which is the second behaviour you want from a gate. Our constraint was a check executing through a shell inside a container we did not want to add a dependency to.

Does set -e protect against this?

No. set -e aborts on a non-zero exit status, and this check exited zero because the pattern genuinely matched. There was no error to abort on. set -e protects you from commands that fail; it has nothing to say about a command that succeeds at answering the wrong question. The same goes for set -u and pipefail.

Isn't this what a Docker healthcheck or a Kubernetes readiness probe is for?

Partly. A container healthcheck would keep polling and eventually mark the container unhealthy, which beats a one-shot gate. It would not have fixed the script. Our gate ran its own poll and reached its own verdict, and a container marked unhealthy still needs something to read that state and refuse the deploy. The probe emits a signal. Something still has to act on it correctly.

How is this different from a flaky test?

A flaky test is non-deterministic and announces itself by changing its answer. This was perfectly deterministic. It returned the same wrong answer on every run, which is exactly why it survived. Deterministically wrong is much harder to find than intermittently wrong, because nothing ever draws your attention to it.

Would code review have caught it?

It did not. The pattern *connected* reads as obviously correct right up until you say the failure string out loud next to it. The reliable catch is not review but a negative test: the reviewer asks to be shown the gate failing, and there is nothing to show.

Talk to us about what your gates actually assert

If you are running agents, deploys or automations where a green check is the only thing between you and a silent outage, it is worth an hour of someone's attention. Get in touch and we will go through what your health checks are asserting, and what they would report if the thing they watch stopped working.