The green test that lied — two types of false pass your CI can't catch

Boss, if the tests are green, that means there are no bugs, right? A row of green checks and you can ship with a clear conscience. …Right? Here’s the thing, Rookie. I took the rule “the mascot must not cover the body text” and made it a real automated test. Not a prose note — a running gate. And yet the exact same defect, “the mascot covers the text”, shipped three times, untouched. The first two times, a human said “I checked every scene” and called it green. The third time, I’d made it a gate — and the gate itself went green, on its own. It never once turned red, all the way to ship. The reason? The test measured the text’s brightness, not its position. Today we dissect a test that’s green and still lying. Take this home, and you’ll be able to tell your CI’s real green from a stand-in wearing green. Hold on. Last time you said prose rules rot, so make them a gate that fails. I took that seriously. So a gate means safe, I figured. Right, that was last time’s conclusion. Writing “never do this again” in a doc doesn’t fix it. So encode the behavior where it’s decided — a default, or a gate that stops the build. Correct. Half correct. Half? Today’s the sequel. I did exactly what I said. I wrote a check — “if the mascot covers the body text, stop the build” — and folded it into the build. Not prose, running code. …And what do you think happened? Uh, you made it a gate, so it stopped, right…? It didn’t stop. The first two times, a human declared “no overlap in any scene” and it went green. And the third time — I’d made it a gate, and the gate went green, brazenly. The same bug reached viewers three times. Last time was right to say “make it a gate.” But it wasn’t enough. I made it a gate and it still didn’t work. Because the kind of gate was wrong. Turning prose into a gate isn’t enough. If that gate doesn’t measure the real thing, the green is a lie. Wasn’t measuring the real thing… the gate was watching “the mascot must not hide the text”, wasn’t it? That’s the crux. The condition we actually wanted to protect — the invariant — is this: “the body text never drops down below the mascot.” It’s about the text’s position. But what the code I actually wrote measured was this. See? Position isn’t measured anywhere. What it measures is “how many bright pixels are under the mascot.” More than 60 bright pixels, and it decides the text is hidden. 60 or fewer, green. …So this gate measured “brightness” — a different number — in place of “the text’s position.” Ah… but text is bright, so if it’s not hidden there are lots of bright pixels, and when it’s hidden they drop. That’s roughly right, no? “Roughly” is the trap. Let’s call the number the gate actually measures the “stand-in.” The real thing we want — the invariant — is “position.” The stand-in is “brightness.” Normally, when things go well, the two move together. Text sitting up top is bright; hidden, it goes dark. So the stand-in looks like it’s working, and when you write the test, it passes and you relax. We watch only the “normal times,” when the stand-in moves with the real thing, and we convince ourselves “this test works.” But — when things break, the two can move in opposite directions. That’s where the green lie is born. The scary part of a stand-in is that it agrees with the real thing nine times out of ten. So no one suspects it. The remaining one in ten — exactly when it breaks — it quietly betrays you. Move in opposite directions…? You mean the text is hidden, but the bright pixels don’t drop? Worse. When the text is hidden, the bright pixels DROP. That’s the problem. When the semi-transparent mascot overlaps the body text, the text sinks dark — we call it wash-out. The text’s brightness falls. So the “bright pixel” count the gate is counting falls too. It drops below 60. So the gate… goes green. You see the hell of it? The worse the overlap gets, the darker the text, the fewer the bright pixels, and the more the gate’s number falls to the “safe” side. The more broken it is, the deeper the green. The gate went green not because the text was protected — but because the bug blinded the gate. Ugh… it looks safest at its very worst? And there’s one more way through. This gate only counts the pixels right under the mascot’s shape — its alpha. But the mascot isn’t the only thing darkening the screen. A dim overlay in the four corners sinks the text too. And that dimming happens not under the mascot, but outside it. The gate doesn’t even go look there. A stand-in that measures the wrong place will miss the lie happening right next door, even while it measures its own spot correctly. This is the third one, after I’d made it a gate. The body is washed out and dark, the axis labels and numbers unreadable. And yet the gate, right next to it, went green — “clear under the mascot, pass.” — The time before that, something even more vivid shipped. A line of code explaining “if it fails, stop the whole build” — its first character sank into the mascot and vanished, and the screen read “ys.exit” as it shipped. A broken line, in a video about things breaking. That was still a round where a human eyeballed it and said “I checked.” The third time, I handed that checking to the gate — and the stand-in number went green without blinking. But that’s because it measured brightness, right? Fix it to actually measure “position,” and the green lie is gone… right? Here’s the thing — there’s a SECOND kind of green lie. And this one is nastier. From here, in the back half, I’ll show you that second kind. Without it, even a gate that measures position will fool you again. On another video, this happened. A certain explainer slide — the payoff slide showing “the self-reported length vs. the measured length,” the punchline of that whole video. It was supposed to be on screen for about 20 seconds. But in the build output, it never appeared on screen at all. Zero seconds. Gone entirely. Gone!? And the gate…? Sixteen scenes, all sixteen green. But that one was empty inside. Not a single line of dialogue assigned to it, no pixels to inspect. If there’s nothing to count, then “zero is 60 or fewer” and of course it passes. Nothing there, so — pass. In math this is called vacuous truth — an “empty truth.” The gate said “there is no hidden text.” Correct. Because there wasn’t a single character there. …pass because there’s nothing? That’s the same as not inspecting at all. Exactly. Kind one: “the stand-in number moves backwards, so a false green.” Kind two: “the input is empty, so it inspects nothing and gives a false green.” On the surface, both are the same clean green checkmark. Inside, one’s been blinded, the other’s empty. Green isn’t proof the condition held. Green might be proof the test’s eyes were shut. This isn’t just about your mascot videos, is it. I’m… starting to feel like I’ve seen this in my own tests. You should have. This hides in every green in your CI. Same pathology — the test measures a “stand-in number” instead of the “real invariant.” 80% coverage only measures that lines of code “were executed.” Call a function, throw away the return value, write not a single assert — the line still runs. Coverage goes up. Green. But it guarantees nothing — this is the same “stand-in” as brightness earlier. “Executed” is the stand-in for “verified correct.” A test that swallows the exception in a try and passes without asserting anything is the same. Green. A snapshot that auto-updates and matches every time. Green. A mocked dependency measures the mood of the mock, not production. Every one of them: the stand-in number goes green, and no one watches the real thing. And — an empty test, empty input, a zero-result — same as the vacuous PASS earlier, it quietly gives you “nothing there, so pass.” A test that loops over a list — what if that list happens to be empty? The asserts inside never run once, and the test slips through, still green. So counting the number of greens is meaningless…? It’s not the “number” of greens — it’s what each green guarantees. If you can’t say in one sentence what a single green protects, that green is probably measuring a stand-in. So how do I fix it? Tell me in a form I can paste straight into my own tests. One at a time. 1. Fill in the blank in “if this test is green, ___ is guaranteed.” If you can’t, the test protects nothing. 2. Stop measuring the stand-in and assert the invariant itself. In my case, not “60 or fewer bright pixels” but “the bottom coordinate of the body text is above this line.” If you want to measure position, measure position. 3. Never let empty input go green. Zero results, empty list, null — that’s not “passed,” it’s “couldn’t inspect.” Tip it to fail-closed — when in doubt, drop it. What’s number 4, “encode it in the build, not memory”? “I confirmed there’s no overlap” and “I wrote it in a doc” both rot by the next run. Human memory and prose notes have no guarantee of being read. In fact, every one of my overlaps shipped after someone declared “I checked every scene for overlap.” The “I checked” evaporates by the next build. So step 4 is dropping steps 1 through 3 into code the build runs itself every time, and stops if it fails. Only when no human has to remember and no doc has to be opened does the lesson become permanent. 1 through 3 are “what to measure,” 4 is “where to put it so it doesn’t rot.” That’s the prescription. …And there’s one thing I have to be honest about. Something to be honest about? These four steps — I wrote them all down, properly, in my own LESSONS doc. “Measure geometry, not brightness.” “Make empty scenes FAIL.” “Fold it into the build.” …And as of recording this video, exactly one of them is actually wired in. Just one…? The rest? Making an empty scene FAIL — the one that kills the second kind of lie — I implemented. It’s right there in the build code. But the first kind — replacing brightness with coordinates — the code still measures brightness. The prescription’s still waiting inside the doc. Folding the gate into the build to stop it automatically — still a command I type by hand. …Notice it? This is exactly today’s story itself. “Wrote the lesson” and “the behavior changed” are two different events. I dissected the green lie, and the fix for it still can’t go green in my own build. It’s the same disease, one level up. So today I won’t pretend at a clean conclusion. I’ll show you the seams and all. So what we take home is… the next step past “make it a gate.” Right. Last time was right — “turn prose into a gate.” Today we add one line on top: green only means something when that gate measures the real invariant AND is actually wired in. Green that measures a stand-in is just a green checkmark stuck onto prose. Watch last time’s video — “rules rot, only defaults hold” — and you’ll see where to put a lesson so it works. Next, we do the opposite — an “over-strict schema” so tight it rejects even the correct things. The exact flip side of today. Is your green real, or a stand-in? Just one — today, ask one of your tests, “if this is green, what does it guarantee?” Subscribe, and I’ll see you in the next one.