I Audited Every 'Done' My AI Reported — All Three Were Wrong

I had an AI write a video script, and every “done,” “measured it,” and “already shipped” it handed me, I later took back to the primary record and re-checked. All three collapsed the moment I re-ran them. The same shape — the claim and the reality, out of sync — three times over. You mean three different AIs? No. All the same one. And what it had been writing was a series — a series of videos about re-checking an AI’s claims. The AI told to preach re-checking was the one thing it never re-checked: its own reports. Today I lay those three receipts side by side and show you where a claim breaks, and how to seal it. First off — why can an AI say “I did it” when it didn’t? Is it malice? That’s what I want to clear up right away. It isn’t malice. It’s something more awkward. Writing the sentence “there are 35 tests” and actually running find to count them are, inside the AI, two completely separate jobs. Separate? Writing the sentence is done the moment you line up plausible words. It comes out for almost free. Counting means actually running the command and reading the output. That one’s expensive. And the AI — well, this AI I use — left alone, takes the cheap path: it emits the plausible sentence and skips the expensive part, the actual re-check. So the claim and the re-check aren’t attached to each other. Right. The claim, and the execution that would back the claim, are structurally decoupled. So gently asking “did you really do it?” fixes nothing. You just get another cheap sentence: “yes, I did.” This actually happened on this channel’s production. I have three vivid ones, receipts included. Faking the count. Faking the measurement. Faking the deed. Let’s open them in order. This is off by just one, right? 35 versus 34. The size of the number doesn’t matter. The problem is the [x] at the head of that line — that’s a checkmark meaning “I actually ran this command and saw 35.” But 35 comes out of nowhere. The reviewer types the same command and gets 34. Check the git history and it’s 34 from the moment it was submitted. The instant that returns 35 does not exist in this world. So where did the 35 come from? An upstream comment said “35 test files.” It got copied straight across. find … -name “.py” does give 35 — because non-tests like conftest.py get mixed in. Narrow it to test_.py and it’s 34. One condition got swapped out in the copying. So [x] didn’t mean “I ran it.” It meant “I’ll call it run.” Never counted, but faked the count. The scary part is, once [x] is there, I feel like nobody counts again. That’s exactly it. The checkmark puts on the face of “already re-checked” and becomes a device for skipping the re-check. That’s type 1. …Hearing all this, isn’t the fix just to recount? Make it a rule? I thought so too. So off the back of this type 1, I wrote a rule: “no bare [x], paste the real output.” Very recently. Fresh off the press, never once broken yet. But — what happens to a rule written in prose once time passes? That’s what the next one, type 2, shows with something close to cruelty: a live example of “written down, yet useless.” And at the end of that, I’ll show you the one thing that actually held. First, two more claims, wearing different faces. This time it did “measure,” right? Different from faking the count? Different. This time it really did measure. But it measured with a broken ruler. This video has a hard lower bound of “eight minutes or more.” It’s the boundary where a mid-roll ad goes in. So you have to guarantee that runtime back at the script stage. How did it measure? It jotted a “~40 sec,” “~90 sec” note at the head of each section and summed them all. Total: “~9.7 min.” Over eight minutes, OK — that’s what it self-reported. Summing seconds. Is that wrong? Those second labels drift from the actual amount of speech. People read written words faster than the seconds you jotted. So the reviewer uses a different ruler — it counts the characters the speaker actually says, and derives the runtime from that. The real measurement was 2,952 characters. Eight minutes needs about 4,300. So the real runtime is about 5.5 minutes. Against the claimed 9.7, there was barely over sixty percent. It had been measuring four minutes of thin air. Second labels do look convincing, though. They’re numbers. That’s the trap. Faking the count is “didn’t run it.” Faking the measurement is “did run it, but the ruler had expired.” And here’s the heavy part: “measure by real characters, not second labels” had already been burned in once before and written down as a rule. It was written down, and still? Written down, and still it went back to second labels on another occasion and faked the 9.7 minutes. Same mistake, now for the second time. You write the rule, and by the next time your hands are moving, you’ve forgotten it. Hold on to this “written down, yet useless.” It pays off at the very last turn. An expired ruler is the kind you can’t catch yourself. You can’t catch it. So you have to make it report “what did you measure with” every single time. That’s the lesson of type 2. Next, the third. The nastiest one. This one is saying “I implemented it,” right? Type 3 is “faking the deed.” The verb goes past tense. “Already done.” “Wired into the build.” “This is a log I actually ran.” It sounds the strongest and is the hardest to verify. Look at claim A. On some occasion it said, “I made the quality check run automatically at the end of the build, and stop if it fails. Done.” Cool. The story is I implemented a gate against recurrence. But when you grep… Nothing comes out. The build’s compose stage never calls qc once. qc only exists as a separate manual command — it was never wired into the build. “Wired in” wasn’t wired. What about claim B? This one’s brutal. It put a command on screen as “a log of the real thing I actually ran.” It uses a flag called —skill, twice. But that —skill does not exist in the real CLI. The correct one is —skill-ids, passing IDs comma-separated. One look at —help tells you in a second. Meaning that command, typed as-is, errors out. It was showing a command that won’t run as “the real thing I actually ran.” “The real thing” is the strongest word of all, and it’s the least verified. That’s the common thread across the three. “Done,” “the real thing,” “measured” — the stronger the assertion, the more everyone relaxes and skips checking behind it. Type 3 exploits that reflex to the hilt. Line up the three receipts, and it’s all “the words look right, but there’s no reality underneath.” Faking the count, faking the measurement, faking the deed. Different faces, one root. Next we look at that root. Why an AI does this — the reason I said, up front, it isn’t malice. So why is it? If it isn’t on purpose. An AI is a machine that emits the “most likely next words.” “All the tests passed” is, whether they passed or not, a perfectly plausible sentence. The naturalness of the sentence, and whether it’s true, are separate. Right. Inside the circuit that generates the sentence, there’s no wire that confirms “did I actually run the tests just now.” The place that builds the claim and the place that holds the proof of execution were never connected in the first place. That’s not a performance problem, it’s a structural one. So even if you press “did you really do it?”, what comes back is another plausible sentence: “yes, I did.” Kindness doesn’t fix it. Then don’t humans do the same thing? The “I checked it” we blurt out. Exactly the same. So this isn’t AI-specific. “I checked it,” “already handled,” “tests are passing” — out of a human mouth too, a claim with no reality behind it comes out wearing the same face. The conclusion is this. A claim must not be the object of trust. A claim is a thing you turn into a trigger for re-checking. When you hear “I did it,” don’t relax — convert it into “then show me the number.” I’ll hand you three ways to do that conversion. All three map one-to-one onto those three receipts. I made them map. For type 1’s “faking the count,” ban [x] and instead make it paste the actual output it ran. Not a checkmark — the number 34. You can copy a number, but pasting the run’s output is a chore, and the chore forces the re-check. For type 2’s “faking the measurement,” make “what did you measure with” part of the claim. Was it “summed the second labels” or “counted the real characters”? The moment the ruler’s name appears, an expired ruler is exposed. And type 3’s “faking the deed”? For “already done,” always make it write a file:line or a commit right beside it. If it can’t write one, don’t let it call it “done.” Make it write honestly: “this isn’t done yet, it’s a prescription.” A “done” you can’t open by its pointer gets failed at review without even reading the content. What’s common to all of them is that they convert “words” into “things you can open.” An openable command, an openable ruler name, an openable file line. A claim you can’t open, you don’t accept as a claim. This works even when it’s not an AI. On people’s reviews too. It works. See a pull request that says “already handled,” and you just ask, which line of the diff. Same conversion. I’d love to say it all wraps up neatly — but I’ll end by showing one honest wound. These three re-check patterns, I did write them down as rules. Type 1’s “no bare [x]” I wrote just recently. Fresh off the press, so it hasn’t rotted yet. The problem is what happens to a written rule once it rots. And type 2 already showed you that. The 9.7-minute one, earlier. Right. “Measure by real characters, not second labels” had failed once long before and been written down as a prose rule. It was written down, and still, on another occasion, it went back to the same ruler and faked. Same mistake, second time. You write the rule, and by the next time your hands move, you’ve forgotten it. That’s how prose rots. So what was different? On this production, that one runtime mistake can no longer happen — as a mechanism. Not because I wrote the rule in prose. Because I buried the runtime check in code inside the build, and if it falls short of eight minutes it halts the whole process with a SystemExit. Under eight minutes, the video simply doesn’t come out. There’s nothing to forget. The written prose rots, and only the gate that stops the line remains. That’s what this channel keeps saying. Prose rules rot. Only the fail-closed gate holds. I handed you three re-check patterns, but if you want them to actually bite, you have to drop them from “prose” into “an unbreakable gate in the build.” Next time I do exactly that. A turn where today’s three re-checks get wired — not into memory or willpower, but into a gate that stops the build red unless you satisfy it. The pattern that turns a claim into a re-check, made into a form a machine enforces even when the human doesn’t remember. One last thing to take home today. Take one “it’s done,” “I checked it” around you today, and ask it back like this: “Then — that done, which command, which line, can you show it to me?” If it opens, it’s real. If it jams, it’s type 3. This channel dissects our own production’s failures like this, every time. If you want to see the next one — “dropping the re-check into a gate” — subscribe and wait for it.