Your probe is measuring something else

Seven times a number lied to me while building a racing game, and what each one cost.

August 2026

The expensive mistakes on this project have almost never been wrong fixes. A wrong fix announces itself: you ship it, the thing is still broken, you back it out, and you've lost an afternoon.

The expensive ones are right numbers about the wrong quantity. Those don't announce anything. The measurement is easy to build, it produces a plausible value, it moves in the direction your hypothesis predicts, and nothing looks wrong until a second instrument disagrees with the first, usually days later, after you've built three things on top of it.

Here are seven of them, with the actual numbers. If you take one thing from this: before you believe a measurement, ask what it would read if the effect you're looking for didn't exist at all. Then go build that column and print it.

01Measure what the player said, not the thing you plan to fix

The report was: "big correction at the start of every multiplayer race, and even at 20ms ping it feels bad."

The obvious instrument is the reconciliation residual, the distance the car gets displaced when the client adopts authoritative server state. I built it. It read about 0.5 m before the fix and about 0.5 m after, at every ping tier. On that number, the fix did nothing and the hypothesis was dead.

The metric that actually matched the complaint was milliseconds after the green light spent below half launch speed: 14 / 49 / 112 / 252 ms at 20 / 60 / 130 / 250 ms RTT. All of them went to zero.

The player was never seeing a teleport. The perfect-start boost was being granted, reconciled away, and granted again, so the car simply did not go when the key was pressed. Half a metre of position error is invisible. Half a metre spent standing still at the start line is the entire complaint.

The internal term you intend to change is the easy thing to instrument, which is exactly why it's the default and exactly why it's wrong. Write down the sentence the player said, and pick the quantity that sentence is about: time stalled, speed lost, input ignored. Keep the internal term as a diagnostic column, never as the headline.

02If two independent fixes both move it by zero, the instrument is broken

I had a finding: the road surface was "33× rougher off the centreline than on it." It scaled beautifully: cleanly, monotonically, with offset. It survived a change to the curve continuity (0% effect) and a true arc-length remap (2% effect) before anyone thought to check the probe rather than propose a third mechanism.

The probe was measuring its own sampling path. It walked a path offset from the road's node polygon, which kinks at every node, so it was reading the kinks in its own ruler. Walking the refined curve instead gave a flat 0.00131 at every offset. There was no roughness. There had never been any roughness.

That cost a day, and the exact trap was already written in a header comment in a sibling probe file.

The thing worth internalising: a cleanly scaling artifact is more convincing than a real effect, because nothing physical is fighting it. Real signals are noisy and have exceptions. My fake one was perfect, and the predicted magnitude matched to within 10%, which felt like confirmation and was coincidence.

Two independent fixes moving a signal by ~0% is not a mystery to solve with a third hypothesis. It's an instrument to audit.

03A control has to remove the effect and nothing else

Building a control column isn't enough, because most controls remove more than you think.

The question was whether a wet road visibly reflected the neon rails above it. Four metrics, each principled, each wrong:

That last one is the interesting failure. Turning the rails off to get a baseline also removes the rails' own pixels from the crop, so the "reflection" I was measuring was partly just the rails themselves.

What worked was a double difference: rails-on minus rails-off, at the test wetness minus at zero wetness. Same scene, same rails, same crop, only the specular term removed.

Ask what your control frame changes besides the effect. If the answer includes "also the source", you need a second baseline that keeps the source and kills only the mechanism.

04A saturated signal is invariant to its own gain

This one is nasty because the experiment designed to convict a term is exactly the experiment that exonerates it.

Chasing a suspension artifact, I turned the suspension load term's gain down by 50×, suspLoad=0.02 versus the default. The render looked identical. That was recorded as a decisive negative: not this term.

The term was running 10–14× past its tanh travel limit. It was pinned to the stop at 1× and still pinned at 0.02×. Of course they looked the same. The clipping was the bug, and calling it innocent cost four more wrong hypotheses and a full day of someone else's driving time.

If a value feeds a tanh, a clamp, or a min/max, instrument it before the clamp, and report what fraction of frames are sitting on the stop. Everything downstream has already been flattened; no amount of screenshotting recovers it.

05Ask what a perfect system would score

A netcode probe reported p95 2.5 m of prediction error on opponent cars. Alarming number. It is also exactly 25 m/s × 100 ms, the extrapolation lead the design intends. A perfectly working system scores 2.5 m on that metric.

Before believing any measurement, ask what the value would be if the system were working flawlessly. If that number isn't zero, your metric is measuring design intent, not defect. (The fix was to compare the predicted state against the authoritative one at the same tick. The code already did this correctly for the local car; the opponent column had to grow its own.)

06Rejecting a contaminated quantity doesn't remove it from your denominator

A regression gate asserted that a particular driving technique raised boost uptime across the whole track catalogue. It flipped sign one day, with nothing wrong with the technique.

The gate had deliberately chosen boost uptime over lap time, on the correct reasoning that a wall-grazing lap diverges macroscopically between platforms and the tracks that stress this technique hardest are the wall-grazing ones. Right about lap time. But uptime is boost-seconds over lap-seconds, so the rejected contamination walked straight back in through the denominator.

Across all 17 routes, the per-route uptime delta correlated r = −0.83 with how much the forced technique inflated the lap clock, and only r = +0.26 with how much boost charge it actually earned. The lap clock explained 69% of the gate's variance; the technique explained 7%. One route carried the whole thing: forcing the technique took it from 112 wall contacts to 447, and from 80.0 s to 172.5 s, so its charge rose 19% while its uptime fell 24 points.

Two lessons that generalise:

• Prefer a per-distance statistic to a per-second one whenever the treatment can change pace. Charge earned per lap has no clock in it, since one lap is one fixed distance, and on that metric the technique was positive in every single counterfactual.
• Quote your estimator's standard error before asserting a sign. The mean of 17 per-route deltas had sd 10.5 pp and sem 2.5 pp around an effect near +1 pp. The gate was a coin flip with a bias from the day it was written. It didn't break; it just landed on the other side.

07Quote a distribution, and say which number the eye reads

A leftover vertical body movement was summarised as "the residual ~1 cm bob" and filed as a leave-it. That 1 cm was the rms. The peaks were 9.7–13.1 cm, and they were already recorded on the same ticket.

The response was a screen recording of very obvious body movement and the words "there's more than 1cm travel." An rms understates precisely what a viewer notices, so quoting it alone doesn't read as a partial claim. It reads as a false one.

Same failure mode in the physics: a single collision depenetration shove measured 1.26 m, which looked survivable. Seven of them in one tick stacked to 8.82 m, because each resolved against no shared budget. The single-instance measurement understated the real defect by 7×.

Whenever you're describing motion a human will judge by eye, quote peak or p95 alongside rms, and say which one the eye is reading. And measure the compounded case, not one instance of it.

+Bonus: removing the named cause doesn't remove the mechanism

The last one, because it's the subtlest.

Our older test suite documented, in bold, that one layer of its golden-replay checks was not portable across platforms: it ran a physics library compiled to WASM, and the same commit ended a run a hundred metres apart on CI versus a dev machine. Stated cause: the physics library.

The rewritten engine has no such library. Its movement code is plain double arithmetic. So the new suite shipped asserting bit-exact digests on every platform, with a header comment carefully explaining why the old limitation didn't apply. A paragraph of confident reasoning from a true premise.

CI refuted it on the first run. Recorded on Windows, replayed on Linux: four of six routes reproduced exactly, two did not. When the test widened to the full catalogue it was nine of seventeen.

The library was never the mechanism. Contact is a discontinuity no matter who implements it. A wall graze either happens or it doesn't, a last-ulp difference in the approach decides which, and from there the two runs separate macroscopically. Removing the library removed the name in the old explanation and left the physics of the thing completely untouched.

This recurs for a structural reason: an inherited explanation names a component, because a component is what the person debugging it happened to be holding at the time. Before concluding that someone else's documented limitation doesn't apply to you, restate their cause as a mechanism and ask whether the mechanism is still present.

The checklist

Everything above collapses into questions worth asking before you believe a number:

1. Does this metric measure the sentence the player actually said?
2. What would this probe read if the effect didn't exist? (Build that column. Print it.)
3. Does my control remove the effect and nothing else?
4. Is this value clamped or saturated downstream of where I'm reading it?
5. What would a perfect system score here?
6. Is the thing I rejected as contaminated still in my denominator?
7. Am I quoting rms for something the eye judges by its peaks?
8. If I'm citing someone's known limitation as not applying, is their mechanism really absent, or just their component?

And one meta-rule that has saved more time than any of them: when your probe and a human's report disagree by orders of magnitude, the probe is in the wrong domain. A headless test harness is deterministic by design, which is exactly what erases frame-rate, timing-jitter and interpolation bugs from it. Reasoning harder inside a world that provably cannot contain the bug is free to do and never works. Build the capture in the real environment before the second hypothesis, not the sixth.

Dead Air Works is an independent studio building SLIPFAULT, an arcade racing game for PC. More posts on the engineering as it ships.