---
title: "The test was green, the effect was absent"
description: "We switched an old automated system off after confirming the new one produced shipping labels correctly. One module was still in shadow mode, and 114 orders over three days shipped without a gift the customer had already paid for."
date: 2026-08-20
lang: en
canonical: https://goingstellar.ro/research/test-was-green-effect-was-absent/
author: Going Stellar
publisher: GOING STELLAR S.R.L.
topic: Migration post-mortem
license: CC BY 4.0
alternate_ro: https://goingstellar.ro/ro/cercetare/testul-era-verde-efectul-lipsea/
---

# The test was green, the effect was absent

*We switched an old automated system off after confirming the new one produced shipping labels correctly. One module was still in shadow mode, and 114 orders over three days shipped without a gift the customer had already paid for.*

> Migration post-mortem · 2026-08-20 · https://goingstellar.ro/research/test-was-green-effect-was-absent/

## Key points

- A unit test checks whether a function returns the right answer; it cannot see whether anything downstream applies that answer.
- A module left in shadow mode can be present, reached, correct and inert, and a full green test suite will not reveal it.
- Sampling the most visible effect of an old system does not verify a migration, because one green sample says nothing about the other effects.
- The unit of verification for a migration is the effect on an order, not the module and not the test.
- A monitor can measure the wrong thing as easily as the system it watches, so confirm it measures the real effect before trusting it.

## A green test and an absent effect

We switched an old automated system off and let a new one take over. Before the switch we checked the most visible thing the old system did: it produced shipping labels. The new system produced them correctly. We recorded the migration as verified.

One module in the new system had been left in **shadow mode**. It wrote a line recording what it *would have* added to the order, and did nothing else. That module was responsible for adding a small free gift the customer had already paid for as part of the offer.

**114 orders over three days shipped without it.** The function that decided what to add was correct and it was tested. What was missing was execution.

On the same day, the same shape turned up twice more: a routine that only suggested a correction instead of applying it, and a feature marked in a code comment as not yet carried over.

## Why testing the code could not catch it

A unit test asks whether a function returns the right answer. Shadow mode does not change the answer. It changes whether anything downstream happens with the answer.

From inside the test suite the two questions are indistinguishable. The assertion is on a return value, and the return value was right no matter what happened to it afterwards. Nothing in the suite asked whether a caller took that value and wrote it to an order.

Every green test we had was true. It was true about the calculation, not about the world. The gap between *a correct answer* and *an applied answer* is invisible to a test that only ever sees the answer.

## The assumption we gave up

We assumed a migration could be verified by sampling: pick the most visible effect, confirm it works, generalise. That was refuted the same day. The label path was live; the gift path was not. One green sample said nothing about the others.

A subtler assumption did more damage: that a module which exists in the new system is a module that runs. Presence in the repository, resolved imports, a call that is genuinely reached — none of these mean the effect landed. Shadow mode is precisely the state of a module that is present, reached, correct, and inert.

What we believe now is narrower. The unit of verification is not the module and not the test. It is the **effect on an order**.

## The rule we adopted: list the actions, not the files

Write down every effect the old system had on an order. Not modules, not files, not services. Effects — the things that would be different for a customer if they stopped happening.

For each one, name who performs it after the switch-off. If the answer is "nobody", the migration is not finished, whatever the tests say. This is the step that would have caught our failure, and it takes no tooling.

Search the new code for the vocabulary of deferred work: *shadow*, *would*, *log-only*, *dry run*, "not ported", "apply at cutover". Each hit is a debt that falls due on the day you switch the old system off, and the day of the switch is exactly when nobody is reading code comments.

Confirm with the people downstream — the warehouse, the support desk — not only with the database. They handle the physical result. The database answers the question you thought to ask it.

## The watchdog, and how it was wrong too

We now run a daily comparison of candidates against effects for every automation: how many orders qualified, and on how many did the effect actually land. It sends mail only when something is dead or has fallen below a threshold, so silence is the normal state and a message means something.

Its first version reported a working automation as dead. It was reading a history table that happened to be empty, while the real effect was being written to a different column. The automation was fine. The monitor was not.

A monitor can measure the wrong thing exactly as easily as the system it monitors, and it inherits none of the scrutiny the system gets. Confirm that you are measuring the effect before you trust the monitor. A red alert on a healthy path and a green one on a dead path are the same defect wearing different colours.

## Running the same check yourself

Take an order that went through the old system and write down what happened to it, end to end, in the language of outcomes rather than components. Then trace each of those outcomes through the new system on a real recent order, in the live store of record — not in a fixture, not in staging with seeded data.

Grep the new codebase for the deferred-work vocabulary above. Treat every hit as unpaid work until you can point at what pays it. Reading the comment is not enough; find the code that performs the effect.

Then ask the people downstream what they are seeing. Where the parcel and the database disagree, the parcel wins. Our own database was internally consistent throughout — it recorded, correctly, that nothing had been added.

## Limitations

This is one migration on one company's systems. It shows that a shadow-mode module can survive a cutover with a full green test suite; it does not establish how often that happens elsewhere, or in what kinds of stack.

The count of 114 is what we could attribute with confidence. We did not attempt to reconstruct anything beyond the three days, so the figure is a floor for the window we examined rather than a total.

The rule is a checklist, not a proof. Enumerating the effects of an old system is manual work, and it can miss an effect nobody remembers it had. The daily comparison narrows the exposure afterwards, but it does not solve the enumeration problem — and its first version demonstrated that it can be wrong about the very thing it exists to watch.

## Questions and answers

### Why can passing unit tests miss a broken migration?

Passing unit tests can miss a broken migration because a unit test asserts on a return value, not on what happens to that value afterwards. A module in shadow mode returns the correct answer but never writes it to the order. The tests stay true about the calculation while the effect on the customer is absent, so the gap between a correct answer and an applied answer stays invisible to the suite.

### What is shadow mode and why is it risky during a cutover?

Shadow mode is a state in which a module records what it would have done and does nothing else. Shadow mode is risky during a cutover because the module is present in the repository, its imports resolve and its call is reached, yet the effect never lands. When the old system is switched off, any work still in shadow mode silently stops happening for real orders.

### How should I verify that a system migration is complete?

Verify a system migration by listing every effect the old system had on an order, then naming who performs each effect after the switch-off. If the answer for any effect is nobody, the migration is not finished, whatever the tests say. Trace each effect on a real recent order in the live store of record, not in a fixture or a seeded staging environment.

### Which words in code signal unfinished migration work?

Words that signal unfinished migration work include shadow, would, log-only, dry run, "not ported" and "apply at cutover". Each match in the new codebase is a debt that falls due on the day the old system is switched off. Treat every hit as unpaid work until you can point at the code that actually performs the effect; reading the comment is not enough.

### Why should I ask warehouse and support staff when checking a migration?

Warehouse and support staff should be asked because they handle the physical result, while a database only answers the question you thought to ask it. A database can stay internally consistent while correctly recording that nothing was added to an order. Where the parcel and the database disagree, the parcel wins, so people downstream often see a missing effect first.

### Can a daily monitoring check be wrong about a healthy automation?

A daily monitoring check can be wrong about a healthy automation if it reads the wrong source. In one case a first version reported a working automation as dead because it read an empty history table while the real effect was written to a different column. A red alert on a healthy path and a green one on a dead path are the same defect, so confirm the monitor measures the effect.

## How to cite

**Cite this as:** Going Stellar (2026). *The test was green, the effect was absent*. GOING STELLAR S.R.L.. https://goingstellar.ro/research/test-was-green-effect-was-absent/

```bibtex
@misc{goingstellar2026test,
  title  = {The test was green, the effect was absent},
  author = {{Going Stellar}},
  year   = {2026},
  url    = {https://goingstellar.ro/research/test-was-green-effect-was-absent/},
  note   = {2026-08-20}
}
```

Licence. Free to quote with attribution.

## Also available

- This page as HTML: https://goingstellar.ro/research/test-was-green-effect-was-absent/
- Same article in Romanian: https://goingstellar.ro/ro/cercetare/testul-era-verde-efectul-lipsea/