---
title: "The data failures that cost most never raised an alarm"
description: "Our costliest data failures ran for days without an error, an alert, or a broken page. Each kept producing numbers on schedule; only the freshness of the data was wrong. The mechanisms we used to catch them are general; the intervals inside them are ours."
date: 2026-08-28
lang: en
canonical: https://goingstellar.ro/research/data-failures-that-cost-most-never-raised-an-alarm/
author: Going Stellar
publisher: GOING STELLAR S.R.L.
topic: Data quality / monitoring
license: CC BY 4.0
alternate_ro: https://goingstellar.ro/ro/cercetare/cele-mai-scumpe-defectiuni-de-date-n-au-declansat-nicio-alarma/
---

# The data failures that cost most never raised an alarm

*Our costliest data failures ran for days without an error, an alert, or a broken page. Each kept producing numbers on schedule; only the freshness of the data was wrong. The mechanisms we used to catch them are general; the intervals inside them are ours.*

> Data quality · monitoring · 2026-08-28 · https://goingstellar.ro/research/data-failures-that-cost-most-never-raised-an-alarm/

## Key points

- The costliest data failures keep producing output on schedule; only the freshness of the data is wrong, and often nothing checks freshness.
- A job that exits zero proves that a process reached its end, not that the data changed.
- A job that never starts cannot report its own failure, so the alarm has to live outside the job: make each job ping on success and treat a missing ping as the alarm.
- When one credential covers every account on a platform, its expiry shows up as every account frozen on the same day, while the log names a single account at random.
- A monitor can measure the wrong thing as easily as the system it watches, so confirm that it measures the effect before trusting it.

## Nothing crashed

We went looking for the failures that had cost us the most, expecting to find outages. Instead we found incidents in our own data plumbing that never threw an error, never paged anyone, and never stopped producing numbers on schedule. The numbers were simply wrong.

The failure mode we had built defences for was the loud one. A job exits non-zero. A connection is refused. A page returns a stack trace. Those get fixed quickly because somebody notices immediately. The failures described here kept running because every surface anyone looked at was green.

The shape they share is worth stating before the detail. Every one of them *produced output*. The pipeline ran. Only the freshness of the data was wrong, and nothing in the system was checking freshness.

## Different failures, one shape

Every advertising figure for one platform flowed through a single long-lived OAuth token, with a life of roughly **sixty** days, covering every advertising account we run. When it expired, everything stopped at once, and it stopped quietly. Reports still rendered. They just stopped moving forward.

The signature is worth memorising, because it misleads on first reading. Every account's last sync timestamp was frozen on the same day, and the sync log reported a failure naming **one** account at random while in fact all of them had failed. Read the log alone and you conclude you have a single broken account, not a dead credential underneath the whole platform.

Another failure was an absence. A scheduled job for a second platform stopped running and nobody knew, for a period we still cannot state. There is no record of when it stopped, because a job that never starts writes nothing at all, not even a failure line.

The expensive one was a profitability table that went unsynchronised for **23** days. The monthly figures it fed were badly wrong for the whole of that time, and no page said so. The dashboards rendered as usual, at the usual speed, in the usual layout, and people read decisions off them.

## Exit zero is not evidence

A job that exits zero tells you a process reached its end. It does not tell you that the data changed. Those are different claims, and almost all of our monitoring was asserting the first while everyone downstream assumed the second.

The absence case is worse. A job that never starts cannot report its own failure, so no amount of log inspection will surface it. You can only detect it by noticing that something expected did not arrive, which means the alarm has to live outside the job.

So we moved the assertion from the job to the output. Is this table fresher than N hours? That question has an answer regardless of which component broke, regardless of how it broke, and regardless of whether any component ran at all.

## What we changed

Watchdogs now check output against a service level rather than checking that a job exited zero. For each table that feeds a decision we wrote down the maximum age at which it is still trustworthy, and the watchdog compares the newest row against that age. It is a claim about data, not about processes.

Each scheduled job pings on SUCCESS, so the absence of a ping is itself the alarm. This is the only way to cover the job that never starts. The system is no longer relying on a failing component to describe its own failure.

Independent sources of the same quantity are reconciled against each other, and the drift history is stored. Email goes out only on NEW drift. This part is about human attention rather than engineering: an alert that fires every day is an alert nobody reads, and a channel full of known noise hides the one line that matters.

Token expiry is now checked ahead of time rather than discovered by an outage. Where a single credential covers every account on a platform, its expiry date is a scheduled event we can see coming, not a symptom we diagnose afterwards.

## We were wrong about the monitor itself

The first version of one of these checkers reported a working automation as dead. It was reading a history table that happened to be empty, while the real effect was being written to a different column entirely. The automation was fine. The checker was not.

The assumption underneath, which we had not noticed making, was that the place recording an action is the same place that shows its effect. Here it was not. We came close to repairing something that already worked, on the authority of a monitor we had just written and had not tested.

A monitor can measure the wrong thing just as easily as the system it monitors, and it is more dangerous when it does. A monitor is the thing you consult precisely when you distrust everything else, so a wrong one has no second opinion above it. *Confirm first that you are measuring the effect, then trust the monitor.*

## Running the same check yourself

Start from effects, not from files or jobs. For each automation, write down what it changes about a real record. Then find the column or table where that change actually lands. That is the thing to measure, and it is often not the log, the run table or the history table you would reach for first.

For every table that feeds a decision, take the newest timestamp and compare it to now, against a freshness limit you have written down in advance. Then repeat the comparison per entity, not just overall. The credential failure above only reveals itself as a pattern if you group by account and see every account frozen on the same day.

Invert one alarm as an experiment. Take your most important scheduled job, make it ping on success, then stop it deliberately and confirm that you hear about it. If you do not, you have learned something more useful than a passing test.

Finally, pick a quantity you can obtain from independent sources, store the difference between them over time, and alert only when that difference changes.

## Limitations

This is one company's data plumbing. The mechanisms are general and can be copied directly. The specific intervals and thresholds are not: our freshness limits come from how often our own pipelines run and how fast our own decisions are made, and copying those numbers would tell you nothing about your systems.

Freshness monitoring catches staleness, not wrongness. Data can arrive on time, pass every watchdog, and still be misattributed, mismapped or double-counted. Nothing described here would detect that, and we do not claim these were the only silent failures we have, or that we now catch all of them.

We also give no figure for what any of this cost. For the job that stopped, we do not know the period it was down. For the rest, the honest accounting would require reconstructing decisions people made from wrong screens, and we cannot check that, so we are not publishing a number for it.

## Questions and answers

### How do you detect a data pipeline that fails silently?

Detecting a data pipeline that fails silently means checking output rather than job status. For each table that feeds a decision, write down the maximum age at which it is still trustworthy and compare the newest row against it. Make every scheduled job ping on success, so a missing ping raises the alarm. Reconcile independent sources of the same quantity and alert only on new drift.

### Why is an exit code of zero not enough to monitor a scheduled job?

An exit code of zero is not enough to monitor a scheduled job because it only shows that a process reached its end, not that the data changed. Most monitoring asserts the first claim while everyone downstream assumes the second. A job that never starts writes nothing at all, not even a failure line. The better question is whether a table is fresher than a set number of hours.

### What does an expired shared OAuth token look like in sync logs?

An expired shared OAuth token looks like a single broken account in sync logs, which is misleading. Every account's last sync timestamp freezes on the same day, while the sync log names one account at random as the failure. Reports keep rendering but stop moving forward. Grouping freshness by account reveals the pattern, and checking token expiry in advance turns it into an event you can see coming.

### Can a data monitor itself be wrong?

A data monitor can be wrong as easily as the system it watches, and a wrong monitor is more dangerous because nothing sits above it as a second opinion. One freshness checker reported a working automation as dead because it read an empty history table, while the real effect was written to a different column. The safe practice is to measure where an automation's change to a real record actually lands.

### Does freshness monitoring catch incorrect data?

Freshness monitoring catches stale data, not incorrect data. Data can arrive on time, pass every watchdog and still be misattributed, mismapped or double-counted. The mechanisms, namely freshness limits, success pings and reconciliation of independent sources, can be copied directly. The intervals and thresholds cannot, because they depend on how often a given organisation's pipelines run and how fast its decisions are made.

## How to cite

**Cite this as:** Going Stellar (2026). *The data failures that cost most never raised an alarm*. GOING STELLAR S.R.L.. https://goingstellar.ro/research/data-failures-that-cost-most-never-raised-an-alarm/

```bibtex
@misc{goingstellar2026data,
  title  = {The data failures that cost most never raised an alarm},
  author = {{Going Stellar}},
  year   = {2026},
  url    = {https://goingstellar.ro/research/data-failures-that-cost-most-never-raised-an-alarm/},
  note   = {2026-08-28}
}
```

Licence. Free to quote with attribution.

## Also available

- This page as HTML: https://goingstellar.ro/research/data-failures-that-cost-most-never-raised-an-alarm/
- Same article in Romanian: https://goingstellar.ro/ro/cercetare/cele-mai-scumpe-defectiuni-de-date-n-au-declansat-nicio-alarma/