---
title: "How to catch silent data failures: freshness watchdogs, heartbeats and reconciliation"
description: "A silent data failure is a pipeline that reports success while its data is stale or wrong. We catch them with three scheduled checks: a watchdog that reads the newest row instead of the job status, heartbeats that fire only on success, and a daily reconciliation of the same number from two independent sources. The checks themselves were wrong several times, and those mistakes are part of the method."
date: 2026-10-11
lang: en
canonical: https://goingstellar.ro/methods/catch-silent-data-failures/
author: Going Stellar
publisher: GOING STELLAR S.R.L.
topic: Engineering / Monitoring
license: CC BY 4.0
alternate_ro: https://goingstellar.ro/ro/metode/cum-prinzi-erorile-tacute-de-date/
---

# How to catch silent data failures: freshness watchdogs, heartbeats and reconciliation

*A silent data failure is a pipeline that reports success while its data is stale or wrong. We catch them with three scheduled checks: a watchdog that reads the newest row instead of the job status, heartbeats that fire only on success, and a daily reconciliation of the same number from two independent sources. The checks themselves were wrong several times, and those mistakes are part of the method.*

> Engineering · Monitoring · 2026-10-11 · https://goingstellar.ro/methods/catch-silent-data-failures/

## Key points

- A job that exits successfully does not prove that the data is fresh; check MAX(date) in the destination table against a deadline.
- A freshness check must also detect absence: declare every expected source, because a source that stops completely produces no rows to report on.
- Heartbeats should fire only on a successful run, and you must know which exit codes a script returns when it succeeds before chaining a heartbeat with &&.
- Reconcile the same quantity from two independent sources and alert only when a key moves from OK to drift, otherwise permanent historical drift becomes daily spam.
- An empty result may mean "does not exist" only if every source that could know was asked; otherwise the honest answer is "unknown".

## What a silent data failure is

A silent data failure is a pipeline that keeps reporting success while the data it produces is stale, partial or wrong. You catch silent data failures by checking the **output** of each pipeline (the newest row in the destination table, the count of records produced today) against a deadline, by making every scheduled job send a heartbeat only when it succeeds, and by reconciling the same figure from two independent sources every day.

Our costliest data failures were not crashes. A profitability table went unsynchronised for **23** days and the monthly figures built on it were wrong for the whole period, while every job around it exited cleanly. A crash gets fixed the same hour because someone sees it. A false success gets read, believed and acted on.

## Step 1: a freshness watchdog that reads the data, not the job status

A freshness watchdog is a scheduled script that asks, for every important table, when the data last moved, and compares the answer with a service-level deadline. The question is `MAX(date)` or `MAX(created_at)` in the destination table, never "did the job run". A job log is written when the sync fails too, so the age of the log proves nothing about the age of the data.

**A plain freshness check is not enough.** In the 23-day incident the month *was* present in the table, because marketing cost came from a different source and kept the month alive. Only revenue and deliveries were zero. The check that catches it is a consistency rule: a current month with marketing cost above zero but revenue or deliveries at zero is a false profit-and-loss statement, not a quiet month.

**Watch for absence, not only presence.** Our first version grouped the sync log by source, so a source that stopped completely simply vanished from the report. Declare the expected sources explicitly, give each a deadline (ours is **120** minutes for scheduled syncs) and treat total silence as critical. We proved the fix with a phantom source, which turned red.

**Compare like with like on time.** A job once produced **414** shipping labels one day and **0** the next while the report stayed green; a whole-day threshold stays silent early in the morning, when the watchdog runs. The check now compares today up to the current hour with the average of the last **7** days up to the same hour.

## Step 2: heartbeats, the dead-man switch for jobs that never start

A heartbeat is a ping that a scheduled job sends at the end of a *successful* run; a separate check flags any job whose last ping is overdue. Heartbeats catch the failure a freshness check cannot see: a job that never starts writes nothing at all, not even a failure line.

We wired heartbeats onto **9** pipeline jobs and flag a job as overdue after twice its interval plus **30** minutes. The job that created every shipping label had no heartbeat, although **20** smaller jobs did. Check that the most important job is on the list before trusting the list.

**Know your exit codes before you write `&& heartbeat`.** An audit script of ours exited with code 1 when it found deviations, its normal outcome, so a heartbeat chained with `&&` almost never fired. The watchdog called a perfectly healthy job critical for **8** days. We now accept codes 0 and 1 for that job and reserve a distinct code for "did not run, lock held".

## Step 3: reconcile the same number from two independent sources

Reconciliation means computing the same quantity from two systems that do not share an input, and alerting when they drift apart. We reconcile delivered orders as counted by the profitability engine against the shipping records, and marketing cost as recorded in a spreadsheet against the data warehouse. Delivered orders were the metric that had been undercounted **1.6**× in an earlier month, and a count of deliveries is independent of currency and tax rules, so drift in it is easy to read.

On the first run, delivered orders matched within **2**% for every business line.

**Alert on the transition, not the state.** Closed months that drifted in the past stay above the threshold forever. Re-sending them daily is spam, and spam teaches people to ignore the channel. The reconciliation stores every snapshot and emails only when a key moves from OK to drift; the first observation of a new key is a silent baseline.

## The traps that look like success

**An HTTP 200 on acceptance is not delivery.** A social posting job logged HTTP 200 on every run while **17** of **17** posts failed on one network. The 200 meant "accepted"; publishing happened later and failed per destination, because a cache of connected accounts had no expiry. Read the final status per destination.

**A 200 from your own webhook proves only that the handler did not throw.** We declared a webhook-to-queue chain verified using a row we had inserted by hand. The new handler never committed, so real webhooks got 200 and the queue stayed empty. Send real traffic in at the upstream end and count rows at the downstream end.

**An empty result has to say whether it searched.** A lookup queried a second source only with the keys returned by the first, which covered about 90% of the volume. On **106,049** orders the first source did not know, the function answered "searched, nothing found" with a success flag. Empty may mean "does not exist" only if every source that could know was asked; otherwise the answer is "unknown".

**Every failure word can contain a success word.** A status matcher checked substrings in order, success first. In Romanian the word for "not delivered" contains the word for "delivered", and in English "undelivered" contains "delivered", so failed parcels came out as delivered and final. Check negations first, and gate the matcher with a test file; ours has **21** cases and runs on every deploy.

**A table that exists is not a feature that works.** A screen, its table and a CHECK constraint were all in place, yet the table had **0** rows, a key column was empty on all **194** records and cost was empty on all **3,920** products. Before fixing a screen, ask who *writes* to it: `COUNT(*)` on the table, `COUNT(column)` on each column.

**One expiring credential can stop a whole platform.** Every advertising account on one platform synced through a single OAuth token lasting about **60** days. When it expired, all accounts froze on the same day while the error log named one account at random, and the token stayed dead for **11** days. Alert on expiry in advance and prefer credentials that do not expire.

## Where we were wrong

**Our fix made a loop invisible.** A sync loop rewrote about **41,000** unchanged rows every cycle and logged its summary only when something was written. We stopped the waste, a healthy cycle then wrote zero rows, and the loop vanished from the log. A review caught it, not us. The summary also could not tell "nothing new" from "could not read": on one day **18** of **23** connections had expired keys and returned zero orders without an exception. The heartbeat line is now always written, with a separate counter for empty sources.

**We measured the length of an error.** To avoid printing a secret, we masked the output of a secret lookup and counted characters. There were **63**, a plausible length for a credential. They were the error message saying the knowledge base was unreachable. Check the exit code and stderr; never measure a property that is equally compatible with success and failure.

**Our watchdog was wrong more often than the automations.** In one day, three red checks were all false. One filtered on the date a backfill wrote the rows instead of the event date and reported **2**% coverage where the truth was **99**%; another demanded an effect before an automation's deliberate **120**-minute grace period had passed. We split the checks rather than loosen them: a checker that cannot fail is not a checker.

**A backfill would have overwritten human decisions.** It selected rows whose value differed from the target instead of rows with no value, so every run after the first would silently convert choices a person had made. Backfills on a column the interface also writes must filter on `IS NULL`.

## Limits of this method

Freshness, heartbeats and reconciliation catch data that stopped moving or drifted apart. They do not catch a figure that is fresh, consistent across sources, and wrong in both because both share a definition error. Reconciliation is only as strong as the independence of the two sources; if one source filters what the other is asked, the second inherits the first's gaps.

Every threshold here was calibrated on our own history: the label check was backtested on **9** real days, and its first variant gave **3** false positives on a weekend. A quiet report says the checks are quiet, not that nothing can break.

## Checklist

**1. Output.** For every table that feeds a decision, check `MAX(date)` against a deadline, and add a consistency rule for months that exist but are hollow.

**2. Absence.** Declare the expected sources; total silence is critical.

**3. Heartbeats.** Ping on success only, know each script's success exit codes, and make sure the most important job is wired.

**4. Reconciliation.** Compare one quantity from two independent sources daily; alert on OK-to-drift transitions only.

**5. Proof.** Verify chains end to end with real traffic, read final status per destination, and make empty results say whether they searched.

**6. The checker.** Before declaring an automation dead, check the denominator, the event date and the grace period.

## Questions and answers

### What is a silent data failure?

A silent data failure is a data pipeline that keeps reporting success while the data it produces is stale, incomplete or wrong. Nothing crashes and no alert fires, so people keep reading the numbers and acting on them. In our case a profitability table stayed unsynchronised for 23 days while every surrounding job exited cleanly, and the monthly figures built on it were wrong for the whole period.

### How do I check whether a data pipeline is actually working?

Check the output, not the job status. Query the newest row in the destination table, for example MAX(date) or MAX(created_at), and compare it with a deadline you have agreed for that table. A job log is written even when the sync fails, so the age of the log says nothing about the age of the data. Also count the records produced today against the same hour on previous days.

### What is a heartbeat or dead-man switch for cron jobs?

A heartbeat is a ping that a scheduled job sends at the end of a successful run. A separate check flags any job whose last ping is overdue, for example after twice its interval plus 30 minutes. Heartbeats catch jobs that never start at all, which write nothing and therefore never appear in a freshness check or an error log.

### Why does my monitoring keep raising false alarms?

Usually the checker measures something the automation cannot produce. Common causes are filtering on the date a row was written instead of the event date, counting items that can no longer be acted on, ignoring a deliberate grace period, or chaining a heartbeat to an exit code that is non-zero on success. Fix the measurement rather than loosening the threshold, because a checker that cannot fail is useless.

### Is an HTTP 200 response proof that data was written or published?

No. A 200 from a scheduling API usually means the request was accepted, and publishing can still fail later for each destination. A 200 from your own webhook means only that the handler did not throw; it may not have committed anything. Verify by reading the final status per destination, or by counting rows at the downstream end after sending real traffic in at the upstream end.

### How do I reconcile data between two systems?

Compute the same quantity from two systems that do not share an input, for example delivered orders from the shipping records and from the profit engine, and compare them daily. Store each snapshot so you can see drift over time, treat the first observation of a key as a silent baseline, and alert only when a key moves from OK to drift.

## How to cite

**Cite this as:** Going Stellar (2026). *How to catch silent data failures: freshness watchdogs, heartbeats and reconciliation*. GOING STELLAR S.R.L.. https://goingstellar.ro/methods/catch-silent-data-failures/

```bibtex
@misc{goingstellar2026catch,
  title  = {How to catch silent data failures: freshness watchdogs, heartbeats and reconciliation},
  author = {{Going Stellar}},
  year   = {2026},
  url    = {https://goingstellar.ro/methods/catch-silent-data-failures/},
  note   = {2026-10-11}
}
```

Licence. Free to quote with attribution.

## Also available

- This page as HTML: https://goingstellar.ro/methods/catch-silent-data-failures/
- Same article in Romanian: https://goingstellar.ro/ro/metode/cum-prinzi-erorile-tacute-de-date/