Home/Research

Post-mortem

The ignore pattern that deleted a live module

A data directory was ignored without a leading slash, so the pattern also matched a source package deeper in the tree. The package was never committed, a mirror deploy removed it from the server, and the only surviving description of what it did was its test suite.

Published 2026-08-27

What broke

A service stopped starting and went into a restart loop. The immediate cause was ordinary: a live route imported a module, the module was not on disk, and the import failed with ModuleNotFoundError before the application could serve anything. Nobody had deleted the file on purpose.

The chain runs from an ignore pattern to a deploy script. The ignore file listed a directory of exported data files. The pattern was written without a leading slash. Somewhere deeper in the source tree there was a package with the same directory name, and the pattern matched that too, because the pattern matches a name, not a location.

So the package was never committed. It existed on the disk of the person who wrote it, and on the server, where an older archive-based deploy had once placed it. Both copies looked healthy for as long as nobody rebuilt the server from a clone.

A pattern without a leading slash matches at any depth

This is documented behaviour of the standard ignore-file syntax, not a bug and not an edge case. If a pattern contains a slash at its start or in the middle, it is anchored: it applies relative to the directory of the ignore file. If it contains no such slash, it may match at any level below that directory. A bare directory name is therefore a wildcard over the whole tree.

That is the right default for things like build output and editor droppings, which appear anywhere. It is the wrong default for one specific directory that holds exported data at the root of the repository, because the same word is a perfectly reasonable name for a package of code.

The failure mode is quiet by design. An ignored file is not reported as untracked, so the normal status output stayed clean for as long as the package was ignored. There was no warning to miss. The tooling did exactly what it was told.

A mirror deploy makes the deploying machine the source of truth

The deploy script removes the source directories on the server and unpacks an archive built from the deploying machine's disk. That is a mirror, not a merge. Anything present on the server and absent from the archive is gone after the deploy, whether it was stale rubbish or a module a live route depends on.

The deploying machine had cloned from the repository, so its disk held exactly the tracked files. The ignored package was not among them. The archive was therefore correct with respect to version control and wrong with respect to reality, and the deploy propagated that difference onto the server.

Stated generally: a deploy that mirrors a disk silently deletes any real file that is present on the server but absent from the deploying machine. The machine doing the deploying becomes the definition of the application. If that machine is missing something, the deploy is a deletion with extra steps.

We expected to undelete the file. We were wrong

The first plan was recovery from the server. This is the plan most people reach for, and it assumes the deleted data is still sitting in unreferenced blocks waiting to be reassembled. On modern filesystems that assumption no longer holds: the block map is zeroed on unlink, so the pointers from the inode to the data are destroyed before the data is. The usual forensic recovery tool returned no recoverable inodes at all — not a damaged partial reconstruction, nothing.

What saved the module was the tests. They were tracked, because no ignore pattern matched their path. There were 32 tests in one file and 6 in another, and together they described the exact behaviour required: inputs, outputs, error cases, the shape of every value the rest of the system expected back. The module was rebuilt from its test suite used as a specification.

The lesson we took from this is not about the ignore file. It is that a test suite is the only description of a module that survives the module. Documentation drifts, comments are lost with the file they annotate, and a recovery tool can return nothing. A test that runs is an executable statement of intent that lives in a different file. That is a stronger argument for writing tests than any coverage target we have ever been given.

Two preventions, both cheap

Anchor ignore patterns for data directories to the repository root with a leading slash. The pattern then applies to the one directory it was written for and cannot match a same-named directory inside the source tree. This removes the entire class of accident.

Before any deploy that mirrors a disk onto a server, import the application's entry module on the machine doing the deploying. An import walks the real dependency graph and touches every module a live route needs. If the import fails locally, it will fail on the server too — and you find out before the server does, on a machine where fixing it is a checkout rather than an incident.

Neither prevention needs a new tool, a policy document or anyone's agreement. The reason they were not there is the reason such precautions are usually missing: until the day the ignore pattern met the deploy script, each looked harmless on its own.

How to run the same check

List every file the repository is currently ignoring that is also present on the server, and ask of each one: is this data or is this code? Data is expected to be ignored and expected to differ between machines. Code that exists only on the server is a module you do not have, and the next mirror deploy will remove it. On the repository side, git ls-files --others --ignored --exclude-standard enumerates what is being ignored; compare that list against the deployed tree.

Then re-read every ignore pattern that lacks a leading slash and decide whether it was meant to match at any depth. Some are: caches and compiled artefacts belong everywhere. Named directories that sit at the root of the repository almost never are. For any file you are unsure about, git check-ignore -v names the exact pattern and line that excluded it, which turns the question from a guess into a lookup.

Finally, run the import check by hand before wiring it into the deploy, so you see what a clean pass looks like on a machine you trust. A pass proves the dependency graph is fully present on the disk about to become the source of truth.

Limitations

The recovery outcome is specific to the filesystem. On a filesystem that does not zero the block map on unlink, an undelete may well succeed, and the conclusion about tests would have arrived as a preference rather than as the only option left. Do not read "recovery is impossible" into this; read "do not plan on it".

The pattern-matching behaviour is not specific to us. It is the documented behaviour of the standard ignore-file syntax, and it generalises to any repository using it.

This is one incident in one codebase. The mechanism is general, the two preventions are cheap enough to apply without evidence, but the frequency is not something a single case can tell you.

Questions and answers

Why does a gitignore pattern match folders deeper in the tree?

A gitignore pattern matches folders deeper in the tree when it has no slash at its start or in the middle. Such a pattern may match at any level below the directory of the ignore file, so a bare directory name works as a wildcard over the whole tree. Anchoring the pattern with a leading slash limits it to the one directory at the repository root.

How can a deploy delete a file that nobody deleted?

A mirror deploy can delete a file that nobody deleted because it removes the source directories on the server and unpacks an archive built from the deploying machine's disk. Anything present on the server but absent from that archive is gone afterwards. If an ignored package was never committed, a machine that cloned the repository does not have it, and the deploy removes it.

Can I recover a deleted source file from a Linux server?

Recovering a deleted source file from a server should not be relied on. On modern filesystems the block map is zeroed on unlink, so the pointers from the inode to the data are destroyed. In the incident described, the usual forensic recovery tool returned no recoverable inodes at all. On a filesystem that does not zero the block map, an undelete may succeed, but do not plan on it.

How can tests help rebuild a lost module?

Tests can rebuild a lost module because a test suite describes inputs, outputs, error cases and the shape of every value the rest of the system expects. Tests live in a different file from the module, so they can survive when the module is lost. In the incident, the module was rebuilt by using its own tracked test suite as a specification.

How do I check which files git is ignoring before a deploy?

Checking which files git is ignoring starts with git ls-files --others --ignored --exclude-standard, which lists ignored files; compare that list against the deployed tree and ask whether each file is data or code. Code that exists only on the server will be removed by the next mirror deploy. For any uncertain file, git check-ignore -v names the exact pattern and line that excluded it.

All research