Backup Testing: A Backup You Have Never Restored Is Not a Backup

Backup testing means restoring an archive to a clean place and checking what comes out. It does not mean confirming the backup job exited 0, or that a file landed in cloud storage. We learned the difference twice in ten days while building the Neogen AI OS, our own multi-agent operating system, and both times the only thing that told the truth was a restore.

The short version: on 31 July 2026 our nightly off-site backup had been encrypting and uploading a frozen copy of a directory nothing wrote to anymore. On 9 August a restore drill found an archive that passed its own manifest check and could not be extracted. Neither fault produced an error. Both are now caught before upload.

What is backup testing, and why is a successful backup job not proof?

A successful backup job proves the archiver ran. It says nothing about whether the archive holds the right files, whether it can be unpacked, or whether it can be decrypted without the machine it came from. Backup testing answers those questions, and the only complete version of it is an actual restore.

NIST's contingency planning guide, SP 800-34 Rev. 1, sets the bar in one sentence: “Backup tapes should be tested regularly to ensure that data are being stored correctly and that the files may be retrieved without errors or lost data.” The word doing the work is retrieved. Both of our failures stored data without complaint. One stored the wrong data. The other stored data that could not be retrieved.

How did our nightly backup archive a dead directory for days?

It kept backing up the path it was written for after that path stopped being where the data lived. We had split the system from one shared container into one container per agent, and each agent's data moved to its own directory. Nobody told the backup script, so it went on reading the old shared profiles folder.

We found it on 31 July during a routine disk and cache audit, not because anything alerted. The old folder had last changed at 19:18 on 29 July. The live founder agent's directory had 63 files modified in the previous two hours. Every night at 03:00 the job had encrypted and uploaded a snapshot of a system that no longer ran.

What that meant in practice:

• The four live agents' config, auth, memory, sessions, skills and scheduled jobs were not in any backup.
• Neo, one of the four, had never been backed up at all. He was built after the snapshot froze.
• The shared data store was still covered, only because its path had not moved in the split.
• Fourteen retained remote copies, about 1.6GB of Google Drive, were holding the corpse.

There was a signal, and nobody was reading it. The archive jumped from 10MB to 116MB on 30 July, because the cache exclusions were also written for the old layout and had stopped matching. A tenfold size change is not a quiet event. But the only health check was whether tar succeeded, and tar succeeded every night.

What should a backup preflight check before it uploads?

A preflight asserts that everything a restore would need is actually in the archive, and refuses to upload if anything is missing. Pointing the script at the new paths fixed that day's layout. The preflight is what stops the next migration repeating it, because paths rot and a check that refuses to upload does not.

Ours, rewritten the same day, runs before the archive is allowed to spend a retention slot. It asserts:

• a config file and an auth file for every agent in a single AGENTS list at the top of the script
• the shared data store
• the approval gate's audit trail
• the agent message bus
• a 20MB size floor, which the frozen 10MB archive would have failed on night one

If any assertion fails, the script deletes the archive and exits non-zero. Adding an agent is a one-line change to the list, and forgetting to make it now breaks the job loudly instead of quietly. Retention changed too. Pruning used to delete everything in the Drive folder past fourteen files, which would have rotated the only pre-split copies away within a fortnight. It now prunes by filename prefix, so the six historical archives are kept for good.

The rewrite also turned up a glob bug worth writing down: an exclude pattern of */cache does not match .cache. A single unexcluded speech-to-text model blob was 145MB of the archive, and 271MB of a 317MB candidate was cache. The final archive came to 122MB and 4,902 files.

Then we restored it. The drill decrypted and extracted all 4,902 files and checked the newest file for each agent. All four were timestamped that morning. That is the corpse test, and the old job would have failed it on its first night.

If you run agents or automations that have never been rebuilt from a backup, that gap is part of what an AI operating system build with us covers: the backup, its preflight and its restore drill ship as part of the system, not as a fix after the first scare.

What did the restore drill find that every other check passed?

An archive that tar would write but refused to unpack. On 9 August an external audit scored the OS 5 out of 10 and flagged two backup gaps: the host control plane was missing, and nothing tested restores on a schedule. The 31 July drill had been a one-off. Fixing the first gap was routine. The second paid for itself on its first run.

The control-plane gap was real. The job already carried the agents, the data store, the audit trail, the mesh config and a proper database dump of the control plane. It did not carry what makes those files run again: the dispatcher and watchdog, all seven systemd units, the directory holding every write credential in the system, the console source, and the whole agent-bus message history, which lived in a Docker volume rather than an ordinary path. Agent state on a bare server is not a recovered system.

The expanded archive was built, verified against its own manifest, encrypted and uploaded. Then the drill decrypted it and could not extract it.

Two paths shared a final name: the watchdog's state directory and the watchdog script, sitting in two different system folders. Passed to tar as separate change-directory-and-name pairs, both landed at the archive root under the same name, and the dispatcher's two paths collided the same way. tar wrote both without complaint. tar refused to unpack them.

Every required path was in the manifest, and every one genuinely existed. The archive still could not be restored. Without the drill, we would have discovered that during the emergency the backup exists for.

The host control plane is now staged into one directory whose paths spell out where each file goes back to, with a RESTORE.md beside them, and the preflight now lists the archive with tar tzf before upload. After the fix the drill ran 40 checks with 0 failures.

Where should the backup encryption key live?

Anywhere except the machine being backed up. Our archives on Google Drive are AES256-encrypted, and the only copy of the passphrase was a file on the box being backed up. Lose that host, which is the one scenario an off-site backup exists for, and all fourteen remote archives become unreadable.

NIST's guidance on encrypted backup media is blunt: keys “should be stored separate from, but accessible to, the primary encrypted backup data.” Ours was neither separate nor, in the disaster case, accessible.

This finding outranks everything else in this post, and the passing drill hid it. The drill proved the box could restore itself using a file that dies with the box. Escrowing the key off-box is a human action, and our build log for that day closes with it still open, along with the drill it unlocks: pulling an archive back from Drive and restoring it on a different machine.

How often should a restore drill run?

Often enough that a broken backup cannot survive long unnoticed. Ours runs on a weekly timer, Sundays at 04:40, and the watchdog alerts on two conditions: a failed verdict, or a drill that has not run in nine days. The second matters as much as the first, because a drill that silently stops running looks exactly like one that keeps passing.

That absence alert is the same pattern that caught a six-day silent outage in our agent mesh, written up in our post on AI agent observability. A check that only fires on errors cannot see a job that has stopped producing anything.

The primitive: restore is the only test of a backup

Every check we had before 9 August measured the act of backing up. The job exited 0, the file uploaded, the manifest matched. None of them measured what a backup is for. The rule we run on now is easy to state and tedious to build: a backup is unproven until it has been restored, somewhere other than its source, recently, on a schedule that alerts when it stops. Anything short of that is a record of an upload.

Frequently asked questions

Is a checksum enough to trust a backup?

No. A checksum proves the archive you downloaded is the archive you uploaded. It cannot tell you the archive held the right directory, and it cannot tell you tar will unpack it. Our 9 August archive would have passed any checksum, because it was byte for byte what we wrote. What we wrote was the problem.

How do you back up a self-hosted n8n instance?

Back up the database and the encryption key together, and test them as a pair. n8n encrypts saved credentials with an instance key, so a restored database without that key brings back workflows but not working credentials. Restore both to a fresh instance and run one workflow that uses a credential. More on running it yourself in our guide to self-hosted n8n automation.

Can a restore drill run on the same machine it backs up?

It can prove the archive is complete and extractable, which is how ours found the name collision. It cannot prove disaster recovery, because a same-box drill can quietly depend on things that only exist on that box, like our passphrase file. The full test pulls an archive from off-site storage onto a different machine.

What should a restore drill check besides “it extracted”?

Check freshness and substance. We look at the newest modification time for each agent, which should be hours old rather than the date of some past migration, and spot-check that critical files exist and are not empty: each agent's config and auth file, the access-control file and the approval audit log. Extraction alone would have passed our frozen archive.

If you want the backups under your automations restored by people who broke their own first, talk to us about your setup.