The Backup That Died Silently, and the Mirror That Worked Too Well

In the first post of this series, I wrote that my host workspace synced nightly to the Synology NAS, the NAS synced to cloud storage, and a 3-2-1 backup chain let me sleep easy. In the August update, I repeated that the 3-2-1 chain was still humming along nicely.

Three weeks later I actually checked it, and found two uncomfortable truths back to back:

1. Neither local backup job had been working when I wrote that August update.
2. Once I fixed the local backup job, the downstream cloud-sync leg dutifully uploaded plaintext .env files and unencrypted firewall config backups to cloud storage.

Run a home lab long enough and you learn that the only thing worse than a broken backup is a working backup pointed at the wrong folder.

Act I: The Two Silent Failures

The host backup setup on REX had two parts: a desktop sync client mirroring the F: workspace drive to the NAS in real time, and a nightly 1.45am scheduled task taking Hyper-V VM checkpoints ahead of a 2am copy.

Both had died quietly without raising a single alert.

The Crash-Looping Sync Client

On 23 August, two concurrent AI coding sessions accidentally clobbered a section of my networking runbook. No drama, I thought - I’ll just grab yesterday’s version from the Btrfs snapshot on the NAS.

The newest file on the NAS was eight days old.

Checking the Windows Application event log showed why: the vendor’s desktop sync client had been crashing on startup inside a shell extension DLL (0xc0000005 access violation) since 15 August. Every time it died, the tray icon vanished without a pop-up. Because nothing checked whether the target folder on the NAS was actually getting fresh writes, an eight-day backup outage looked completely normal from the outside.

The Four-Bug Checkpoint Script

That prompted an immediate check on the nightly Hyper-V VM backup job. According to Windows Task Scheduler, LastTaskResult was 0 (success). According to the backup share on the NAS, not a single checkpoint had landed since May.

Opening the script explained how a job can fail for three months while returning exit code zero:

1. Wrong VM names - The script still referenced an early draft of the VM names (Kali-Linux with a hyphen instead of KaliLinux), so Get-VM failed on three of the four VMs.
2. A UTF-8 BOM in a bash heredoc - The script piped a multi-line bash snippet over SSH to the NAS. PowerShell’s default encoding stuck a byte-order mark (\ufeff) on the first line, turning #!/bin/bash into a command-not-found error on the Linux side.
3. Unquoted paths with spaces on the remote cleanup command.
4. An interactive SSH password prompt hanging in a background scheduled task until it timed out, followed by a bare exit 0 at the end of the wrapper script.

Every error got swallowed, and Task Scheduler happily reported green.

Rebuilding the Local Backup Chain

I ripped out the desktop sync client and the checkpoint script and replaced them with two simple nightly jobs running as elevated scheduled tasks:

• 2am - Workspace Mirror - An incremental robocopy /MIR pass from the host workspace to a Btrfs-snapshotted share on the NAS, skipping transient build caches (node_modules, .venv, __pycache__).
• 3am - Cold VHDX Copy - Rather than trusting live checkpoints, each Linux VM shuts down cleanly in sequence, copies its virtual disk over the 10GbE storage VLAN to a separate un-snapshotted NAS share (avoiding Btrfs copy-on-write fragmentation on big disk images), and boots back up. Each VM is offline for about ninety seconds at 3am.

Most importantly, both jobs write a small _backup-status.json heartbeat file with the timestamp, duration, byte count, and exit code into the target share. At 7am, Homer’s morning report script reads that JSON off the NAS and flags any job whose timestamp is more than 26 hours old or whose exit code isn’t zero.

A backup is only real if an independent check verifies the files actually landed.

Act II: The Mirror That Worked Too Well

Three weeks after the new 2am workspace mirror went live, a routine check of the NAS backup share turned up the second problem.

That NAS backup share had an active Cloud Sync task mirroring its contents to cloud storage as Leg 2 of the 3-2-1 chain. While the old desktop client had only synced a narrow set of subfolders, the new robocopy mirror covered the whole workspace tree - which included:

• Per-server .env files created during the August MCP refactor (to work around Windows env vars not crossing wsl -e)
• Unencrypted weekly config dumps pulled from OPNsense (config.xml, complete with pre-shared keys and password hashes) and the Aruba controller
• Local AI session transcripts (.claude history files), which occasionally captured command outputs containing tokens during debugging

The cloud drive was private to my account and nothing had been touched externally, but having plaintext .env files, firewall config dumps, and session logs sitting in cloud storage broke my own rules on secret hygiene.

Closing the Loop: Pre-Encryption and the vault CLI

Fixing it properly meant assuming any folder backed up to the NAS might eventually be synced off-site, and making sure a raw filesystem copy of the workspace holds zero usable secrets.

1. Encrypting Appliance Backups at Capture

An OPNsense config.xml or Aruba running config is effectively a master key to the network. Instead of writing config.xml to disk and relying on folder permissions downstream, the weekly backup script now pipes the downloaded config straight through age using my public recipient key before a single byte touches the disk.

Only .config.xml.age ever lands on the filesystem. Copy the backup folder to a USB drive or a cloud bucket and all you get is ciphertext.

2. Hard Exclusion Gates on Mirrors and Syncs

Both the nightly NAS mirror and the cross-machine workspace sync now enforce strict filter lists blocking .env*, *.key, *.pem, devices*.json, and .claude/ transcript folders, backed by an automated check that verifies zero matching files exist in the mirror target.

3. Replacing .env Files with a Session-Cached vault CLI

The last piece was getting rid of the per-server .env files that the MCP servers read at startup. Keeping plaintext .env files on the WSL2 disk just to cross the Windows-to-Linux process boundary was a lazy shortcut.

To kill them off without typing an age passphrase every time an MCP server boots, I wrote a small Python CLI (vault) around the secrets.age file:

1. Unlock once per login session - Running vault unlock asks for the age passphrase once and caches the decrypted secret map in the OS credential store (protected by DPAPI on Windows, or a 0700 RAM-backed tmpfs directory on Linux).
2. Process-scoped injection (vault exec) - MCP server wrappers and scripts run under vault exec, which reads the needed keys from the session cache and injects them straight into the child process’s memory environment.
3. Ephemeral SSH pipes (vault ssh) - Automation SSH private keys live inside the age vault too. When a tool needs to SSH into an appliance, vault ssh feeds the key through an ephemeral pipe or short-lived 0600 tmpfs file for the duration of the command and wipes it on exit.
4. Zero secrets in AI transcripts - Because agents run commands through vault exec -- <command> instead of reading .env files or passing tokens on the command line, secret values never enter the LLM’s context window or session logs.

To close it out, I rotated every credential that had touched the old mirror - API keys, RADIUS shared secrets, service account passwords, and SSH keys - across the firewall, switch, wireless controller, NAS, and VMs.

What Survived

Those two incidents - eight days of silent backup failure followed by three weeks of over-eager cloud syncing - are really the same lesson. Automation does what you wrote, not what you meant.

If you don’t monitor the heartbeat, your backup job will eventually fail in silence while returning exit 0. And if you don’t encrypt secrets before they hit the disk, your backup job will eventually do its job and copy them somewhere you didn’t want them to go.