logrotate-bug-restarted-my-fleet-at-midnight
TL;DR — Five of my Go backends restarted every single midnight for about three weeks and I never noticed. A logrotate postrotate script ran systemctl reload on binaries that have no reload handler, so systemd delivered a SIGHUP, and SIGHUP's default disposition is "die." Restart=always resurrected each one in about five seconds, which is exactly fast enough to hide a chronic crash from every health check I have. The fix was one word: copytruncate. The lesson is a counter called NRestarts.
The discovery
I wasn't looking for this. I was doing a routine log sweep across the fleet — the kind of boring hygiene pass that never finds anything — when I noticed something odd in one backend's logs: a clean startup banner. At midnight. Then I scrolled back a day. Another startup banner, also at midnight. Every midnight, going back weeks, like the service had a bedtime.
systemctl status said everything was fine. Green dot, active (running). But two lines down:
NRestarts=22
Twenty-two restarts. One per midnight, for three weeks, on a service I believed had weeks of uptime. Four sibling services told the same story.
The cause: reload is not restart
Each product on my box gets provisioned by a setup script from a shared template. That template wrote a logrotate config that looked completely reasonable:
postrotate
systemctl reload <service>-backend > /dev/null 2>&1 || true
endscript
The intent is standard sysadmin folklore: after rotating the log file, tell the daemon to reopen its file handles so it doesn't keep writing to a deleted inode. For nginx or Postgres, reload does exactly that.
My backends are plain Go binaries. They have no ExecReload= in their unit files and no SIGHUP handler in the code. So here's what systemctl reload actually does in that case: systemd has no reload command to run, falls through to sending the unit SIGHUP — and a process with no SIGHUP handler terminates. That's the POSIX default. "Reload" quietly became "kill."
So every midnight, logrotate rotated the logs, then politely asked five services to reload, and all five dropped dead.
The accomplice: Restart=always
Here's the part that actually bothers me. Every unit has Restart=always, because of course it does — that's the responsible setting. And it worked flawlessly: each service came back in roughly five seconds. Total user-facing damage per night, maybe five seconds of connection resets at 00:00, when approximately nobody is using my products anyway.
Which means Restart=always didn't just mitigate the bug. It concealed it. My uptime monitors probe every minute or so; a five-second blip at midnight sails straight through. Health checks passed. Dashboards were green. The only witnesses were a startup banner in the logs and NRestarts silently incrementing — and NRestarts appears on exactly zero default dashboards. It's the odometer of crashes, it resets when you manually restart the unit, and nobody looks at it. Twenty-one midnights times five seconds is under two minutes of total outage across three weeks. That is simultaneously "nothing" and "your fleet crashed 100+ times and you had no idea."
The || true at the end of the postrotate line deserves a dishonorable mention too: even if the reload had errored loudly, we'd have thrown the evidence away.
The fix: one word
The correct fix for "daemon keeps writing to a rotated file" when the daemon has no reopen mechanism is logrotate's copytruncate: copy the log, then truncate the original in place. The file handle never changes, so the process never needs to be told anything, so there is nothing to reload, so there is nothing to accidentally kill.
- create 0640 <service> <service>
- sharedscripts
- postrotate
- systemctl reload <service>-backend > /dev/null 2>&1 || true
- endscript
+ copytruncate
Net diff: minus five lines, plus one. (Yes, copytruncate can drop a few log lines in the copy-truncate window. For app logs on my fleet, that trade is free.)
I fixed the live configs on the box, then — and this is the part that matters — fixed the provisioning template in the repo, because the template is where the bug actually lived. Those five services didn't independently develop the same disease. They inherited it. Every product provisioned from that template got the broken postrotate stanza for free, and every future product would have too. A bug in a template isn't one bug; it's a bug subscription.
Verification was pleasantly binary: wait for midnight. The next morning, every service showed an unbroken uptime through 00:00 and NRestarts counters that stayed put. Three weeks of nightly murders, ended by one word.
What I actually learned
Reload is not restart, and on the wrong process it's not even reload. systemctl reload on a unit with no reload handler degrades to SIGHUP, and SIGHUP's default is death. If you write reload in any automation, you owe it thirty seconds of checking what the target actually does with it.
Restart=always hides chronic failure by design. It converts crashes into blips, and blips into nothing. It's still the right setting — but it means "is the service up?" and "is the service healthy?" are different questions, and only the first one is on your dashboard.
Check NRestarts. It's the one number that can't be fooled by a fast recovery. A quick systemctl show -p NRestarts '*.service' sweep is now part of my regular ops loop. Any nonzero value on a service I didn't deliberately restart is a finding.
Templates propagate bugs at provisioning speed. Shared setup scripts are how a solo operator runs a fleet at all — but they mean every mistake ships fleet-wide, silently, forever. When you find a bug in one provisioned service, the second question is always "who else inherited this?" The answer here was: everyone.
The whole incident cost me almost nothing in downtime and a small existential crisis about what else Restart=always is hiding from me. I've since audited every logrotate config on the box. They all say copytruncate now. Midnight is boring again, the way midnight should be.