Six lessons from automation that runs while you sleep

Most of what I run at home runs without me. Jobs fire on timers, do work, publish results, and tell me only if something interesting happened. That’s the point of automation and it’s also the danger of it, because a job nobody watches can be broken for two weeks before anyone notices.

These all cost me something. None of them are clever. Most read as obvious and weren’t obvious until they’d happened to me.

1. A monotonic timer can stop firing and say nothing

The first one is specific enough to be worth naming exactly, because it’s a trap in a very widely used tool.

Systemd timers come in two broad shapes. Calendar timers fire at wall-clock times — every hour on the hour, every Tuesday at five. Monotonic timers fire relative to an event, so fifteen minutes after boot, or an hour after the last run.

Monotonic looks simpler and for a lot of jobs it is. But a monotonic timer can end up in a state where it has elapsed and has no next trigger scheduled, and when that happens it just stops. No error, no failed unit, no alert. The job never runs again.

I lost sixteen days of data collection to exactly that before noticing the numbers had gone flat. What I’d suggest is calendar timers with catch-up enabled for anything whose job is to happen regularly, and monotonic reserved for things genuinely tied to an event.

The general version is to prefer scheduling primitives that fail loudly. A job that runs and errors is visible. A job that silently stops existing isn’t.

2. Your success message will lie to you eventually

I had a fetch step that printed how many new items it had found, and it printed the same number every run, forever, because it was counting the wrong thing — total candidates after de-duplication rather than genuinely new ones.

The consequence wasn’t the wrong number. It was that “quiet” and “broken” became indistinguishable. When somebody eventually asked why nothing had been published for three days, the logs said the pipeline had found plenty of work each time. It hadn’t. I couldn’t tell a slow news week from a dead feed, so I couldn’t tell there was nothing wrong — which, as it turned out, there mostly wasn’t, but three other real bugs were hiding behind that one bad counter.

Make the summary line report the thing you’d actually use to decide whether to investigate. If it can’t distinguish “nothing happened” from “nothing worked”, it’s worse than no summary at all.

3. Never let a raw exception reach a log

A library that raises on a failed HTTP request will usually put the request URL in the exception message, and if that URL contains an API token — which for a lot of messaging and notification APIs it does, in the path — then an unhandled error prints your credential.

On an interactive run you see it and shrug. On a scheduled run it goes to the system journal, where it sits in plain text in a file that gets rotated into backups.

Every outbound call of mine now goes through a small wrapper that catches, discards the original exception, and re-raises with only the method name and the service’s own error description. It took twenty minutes to write and closed a hole I wouldn’t have found by looking.

4. A deterministic check that is too broad fails with total confidence

I had a validation step whose job was to reject any text claiming a vulnerability was being exploited when the source didn’t support it, and it was implemented as a substring search for the word “exploit”.

Which meant it rejected “no known exploitation”. And “not currently exploited”. Both of which are true, and both of which are precisely what careful writing says.

It sat there quietly discarding correct output with nothing anywhere indicating that the check was the thing that was wrong. The failure looked like a generation problem, so I spent time on the generator.

The lesson isn’t really “handle negation”, though it’s now negation-aware. It’s that a deterministic guard which is too broad is worse than a fuzzy one, because it fails with complete confidence and produces no reasoning you can inspect. If you write a rule-based check, probe it in both directions — the things it should catch, and the things it must not.

5. Approval gates must not block

An early version of a review step posted a message and then waited, in-process, for somebody to approve it, with a six-hour timeout that felt generous.

It fails on the most common real case there is. The job runs at half past midnight, everyone’s asleep, and by seven in the morning the process has already given up, having sat idle for six hours holding a finished piece of work.

Approval is durable state on disk now. The job posts the request, writes a record, and exits, and a separate small job runs periodically, matches approvals to pending records and acts on them. A response at any hour works, including after a reboot, because nothing is waiting in memory.

The related trap is that tapping approve twice delivers two events, and if both arrive in the same batch, naive handling does the work twice. Claim the token before doing any work, and re-check that the pending record still exists.

6. Write down how it works somewhere that is not the machine

This is the expensive one.

I had a box holding the only copy of a system’s configuration and the credentials it used, and it died. The code existed in a repository; the thing that made the code run didn’t.

Everything lives in version control now — the scripts, the unit files, the configuration, and the documentation of why each odd decision is the way it is. Secrets stay out and go somewhere they can be reissued. The test I apply is whether, if this machine vanished tonight, rebuilding it would be a boring afternoon or a research project.

The related discipline is writing down the reasons rather than just the settings. Half the notes I keep are of the form “this looks wrong and must stay this way, here is what happens if you fix it”. Those have saved me from myself more than once, because six months later I’m a different person with less context and an appetite for tidying up.

The thread running through them

Every one of these was invisible rather than loud. The timer that stopped, the counter that lied, the check that rejected correct output, the gate that gave up while everyone slept — none of them raised an error, and some of them looked like success.

So if there’s one thing to take from it: when you build something that runs unattended, the effort goes into making failure visible rather than into making it unlikely. You won’t prevent it. You can arrange to find out.