2026-08-28 · article
Two ways a self-built app fails
Disk full but df shows free space. A cron job that runs green and writes nothing. An SSL certificate that did not auto-renew. The operational failures that take down an app you built yourself, what each symptom actually means, and the three fixes that cover most of them.
One stops everything, and you find out the same hour. The other reports success while it quietly stops doing part of the job, and you hear about it from a customer. The second kind is the one that costs you.
An alert told me a machine was over 80 percent full. I went looking, and the databases turned out to be small. The logs were 85 gigabytes. Nothing was broken yet. A few more weeks and the disk fills, the app stops, and no error gets written anywhere, because there is no room left to write one.
The code was fine. Log rotation was never configured, because nothing asks you to configure it. The default is to keep growing.
What follows is organised by symptom, because that is how these arrive. If you searched a phrase and landed here, the phrase is probably below.
Symptoms, and what each one actually is
“The disk is full, but df shows plenty of free space.”
What it is: On a container or an overlay filesystem, the reported free space can describe the virtual layer rather than the volume your writes actually land on. The scratch volume fills while the number you are checking stays healthy.
What it does next: Writes start failing with no space left on device. Processes that use temporary files to signal they are alive stop signalling. The supervisor kills them and cannot restart them. The result is a process that is running and serving nothing, with no log line explaining it, because logging needs disk too.
What to check: The real mount the container writes to, not the root filesystem. Then set an alert on it at 70 percent, not 90, because the useful part is the weeks of warning.
“My logs grew to tens of gigabytes on their own.” · “sqlite grew to 17Gb” · “where do I delete logs?”
What it is: Rotation is almost never on by default. Docker’s default json-file logging driver writes every line of container output to a file with no size limit and no rotation until you configure one. Workflow and automation tools usually store execution history as database rows, with pruning switched off.
The trap: Turning on pruning does not always give the space back. On SQLite the file keeps its size until you compact it, and compacting can need as much free space again as the database currently occupies, which is the one thing you no longer have. People discover this while already at 95 percent, and the repair becomes the outage.
What to check: Every service that writes output, not only the noisy one. Set the limit before you need it, and if you are already close to full, get space first and compact second.
“The job runs every night, finishes green, and writes nothing.”
What it is: Something quiet upstream. A token expired, a field got renamed, an API changed the shape of its response, or a filter stopped matching anything. Nothing threw an exception, so from the system’s point of view nothing went wrong.
Why no alert fired: Error handling fires on unhandled exceptions. This is precisely the case where there is no exception. The run succeeded. It just did not do anything.
What to check: The output, not the exit status. Not whether the job finished, but whether it wrote the rows it was supposed to write. If it finished and the count is zero, that is worth a look.
“The SSL certificate expired and did not renew. It worked fine for months.”
What it is: An automatic renewal that stopped being automatic, usually after a DNS change, a domain move, or a platform migration that nobody connected to certificates at the time. The failure surfaces up to ninety days after the change that caused it.
Why it is worse than it looks: Your users see a browser security warning, which is the single worst thing a small business can show a customer, and it happens on a day when nothing was deployed.
What to check: Expiry dates on every certificate you own, with a reminder at thirty days. If it renewed silently for a year and then stopped, the renewal has probably been failing for weeks already.
“I have backups.”
What it is: Usually a file. A backup you have never restored is an assumption, and the assumption is tested for the first time on the worst day you will have.
What goes wrong: The dump is of the wrong database. The retention window is shorter than the time it took to notice. The restore works but takes six hours, which you find out six hours into an outage. Or the platform’s own rollback fails and there is nothing underneath it.
What to check: Restore one somewhere else, once, before you need it. Write down how long it took. That number is the one you will want during the outage.
Can the assistant that built the app fix these?
Not by default, and this is worth understanding before you spend an evening asking it to.
A coding assistant edits code. An expired certificate, a full disk and a job that ran and did nothing are none of them in the code. It can act on the server, but only if it can reach the machine your app is deployed on and you ask it to do that explicitly. Left to guess, it will keep editing files that were never the problem, confidently, for as long as you let it.
Three things I would do first
- Cap the logs. A size limit and a number of files to keep, on every service that writes them. One line of configuration in most places, and it is the single highest ratio of risk removed to effort spent on this list.
- Set up a database backup, then restore it somewhere else once. Before you need it, to confirm it works and to see how long it takes.
- Check the result, not the run. Not whether the job finished, but whether it wrote what it was supposed to write. A finished run with a count of zero should be a question, not a success.
The part that is actually expensive: Not the tooling, and not the tokens, which come out of a subscription you already pay for. The cost is who ends up doing the looking. In a small company that is the owner, on an evening, on a problem that has nothing to do with the business.
Honest scope: you can do all of this yourself. Each item is an afternoon, and none of it is complicated. What it is not is a one-time job, which is the whole reason it gets skipped. Everything here comes from running systems that other people depend on, and from getting some of it wrong first.
Would rather spend that time on the business? Bring the situation to an external assessment and leave with a written recommendation. View the decision session.