The disk filled at the worst possible hour
It had been filling for days and nothing was watching the trend, so the first notification anyone received was the outage itself.
What actually happens
Disk-full outages are the most preventable incidents in any estate, and they keep happening, because the alert is set on the wrong quantity.
A volume grows steadily over several days. Log rotation is not keeping up with a change in verbosity, or a retention policy quietly stopped applying, or a new feature is writing more than anyone modelled.
The alert on that volume is set at a percentage, typically somewhere between eighty and ninety. That alert is not wrong, it is just late by design: it fires when the situation is already nearly out of time, and by convention it fires at whatever hour the growth curve happens to cross it.
That hour is usually a bad one, because sustained growth curves tend to cross thresholds during the quiet period when nobody is watching and nothing else is competing for the disk.
The service fails when writes start failing. The failure mode is rarely graceful: partial writes, corrupted state, and a restart sequence that has to be done carefully rather than quickly.
The remediation is almost always trivial. Clear a directory, extend the volume, fix the rotation policy. It is trivial at 3am and it is equally trivial at 3pm four days earlier, which is the entire point.
A percentage threshold answers "am I nearly full". Nobody set an alert on the question that would have prevented this, which is "when will I be full".
The same week, two ways
Measured in minutes this looks unimpressive. Measured in days of warning it is the whole argument.
Illustrative, not measured. The times below model a scenario built from the patterns we see in production estates. They are not timings recorded at a named customer. The point is the shape of the clock, not the totals: check it against your own last ten incidents.
Today, alerting at the threshold
- Day 1Volume growth rate increases after a logging change. Utilisation well below threshold.
- Day 1 ↓ Day 4WaitingGrowth continues steadily. No alert, because no threshold is crossed.
- Day 4 02:40Threshold crossed. Alert fires. On-call paged.
- Day 4 02:55Writes begin failing before anyone can act. Service degrades.
- Day 4 03:30Directory cleared, volume extended. Service restarted carefully.
- Day 4 04:10Recovered. Rotation policy fixed the following morning.
~90 minutes of outage · after 4 days of visible warning
With Sentinel projecting time to full
- Day 1Sentinel detects the growth rate change against this volume own history.
- Day 1Log rotation configuration, retention policy, recent deploys and write patterns correlated.
- Day 1Cause: a logging verbosity change increasing write volume beyond what rotation reclaims.
- Day 1Projection attached: at the current rate, this volume reaches capacity in approximately 3 days.
- Day 1Action Ticket executes the reclaim under policy, and raises the rotation policy fix for the owning team.
- Day 2Rotation policy corrected in business hours. No outage, no page, no restart sequence.
3 days of warning · and the fix happened in daylight
The remediation is identical in both columns. What differs is whether it happened during a planned working hour or during a service outage at three in the morning.
The reason threshold alerting fails here is arithmetic rather than technical. A threshold is a position, and the quantity that predicts an outage is a rate. Position tells you where you are, rate tells you when you arrive.
Sentinel monitors the growth rate and publishes a time-to-full projection. That projection is what makes the ticket actionable days early: the owning team is not asked to act on a volume that is sixty percent full, they are shown a date.
This is the easiest pattern on the site to validate against your own estate. Pull your last four disk-full incidents and plot utilisation for the preceding week. In almost every case the slope is visible and unambiguous well before the threshold.
Why the number is what it is
The remediation is identical in both columns. What differs is whether it happened during a planned working hour or during a service outage at three in the morning.
The reason threshold alerting fails here is arithmetic rather than technical. A threshold is a position, and the quantity that predicts an outage is a rate. Position tells you where you are, rate tells you when you arrive.
Sentinel monitors the growth rate and publishes a time-to-full projection. That projection is what makes the ticket actionable days early: the owning team is not asked to act on a volume that is sixty percent full, they are shown a date.
This is the easiest pattern on the site to validate against your own estate. Pull your last four disk-full incidents and plot utilisation for the preceding week. In almost every case the slope is visible and unambiguous well before the threshold.
Sentinel monitors the growth rate and publishes a time-to-full projection.
Who decides to press go
Reclaiming space and changing a rotation policy are different classes of action and are gated differently.
Reclaiming space from an approved reclaim list runs under policy: pre-check, execute, post-check, verify, with automatic rollback. Nobody is paged.
That is raised for the owning team with the projected growth, the compliance implications of the retention change and the rollback path attached, and it waits for approval.
The split matters because reclaiming temporary files is safe and changing what data you keep is a decision with regulatory weight. Those should never be the same action.
Volume growth rate increased against this volume own history. Utilisation still well inside threshold. No alert would fire for days.
Log rotation configuration, retention policy, recent deploys and write patterns correlated. Logging verbosity change identified as the cause of the increased write volume.
Reclaim executed under policy from the approved list. Rotation policy fix raised for the owning team with a time-to-full projection attached.
Growth rate promoted to a monitored quantity alongside utilisation. Time-to-full projection published per volume. Verbosity changes added to the pre-deploy capacity check.
This is a platform capability, not a published customer deployment for this exact scenario. The mechanism, which is correlated investigation followed by governed MOP execution with pre-check, post-check, rollback and approval gating, is running in production today; see governed day-2 operations across 2,000+ nodes, infrastructure observability across a large server estate and closed-loop network automation. The timings shown are modelled, not measured.
If the action carries no service impact
Reclaiming space from an approved reclaim list runs under policy: pre-check, execute, post-check, verify, with automatic rollback. Nobody is paged.
If it changes a retention or rotation policy
That is raised for the owning team with the projected growth, the compliance implications of the retention change and the rollback path attached, and it waits for approval.
What Sentinel did, step by step
- ObserveVolume growth rate increased against this volume own history. Utilisation still well inside threshold. No alert would fire for days.
- InvestigateLog rotation configuration, retention policy, recent deploys and write patterns correlated. Logging verbosity change identified as the cause of the increased write volume.
- ActReclaim executed under policy from the approved list. Rotation policy fix raised for the owning team with a time-to-full projection attached.
- OptimizeGrowth rate promoted to a monitored quantity alongside utilisation. Time-to-full projection published per volume. Verbosity changes added to the pre-deploy capacity check.
Bring us your last disk-full outage
We will plot the week before it and show you how many days of warning there were.