Product
Inside Sherlock: How We Verify a Fix Actually Held
Written by
Jayesh Verma
Published
August 2026
Reading time
9 min read
A closure task is a set of independent mechanical checks with their criteria written before the action ran. There is deliberately no model involved at evaluation time.
Here is a sentence that appears in incident records everywhere and means less than it looks: the remediation completed successfully.
It can be true while the fault is still present. What it asserts is that a command ran and exited without error. Whether the condition that caused the incident actually cleared is a different question, and in most estates nobody asks it, because asking it means going back and looking at a metric twenty minutes later when three other things are on fire.
Sherlock exists for that one question. Did it hold? Not did the command return zero. Did the thing that was wrong stop being wrong, measured against live signal, judged by a criterion written before anyone touched anything.
A closure task is not a check, it is several
One incident rarely reduces to one assertion. A cache node restarted to recover payment latency needs the latency to recover, the error rate to fall, and the node to still be in the pool ten minutes later. Any one of those passing on its own tells you very little.
So closure is submitted as a task containing several independent leaf checks. Each leaf carries its own mechanical criterion, something on the order of p99 below 100ms across two consecutive samples. The leaves are evaluated independently and in parallel. There are deliberately no conditional edges between them, no if-this-then-check-that, because a branching evaluation tree is a small program and a small program is something else to debug at the exact moment you need a trustworthy answer.
Who owns which decision
The split matters more than it sounds, because it is what stops the verifier from being able to talk itself into a pass.
- Sentinel authorsDecides which checks are worth running, which connector each one uses, what each leaf asserts, and what policy across the leaves counts as verified. All of it before the remediation runs.
- Sherlock executesOwns the connectors, the scheduling and the retries. Runs each leaf and marks it passed or failed against that leaf own rule. It never decides the incident-level verdict.
- The policy decidesAll leaves, or a stated minimum, or a weighting. Written down in advance as part of the task, evaluated mechanically, and recorded with the result.
Nothing in that chain gets to reconsider. The component that chose the criteria is not the component that grades them, and the grading is arithmetic rather than judgement.
Failure reopens, it does not close
When the policy is not met, the incident does not quietly close with a note. It reopens, as a child incident, carrying the evidence: which leaves passed, which failed, what each one asserted and what it actually measured. The remediation that ran is on the record as an attempt rather than a resolution.
That is a small design decision with a large consequence. Once failed closure is a recorded state rather than an absence, the estate accumulates data about which fixes do not hold, and top-offender analysis becomes possible. That is where the reduction in repeat incidents in our North America telecom deployment came from, and it is bookkeeping rather than cleverness.
The number this changes
Most cockpits report mean time to resolve measured to the moment the remediation completed. That is the number that makes automation look best, and it is measuring the wrong event.
| What is measured | What it tells you |
|---|---|
| Time to execution | How quickly a procedure ran. Says nothing about whether the fault cleared. |
| Time to verified closure | How long until independent checks confirmed the condition held. Always the larger number. |
| Failed closure rate | How often a fix that reported success did not hold. The most useful and least published number in operations. |
We hold ourselves to the middle row internally, and it is not flattering. Verified closure is slower than execution by definition, and publishing the faster number would be the easiest and most common form of dishonesty in this category. Our rule for which numbers get published at all is in how we decide what to publish as a number.
Why no AI in the evaluation
This is the part people find counter-intuitive from a company that sells autonomous operations. Models are genuinely good at the upstream half: reading an incident, understanding which signals would indicate recovery, and proposing checks a tired engineer would not have thought to write.
They are the wrong tool at the point where the answer becomes evidence. A verdict on whether a fix held has to be reproducible, explainable to somebody who was not there, and identical on the second reading. A mechanical assertion against a metric satisfies all three. A model asked to judge the same evidence twice does not reliably satisfy any of them, and the moment you cannot reproduce a verdict, the audit trail underneath it is decoration.
Sherlock is the third component of the pattern described in introducing the SRE Agent, and the post-check stage of an Action Ticket.
Frequently asked questions
Why is there no model at evaluation time?
Because a verdict on whether a fix held has to be reproducible by anyone reading the record a year later, and a model asked the same question twice may not answer it the same way. Each leaf check is a mechanical assertion against a metric or a state, and the aggregate verdict is a stated policy across those checks. Models are useful upstream, choosing which checks are worth running; they are a liability at the point where the answer becomes an audit record.
What happens if the checks pass but the problem is still there?
Then the criteria were wrong, and that is a real failure mode we do not pretend away. A weak closure criterion closes an incident that has not held, which is the original problem wearing a better interface. The mitigation is that the criteria are visible, versioned and attached to the record, so a recurrence can be traced back to the assertion that was too loose rather than argued about.
How long does verification take?
As long as the criteria say. Some assertions are immediate, others need consecutive samples over a window before they mean anything, and a fix that looks good for thirty seconds and fails at four minutes is exactly the case a short window would miss. The incident stays open until the window closes, which is why time to verified closure is a different and larger number than time to execution.
Does this only apply to automated fixes?
No, and applying it to human fixes is often the more useful direction. The same closure task can be raised after an engineer resolves something by hand, which answers "did the thing you did at 03:00 actually hold" without anyone having to remember to check in the morning.