Evidence
The Same Fault, Every Week, Until the Loop Closed
Written by
Amber Jain
Published
August 2026
Reading time
9 min read
A Tier-1 telecom operator in North America, seven data domains that never agreed, and a fourth stage almost nobody builds: refusing to close an incident until the numbers say it held.
Every network operations team has a fault it has fixed more than once. Somebody resolves it at 03:00, writes a short note, and closes the ticket. Six weeks later a different engineer meets the same symptom, does not find the note, and rediscovers the fix. Nothing about that sequence is anyone failing. It is what happens when the memory of an estate lives in tickets nobody searches and heads that go home at six.
This operator ran a network that told its story across seven separate data domains: faults, performance counters, configuration, inventory and topology, telemetry, syslogs and OSS streams. No two of them described an event the same way. Recognising that an anomaly in one domain and a counter in another were the same network problem was manual pattern-matching, done by people, under time pressure, at whatever hour the problem chose.
The result was a specific and very common failure: issues were closed on hope, and reopened on evidence. Nothing measured whether a fix had actually held, so the only reliable signal that it had not was the fault returning.
Four stages, and only one of them is unusual
What was built is a loop with four configurable stages. The first three are what most of this category calls self-healing.
- Discovery. ML models and custom rules run across all seven domains, looking for symptoms and anomalies rather than waiting for an alarm to name the problem. This is the difference between watching for something to break and watching for something to change.
- Diagnosis. Automated root cause through trees assembled from rules, ML models and the operator own knowledge documents. That last source is the important one. It takes diagnostic experience that existed as tribal knowledge and makes it executable and repeatable, which means available at 03:00 to whoever is actually on call.
- Remedial action. Auto-correction driven by the identified cause: trouble tickets, change requests, workflow orchestration, notifications and custom procedure execution, running autonomously or holding for a human exactly as the operator configured it.
- Feedback and closure. Quantitative metrics decide the outcome. The loop writes a terminal state of closed, on hold or failed. No issue is closed on hope.
Stage four is the one almost nobody builds, and it is the one that makes the other three mean anything.
Why closure is the hard part
Consider what "the automation fixed it" usually means. A procedure ran. It returned successfully. A workflow marked the ticket resolved. Every one of those statements can be true while the fault is still present, because what was verified is that the command completed, not that the condition cleared.
A closure criterion is a different object. It is a measurable assertion about the estate, defined before the action runs, and evaluated afterwards against live signal. Not "the restart succeeded" but "the counter recovered and stayed recovered across consecutive samples." When that assertion fails, the loop records a failure with the evidence attached rather than a resolution, and the incident stays open. This is the same principle as the post-check in an Action Ticket, applied at the level of a whole incident rather than a single procedure.
The side effect is the one people underestimate. Once closure is measured, repeat offenders become visible as data. Top-offender and persistent-root-cause analytics surface at estate level, so the fault you have fixed four times stops being an anecdote somebody half-remembers and becomes a ranked entry on a list. That is where the reduction in repeats comes from, and it is not clever remediation. It is bookkeeping that nobody was doing.
What was measured
| Figure | What it is |
|---|---|
| 7 data domains correlated | Fault, performance, configuration, inventory and topology, telemetry, syslogs, OSS streams. |
| ~60% of recurring faults resolved end to end | Discovered, diagnosed, acted on and verified closed by the loop, without a human in the path. |
| ~50% faster to root cause | Against the operator prior manual correlation process. |
| ~35% fewer repeat incidents | Attributed to top-offender and persistent-root-cause analytics, which only exist once closure is measured. |
| Full provenance on every automated fix | RCA tree traversal, node-level drilldown and the complete action trail. |
All five are from this one deployment, a Tier-1 telecom operator in North America. They are not the India estate figures and they are not the enterprise patching figures, and we do not add them together. The reasoning, including the occasion we got it wrong, is in how we decide what to publish as a number.
Provenance is what makes autonomy acceptable
Every automated correction records which RCA path fired, what evidence supported it, what actions ran, and how the loop closed. That turns "the system fixed it" from a claim into a statement with a trail behind it, and it is the difference between an operations team tolerating automation and trusting it.
It is also the answer to the question a regulator or an incident review will eventually ask, which is never "was the model accurate" but always "show me what happened and why." A black box that is right most of the time fails that question. A traceable loop that is right most of the time passes it, and the traceability is the cheaper of the two things to build.
The full before-and-after is in the case study. The other deployments in this series are a 27,000 device NOC in India and a North America enterprise patching estate.
Frequently asked questions
Which deployment produced these numbers?
One: a Tier-1 telecom operator in North America running a four-stage closed loop across seven network data domains. It is a different customer from the India telecom estate and from the North America enterprise patching estate we also publish, and the three sets of figures never appear in the same sentence. That rule, and the time we broke it, is set out in our evidence policy.
Is this the same as auto-remediation?
Auto-remediation is the third stage of four. The difference this deployment demonstrates is the fourth: the loop does not record an incident as resolved because a procedure returned successfully. It evaluates quantitative closure metrics and writes a terminal state of closed, on hold, or failed. An action that ran and did not hold is a failure with evidence, not a closed ticket.
Who writes the RCA logic?
The operator, mostly. The RCA trees combine rules, ML models and the operator own knowledge documents, and every stage including data sources, trees, actions and closure metrics is configured through a drag-and-drop modeller in their hands rather than ours. That matters more than it sounds: diagnostic experience that lives in three senior engineers heads is the single most fragile asset in any NOC.
What happens to the faults it cannot resolve?
They reach an engineer with the full provenance attached: which RCA path fired, what evidence supported it, what actions ran, and how far the loop got. That is the intended outcome rather than a gap. Around 60 percent of recurring faults resolved end to end means around 40 percent did not, and those are the ones where the evidence trail earns its keep.