Customer Case Study · Tier-1 · North America
Day-2 operations at node scale, governed end to end
How a Tier-1 customer in North America put ProcBot in charge of day-2 operations across more than two thousand live nodes: scalable health checks, log capture with Sentinel analysis on top, SSH access under RBAC, and mass patching that runs as governed MOPs, prechecked, postchecked, audited, and verified after every apply, with roughly one hundred thousand nodes forecast to follow.
- 2,000+nodes live today under ProcBot day-2 automationCounted, live today
- ~100knodes forecast to run this automation at target scaleFORECAST, not a result
- 100%of MOP runs carry precheck, postcheck and a full audit trailEvery run, not a sample
- ~70%faster mass patch rollout versus the manual approachAgainst the prior manual rollout
- ~60%less engineer effort consumed by routine day-2 operationsEngineer time returned
- Automatedpost-apply verification: applied or not, errors flagged, success and failure rates reportedCompiles per cycle, unassisted
The Challenges
Day-2 work, health checks, log collection, process verification, and above all patching, had no platform. There was no solution for rolling out mass patching across the server estate: multiple engineers did it manually, or through scattered shell scripts that nobody governed. Whether a patch actually applied correctly was a question answered node by node, if it was answered at all.
- No mass patching capability
Patch rollouts at estate scale simply did not exist as a repeatable operation; each cycle was a project.
- Manual, engineer-heavy execution
Multiple engineers walked servers by hand, node after node, cycle after cycle.
- Ungoverned scripts
Where automation existed, it was ad-hoc shell scripting: no approvals, no audit trail, no rollback story.
- No verification after apply
Whether a patch landed correctly, and what broke if it did not, was unknown until something failed later.
- Health and logs out of reach at scale
Checking node health, capturing logs and reading process status across thousands of servers was not feasible manually.
- Scale ahead, method behind
The estate was heading toward one hundred thousand nodes, on a process that strained at two thousand.
The architecture, and what wraps every run
Running a command across two thousand machines was never the hard part. Two things were: knowing afterwards what state two thousand machines are in, and finishing before the maintenance window closes. The first is answered by wrapping every run in a precheck and a postcheck. The second is answered by the execution engine, and it is the reason the path from two thousand nodes to a hundred thousand is the same procedures rather than a proportionally larger team.
ProcBot does not walk an estate node by node. When a MOP is dispatched, a planner sizes the run: it derives how many containerised execution pods to start from the number of target devices and endpoints, the concurrency limits you have set and the window available, and the pods work through their slices concurrently. The sizing is dynamic and it is deliberately not one pod per device. Each pod takes a slice of many devices and works through it, so a run across thousands of nodes starts tens of pods rather than thousands, and the same engine handles a cycle of eleven nodes without spinning up machinery it does not need. A cycle covering eleven nodes and a cycle covering two thousand run the same certified procedure and differ only in how wide the fan-out is. The practical consequence is that wall-clock time for a rollout tracks the slowest individual node rather than the sum of all of them, which is what turns mass patching from a project into an operation.
Governance on every touch: RBAC on access, approvals on execution, audit and logs on everything ProcBot does. The estate gained mass automation and lost nothing in control, the opposite of the ungoverned scripts it replaced.
What was switched on, and what was not
This estate had been burned by ungoverned shell scripts, which is the specific reason the governance is the feature rather than the tax. A team that has lost a night to an unaudited script does not respond to a proposal for faster unaudited scripts.
Note what the left-hand column does not contain: the platform choosing which nodes to patch, or when. Mass rollout became an operation the platform performs, not a decision it makes.
Runs without a human
- Continuous health sweeps across the estate, at the scale one node used to be checked
- Log capture at scale, with Sentinel analysing for patterns and anomalies
- Reading process status directly from nodes over SSH under RBAC, every session logged
- Execution of an approved MOP, wrapped in precheck, apply and postcheck
- Sizing the run: how many execution pods to start, derived from the device and endpoint count
- Compiling per-node outcomes and success and failure rates into a rollout report
Does not run without a human
- Approval to execute. Approvals sit on execution, not on the platform judgement of its own readiness
- Authoring or certifying a MOP. Procedures are written and approved by your engineers
- Anything outside the approved MOP set, at any scale
- Proceeding when a precheck fails. The run halts and records why rather than continuing on a best guess
What a patch cycle looks like now
The same six steps run whether the cycle covers eleven nodes or two thousand. That property is the whole point, and it is why the path to a hundred thousand is the same MOPs rather than a proportionally larger team.
The Sentinel loop, as it runs here
Every Opstral deployment runs the same four-stage loop: Observe, Investigate, Act, Optimize. It is the methodology rather than a feature list, and the point of setting it out per deployment is that you can see which stages carried the weight in this one and which did not.
- ObserveTake in every signal the estate produces, normalised and correlated as it arrives rather than after somebody goes looking.HereA MOP is authored and approved by a named engineer, and the engine sizes the run against the estate before anything starts. Observation here is of the fleet’s readiness rather than of an incident.
- InvestigateWork the signal into a probable cause with the evidence attached, before anyone is notified.HereLogs from every run are captured and analysed rather than archived, which is what turns a patch cycle from an event you survive into a dataset you learn from.
- ActRun the approved procedure where the blast radius allows it, or hand a named human the plan, the evidence and the rollback.HereThis deployment is concentrated here, and deliberately so. Precheck validates each node, apply runs in parallel under RBAC over SSH with a full audit trail, and postcheck asserts the result node by node.
- OptimizeFeed the outcome back so the next run of the loop is better informed than the last.HereThe cycle reports itself, so the next run is planned against what the last one actually did rather than against what the runbook says it should do.
This is our execution study rather than our detection study. If you are looking for the Observe and Investigate stages carrying the weight, the carrier-scale network observability deployment is the better read.
- A MOP is authored and approved
Your engineers write the procedure and it enters the approved set by a deliberate act. Nothing enters that set because the platform performed well recently.
Named engineerObserve - The engine sizes the run before it starts
The planner reads how many devices and endpoints this MOP targets, together with your concurrency limits and the window available, and starts the number of containerised execution pods that run needs. Each pod takes a slice of many devices rather than one device each, so the count is tens rather than thousands. Nobody chooses a batch size and nobody splits the estate into tranches by hand, which is what the previous process required and what made every cycle a project.
PlatformAct - Precheck validates the node before anything touches it
The node is checked and the result recorded. This is what makes the postcheck meaningful: without a recorded starting state, verifying the outcome tells you a command completed rather than that the change landed correctly.
PlatformAct - Apply runs in parallel, under RBAC, over SSH, fully audited
Every pod executes the identical certified procedure against its own slice, concurrently rather than one node after another. Each session is accountable and each step is logged as it happens. No pod runs a shortened version of the procedure: the precheck and postcheck travel with the MOP, not with the scheduler.
PlatformAct - Postcheck asserts the result, node by node
Applied properly or not, with errors flagged and what happened recorded. One hundred percent of runs carry this. It is the difference between a rollout you have to investigate and a rollout you can read.
PlatformAct - Logs are captured and analysed rather than archived
Node logs are collected at estate scale and passed to Sentinel, so patterns and anomalies surface instead of sitting unread on disk until somebody has a reason to go looking.
PlatformInvestigate - The cycle reports itself
Success rates, failure rates and per-node outcomes compile automatically. Previously this lived in spreadsheets and memory, which is a polite way of saying it lived nowhere once the people who ran the cycle moved on.
PlatformOptimize
The Impact
Mass patching went from nonexistent to routine. Two thousand nodes run under governed day-2 automation today, engineers stopped walking servers, and the path to one hundred thousand nodes is the same MOPs at larger scale, not a bigger team.
| Dimension | Before | After · ProcBot-governed |
|---|---|---|
| Mass patching | No solution; manual, engineer-by-engineer rollouts | MOP-driven rollout across 2,000+ nodes, scalable by design |
| Automation governance | Ad-hoc shell scripts, ungoverned | Precheck, postcheck, approvals, audit and logs on every run |
| Node access | Individual sessions, unaccountable | SSH under RBAC, process status read directly, every action logged |
| Verification | Applied or not, discovered later | Automated post-apply analysis, errors flagged, outcomes reported |
| Rollout visibility | Spreadsheets and memory | Success and failure rates compiled automatically per cycle |
| Scale trajectory | Straining at 2,000 nodes | ~100,000 nodes forecast on the same governed automation |
Patching two thousand nodes by hand is a staffing problem; patching a hundred thousand is an impossibility. Making the MOP the unit of work, with precheck, postcheck and audit built in, turned mass rollout from something the team dreaded into something the platform reports on.
At a glance
- Industry
- Tier-1 Enterprise (North America)
- Geography
- North America
- Engagement
- Production deployment
- Scope
- Day-2 Server Operations & Mass Patching
- Platform components
- ProcBot distributed MOP execution engine, pod-based parallel runs, SSH under RBAC, Sentinel log analysis
- Evidence basis
- Node counts are as deployed and forecast by the customer. Improvement percentages are indicative rather than measured. Customer identity withheld by request.
Make day-2 operations a governed capability
Scalable health checks, Sentinel-analyzed logs, and mass patching with built-in verification, executed by ProcBot inside your environment.
Customer identity withheld by request and referred to throughout as a "Tier-1 customer in North America." Node counts are as deployed and forecast by the customer; improvement percentages are indicative, reflecting target and expected outcomes, and vary by estate scope and rollout phase.