The reconciliation job failed and told nobody

What actually happens

A job that fails loudly is a nuisance. A job that fails quietly is a data quality incident wearing a disguise.

A reconciliation job runs as part of the nightly pipeline. It exits without completing its work and without raising anything that the operations team would see, because the orchestrator records it as finished and the failure lives in an exit code and a log line.

Downstream jobs do not wait, because from their perspective the upstream dependency completed. They run against incomplete data and produce output that is internally consistent and wrong.

That output feeds reporting. The reporting is used. Nobody questions it, because it looks exactly like the reporting that was correct yesterday, and the only signal that anything is off is a set of numbers that will not tie out at a point somewhat later in the process.

When finance raises it, the incident is no longer a failed job. It is a lineage problem: which datasets are affected, which reports drew from them, who has already seen the numbers, and what has to be reissued. That work is the expensive part and it is entirely caused by the delay.

The recovery is roughly twice the work it would have been at the time of failure, because reprocessing one job is straightforward and reprocessing a corrupted chain is not.

The job took seconds to fail. The corruption took hours to spread. Everything expensive about this incident happened in the gap between those two events.

The same night, two ways

The comparison here is not about repair speed. It is about how far the corruption travelled before anyone noticed.

The important number in this comparison is not the fifteen hours. It is the fact that in the second column no incorrect report was ever published, so there is no reissue, no recipient list and no conversation with finance about which numbers to trust.

Holding the downstream chain is the whole mechanism. It is a reversible action taken in the first five minutes, and it converts a data quality incident into a delayed job.

Sentinel treats job outcome as a first-class signal rather than as orchestration metadata. An exit code that indicates incomplete work is an event, and the dependency graph is already known, so the blast radius is computed before anything downstream is released.

This is the cheapest pattern on the site to verify against your own estate. Look at your last three data quality incidents and check what time the originating job actually failed. The gap between that timestamp and the first human question is the number this addresses.

Who decides to press go

Holding a downstream chain and replaying a reconciliation are different classes of action, and they are gated differently.

Holding the downstream schedule runs under policy. It is fully reversible, it corrupts nothing, and releasing the hold is a single step once the upstream is correct.

The re-extract and replay MOP is raised and held with the affected dataset list, the record count expectations and the rollback path attached. The data platform owner approves before anything is reprocessed.

The asymmetry is deliberate. Stopping something is safe and Sentinel does it immediately. Reprocessing something is not, and it waits for a name.

Reconciliation job exits incomplete. Orchestrator reports the job as finished. No operator notification fired. Downstream jobs scheduled to run.

Job history, input record counts, dependency graph and downstream schedule correlated. Source extract delivered a partial file, leaving nothing to reconcile against.

Downstream chain held before execution. Affected dataset list attached. Re-extract and replay MOP staged and held for the data platform owner.

Record count validation added at extract rather than at reconciliation. Job exit codes promoted to first-class signals across the pipeline. Partial-file signature added to the watch list.

This is a platform capability, not a published customer deployment. The mechanism, which is treating a job outcome as a signal and then executing a governed MOP with pre-check, post-check, rollback and approval gating, is running in production today; see governed day-2 operations across 2,000+ nodes. The finance pipeline scenario above applies that same mechanism to a data workload. The timings shown are modelled, not measured.