Notification delivery and recovery verified

Gap closed: rules now have delivery evidence

The preceding review verified alert expressions and runtime telemetry, but did not exercise an Alertmanager receiver. The new executable gate starts checksum-pinned Prometheus 3.5.0 and Alertmanager 0.28.1, replays the collector's recorded metric contract, and injects three synthetic failures.

All nine repository rules load without test-time rewrites. The three exercised alerts retain their full two-minute pending period. Each test checks pending state, a delivered firing notification, a delivered resolved notification, stable fingerprints, severity, job/instance labels and matching runbook annotations. The receiver deliberately returns HTTP 503 once; Alertmanager successfully retries the same alert.

These timings are local fixture observations, including the configured pending period. They are not deployment SLOs.

Defect fixed: an incomplete scrape could look healthy

The missing-telemetry rule checked only the health-success flag. If a host exported that flag but dropped its last-success timestamp, the missing-telemetry alert remained silent and the freshness alert had no timestamp to evaluate.

The rule now requires both metrics for the same job and instance. Six new assertions cover healthy and pending states, firing, recovery, and isolation from a healthy second host. The previous rule fails the new firing assertion; the corrected rule passes all 37 assertions. The live pipeline also verifies firing and recovery when only the timestamp disappears.

Reproduced failure with the previous rule · Corrected rule results · Regression cases

Implementation and validation

The repeatable exercise validates official archive checksums before execution, binds all listeners to loopback, disables Alertmanager cluster gossip, bounds transition waits, and stops its subprocesses when finished. It retains generated configurations, logs, source hashes and webhook payloads. CI and release-evidence workflows run the gate and retain failure diagnostics.

The local Windows run passed all three scenarios and the receiver retry. Both native configuration checks passed, including all nine rules. All 37 promtool rule assertions passed. Three harness tests also passed: bootstrap failure replaces stale success evidence, interruption is recorded and propagated, and a corrupt cached executable archive is rejected before extraction. The documentation checker reported zero errors. Python compilation and both workflow YAML parses passed.

The 46 runtime module source/build/API files recorded by the preceding review are unchanged. Its 96 passing module tests remain historical evidence; they were not rerun for this tooling and documentation change. Docker's local engine was unavailable, so this exercise used native binaries. Linux hosted CI is configured but has not been executed here.

Executed scenarios and tool hashes · Received webhook payloads · Prometheus log · Alertmanager log · Verification and source hashes

Score reassessment

Observability remains 9.0 / 10. This closes a local delivery-validation gap and adds a regression gate. The remaining deployment evidence is still required before awarding the final point. Other categories are carried forward without a new assessment.

The unchanged six-category total is 53 / 6 = 8.8333, rounded to 8.8 / 10. Scores are engineering judgment, not production certification. Machine-readable scores.

1. Instrument and verify the deployed host's queue, connection pool and distributed trace context.
2. Agree deployment SLOs and validate alert thresholds under representative load.
3. Verify protected telemetry access, retention and redaction in the deployed backends.
4. Observe firing and resolved delivery through the intended on-call route with its real routing, authentication and escalation settings.

The receiver is synthetic and local. No external recipient, production service or new database outage was involved.