Escalation & Alert Routing for Excursion Events
A scored excursion is a decision waiting for a human. The rule engine can quantify magnitude and persistence, but it cannot open a deviation investigation, sign a quarantine release, or authorise the destruction of a condemned biologic — those are quality-system acts performed by named, accountable people. Escalation and alert routing is the connective tissue that carries a machine-scored event to the right person, at the right severity, inside a bounded time window, and records the whole journey as evidence. Done badly it produces alert fatigue and unactioned pages; done correctly it is the mechanism by which a temperature deviation becomes a documented, owned, and eventually closed corrective action. The governing anchor is 21 CFR Part 11 §11.10(e): every notification, acknowledgement, reassignment, and timeout in this pipeline is a controlled operation that must survive as a secure, time-stamped, tamper-evident record.
Problem Statement: When a Score Is Not Yet an Action
A duration-weighted score routed to a QUARANTINE state is worthless if the page it generates lands in a muted channel at 03:00 and is never acknowledged. Three failure patterns push cold chain teams to treat escalation as a validated control rather than a convenience integration:
- Alert storms bury the signal. A single compressor failure in a multi-zone room can breach thresholds for forty pallets in ninety seconds. If each breach fans out to an unfiltered page, the on-call pharmacist receives forty notifications for one physical fault and the one genuinely distinct event elsewhere in the facility is lost in the noise. ICH Q10 expects the pharmaceutical quality system to route quality events to management for review; a channel that has been silenced because it cries wolf cannot discharge that expectation.
- Unacknowledged events have no owner. §11.10(e) requires records of operator actions, but an event that is delivered and then ignored generates no action to record. Without an acknowledgement contract — a defined person, a defined deadline, and an automatic escalation when the deadline lapses — an excursion can sit in limbo while a biologic degrades, and the audit trail shows only that the system alerted, not that anyone responded.
- Ad-hoc routing is unprovable. If severity-to-recipient mapping lives in a maintainer’s head or a mutable spreadsheet, an investigator can never reconstruct why a critical excursion went to a junior technician instead of the quality-on-call tier. Under ICH Q10 the escalation logic is part of the change-controlled quality system, and its routing table must be a versioned, evidenced artifact.
Escalation routing sits immediately downstream of the scoring layer inside the broader temperature excursion detection and automated rule engines platform. It consumes the disposition emitted by duration-based scoring for temperature excursions, suppresses the redundant alarms that multi-sensor correlation to reduce false positives has already flagged as physically consistent, and hands a single owned, acknowledged event to CAPA routing automation for temperature excursions in the quality-management layer.
Concept & Specification: The Escalation Record
Escalation is a state machine over a single logical event, not a fire-and-forget message. The unit of work is an escalation record: the immutable identity of the excursion, the severity that governs its routing, the tier it currently sits at, the acknowledgement deadline, and an append-only ledger of every transition. Two design commitments make the record inspection-ready. First, the routing decision is derived deterministically from the score and product criticality, never chosen by hand, so a replay produces the same tier assignment. Second, every state change — raised, notified, acknowledged, escalated, resolved — appends a hash-linked ledger entry rather than mutating a status field, so the history of who was told what and when cannot be quietly rewritten.
The persisted escalation record carries the following fields. The Regulatory anchor column states why each field must exist for the record to survive an inspection.
| Field | Type | Constraint | Regulatory anchor |
|---|---|---|---|
escalation_id |
string (UUID) | Immutable, unique per event | §11.10(e) attributable, traceable record |
excursion_id |
string (UUID) | Foreign key to the scored event | §11.10(e) links notification to its cause |
fingerprint |
string (sha256) | Dedup key: device+zone+window+rule | ICH Q10 suppress duplicate quality events |
severity |
enum | INFO / WARNING / CRITICAL |
ICH Q9 proportional response tier |
tier |
int | 0…N, current on-call level | §11.10(e) records escalation path |
ack_deadline_utc |
string (ISO 8601, UTC) | Contemporaneous, timezone-aware | §11.10(e) time-boxed accountability |
acknowledged_by |
string | null | Authenticated operator identity | §11.10(g) authority check |
state |
enum | RAISED/NOTIFIED/ACK/ESCALATED/RESOLVED |
ICH Q10 event lifecycle |
config_version |
string (sha256) | Fingerprint of routing policy | §11.10(k) change control over routing |
prev_hash |
string (sha256) | Links to prior ledger entry | §11.10(e) tamper-evident chain |
entry_hash |
string (sha256) | SHA-256 of canonical entry | ALCOA+ enduring, tamper-evident |
Deduplication deserves its own note because it is where escalation most often breaks compliance in the wrong direction. Suppressing a duplicate must never suppress a distinct event, so the fingerprint is computed from the physical and logical coordinates of the excursion — device, storage zone, time bucket, and the rule that fired — and is derived, not stored by the alerter. The full derivation and its collision properties are covered in deduplicating excursion alerts with fingerprinting. The mapping from severity to concrete on-call channels and acknowledgement tracking is developed in routing excursion alerts by severity in Python.
Architecture Diagram: The Escalation Ladder
A scored event enters the router, is fingerprinted for suppression, mapped to a severity, and placed on tier 0 of an escalation ladder. Each tier has a bounded acknowledgement window; if the deadline lapses without an authenticated acknowledgement, the event climbs one rung and notifies the next on-call level. An acknowledgement at any rung halts the climb and moves the event toward resolution, which hands a single owned record to the CAPA layer. Every rung transition appends one ledger entry.
Production Python Implementation
The module below is a complete, runnable escalation engine. It fingerprints an incoming disposition to suppress duplicates within a live window, maps severity deterministically from the score and product criticality, drives a per-event state machine with acknowledgement deadlines, and appends every transition to a hash-linked ledger. Notification transport is abstracted behind a channel protocol so the escalation logic — the part an auditor scrutinises — stays free of vendor specifics. Each block cites the clause it satisfies.
from __future__ import annotations
import hashlib
import json
import logging
import uuid
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Optional, Protocol
logger = logging.getLogger("excursion.escalation")
class Severity(str, Enum):
INFO = "INFO"
WARNING = "WARNING"
CRITICAL = "CRITICAL"
class EscalationState(str, Enum):
RAISED = "RAISED"
NOTIFIED = "NOTIFIED"
ACK = "ACK"
ESCALATED = "ESCALATED"
RESOLVED = "RESOLVED"
@dataclass(frozen=True)
class Tier:
name: str
channel: str # abstract channel key resolved by the transport layer
ack_window: timedelta # bounded acknowledgement SLA for this rung
@dataclass(frozen=True)
class RoutingPolicy:
# The escalation policy is a change-controlled artifact under ICH Q10; its
# fingerprint is bound to every ledger entry so a reviewer can prove which
# routing table produced a given escalation path (21 CFR Part 11 §11.10(k)).
tiers: tuple[Tier, ...]
dedup_window: timedelta = timedelta(minutes=10)
# Score boundaries that map a disposition onto a response severity. These are
# deliberately separate from the scoring engine's own state boundaries.
warning_at: float = 26.0
critical_at: float = 76.0
def severity_for(self, score: float, critical_product: bool) -> Severity:
# ICH Q9 proportionality: response tier scales with quantified risk, and a
# narrow-therapeutic-index product is escalated one band harder.
if score >= self.critical_at or (critical_product and score >= self.warning_at):
return Severity.CRITICAL
if score >= self.warning_at:
return Severity.WARNING
return Severity.INFO
def fingerprint(self) -> str:
payload = json.dumps(
{
"tiers": [(t.name, t.channel, t.ack_window.total_seconds()) for t in self.tiers],
"dedup_window_s": self.dedup_window.total_seconds(),
"warning_at": self.warning_at,
"critical_at": self.critical_at,
},
sort_keys=True, separators=(",", ":"),
)
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
class NotificationChannel(Protocol):
def send(self, channel: str, subject: str, body: str) -> str:
"""Deliver a notification and return a transport receipt id."""
def event_fingerprint(device_id: str, zone: str, rule_id: str, at: datetime,
bucket: timedelta) -> str:
# A stable dedup key over the PHYSICAL + LOGICAL coordinates of the excursion.
# Bucketing the timestamp collapses a burst from one fault into one key while
# a different device, zone, or rule yields a distinct key — suppression must
# never drop a genuinely separate quality event (ICH Q10 event handling).
epoch = int(at.timestamp() // bucket.total_seconds())
raw = f"{device_id}|{zone}|{rule_id}|{epoch}"
return hashlib.sha256(raw.encode("utf-8")).hexdigest()
@dataclass
class Escalation:
excursion_id: str
fingerprint: str
severity: Severity
policy: RoutingPolicy
escalation_id: str = field(default_factory=lambda: str(uuid.uuid4()))
tier: int = 0
state: EscalationState = EscalationState.RAISED
ack_deadline_utc: Optional[datetime] = None
acknowledged_by: Optional[str] = None
_prev_hash: str = "0" * 64
ledger: list[dict] = field(default_factory=list)
def _append(self, transition: str, now: datetime) -> None:
# §11.10(e): every state change is a secure, time-stamped entry that does
# not obscure prior entries. Each record commits to its predecessor's
# digest, so a retrospective edit breaks the chain (tamper evidence).
entry = {
"escalation_id": self.escalation_id,
"excursion_id": self.excursion_id,
"fingerprint": self.fingerprint,
"transition": transition,
"severity": self.severity.value,
"tier": self.tier,
"state": self.state.value,
"ack_deadline_utc": self.ack_deadline_utc.isoformat() if self.ack_deadline_utc else None,
"acknowledged_by": self.acknowledged_by,
"config_version": self.policy.fingerprint(),
"recorded_at": now.isoformat(), # contemporaneous UTC stamp §11.10(e)
"prev_hash": self._prev_hash,
}
canonical = json.dumps(entry, sort_keys=True, separators=(",", ":"))
entry["entry_hash"] = hashlib.sha256(canonical.encode("utf-8")).hexdigest()
self._prev_hash = entry["entry_hash"]
self.ledger.append(entry)
def notify(self, channel: NotificationChannel, now: datetime) -> None:
tier = self.policy.tiers[self.tier]
self.ack_deadline_utc = now + tier.ack_window
receipt = channel.send(
tier.channel,
subject=f"[{self.severity.value}] excursion {self.excursion_id}",
body=f"Acknowledge by {self.ack_deadline_utc.isoformat()} (tier {tier.name}).",
)
self.state = EscalationState.NOTIFIED
logger.info("notified %s tier=%s receipt=%s", self.escalation_id, tier.name, receipt)
self._append("notified", now)
def acknowledge(self, operator_id: str, now: datetime) -> None:
# §11.10(g) authority check: acknowledgement must carry an authenticated
# operator identity; an anonymous ack is not an accountable action.
if not operator_id:
raise ValueError("acknowledgement requires an authenticated operator id")
if self.state == EscalationState.RESOLVED:
raise ValueError("cannot acknowledge a resolved escalation")
self.acknowledged_by = operator_id
self.state = EscalationState.ACK
self._append("acknowledged", now)
def escalate_if_overdue(self, channel: NotificationChannel, now: datetime) -> bool:
# Time-boxed accountability: an unacknowledged deadline climbs one rung.
if self.state in (EscalationState.ACK, EscalationState.RESOLVED):
return False
if self.ack_deadline_utc is None or now < self.ack_deadline_utc:
return False
if self.tier + 1 >= len(self.policy.tiers):
# Top rung reached: hold at max tier, re-notify, do not silently drop.
self.state = EscalationState.ESCALATED
self._append("escalated_top", now)
self.notify(channel, now)
return True
self.tier += 1
self.state = EscalationState.ESCALATED
self._append("escalated", now)
self.notify(channel, now)
return True
def resolve(self, now: datetime) -> dict:
# An event is only resolvable once owned — proves an action was recorded,
# not merely that the system alerted (ICH Q10 / §11.10(e)).
if self.acknowledged_by is None:
raise ValueError("cannot resolve an unacknowledged escalation")
self.state = EscalationState.RESOLVED
self._append("resolved", now)
return self.ledger[-1]
class EscalationRouter:
def __init__(self, policy: RoutingPolicy, channel: NotificationChannel):
self.policy = policy
self.channel = channel
self._active: dict[str, Escalation] = {} # fingerprint -> escalation
def raise_event(self, excursion_id: str, device_id: str, zone: str, rule_id: str,
score: float, critical_product: bool, now: datetime) -> Optional[Escalation]:
fp = event_fingerprint(device_id, zone, rule_id, now, self.policy.dedup_window)
existing = self._active.get(fp)
if existing is not None and existing.state != EscalationState.RESOLVED:
# Duplicate within the live window: attach to the open event, do not
# raise a second page (suppresses alert storms without losing signal).
existing._append("deduplicated", now)
logger.info("suppressed duplicate for fingerprint=%s", fp)
return None
severity = self.policy.severity_for(score, critical_product)
esc = Escalation(excursion_id, fp, severity, self.policy)
esc._append("raised", now)
esc.notify(self.channel, now)
self._active[fp] = esc
return esc
def sweep(self, now: datetime) -> None:
# Called on a fixed cadence to enforce acknowledgement SLAs.
for esc in list(self._active.values()):
esc.escalate_if_overdue(self.channel, now)
The router keeps a single open escalation per fingerprint, so a compressor fault that breaches forty pallets in one zone raises one page and forty suppressed-duplicate ledger entries rather than forty independent escalations. The suppression itself is recorded — an inspector can see that duplicates were collapsed deliberately, not lost. When the event resolves, the router hands the terminal ledger entry to the quality system, where mapping excursion severity to CAPA priority in Python translates the severity into a corrective-action priority and auto-generating CAPA records from excursion events opens the formal record.
Configuration & Deployment
Routing behaviour is driven entirely by the RoutingPolicy, which keeps the escalation logic version-controlled and re-validatable. Treat the policy as a controlled document under your ICH Q10 pharmaceutical quality system: any change to tiers, acknowledgement windows, dedup window, or severity boundaries is a change-management event that may require re-validation, because it alters who is accountable for a given class of excursion.
| Variable | Example | Purpose | Regulatory anchor |
|---|---|---|---|
ESC_TIER_ACK_WINDOWS |
900,600,300 |
Per-rung acknowledgement SLA (seconds) | §11.10(e) time-boxed accountability |
ESC_DEDUP_WINDOW_S |
600 |
Suppression window for one fingerprint | ICH Q10 duplicate-event handling |
ESC_WARNING_AT / ESC_CRITICAL_AT |
26 / 76 |
Score-to-severity boundaries | ICH Q9 proportional response |
ESC_SWEEP_INTERVAL_S |
30 |
Cadence of the overdue sweep | §11.10(e) reliable escalation timing |
ESC_LEDGER_DSN |
append-only store DSN | Hash-chained record sink | §11.10(e) secure audit trail |
ESC_CHANNEL_SECRET |
secrets-manager ref | Transport credential, never inline | §11.10(g) authorised transmission |
Run the sweep on a dedicated scheduler rather than piggybacking on telemetry ingestion, so a lull in readings can never stall escalation timing — an overdue acknowledgement must fire on wall-clock, not on the next payload. Keep notification transport idempotent by threading a receipt id through the channel, so a retried send after a broker blip does not double-page a technician. Load channel credentials from a secrets manager and fail closed if a credential is missing: an escalation engine that cannot authenticate to its paging provider must raise a monitoring alarm, never silently swallow a CRITICAL.
Verification & Testing
An escalation engine that governs who responds to a product-integrity event is GxP-relevant software, so its tests are validation evidence, not developer convenience. Build the suite around the transitions an inspector will probe:
- Deterministic severity tests. Assert that a given score and product-criticality flag always map to the same
Severity, including the exact boundary values (26 and 76), so routing under ICH Q9 is reproducible. - Deduplication tests. Raise two events with identical device, zone, rule, and timestamp bucket and assert the second is suppressed with a
deduplicatedledger entry; then vary one coordinate and assert a distinct escalation is raised. This proves suppression never drops a separate event. - Escalation-timeout tests. Advance a controlled clock past an acknowledgement deadline and assert the event climbs exactly one rung and re-notifies, demonstrating time-boxed accountability under §11.10(e).
- Authority tests. Assert that
acknowledge("")raises rather than recording an anonymous acknowledgement, satisfying §11.10(g). - Ledger-integrity tests. Mutate one stored entry and assert the recomputed
entry_hashno longer matches the successor’sprev_hash, demonstrating tamper-evidence for §11.10(e). - CSV protocol hooks. Expose a fixture loader that replays an OQ scenario CSV (event, injected clock, expected tier, expected recipient) so Operational Qualification can be executed and signed against documented expected escalations.
from datetime import datetime, timedelta, timezone
class _RecordingChannel:
def __init__(self):
self.sent = []
def send(self, channel, subject, body):
self.sent.append((channel, subject))
return f"receipt-{len(self.sent)}"
def _policy():
return RoutingPolicy(tiers=(
Tier("Technician", "chat", timedelta(minutes=15)),
Tier("Supervisor", "sms", timedelta(minutes=10)),
Tier("Quality", "phone", timedelta(minutes=5)),
))
def test_timeout_climbs_one_rung():
ch = _RecordingChannel()
router = EscalationRouter(_policy(), ch)
t0 = datetime(2026, 7, 14, 2, 0, tzinfo=timezone.utc)
esc = router.raise_event("EXC-9", "DEV-1", "ZONE-A", "R-HI", 82.0, True, t0)
# Past the 15-minute tier-0 SLA the event must climb to tier 1 (§11.10(e)).
esc.escalate_if_overdue(ch, t0 + timedelta(minutes=16))
assert esc.tier == 1
assert esc.severity is Severity.CRITICAL
def test_duplicate_is_suppressed_not_dropped():
ch = _RecordingChannel()
router = EscalationRouter(_policy(), ch)
t0 = datetime(2026, 7, 14, 2, 0, tzinfo=timezone.utc)
first = router.raise_event("EXC-1", "DEV-1", "ZONE-A", "R-HI", 40.0, False, t0)
second = router.raise_event("EXC-2", "DEV-1", "ZONE-A", "R-HI", 40.0, False, t0)
assert first is not None and second is None # storm collapsed to one page
assert first.ledger[-1]["transition"] == "deduplicated"
Known Failure Modes & Mitigations
| Failure mode | Symptom | Mitigation | Regulatory anchor |
|---|---|---|---|
| Over-broad fingerprint | Distinct events silently merged | Include rule_id and zone in the key; test with varied coordinates | ICH Q10 complete event handling |
| Sweep stalled on ingestion lull | Deadlines never fire | Run the overdue sweep on a wall-clock scheduler, not per-payload | §11.10(e) reliable escalation |
| Paging provider outage | CRITICAL never delivered |
Fail closed, raise a meta-alarm, retain the unacked event | §11.10(e) record protection |
| Anonymous acknowledgement | Action not attributable | Require authenticated operator id on acknowledge |
§11.10(g) authority check |
| Policy drift across nodes | Divergent routing for same score | Fingerprint the policy into every ledger entry | §11.10(k) change control |
| Double-page on retry | Duplicate notifications, alarm fatigue | Thread an idempotent transport receipt through the channel | ICH Q9 proportional response |
When the paging provider is unreachable, hold the escalation at its current rung, re-notify on the next sweep, and raise a separate infrastructure alarm rather than marking the event resolved; a delivery gap with a documented meta-alarm is defensible, a CRITICAL quietly abandoned is not.
Compliance Q&A
Does suppressing duplicate excursion alerts risk hiding a real event from the audit trail?
Only if suppression discards the duplicate silently. The compliant pattern keeps one open escalation per fingerprint and appends a deduplicated entry to the ledger for each suppressed copy, so an inspector can see that duplicates were collapsed against a live event rather than lost. The fingerprint is derived from device, zone, rule, and time bucket, so a genuinely distinct event — a different zone or a different rule — yields a new key and always raises its own escalation.
Is an unacknowledged escalation a 21 CFR Part 11 §11.10(e) audit finding?
An escalation that is delivered and then ignored, with nothing recorded, is a finding because §11.10(e) expects records of operator actions and there is no action to record. The engine avoids this by attaching an acknowledgement deadline to every notification and climbing to the next on-call tier when the deadline lapses, so the ledger always shows either a timely acknowledgement or a documented escalation path, never an event that simply evaporated.
Where does escalation routing sit relative to CAPA under ICH Q10?
Escalation routing is the accountability layer that precedes CAPA. It ensures a scored excursion is owned by a named, authenticated person within a bounded time window, then hands that single owned record to the corrective-and-preventive-action process. ICH Q10 expects quality events to reach management review; the escalation ladder is the mechanism that guarantees the event climbs to the appropriate quality tier before a CAPA record is opened.
Related
- Duration-Based Scoring for Temperature Excursions — produces the disposition this router escalates.
- Deduplicating Excursion Alerts with Fingerprinting — the stable event key that suppresses alert storms.
- Routing Excursion Alerts by Severity in Python — maps severity to concrete channels and acknowledgement tracking.
- Multi-Sensor Correlation to Reduce False Positives — filters spurious spikes before they escalate.
- CAPA Routing Automation for Temperature Excursions — receives the owned record this pipeline resolves.
For architectural context, this page sits under Temperature Excursion Detection & Automated Rule Engines.