Escalation & Alert Routing for Excursion Events

A scored excursion is a decision waiting for a human. The rule engine can quantify magnitude and persistence, but it cannot open a deviation investigation, sign a quarantine release, or authorise the destruction of a condemned biologic — those are quality-system acts performed by named, accountable people. Escalation and alert routing is the connective tissue that carries a machine-scored event to the right person, at the right severity, inside a bounded time window, and records the whole journey as evidence. Done badly it produces alert fatigue and unactioned pages; done correctly it is the mechanism by which a temperature deviation becomes a documented, owned, and eventually closed corrective action. The governing anchor is 21 CFR Part 11 §11.10(e): every notification, acknowledgement, reassignment, and timeout in this pipeline is a controlled operation that must survive as a secure, time-stamped, tamper-evident record.

Problem Statement: When a Score Is Not Yet an Action

A duration-weighted score routed to a QUARANTINE state is worthless if the page it generates lands in a muted channel at 03:00 and is never acknowledged. Three failure patterns push cold chain teams to treat escalation as a validated control rather than a convenience integration:

  • Alert storms bury the signal. A single compressor failure in a multi-zone room can breach thresholds for forty pallets in ninety seconds. If each breach fans out to an unfiltered page, the on-call pharmacist receives forty notifications for one physical fault and the one genuinely distinct event elsewhere in the facility is lost in the noise. ICH Q10 expects the pharmaceutical quality system to route quality events to management for review; a channel that has been silenced because it cries wolf cannot discharge that expectation.
  • Unacknowledged events have no owner. §11.10(e) requires records of operator actions, but an event that is delivered and then ignored generates no action to record. Without an acknowledgement contract — a defined person, a defined deadline, and an automatic escalation when the deadline lapses — an excursion can sit in limbo while a biologic degrades, and the audit trail shows only that the system alerted, not that anyone responded.
  • Ad-hoc routing is unprovable. If severity-to-recipient mapping lives in a maintainer’s head or a mutable spreadsheet, an investigator can never reconstruct why a critical excursion went to a junior technician instead of the quality-on-call tier. Under ICH Q10 the escalation logic is part of the change-controlled quality system, and its routing table must be a versioned, evidenced artifact.

Escalation routing sits immediately downstream of the scoring layer inside the broader temperature excursion detection and automated rule engines platform. It consumes the disposition emitted by duration-based scoring for temperature excursions, suppresses the redundant alarms that multi-sensor correlation to reduce false positives has already flagged as physically consistent, and hands a single owned, acknowledged event to CAPA routing automation for temperature excursions in the quality-management layer.

Concept & Specification: The Escalation Record

Escalation is a state machine over a single logical event, not a fire-and-forget message. The unit of work is an escalation record: the immutable identity of the excursion, the severity that governs its routing, the tier it currently sits at, the acknowledgement deadline, and an append-only ledger of every transition. Two design commitments make the record inspection-ready. First, the routing decision is derived deterministically from the score and product criticality, never chosen by hand, so a replay produces the same tier assignment. Second, every state change — raised, notified, acknowledged, escalated, resolved — appends a hash-linked ledger entry rather than mutating a status field, so the history of who was told what and when cannot be quietly rewritten.

The persisted escalation record carries the following fields. The Regulatory anchor column states why each field must exist for the record to survive an inspection.

Field Type Constraint Regulatory anchor
escalation_id string (UUID) Immutable, unique per event §11.10(e) attributable, traceable record
excursion_id string (UUID) Foreign key to the scored event §11.10(e) links notification to its cause
fingerprint string (sha256) Dedup key: device+zone+window+rule ICH Q10 suppress duplicate quality events
severity enum INFO / WARNING / CRITICAL ICH Q9 proportional response tier
tier int 0…N, current on-call level §11.10(e) records escalation path
ack_deadline_utc string (ISO 8601, UTC) Contemporaneous, timezone-aware §11.10(e) time-boxed accountability
acknowledged_by string | null Authenticated operator identity §11.10(g) authority check
state enum RAISED/NOTIFIED/ACK/ESCALATED/RESOLVED ICH Q10 event lifecycle
config_version string (sha256) Fingerprint of routing policy §11.10(k) change control over routing
prev_hash string (sha256) Links to prior ledger entry §11.10(e) tamper-evident chain
entry_hash string (sha256) SHA-256 of canonical entry ALCOA+ enduring, tamper-evident

Deduplication deserves its own note because it is where escalation most often breaks compliance in the wrong direction. Suppressing a duplicate must never suppress a distinct event, so the fingerprint is computed from the physical and logical coordinates of the excursion — device, storage zone, time bucket, and the rule that fired — and is derived, not stored by the alerter. The full derivation and its collision properties are covered in deduplicating excursion alerts with fingerprinting. The mapping from severity to concrete on-call channels and acknowledgement tracking is developed in routing excursion alerts by severity in Python.

Architecture Diagram: The Escalation Ladder

A scored event enters the router, is fingerprinted for suppression, mapped to a severity, and placed on tier 0 of an escalation ladder. Each tier has a bounded acknowledgement window; if the deadline lapses without an authenticated acknowledgement, the event climbs one rung and notifies the next on-call level. An acknowledgement at any rung halts the climb and moves the event toward resolution, which hands a single owned record to the CAPA layer. Every rung transition appends one ledger entry.

Severity-routed escalation ladder with acknowledgement SLAs and an audit hash chain A scored excursion is fingerprinted, deduplicated, and severity-mapped, then climbs a four-tier on-call ladder on acknowledgement-deadline lapse; an acknowledgement halts the climb and resolves the event into the CAPA layer, and every transition appends a tamper-evident SHA-256 ledger entry. INTAKE & ROUTING Scored disposition score · state · SKU Fingerprint · dedup device·zone·window·rule Severity map INFO · WARNING · CRITICAL Enter ladder tier 0 · start SLA ESCALATION LADDER · ACK-DEADLINE LAPSE CLIMBS ONE RUNG timeout timeout timeout Tier 0 · Technician chat · 15 min ack shift on duty Tier 1 · Supervisor SMS · 10 min ack on-call rota Tier 2 · Quality phone · 5 min ack QA on-call Tier 3 · QA mgmt phone · page-all site management authenticated ack halts climb RESOLVED · owned acknowledged_by set CAPA routing single owned record APPEND-ONLY LEDGER · 21 CFR Part 11 §11.10(e) prev_hash prev_hash raised → notified tier · severity · deadline entry_hash = SHA-256 escalated → acked acknowledged_by · UTC entry_hash = SHA-256 resolved → CAPA config_version · UTC entry_hash = SHA-256

Production Python Implementation

The module below is a complete, runnable escalation engine. It fingerprints an incoming disposition to suppress duplicates within a live window, maps severity deterministically from the score and product criticality, drives a per-event state machine with acknowledgement deadlines, and appends every transition to a hash-linked ledger. Notification transport is abstracted behind a channel protocol so the escalation logic — the part an auditor scrutinises — stays free of vendor specifics. Each block cites the clause it satisfies.

python
from __future__ import annotations

import hashlib
import json
import logging
import uuid
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Optional, Protocol

logger = logging.getLogger("excursion.escalation")


class Severity(str, Enum):
    INFO = "INFO"
    WARNING = "WARNING"
    CRITICAL = "CRITICAL"


class EscalationState(str, Enum):
    RAISED = "RAISED"
    NOTIFIED = "NOTIFIED"
    ACK = "ACK"
    ESCALATED = "ESCALATED"
    RESOLVED = "RESOLVED"


@dataclass(frozen=True)
class Tier:
    name: str
    channel: str          # abstract channel key resolved by the transport layer
    ack_window: timedelta  # bounded acknowledgement SLA for this rung


@dataclass(frozen=True)
class RoutingPolicy:
    # The escalation policy is a change-controlled artifact under ICH Q10; its
    # fingerprint is bound to every ledger entry so a reviewer can prove which
    # routing table produced a given escalation path (21 CFR Part 11 §11.10(k)).
    tiers: tuple[Tier, ...]
    dedup_window: timedelta = timedelta(minutes=10)
    # Score boundaries that map a disposition onto a response severity. These are
    # deliberately separate from the scoring engine's own state boundaries.
    warning_at: float = 26.0
    critical_at: float = 76.0

    def severity_for(self, score: float, critical_product: bool) -> Severity:
        # ICH Q9 proportionality: response tier scales with quantified risk, and a
        # narrow-therapeutic-index product is escalated one band harder.
        if score >= self.critical_at or (critical_product and score >= self.warning_at):
            return Severity.CRITICAL
        if score >= self.warning_at:
            return Severity.WARNING
        return Severity.INFO

    def fingerprint(self) -> str:
        payload = json.dumps(
            {
                "tiers": [(t.name, t.channel, t.ack_window.total_seconds()) for t in self.tiers],
                "dedup_window_s": self.dedup_window.total_seconds(),
                "warning_at": self.warning_at,
                "critical_at": self.critical_at,
            },
            sort_keys=True, separators=(",", ":"),
        )
        return hashlib.sha256(payload.encode("utf-8")).hexdigest()


class NotificationChannel(Protocol):
    def send(self, channel: str, subject: str, body: str) -> str:
        """Deliver a notification and return a transport receipt id."""


def event_fingerprint(device_id: str, zone: str, rule_id: str, at: datetime,
                      bucket: timedelta) -> str:
    # A stable dedup key over the PHYSICAL + LOGICAL coordinates of the excursion.
    # Bucketing the timestamp collapses a burst from one fault into one key while
    # a different device, zone, or rule yields a distinct key — suppression must
    # never drop a genuinely separate quality event (ICH Q10 event handling).
    epoch = int(at.timestamp() // bucket.total_seconds())
    raw = f"{device_id}|{zone}|{rule_id}|{epoch}"
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()


@dataclass
class Escalation:
    excursion_id: str
    fingerprint: str
    severity: Severity
    policy: RoutingPolicy
    escalation_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    tier: int = 0
    state: EscalationState = EscalationState.RAISED
    ack_deadline_utc: Optional[datetime] = None
    acknowledged_by: Optional[str] = None
    _prev_hash: str = "0" * 64
    ledger: list[dict] = field(default_factory=list)

    def _append(self, transition: str, now: datetime) -> None:
        # §11.10(e): every state change is a secure, time-stamped entry that does
        # not obscure prior entries. Each record commits to its predecessor's
        # digest, so a retrospective edit breaks the chain (tamper evidence).
        entry = {
            "escalation_id": self.escalation_id,
            "excursion_id": self.excursion_id,
            "fingerprint": self.fingerprint,
            "transition": transition,
            "severity": self.severity.value,
            "tier": self.tier,
            "state": self.state.value,
            "ack_deadline_utc": self.ack_deadline_utc.isoformat() if self.ack_deadline_utc else None,
            "acknowledged_by": self.acknowledged_by,
            "config_version": self.policy.fingerprint(),
            "recorded_at": now.isoformat(),  # contemporaneous UTC stamp §11.10(e)
            "prev_hash": self._prev_hash,
        }
        canonical = json.dumps(entry, sort_keys=True, separators=(",", ":"))
        entry["entry_hash"] = hashlib.sha256(canonical.encode("utf-8")).hexdigest()
        self._prev_hash = entry["entry_hash"]
        self.ledger.append(entry)

    def notify(self, channel: NotificationChannel, now: datetime) -> None:
        tier = self.policy.tiers[self.tier]
        self.ack_deadline_utc = now + tier.ack_window
        receipt = channel.send(
            tier.channel,
            subject=f"[{self.severity.value}] excursion {self.excursion_id}",
            body=f"Acknowledge by {self.ack_deadline_utc.isoformat()} (tier {tier.name}).",
        )
        self.state = EscalationState.NOTIFIED
        logger.info("notified %s tier=%s receipt=%s", self.escalation_id, tier.name, receipt)
        self._append("notified", now)

    def acknowledge(self, operator_id: str, now: datetime) -> None:
        # §11.10(g) authority check: acknowledgement must carry an authenticated
        # operator identity; an anonymous ack is not an accountable action.
        if not operator_id:
            raise ValueError("acknowledgement requires an authenticated operator id")
        if self.state == EscalationState.RESOLVED:
            raise ValueError("cannot acknowledge a resolved escalation")
        self.acknowledged_by = operator_id
        self.state = EscalationState.ACK
        self._append("acknowledged", now)

    def escalate_if_overdue(self, channel: NotificationChannel, now: datetime) -> bool:
        # Time-boxed accountability: an unacknowledged deadline climbs one rung.
        if self.state in (EscalationState.ACK, EscalationState.RESOLVED):
            return False
        if self.ack_deadline_utc is None or now < self.ack_deadline_utc:
            return False
        if self.tier + 1 >= len(self.policy.tiers):
            # Top rung reached: hold at max tier, re-notify, do not silently drop.
            self.state = EscalationState.ESCALATED
            self._append("escalated_top", now)
            self.notify(channel, now)
            return True
        self.tier += 1
        self.state = EscalationState.ESCALATED
        self._append("escalated", now)
        self.notify(channel, now)
        return True

    def resolve(self, now: datetime) -> dict:
        # An event is only resolvable once owned — proves an action was recorded,
        # not merely that the system alerted (ICH Q10 / §11.10(e)).
        if self.acknowledged_by is None:
            raise ValueError("cannot resolve an unacknowledged escalation")
        self.state = EscalationState.RESOLVED
        self._append("resolved", now)
        return self.ledger[-1]


class EscalationRouter:
    def __init__(self, policy: RoutingPolicy, channel: NotificationChannel):
        self.policy = policy
        self.channel = channel
        self._active: dict[str, Escalation] = {}  # fingerprint -> escalation

    def raise_event(self, excursion_id: str, device_id: str, zone: str, rule_id: str,
                    score: float, critical_product: bool, now: datetime) -> Optional[Escalation]:
        fp = event_fingerprint(device_id, zone, rule_id, now, self.policy.dedup_window)
        existing = self._active.get(fp)
        if existing is not None and existing.state != EscalationState.RESOLVED:
            # Duplicate within the live window: attach to the open event, do not
            # raise a second page (suppresses alert storms without losing signal).
            existing._append("deduplicated", now)
            logger.info("suppressed duplicate for fingerprint=%s", fp)
            return None
        severity = self.policy.severity_for(score, critical_product)
        esc = Escalation(excursion_id, fp, severity, self.policy)
        esc._append("raised", now)
        esc.notify(self.channel, now)
        self._active[fp] = esc
        return esc

    def sweep(self, now: datetime) -> None:
        # Called on a fixed cadence to enforce acknowledgement SLAs.
        for esc in list(self._active.values()):
            esc.escalate_if_overdue(self.channel, now)

The router keeps a single open escalation per fingerprint, so a compressor fault that breaches forty pallets in one zone raises one page and forty suppressed-duplicate ledger entries rather than forty independent escalations. The suppression itself is recorded — an inspector can see that duplicates were collapsed deliberately, not lost. When the event resolves, the router hands the terminal ledger entry to the quality system, where mapping excursion severity to CAPA priority in Python translates the severity into a corrective-action priority and auto-generating CAPA records from excursion events opens the formal record.

Configuration & Deployment

Routing behaviour is driven entirely by the RoutingPolicy, which keeps the escalation logic version-controlled and re-validatable. Treat the policy as a controlled document under your ICH Q10 pharmaceutical quality system: any change to tiers, acknowledgement windows, dedup window, or severity boundaries is a change-management event that may require re-validation, because it alters who is accountable for a given class of excursion.

Variable Example Purpose Regulatory anchor
ESC_TIER_ACK_WINDOWS 900,600,300 Per-rung acknowledgement SLA (seconds) §11.10(e) time-boxed accountability
ESC_DEDUP_WINDOW_S 600 Suppression window for one fingerprint ICH Q10 duplicate-event handling
ESC_WARNING_AT / ESC_CRITICAL_AT 26 / 76 Score-to-severity boundaries ICH Q9 proportional response
ESC_SWEEP_INTERVAL_S 30 Cadence of the overdue sweep §11.10(e) reliable escalation timing
ESC_LEDGER_DSN append-only store DSN Hash-chained record sink §11.10(e) secure audit trail
ESC_CHANNEL_SECRET secrets-manager ref Transport credential, never inline §11.10(g) authorised transmission

Run the sweep on a dedicated scheduler rather than piggybacking on telemetry ingestion, so a lull in readings can never stall escalation timing — an overdue acknowledgement must fire on wall-clock, not on the next payload. Keep notification transport idempotent by threading a receipt id through the channel, so a retried send after a broker blip does not double-page a technician. Load channel credentials from a secrets manager and fail closed if a credential is missing: an escalation engine that cannot authenticate to its paging provider must raise a monitoring alarm, never silently swallow a CRITICAL.

Verification & Testing

An escalation engine that governs who responds to a product-integrity event is GxP-relevant software, so its tests are validation evidence, not developer convenience. Build the suite around the transitions an inspector will probe:

  • Deterministic severity tests. Assert that a given score and product-criticality flag always map to the same Severity, including the exact boundary values (26 and 76), so routing under ICH Q9 is reproducible.
  • Deduplication tests. Raise two events with identical device, zone, rule, and timestamp bucket and assert the second is suppressed with a deduplicated ledger entry; then vary one coordinate and assert a distinct escalation is raised. This proves suppression never drops a separate event.
  • Escalation-timeout tests. Advance a controlled clock past an acknowledgement deadline and assert the event climbs exactly one rung and re-notifies, demonstrating time-boxed accountability under §11.10(e).
  • Authority tests. Assert that acknowledge("") raises rather than recording an anonymous acknowledgement, satisfying §11.10(g).
  • Ledger-integrity tests. Mutate one stored entry and assert the recomputed entry_hash no longer matches the successor’s prev_hash, demonstrating tamper-evidence for §11.10(e).
  • CSV protocol hooks. Expose a fixture loader that replays an OQ scenario CSV (event, injected clock, expected tier, expected recipient) so Operational Qualification can be executed and signed against documented expected escalations.
python
from datetime import datetime, timedelta, timezone


class _RecordingChannel:
    def __init__(self):
        self.sent = []

    def send(self, channel, subject, body):
        self.sent.append((channel, subject))
        return f"receipt-{len(self.sent)}"


def _policy():
    return RoutingPolicy(tiers=(
        Tier("Technician", "chat", timedelta(minutes=15)),
        Tier("Supervisor", "sms", timedelta(minutes=10)),
        Tier("Quality", "phone", timedelta(minutes=5)),
    ))


def test_timeout_climbs_one_rung():
    ch = _RecordingChannel()
    router = EscalationRouter(_policy(), ch)
    t0 = datetime(2026, 7, 14, 2, 0, tzinfo=timezone.utc)
    esc = router.raise_event("EXC-9", "DEV-1", "ZONE-A", "R-HI", 82.0, True, t0)
    # Past the 15-minute tier-0 SLA the event must climb to tier 1 (§11.10(e)).
    esc.escalate_if_overdue(ch, t0 + timedelta(minutes=16))
    assert esc.tier == 1
    assert esc.severity is Severity.CRITICAL


def test_duplicate_is_suppressed_not_dropped():
    ch = _RecordingChannel()
    router = EscalationRouter(_policy(), ch)
    t0 = datetime(2026, 7, 14, 2, 0, tzinfo=timezone.utc)
    first = router.raise_event("EXC-1", "DEV-1", "ZONE-A", "R-HI", 40.0, False, t0)
    second = router.raise_event("EXC-2", "DEV-1", "ZONE-A", "R-HI", 40.0, False, t0)
    assert first is not None and second is None       # storm collapsed to one page
    assert first.ledger[-1]["transition"] == "deduplicated"

Known Failure Modes & Mitigations

Failure mode Symptom Mitigation Regulatory anchor
Over-broad fingerprint Distinct events silently merged Include rule_id and zone in the key; test with varied coordinates ICH Q10 complete event handling
Sweep stalled on ingestion lull Deadlines never fire Run the overdue sweep on a wall-clock scheduler, not per-payload §11.10(e) reliable escalation
Paging provider outage CRITICAL never delivered Fail closed, raise a meta-alarm, retain the unacked event §11.10(e) record protection
Anonymous acknowledgement Action not attributable Require authenticated operator id on acknowledge §11.10(g) authority check
Policy drift across nodes Divergent routing for same score Fingerprint the policy into every ledger entry §11.10(k) change control
Double-page on retry Duplicate notifications, alarm fatigue Thread an idempotent transport receipt through the channel ICH Q9 proportional response

When the paging provider is unreachable, hold the escalation at its current rung, re-notify on the next sweep, and raise a separate infrastructure alarm rather than marking the event resolved; a delivery gap with a documented meta-alarm is defensible, a CRITICAL quietly abandoned is not.

Compliance Q&A

Does suppressing duplicate excursion alerts risk hiding a real event from the audit trail?

Only if suppression discards the duplicate silently. The compliant pattern keeps one open escalation per fingerprint and appends a deduplicated entry to the ledger for each suppressed copy, so an inspector can see that duplicates were collapsed against a live event rather than lost. The fingerprint is derived from device, zone, rule, and time bucket, so a genuinely distinct event — a different zone or a different rule — yields a new key and always raises its own escalation.

Is an unacknowledged escalation a 21 CFR Part 11 §11.10(e) audit finding?

An escalation that is delivered and then ignored, with nothing recorded, is a finding because §11.10(e) expects records of operator actions and there is no action to record. The engine avoids this by attaching an acknowledgement deadline to every notification and climbing to the next on-call tier when the deadline lapses, so the ledger always shows either a timely acknowledgement or a documented escalation path, never an event that simply evaporated.

Where does escalation routing sit relative to CAPA under ICH Q10?

Escalation routing is the accountability layer that precedes CAPA. It ensures a scored excursion is owned by a named, authenticated person within a bounded time window, then hands that single owned record to the corrective-and-preventive-action process. ICH Q10 expects quality events to reach management review; the escalation ladder is the mechanism that guarantees the event climbs to the appropriate quality tier before a CAPA record is opened.

For architectural context, this page sits under Temperature Excursion Detection & Automated Rule Engines.