Versioning Telemetry Schemas Without Breaking Audit Trails

Firmware moves. A new sensor generation adds a battery-voltage field, a probe vendor renames a unit code, a compliance change requires a calibration certificate reference in every reading. Each of these changes the shape of the telemetry payload, and a validation gate that only knows the old shape will reject the new fleet — or, worse, silently coerce it — the moment a rollout begins. This guide shows how to evolve payload schemas so that a firmware rollout is a routine, reversible event rather than a data-integrity incident: additive and explicitly versioned changes, a strict validator per version, and a quarantine queue that isolates unknown versions instead of poisoning the record. It extends the schema validation pipelines for temperature telemetry gate with the versioning discipline that keeps that gate stable across change.

Regulatory hook

Two clauses govern schema evolution. 21 CFR Part 11 §11.10(a) requires validation for consistent intended performance, which a validator cannot demonstrate if it silently accepts a payload shape it was never validated against. §11.10(e) requires that the original entry be preserved in a secure, time-stamped audit trail and that previously recorded information not be obscured — which forbids retrofitting old records to a new shape or dropping fields the old schema captured. Correct versioning satisfies both: every record is validated against the exact schema its firmware emitted, and no historical record is ever rewritten to match a newer contract.

The trap most teams fall into is treating a schema as mutable infrastructure — editing the one validation model in place when a field changes and redeploying. That single edit rewrites the meaning of every record already in the store, because the schema is the contract those records were captured under. An inspector who re-validates a six-month-old reading against the current model and finds it now fails, or now passes with a different interpretation, has grounds to question the integrity of the entire monitored period. Versioning converts an in-place mutation into an additive, auditable event, which is the only way to keep a growing fleet’s payloads defensible while the firmware underneath them changes.

Prerequisites

Python 3.11 or newer, for the discriminated-union validation shown below.
Library: pip install "pydantic>=2.6,<3". Pydantic v2’s discriminated unions route each payload to the correct versioned model.
A schema registry: a version-keyed store of validated schema definitions, each bound to the firmware versions that emit it. A directory of versioned JSON Schema files under change control is sufficient to start.
A quarantine sink: a durable, append-only store separate from the regulated time-series database, for payloads whose version is unknown or fails validation.
Change control: every schema version is a controlled document; introducing a version is a change-management event under your ICH Q10 quality system.

Step-by-Step Implementation

Step 1 — Make the version explicit and mandatory

Every payload must carry a schema_version field, and the validator must reject any payload that omits it. An implicit or inferred version is unvalidatable, because two firmware generations could produce the same field set with different semantics.

python

from typing import Literal, Optional, Union
from pydantic import BaseModel, Field, ValidationError


class ReadingV1(BaseModel):
    # §11.10(a): the version is a required discriminator, never inferred.
    schema_version: Literal["1.0"]
    device_id: str = Field(min_length=8, max_length=32)
    sequence_id: int = Field(ge=0)
    captured_at: str
    temperature_c: float = Field(ge=-90.0, le=60.0)

Step 2 — Evolve additively, never destructively

A backward-compatible change adds optional fields; it never removes a field, never repurposes a field’s meaning, and never tightens a constraint that would reject previously valid data. The new field defaults such that a record which predates it remains valid, so the audit trail of old readings is untouched — the §11.10(e) requirement that previously recorded information is not obscured.

python

class ReadingV2(BaseModel):
    # Additive evolution: battery_v and calibration_ref are OPTIONAL, so the
    # change cannot invalidate any V1 record (§11.10(e) — original preserved).
    schema_version: Literal["2.0"]
    device_id: str = Field(min_length=8, max_length=32)
    sequence_id: int = Field(ge=0)
    captured_at: str
    temperature_c: float = Field(ge=-90.0, le=60.0)
    battery_v: Optional[float] = Field(default=None, ge=0.0, le=5.0)
    calibration_ref: Optional[str] = Field(default=None)

A change that must remove or repurpose a field is not additive; it becomes a new major version with its own model, and both versions run side by side until the old firmware is fully retired. Never mutate ReadingV1 in place — the historical records validated against it must remain re-validatable exactly as captured.

The distinction between additive and breaking is worth stating precisely, because it is the line an auditor will probe. Adding an optional field, widening a numeric range so previously valid readings stay valid, or introducing a new enum value that old firmware never emits are all additive: no record that passed the old schema fails the new one. Removing a field, renaming it, changing its unit, narrowing a range, or making a previously optional field required are all breaking: at least one historically valid record would now be rejected or reinterpreted. Only additive changes may extend an existing major version; everything else forces a new one. This is stricter than typical API versioning precisely because the payloads are regulated records rather than transient request bodies — the cost of a wrong call is not a broken client but a questioned dataset.

Step 3 — Route each payload to its version with a discriminated union

A discriminated union dispatches on schema_version, so each payload is validated against precisely the schema its firmware emitted. There is no “best effort” fallback that would let a mismatched payload slip through. The diagram traces a payload from arrival through the version discriminator to either a versioned model and the regulated store, or — for an unrecognized version — the quarantine sink.

python

class TelemetryEnvelope(BaseModel):
    # Pydantic routes on schema_version; an unknown version raises rather than
    # matching a wrong model (§11.10(a) — validated against the exact contract).
    reading: Union[ReadingV1, ReadingV2] = Field(discriminator="schema_version")


def validate_versioned(raw: dict) -> BaseModel:
    return TelemetryEnvelope(reading=raw).reading

Prove that each known version validates and that a wrong-shaped payload for a known version is rejected:

python

v1 = validate_versioned({"schema_version": "1.0", "device_id": "EDGE-TH01",
                         "sequence_id": 5, "captured_at": "2026-07-14T09:00:00+00:00",
                         "temperature_c": 4.2})
assert v1.schema_version == "1.0"
try:
    validate_versioned({"schema_version": "2.0", "device_id": "EDGE-TH01",
                        "sequence_id": 6, "captured_at": "2026-07-14T09:00:30+00:00",
                        "temperature_c": 4.3, "battery_v": 99.0})  # out of range
    assert False, "expected rejection"
except ValidationError:
    pass

Step 4 — Quarantine unknown versions instead of dropping them

When a payload carries a schema_version the registry does not recognize — a firmware rollout that outran the schema deployment — the reading is neither accepted into the regulated store nor discarded. It is routed to a durable quarantine queue with its raw bytes intact, so no captured reading is lost and none is coerced into a wrong shape. Quarantine is the mechanism that lets the two clauses coexist during the messy window of a rollout: §11.10(a) forbids admitting an unvalidated shape into the regulated stream, while §11.10(e) forbids dropping a genuinely captured reading. Holding the payload verbatim, with a hash and a reason, in a store separate from the time-series database resolves the tension — the reading survives for later promotion without ever contaminating the validated record.

python

import hashlib
import json
from datetime import datetime, timezone


class QuarantineError(ValueError):
    """Payload version is unknown or failed validation; route to quarantine."""


def ingest_or_quarantine(raw: dict, quarantine_sink) -> Optional[BaseModel]:
    try:
        return validate_versioned(raw)
    except ValidationError as exc:
        # §11.10(e): preserve the ORIGINAL bytes and the reason, never coerce.
        record = {
            "raw": json.dumps(raw, sort_keys=True, separators=(",", ":")),
            "reason": str(exc),
            "quarantined_at": datetime.now(timezone.utc).isoformat(),
            "raw_sha256": hashlib.sha256(
                json.dumps(raw, sort_keys=True, separators=(",", ":")).encode()
            ).hexdigest(),
        }
        quarantine_sink.append(record)   # durable, append-only, out of the regulated store
        raise QuarantineError(record["reason"]) from exc

Confirm an unknown version lands in quarantine rather than the regulated store:

python

sink = []
try:
    ingest_or_quarantine({"schema_version": "9.9", "device_id": "EDGE-TH01"}, sink)
    assert False, "expected quarantine"
except QuarantineError:
    # The reading is preserved for review; the regulated stream stays clean.
    assert len(sink) == 1 and sink[0]["raw_sha256"]

Step 5 — Roll out schema-first, retire firmware-last

Deploy the new schema version to the registry and validator before any device emits it, and keep the retiring version registered until the last device on the old firmware has been decommissioned. This ordering means a rollout never produces a payload the platform cannot validate, and a rollback to old firmware never orphans records the platform has forgotten how to read — both of which would breach §11.10(a) consistent performance. Reconcile the quarantine queue after every rollout: any payloads that landed there because the schema deployment lagged the firmware are re-validated against the now-registered version and promoted, with the promotion itself recorded in the audit trail.

Compliance Validation Checklist

Version mandatory — every payload carries schema_version; a missing version is rejected, never inferred (§11.10(a)).
Additive-only within a major — new fields are optional with defaults that keep prior records valid; no field is removed or repurposed in place (§11.10(e)).
Old models frozen — historical schema models are never mutated, so past records re-validate exactly as captured (§11.10(e)).
Exact-version validation — a discriminated union routes each payload to its own schema with no best-effort fallback (§11.10(a)).
Unknown versions quarantined — unrecognized versions go to a durable, append-only quarantine with raw bytes and a reason, never dropped (§11.10(e)).
Schema-first rollout — the schema is registered before any device emits it, and retiring versions stay registered until firmware is fully decommissioned (§11.10(a)).
Quarantine reconciled — post-rollout, quarantined payloads are re-validated and promoted with the promotion recorded in the audit trail (§11.10(e)).
Change controlled — each schema version is a controlled document introduced through change management (ICH Q10).

Troubleshooting

Symptom	Root cause	Fix
Whole fleet rejected right after a firmware push	Schema deployed after firmware, so payloads carry an unknown version	Deploy schema-first; reconcile the quarantine queue and promote the affected records
Old records fail re-validation after a schema change	An existing model was mutated instead of adding a new version	Restore the frozen old model; make the change a new major version running side by side
Silent unit or semantic drift in a field	A field was repurposed rather than added	Treat repurposing as a breaking change; introduce a new version and migrate explicitly
Quarantine queue growing without bound	No reconciliation step after rollouts	Automate post-rollout re-validation and promotion; alarm on quarantine depth
New optional field breaks downstream math	Consumer assumed the field is always present	Default the field and make consumers tolerate its absence for prior versions

Schema validation pipelines for temperature telemetry — the validation gate this versioning discipline keeps stable.
Validating JSON schemas for IoT temperature payloads — the field-level validation each version applies.
Handling network partitions in telemetry ingestion — where replayed readings still clear the versioned gate.
How to map 21 CFR Part 11 requirements to MQTT payloads — the payload contract versions extend.
Async batching strategies for high-volume sensor data — the persistence layer downstream of validation.

This guide supports schema validation pipelines for temperature telemetry, within the broader IoT sensor data ingestion & time-series synchronization section.