54-Reproducibility Checklist Template v1.0 | Chapter 11 — Diagnosis & Rollback (Drift / Bias / Safety)

Home ／ Docs-Technical WhitePaper ／ 54-Reproducibility Checklist Template v1.0

Chapter 11 — Diagnosis & Rollback (Drift / Bias / Safety)

I. Purpose & Scope

• Provide diagnosis workflows and rollback mechanisms when reproduction fails or alignment is not met, covering data/concept drift, measurement & labeling bias, runtime & safety policies, to ensure issues are locatable, degradable, and recoverable while maintaining release compliance.
• For path quantities (arrival/phase), explicitly show gamma(ell) and d ell; record delta_form ∈ {general, factored}; parenthesize all expressions; publication requires p_dim = 1.0 with check_dim_report.json.

II. Inputs & Dependencies

• Depends on: Ch. 4 (Env Lock), Ch. 5 (Data Snapshot), Ch. 6 (Weights/Params/Freshness), Ch. 7 (Scripts & Commands), Ch. 8 (Seeds & Determinism), Ch. 9 (Metrics/Intervals/Gates), Ch. 10 (Reproduction Workflow).
• Cross-volume alignment: Dataset Card Ch. 9 (Bias/Ethics/Privacy), Model Card Ch. 9 (Bias/Ethics/Safety), Error Budget Card Ch. 9 (Threshold Mapping), Pipeline Card Ch. 7/9 (State/Alerts).
• Citations: “volume + version + anchor (P/S/M/I)”, anchor coverage ≥ 90%.

III. Triggers

• Gate failures: any G1–G8 fail or S1–S5 triggered.
• Alignment failures: Chapter 9 decisions (numeric/interval/curve) not satisfied.
• Anomalies: p_dim < 1.0, clock_state!="locked", missing path blocks or unequal arrays, invalid units/dimensions.
• Safety alerts: unauthorized inference, sensitive outputs, load/power breach, abnormal retries/idempotency conflicts.

IV. Diagnosis Workflow

• Env & determinism: compare env_lock.json / preflight_report.json / seed_policy.yaml / rng_state.json; confirm library/driver/container digest and concurrency equality.
• Data & lineage: verify data_refs.yaml / split_manifest.json / lineage_graph.json / checksums.txt; check split leakage, snapshot checksums, and license.
• Path & phase alignment: ensure equal-length gamma_ell/d_ell/n_eff and step Δell ≤ ( c_ref / f_s ) / max(n_eff); align phase in reference window and echo delta_form.
• Metrics & intervals: re-check scorecard.json / compare_spec.yaml; confirm single coverage mode (k/alpha/quantile) and interval-overlap rules; spot-check convergence curves and per-sample residuals.
• Error budget & covariance: trace J/Σ/kernel from model_uq.yaml / uq_summary.json; check PD and bandwidth/correlation assumptions.
• Bias / ethics / safety: use bias_report.md / ethics.md / safety_policy.yaml to localize sampling/measurement/labeling/processing bias and usage constraints.

V. Drift & Bias Localization

• Data drift: KS/χ²/AD, MMD/Energy distance (stratify batch/device/region/slice_k); for path variables, track T_arr/Phi coverage and band width.
• Concept drift: ΔMAE/ΔAUC/Δr_phi over thresholds with non-overlapping CIs; delayed labeling aligned to val/test/holdout.
• Measurement bias: δt_abs/Δτ_ch/σ_y(τ) out of bounds or systematic n_eff residuals.
• Labeling bias: κ < κ_min, large DTW/Hausdorff; trigger re-annotation/adjudication.

VI. Rollback Mechanism (FSM)

1. States: normal → degrade → rollback → recover → normal (events: gate breach / drift confirmed / perf alert / safety violation).
2. Degrade strategies:
   • Model: robust surrogate (Huber/quantile), disable unstable ops, enable conservative intervals.
   • Data: isolate risky slices, tighten gates, shorten windows.
   • Path: switch to fullband/short-window or raise guards (without breaking Δell upper bound).
3. Rollback execution: lock *-lock stable version; verify sha256 + signature; keep I/O contracts & coverage unchanged; canary replay before full rollout.
4. Recovery verification: /validate passes G1–G8, perf/quality meet SLA/SLO & thresholds, then return to normal.

VII. Normative Path Forms

• Arrival (two equivalent):
  T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell )
  T_arr = ( ∫ ( n_eff / c_ref ) d ell )
• Phase accumulation:
  Phi = ( 2π / λ_ref ) * ( ∫ n_eff d ell )

Before diagnosis/rollback, align time → path → phase; arrays len(gamma_ell)=len(d_ell)=len(n_eff)≥2; echo delta_form; run I70-dim_check with p_dim = 1.0.

VIII. Gate Mapping

• G1 Schema completeness: diagnosis & rollback report fields present.
• G2 Citation compliance: anchor coverage ≥ 90%.
• G3 Path conventions: path blocks complete; step compliant.
• G4 Dimensional closure: check_dim_report.json OK.
• G5 Freshness: clock_state="locked".
• G6 Coverage consistency: online/offline intervals match cross-volume mode.
• G7 Covariance consistency: Σ PD and aligned with Error Budget.
• G8 Uniqueness & acyclicity: events/artifacts checksummed; lineage acyclic.
• Trigger S1–S5 (dimension/freshness/path/covariance/citation) to degrade or rollback; tag [Restricted].

IX. Machine-Readable Artifacts
A. diagnosis_report.md (outline)

# Diagnosis Report

- Triggers & symptoms

- Env/seed/data/path checks

- Metric & interval analysis

- UQ & covariance checks

- Bias/ethics/safety notes

- Decision: degrade | rollback | recover plan

B. rollback_fsm.yaml

version: "1.0.0"

states: [normal, degrade, rollback, recover]

transitions:

- { from: normal, to: degrade, when: "gate_alert or drift_alert" }

- { from: degrade, to: rollback, when: "gate_critical or perf_critical" }

- { from: rollback,to: recover, when: "stable_prev_version_ready" }

- { from: recover, to: normal, when: "validate_pass and perf_ok" }

degrade:

strategies: ["robust_surrogate","tighten_gates","isolate_slices"]

rollback:

version_tag: "v1.2.3-lock"

verify: ["checksum","/validate","SLA/SLO"]

recover:

rollout: { canary_percent: 10, steps: 3, pause_s: 600 }

C. alerts.jsonl (sample)

{
  "ts": "2025-09-24T16:10:00Z",
  "level": "critical",
  "event": "gate_fail",
  "gate": "G4",
  "detail": "p_dim < 1.0",
  "action": "rollback"
}

X. Anti-Patterns & Fixes

• Anti: reporting means without intervals/CIs → Fix: add U = k·u_c or quantile bands with convergence diagnostics.
• Anti: T_arr = ∫ n_eff / c_ref d ell (no parentheses) → Fix: parenthesized unified form.
• Anti: detectable drift but no degrade/rollback → Fix: bind automatic FSM actions & approval thresholds.
• Anti: rollback version unsigned/missing checksum → Fix: require signature & checksum verification.
• Anti: unequal path arrays or missing delta_form → Fix: equalize lengths and add metadata.

XI. Cross-References

• Ch. 4 (Env Lock), Ch. 5 (Data Snapshot), Ch. 6 (Weights/Params & Freshness), Ch. 7 (Scripts & Commands), Ch. 8 (Seeds & Determinism), Ch. 9 (Metrics & Gates), Ch. 10 (Reproduction Workflow).
• Model Card Ch. 9 (Bias/Ethics/Safety), Error Budget Card Ch. 9 (Threshold Mapping), Dataset Card Ch. 9 (Bias/Ethics/Privacy), Pipeline Card Ch. 7/9 (State/Alerts).

XII. Checklist

• diagnosis_report.md / rollback_fsm.yaml / alerts.jsonl generated & archived; decision chain auditable.
• Path alignment explicit gamma/measure/delta_form; len(path) ≥ 2, Δell compliant; p_dim = 1.0.
• Data/concept drift, bias & safety checks executed with evidence; UQ/covariance assumptions align with Error Budget, Σ PD.
• /validate passed G1–G8; out-of-band cases degraded/rolled back as [Restricted] with canary & recovery plan.
• Release bundle PTN_EXPORT/ updated with report_manifest.yaml / checksums.txt / SIGNATURE.asc and audit trail.