14-EFT.WP.Methods.Inference v1.0 | Chapter 13: Change Management & Canary Strategy

Home ／ Docs-Technical WhitePaper ／ 14-EFT.WP.Methods.Inference v1.0

Chapter 13: Change Management & Canary Strategy

I. Scope & Objectives

• Define version governance, staged rollout, and rollback playbooks for an inference system across multi-channel canary / stable / LTS, constraining model and pipeline changes to a controlled blast radius with verifiable evidence, and ensuring that SLOs, calibration, and consistency are no worse than baseline.
• Provide an end-to-end change workflow and gate strategy that connects with Chapter 9 cards & signatures (IPC/PC, fingerprint, anchor), Chapter 10 operations & alerting (TS.*), Chapter 12 acceptance (Mx-41 → Mx-44), and Chapter 11 cross-device certificate CertEq.

II. Terms & Symbols

• Versions & channels: ver = major.minor.patch; channel ∈ {canary, stable, LTS}; channel promotion is controlled by Gate (see Chapter 12).
• Canary weights & rings: q ∈ (0,1] is the traffic share; ring set R = {R0, R1, R2, R3} denotes progressively broader cohorts or sites (internal → regional → global).
• Shadow & double-run: shadow (inference only, no serve), double-run (A/B synchronous inference); baseline A vs candidate B.
• Comparison metrics: delta_AB = E_{D}[ L( y, y_hat_B ) - L( y, y_hat_A ) ];
  p_fail = P( delta_AB > tau_noninf ) (estimated via bootstrap).
• Blast radius: blast = q * N_active; budget budget.blast.
• Compatibility flags: flag.compat, flag.schema_guard, flag.kill_switch; migration window [ts_start, ts_end].
• Rollback anchors: anchor_prev, anchor_curr; rollback target anchor_target = anchor_prev.

III. Postulates & Minimal Equations

• P41-91 Channel-steady postulate
  Under EnvLock, fixed IPC/PC, and locked rng.seed, the long-run metrics of the stable channel satisfy the declared SLOs, and any version promoted to stable must pass Gate = true with
  score_LCB >= score_base - tau_noninf (see Chapter 12).
• P41-92 Canary-safety postulate
  At any time during a canary rollout:
  blast <= budget.blast and TS.error <= tau_error and TS.latency_{p99} <= tau_p99;
  otherwise immediately fire flag.kill_switch.
• S42-91 Traffic advancement law
  If, within ring Rk and window W, the following all hold:
• p_fail <= alpha_gate
• TS.error <= tau_error
• ECE_T <= tau_cal
• delta_offon <= tau_cons

then advance traffic by

q_{t+1} = min( q_t * ( 1 + g ), q_t + s_max, q_cap )

where g is the growth factor, s_max the per-step cap, and q_cap the ring or global limit.

• S42-92 Double-run non-inferiority
  H0: delta_AB <= tau_noninf; estimate
  p_fail = P( delta_AB > tau_noninf ) via stratified bootstrap; accept if
  p_fail <= alpha_gate.
• S42-93 Compatibility & migration constraints
  For architecture/feature changes, require schema_in^{new} to be compatible with schema_out^{old}, and keep flag.compat = on during the migration window; a violation implies Gate = false and forced rollback.

IV. Data & Manifest Conventions

• Release ticket must include:
  ver, channel, anchor_curr, anchor_prev, ring_map: R→{q_cap}, s_max, g, W, alpha_gate, tau_noninf, tau_error, tau_p99, tau_cons, tau_cal, budget.blast.
• Per-ring telemetry: collect TS.latency_{p95,p99}, TS.thrpt, TS.error, ACC/AUC/NLL, ECE/MCE, cost.per.req, power.avg, delta_offon, delta_dev, and publish CI_{1-delta}.
• Traceability: every canary batch is written with fingerprint, and bound to PlanId, GateDecision, reasons; sample trails are de-identified with hash(•).

V. Algorithms & Implementation Bindings

1. New prototypes
   • I40-70 plan_rollout(spec:dict) -> RolloutPlan
   • I40-71 run_shadow(rtA:Runtime, rtB:Runtime, stream:any, W:any) -> ShadowReport
   • I40-72 split_traffic(router:any, q:float, ring:str) -> Ack
   • I40-73 compare_streams(repA:any, repB:any, policy:dict) -> {delta_AB:float, p_fail:float, CIs:dict}
   • I40-74 adjust_canary(plan:RolloutPlan, obs:dict) -> {q_next:float, ring_next:str}
   • I40-75 trigger_rollback(anchor_target:str, reason:str) -> Ack
   • I40-76 freeze_channel(channel:str) -> Ack
   • I40-77 write_changelog(ver:str, changes:list, impacts:list) -> LogRef
2. Pseudocode (abridged)
   • plan = I40-70(spec); lock EnvLock and IPC/PC.
   • Run I40-71 (shadow). If p_fail > alpha_gate → abort.
   • Initialize ring = R0, q = q0; loop:
     1. I40-72(router, q, ring); aggregate metrics over window W;
     2. stats = I40-73(repA, repB, policy);
     3. If any TS.* or ECE/MCE or delta_offon breaches → I40-75 (rollback);
     4. Else step = I40-74(plan, stats); advance q, ring;
     5. If ring = R_last and q = 1.0 → finish.
   • I40-77 record changes; after switching to stable, I40-76("canary") freezes the canary channel.

VI. Metrology Flows & Run Diagram

1. Mx-45 Version tiering & channel binding
   Map change types to strategy:
   • major: require full Mx-41 → Mx-44 acceptance, full shadow, ring progression R0→R3, and flag.compat on.
   • minor: shadow + small q steps; schema must be non-breaking; flag.compat optional.
   • patch: fast canary or blue-green; still guarded by TS.*.
2. Mx-46 Shadow & double-run
   Execute I40-71 on production-sourced streams; compute delta_AB, p_fail, and CI_{1-delta}. If pass, enter first ring R0.
3. Mx-47 Ring advancement & gate guards
   Apply S42-91; at each step verify SLOs, calibrated ECE_T, delta_offon, cross-device delta_dev, and cost ceilings.
4. Mx-48 Completion & stabilization
   Upon success, switch channel = stable; continue monitoring for regressions over an observation window; if triggered, execute Mx-49.
5. Mx-49 Rollback & reset
   Call I40-75(anchor_prev); restore router weights, set flag.kill_switch = on; re-validate in R0 before re-starting canary or freezing to LTS.

VII. Verification & Test Matrix

• Advancement safety: in any Rk, ensure Pr( TS.error > tau_error ) <= alpha_gate; construct an upper bound via time-block bootstrap.
• Non-inferiority power: p_fail does not exceed alpha_gate per ring; if power < 1 - beta_err, enlarge W or pause progression.
• Compatibility & rollback drill: in sandbox and pre-prod, verify flag.compat toggling and that I40-75 completes within T_recover <= tau_recover.
• Calibration guard: after temperature scaling, ECE_T and MCE remain no worse than baseline ECE_base through the entire canary.
• Joint consistency: delta_offon <= tau_cons, delta_dev <= tau_dev, and CertEq valid.
• Cost & energy: cost.per.req and power.avg stay within tau_cost without conflicting with latency goals.
• Audit replay: using fingerprint, anchor, replay stepwise advancement and Gate decisions; produce equivalent reasons.

VIII. Cross-References & Dependencies

• Acceptance scores & gates: Chapter 12.
• TS.*, alerting & degradation: Chapter 10.
• Offline/online consistency: Chapter 6.
• Cross-device: Chapter 11.
• Cards & signatures: Chapter 9.
• Model & operator constraints: Chapter 5.
• Calibration & uncertainty: Chapter 7.

IX. Risks, Limitations & Open Questions

• Metric variance and intersecting gates can cause oscillatory advancement; apply robust percentile estimators and lagged stepping for TS.latency_{p99} and TS.error.
• Subtle data drift may be invisible at small q; combine stratified sampling with sensitive-cohort amplification.
• Architecture-level changes (features/schemas/operator families) may pass shadow yet contend for resources at peak load; integrate concurrency load testing.
• Multi-site latency heterogeneity skews SLO composition; aggregate by site weights within each Rk and provide site-level rollback gates.

X. Deliverables & Versioning

1. Deliverables
   • RolloutPlan.yaml (from Mx-45: ring definitions, q schedule, gates, budgets, rollback targets).
   • ShadowReport.json (from Mx-46: delta_AB, p_fail, CIs).
   • RingMetrics.csv (per-ring TS.*/ACC/NLL/ECE/MCE/cost.* with CI_{1-delta}).
   • GatePolicy.json (advancement law, alpha_gate, tau_*, and kill_switch conditions).
   • RollbackPlaybook.md (I40-75 procedure, T_recover, and drill records).
   • ChangeLog.md (semantic versioning, impact surface, and parent fingerprint PC.meta.parent_fingerprint).
   • CertEq.pdf (updated certificate if a cross-device change is involved).
2. Version & channel policy
   • major must enter canary, complete the full Mx-41 → Mx-49 chain before switching to stable, and update LTS compatibility flags within one observation cycle.
   • minor may skip some rings but may not skip shadow and at least one ring of real-traffic validation.
   • patch may use blue-green, but must retain holdback and kill_switch.
   • Any rollback must create a new entry and refresh the AnnBundle and ArchiveReceipt (see Chapter 12).