12-EFT.WP.Methods.Repro v1.0 | Chapter 11 Fault Injection and Rollback Strategies

Home ／ Docs-Technical WhitePaper ／ 12-EFT.WP.Methods.Repro v1.0

Chapter 11 Fault Injection and Rollback Strategies

I. Scope and Objectives

• Design, execute, and audit fault injection (FI) to validate system resilience, observability, and rollback capability under reproducibility goals; ensure every injection runs in an isolation domain that preserves the forensic integrity of EnvLock, PipelineCard, ParamCard, and baseline artifacts.
• Objective: with ts = alpha + beta * tau_mono, seed, and EnvLock fixed, verify that the reproduction gate gate.rep, spectral gates, and run-time SLOs are not unexpectedly violated pre/post injection; if any gate is breached, trigger automatic rollback and compensating transactions, and emit a verifiable audit trail.

II. Terms and Symbols

• FI: a fault-injection unit; FI.plan is the injection plan; FI.ticket is the signed work order for a single injection.
• rollback: the rollback procedure; compensation: compensating transactions.
• safe window: the injection safety window; jointly defined by TS.*, hb, bp, makespan, critical path, and reproducibility metrics.
• z_risk: composite injection-risk indicator; gate.rep, tau_tb, tau_psd, eps_mass, eps_norm are pass gates and conservation thresholds.
• Exception set (extended): E_ENV_DRIFT, E_DATA_MISMATCH, E_TIMEBASE_SKEW, E_NONDETERMINISM, E_SEED_INVALID, E_SCHEMA_MISMATCH, E_HB_LOSS, E_BP_OVERFLOW, E_DEADLINE_EXCEEDED, E_RESOURCE_EXHAUST.
• Collision mandate: T_fil denotes tension only; T_trans denotes transmission coefficient only; never mix n with n_eff. When T_arr is involved, compute both gauges in parallel and publish delta_form.

III. Postulates and Minimal Equations

1. P31-20 Injection-isolation postulate
   Any FI must execute in a sandbox or shadow channel, and must not affect the forensic integrity of the baseline golden run: fingerprint(golden_after) = fingerprint(golden_before).
2. P31-21 Rollback monotonic-improvement postulate
   For the same scenario, rollback must not enlarge the deviation from baseline: delta_rep(after_rollback) <= max( delta_rep(before_rollback) , gate.rep ).
3. S32-36 Risk synthesis and triggers
   • z_risk = w1*( delta_rep / gate.rep ) + w2*( r_tb / tau_tb ) + w3*( bp / bp_max ) + w4*( |hb - hb_ref| / hb_ref ) + w5*( eps_mass / tau_mass )
   • Trigger rollback and compensation when z_risk >= 1 or any hard gate is tripped.
4. S32-37 Injection amplitude scales
   • Data level: inj.data = lambda * y (λ is the amplitude factor)
   • Time-base level: inj.time = {alpha_shift, beta_scale} applied to ts = alpha + beta * tau_mono
   • Randomness level: inj.seed = seed ⊕ mask
   • Concurrency level: inj.sched = drop(hb, p) or delay(critical path, dt)
5. S32-38 Spectral conservation & energy constraint
   • Pre/post injection must satisfy var( x ) ≈ ( ∫ S_xx(f) df ).
   • When the injection targets spectral distortion, measure
     delta_psd = ( ∫ | S_xx^inj(f) - S_xx^ref(f) | df ) / ( ∫ S_xx^ref(f) df )
     and compare against tau_psd.

IV. Data and Manifest Gauges

1. FI.plan minimal fields
   • plan.id, target.stage, inj.type (env/data/time/rng/concurrency/network/io)
   • inj.params (e.g., lambda, alpha_shift, beta_scale, mask, dt, drop.p)
   • safe window (bp_max, hb_ref, T_obs, U_w, ENBW)
   • guard (trigger/rollback conditions and z_risk weights {w1..w5})
   • audit.keys (fingerprint, hash(•), verifier.pk_ref).
2. Audit events
   action = {inject | rollback | compensate | abort}, ts, actor, H_k = hash( H_{k-1} || event ), signature sig_k, and associated FI.ticket.
3. Samples and windows
   State T_obs, window function U_w and ENBW. If T_arr is involved, publish both gauges with delta_form. Path integrals are annotated by gamma(ell) and the measure d ell.

V. Algorithms and Implementation Bindings

1. I30-16 inject_fault(plan:dict) -> FI.ticket
   • Validate EnvLock and the safe window.
   • Apply inj.type/params in the shadow channel.
   • Emit audit event action = inject and return FI.ticket.
2. I30-17 assert_invariants(metrics:dict, guards:dict) -> AssertReport
   • Compute delta_rep, r_tb, delta_psd, eps_mass, eps_norm.
   • Synthesize z_risk and return pass with the list of triggered terms.
3. I30-18 trigger_rollback(ticket:any, policy:dict) -> RollbackReport
   • Route traffic back to stable/LTS.
   • Restore seed, ts, ParamCard, and caches.
   • Audit action = rollback and verify P31-21.
4. I30-19 compensate_txn(txn_id:any, policy:dict) -> CompReport
   • Execute undo/redo per idempotency/compensation contracts.
   • Verify conservation eps_mass <= tau_mass and units via check_dim(expr) = true.
5. I30-20 sandbox_replay(bundle:any, seed:any) -> ReplayReport
   • Reproduce pre/post-injection samples with fixed seed and EnvLock.
   • Produce overlays, R_coef, and a delta decomposition.
6. I30-21 gate_and_cutover(canary:any, stable:any, rule:dict) -> CutoverReport
   • Monitor TS.* and reproducibility metrics under rule.
   • Phase cutover or auto-rollback when gates are satisfied/violated.

VI. Metrology Flows and Run Graph

1. Mx-60 plan-fi
   • Gather risk domains and target SLOs.
   • Produce FI.plan and the safe window.
   • Rehearse with sandbox_replay to validate controllability.
2. Mx-61 execute-and-observe
   • Run I30-16 and monitor TS.* in real time.
   • Periodically run I30-17; on out-of-bounds, trigger I30-18.
3. Mx-62 rollback-and-compensate
   • Execute rollback and compensations via I30-19.
   • Re-check P31-21 and conservation constraints.
   • Append H_k and sig_k.
4. Mx-63 analyze-and-harden
   • Use sandbox_replay for contrasts and root-cause analysis.
   • Extract regression cases and add them to the benchmark suite (Chapter 8).
   • Update PipelineCard/ParamCard and alerting rules.

VII. Verification and Test Matrix

1. Minimal required cases
   • Data-level injection: increase lambda stepwise; delta_rep rises monotonically and R_coef = 1 when lambda = 0.
   • Time-base injection: set alpha_shift, beta_scale; estimate r_tb and verify rollback triggers.
   • RNG injection: mask != 0 yields nondeterminism detected by I30-17 as E_NONDETERMINISM.
   • Concurrency injection: drop(hb,p) triggers E_HB_LOSS; with bp rising below bp_max, system performs a stable rollback.
2. Boundary & extreme scenarios
   • Multi-fault concurrency: env + data + time combined; z_risk must correctly trigger the earliest hard gate.
   • Resource exhaustion: on E_RESOURCE_EXHAUST, makespan spikes; rollback must finish within T_cut.
3. Repro gates
   • In the post-rollback window: delta_rep <= gate.rep, delta_psd <= tau_psd, r_tb <= tau_tb, eps_mass <= tau_mass.
   • Coupled validation with the Chapter 8 score function score.

VIII. Cross-References and Dependencies

• Depends on Chapter 5 EnvLock; Chapter 6 alpha/beta/seed; Chapter 7 cards & signatures; Chapter 8 scoring & gates; Chapter 9 audit trail & release strategies; Chapter 10 cross-site windows & bias budget.
• Core.Threads provides TS.* semantics; Core.Metrology provides S_xx(f), U_w, ENBW; Core.DataSpec provides ingestion and traceability fields.

IX. Risks, Limits, and Open Questions

• Risks
  Coupled injections cause uncontrolled cascades (notably time × concurrency); audit-chain gaps breaking H_k; compensations that miss long-tail side effects.
• Limits
  For strongly nonlinear systems, linear z_risk weighting may not characterize critical regions; some hardware-level injections cannot be perfectly sandboxed.
• Open questions
  Learning adaptive z_risk weights; joint optimization of rollback and score; comparability of injections under dual T_arr gauges and weighting of delta_form.

X. Deliverables and Version Management

1. Deliverables
   • FI.plan (injection types, parameters, gates, and safe window).
   • FI.ticket (execution signature and audit chain H_k, sig_k).
   • AssertReport (delta_rep, r_tb, delta_psd, z_risk, and trigger set).
   • RollbackReport and CompReport (rollback timeline, compensations, conservation audits).
   • ReplayReport (overlays and R_coef) and change recommendations.
2. Version policy
   • Adding injection types or hard gates requires a minor version bump and expansion of the Chapter 8 benchmarks.
   • Rollback playbook updates follow canary → stable → LTS phased promotion.
   • Any failed injection must never be deleted—only appended with state updates—and archived to a long-term forensic channel.