18-EFT.WP.Methods.CrossStats v1.0 | Chapter 14 — Quality Metrics, SLOs & Audits (Statistical Services)

Home ／ Docs-Technical WhitePaper ／ 18-EFT.WP.Methods.CrossStats v1.0

Chapter 14 — Quality Metrics, SLOs & Audits (Statistical Services)

One-line objective: Define an SLI/SLO system, measurement scheme, and audit loop for statistical services so that estimation, testing, and decisioning are monitorable, traceable, and rollback-capable under unified metrology, time-base, and dual T_arr formulations.

I. Scope & Targets

1. Scope
   • Runtime quality governance for statistical services: coverage, error rates, power attainment, drift, latency, reproducibility, and compliance audits.
   • Applicable to online experiments, offline evaluations, and hybrid batch/stream pipelines.
2. Targets
   • Inputs: manifest.stats.*, trace, logs, streaming metrics, and a reference baseline ref.
   • Outputs: audit_report, slo_attainment, violation_events, rollback_plan.
   • Time-base: compute metrics over windows on tau_mono, publish on ts with offset/skew/J; if T_arr appears, record both formulations and delta_form.

II. Terms & Symbols

• Coverage & calibration: coverage_rate, ECE, Brier.
• Errors & power: FPR, FNR, FDR, power, MDE.
• Latency & throughput: latency_ms_p50/p95/p99, throughput, staleness.
• Drift & baseline: W1, KL, psi, drift_level, drift_slope.
• Reproducibility & audit: repro_hash = hash_sha256(blob), audit_completeness.
• Arrival-time consistency: dual T_arr formulations and delta_form.

*III. Axioms P314- **

• P314-1 (Unified Gauges): All quality metrics follow a unified measure and unit; execute check_dim(expr) prior to publication.
• P314-2 (Explicit Windows): The metric window Delta_t and alignment policy must be explicit; implicit sliding is prohibited.
• P314-3 (Dual Formulations in Parallel): For services involving T_arr, record both formulations in parallel and persist delta_form.
• P314-4 (Pre-Frozen SLOs): Freeze SLOs and alert strategy cards before launch; do not relax thresholds ad hoc during operations.
• P314-5 (Reproducibility): Any published conclusion must be fully reproducible via repro_hash and the accompanying manifest.

*IV. Minimal Equations S314- **

• S314-1 (Coverage Estimation)
  coverage_rate = ( 1 / N ) * ∑ 1{ theta_true ∈ CI_i } (where ground truth or synthetic reference exists).
• S314-2 (Calibration Error)
  ECE = ∑_{b=1..B} ( n_b / N ) * | acc_b - conf_b |.
• S314-3 (Error Rates & Power)
  FPR = FP / (FP + TN), FNR = FN / (TP + FN), power = 1 - beta.
• S314-4 (FDR Control)
  FDR = E[ V / max(R,1) ]; for sequential testing, accumulate via an alpha-spending function with alpha_spent ≤ alpha_budget.
• S314-5 (Latency Quantiles)
  latency_ms_pq = quantile( latency_ms, q ).
• S314-6 (Drift Metrics)
  W1(p,q), KL(p||q), psi = ∑ ( (q_i - p_i) * ln(q_i/p_i) ) (binned formulation).
• S314-7 (Arrival-Time Discrepancy)
  delta_form = | ( 1 / c_ref ) * ( ∫ n_eff d ell ) - ( ∫ ( n_eff / c_ref ) d ell ) |.
• S314-8 (Audit Completeness)
  audit_completeness = (#present_required_fields) / (#required_fields).

V. Metrology Flow M30-14 (Define → Instrument → Compute → Decide → Audit → Rollback)

• Define
  Freeze SLO targets such as coverage_min, ECE_max, FPR_max, power_min, latency_p99_max, psi_max.
• Instrument
  At thr/chan and operator boundaries, collect latency, throughput, staleness, and record TraceID.
• Compute
  Within window Delta_t, compute S314-* metrics; when truth is absent, use synthetic insertion or PPC to estimate coverage.
• Decide
  Generate violation_events per contracts C30-14xx, and write results to manifest.stats.audit.*.
• Audit
  Produce a signed report including repro_hash, alpha_spent, drift_report, and evidence links.
• Rollback
  Trigger strategy cards: downgrade gauge, freeze publication, roll back to ref, expand samples or resample to strengthen uncertainty assessment.

VI. Contracts & Assertions C30-14xx

• C30-1401 (Coverage): coverage_rate ≥ SLO.coverage_min (simulation or backtest mode).
• C30-1402 (Calibration): ECE ≤ SLO.ECE_max, Brier ≤ SLO.Brier_max.
• C30-1403 (Error Rates): FPR ≤ SLO.FPR_max and FDR ≤ SLO.FDR_max.
• C30-1404 (Power): power ≥ SLO.power_min or the MDE target is achieved.
• C30-1405 (Latency): latency_ms_p99 ≤ SLO.latency_p99_max, staleness ≤ SLO.staleness_max.
• C30-1406 (Drift): W1 ≤ W1_max ∧ KL ≤ KL_max ∧ psi ≤ psi_max; otherwise trigger recalibration or rollback.
• C30-1407 (Dual-Form Gap): If T_arr is present, assert delta_form ≤ tol_Tarr.
• C30-1408 (Alpha Budget): alpha_spent ≤ alpha_budget (including sequential and multiple testing).
• C30-1409 (Reproducibility & Signature): repro_hash and signature must exist and verify; audit_completeness ≥ 0.98.
• C30-1410 (Robustness): decision_sign_stability ≥ SLO.dec_stability_min (via bootstrap or leave-one-out).

*VII. Implementation Bindings I30- **

• I30-141 collect_sli(stream, spec) -> metrics
• I30-142 compute_slo_attainment(metrics, slo) -> attainment
• I30-143 backtest_coverage(ds, plan) -> coverage_rate
• I30-144 calibration_report(pred, obs, bins) -> {ECE, Brier}
• I30-145 track_alpha_spending(seq_tests) -> {alpha_spent, budget_ok}
• I30-146 drift_monitor(ref, cur, methods) -> drift_report
• I30-147 latency_summary(traces) -> {p50,p95,p99,staleness}
• I30-148 audit_decision(trace, manifest) -> audit_report
• I30-149 reproducibility_hashes(env, code, data_ptrs) -> repro_hash
• I30-14A emit_stats_audit(results, policy) -> manifest.stats.audit
• Invariants: alpha_spent ≤ alpha_budget; metrics.window == Delta_t; sum(w_i)/N ≈ 1 (if weighted); signature verifiable.

VIII. Cross-References

• Dimensions & units: Methods.Cleaning v1.0, Chapter 4.
• Timelines & synchronization: Methods.Cleaning v1.0, Chapter 5.
• Multiple testing & error control: this volume, Chapter 6.
• Drift & distribution alignment: this volume, Chapter 7.
• Audit manifests & signatures: Methods.Cleaning v1.0, Chapter 10, and this volume, Appendix C.

IX. Quality & Risk Control

1. Suggested SLI/SLO Panel
   • Baseline: coverage_rate, ECE, FDR, latency_ms_p99, psi, alpha_spent, audit_completeness, decision_sign_stability.
   • Runtime: throughput, queue.rho, W_q, error_rate_ops.
2. Rollback Strategies
   • Minor breaches: enlarge window, trigger resampling, and use bootstrap to widen intervals.
   • Moderate breaches: freeze new publications and apply robust gauges (HK/Bayesian with more conservative priors).
   • Severe breach / safety threshold: roll back to ref, enforce re-calibration and re-audit.
3. Audit Traceability
   Chain TraceID across manifest, code versions, and data slices; archive after verifying repro_hash consistency.

Summary