15-EFT.WP.Methods.Falsification v1.0 | Chapter 8: Uncertainty & Falsification Confidence

Home ／ Docs-Technical WhitePaper ／ 15-EFT.WP.Methods.Falsification v1.0

Chapter 8: Uncertainty & Falsification Confidence

I. Scope & Objectives

• Define a falsification-confidence framework grounded in uncertainty estimation and calibration, spanning posterior/predictive distributions, intervals & sets, ECE/MCE/NLL/Brier calibration metrics, temperature scaling and monotone regression, conformal coverage, OOD detection, and risk budgeting. Tie these pieces to an executable online gating mechanism with GateDecision ∈ {pass, hold, block}. All procedures operate under a shared time base ts = alpha + beta * tau_mono and a locked environment EnvLock.
• Conflict-name disambiguation
  Significance and Type-II error continue to use alpha_sig and beta_err. This chapter denotes the miscoverage rate by delta_cov (coverage confidence is 1 - delta_cov), to avoid confusion with delta_offon.

II. Terms & Symbols

1. Posterior & predictive
   • p(theta | D), p(y | x, theta), p(y | x, D) = ( ∫ p(y | x, theta) p(theta | D) d theta ).
   • Predictive moments: mean = E[ y | x, D ], var = Var[ y | x, D ], quantile_q.
2. Intervals & sets
   • Confidence interval CI_{1 - delta_cov}, Bayesian credible interval CrI_{1 - delta_cov}, prediction interval PI_{1 - delta_cov}.
   • Conformal prediction set: Pi(x) = { y : S(x,y) ≤ q_{1 - delta_cov} }, where S(x,y) is a nonconformity score.
3. Uncertainty decomposition
   var_total = E_{p(theta|D)}[ var( y | x, theta ) ] + var_{p(theta|D)}( E[ y | x, theta ] ) (aleatoric vs. epistemic).
4. Calibration & metrics
   • ECE, MCE, NLL = ( - 1/N ) * Σ log p_hat( y_i | x_i ), Brier = ( 1/N ) * Σ || p_hat_i - onehot(y_i) ||_2^2.
   • Binning: bins = {B_b}, with conf_b (mean confidence) and acc_b (mean accuracy).
5. Online gating & risk
   Violation probability P( violation | D ); risk budget rho_budget; policy thresholds {tau_pass, tau_hold, tau_block}.
6. Mismatch & drift
   • Consistency: delta_offon = ( norm( y_hat_off - y_hat_on ) / norm( y_hat_off ) ), R_infer = 1 - delta_offon.
   • OOD(x) (out-of-distribution score) with threshold tau_ood.

III. Postulates & Minimal Equations

• P51-12 (Posterior–predictive consistency postulate)
  Under EnvLock and stable data conventions, falsification decisions should depend only on the combination of p(y | x, D) and the calibration map Cal(•), not on implementation minutiae.
• P51-13 (Calibration–gating coupling postulate)
  If ECE ≤ tau_ECE and the gating risk functional is Lipschitz(L) in predicted probabilities, then
  | R_gate( p_hat ) - R_gate( p_true ) | ≤ L * tau_ECE.
• P51-14 (Exchangeable conformal-coverage postulate)
  When labeled and calibration sets are exchangeable, Pi(x) covers the true label with probability at least 1 - delta_cov.
• S52-29 (Posterior predictive)
  p(y | x, D) = ( ∫ p(y | x, theta) p(theta | D) d theta ).
• S52-30 (Variance decomposition)
  var_total = E_{p(theta|D)}[ var( y | x, theta ) ] + var_{p(theta|D)}( E[ y | x, theta ] ).
• S52-31 (ECE & MCE)
  ECE = Σ_{b=1..B} ( |B_b| / N ) * | acc_b - conf_b | ; MCE = max_{b} | acc_b - conf_b |.
• S52-32 (Temperature scaling)
  p'_k = softmax( logit_k / T ), T > 0;
  T* = arg min_T ( - Σ log p'_y ) (on a validation set).
• S52-33 (Conformal quantile calibration)
  q_{1 - delta_cov} = quantile_{1 - delta_cov}( { S(x_j, y_j) }_{j ∈ calib} );
  Pi(x) = { y : S(x,y) ≤ q_{1 - delta_cov} }; guarantees P( y ∈ Pi(x) ) ≥ 1 - delta_cov.
• S52-34 (Coverage estimation & confidence)
  With coverage indicators Z_i ∈ {0,1}, cov_hat = ( 1/N ) * Σ Z_i; Hoeffding bound:
  P( cov_true ≥ cov_hat - epsilon ) ≥ 1 - exp( -2 * N * epsilon^2 ).
• S52-35 (OOD-aware gating)
  If OOD(x) ≥ tau_ood, update delta_cov ← min( 1, delta_cov + delta_boost ) and escalate GateDecision (pass → hold, or hold → block).
• S52-36 (Falsification decision rule)
  Let r = P( violation | D ), then
• r ≥ tau_block → GateDecision = block
• tau_hold ≤ r < tau_block → GateDecision = hold
• r < tau_hold → GateDecision = pass

with {tau_hold, tau_block} constrained jointly by rho_budget and alpha_sig, beta_err.

IV. Data & Manifest Conventions

• Uncertainty.card
  {method ∈ {ensemble, mc_dropout, laplace, bootstrap, bayes}, n_members, rng.seed, rng_family, aleatoric_flag:bool, epistemic_flag:bool}.
• Calibration.card
  {scheme ∈ {temperature, isotonic}, bins:B, binning ∈ {equal-width, equal-mass}, ECE_target, NLL_target, holdout_hash, Cal.sig}.
• Conformal.card
  {score:S(x,y), delta_cov, split ∈ {split, cv+}, calib_size, exchangeability_assumption, Pi.sig}.
• Provenance & signatures
  Outputs include {reliability.csv, ece.json, nll.txt, pi_coverage.csv, ood_thresholds.yaml, Gate.policy, fingerprint, hash(•)}.

V. Algorithms & Implementation Bindings

1. Prototype mapping (extending I50-*)
   • I50-14 calibrate_temperature(logits:any, labels:any) -> {T:float, CalibReport}
   • I50-15 calibrate_isotonic(scores:list, labels:list) -> CalibModel
   • I50-16 conformal_calibrate(scores:list, labels:list, delta_cov:float, mode:str) -> {q:float, Pi}
   • I50-17 estimate_uncertainty(runtime:any, x:any, method:str) -> {mean:float, var:float, meta:dict}
   • I50-18 ood_score(x:any, method:str) -> float
2. Reference flow (ECE computation)
   • Bucket confidences into bins to get B_b.
   • acc_b = ( 1/|B_b| ) * Σ 1[ y_i = argmax p_hat_i ], conf_b = ( 1/|B_b| ) * Σ max p_hat_i.
   • Emit ECE, MCE, and the reliability table/plot.
3. Reference flow (conformal split)
   • Compute S(x_j, y_j) on the calibration set; take q_{1 - delta_cov}.
   • At prediction time, return Pi(x) or PI_{1 - delta_cov}(x); record coverage indicators Z_i.
4. Reference flow (risk coupling)
   • Estimate r = P( violation | D ) or an admissible upper bound r_hat.
   • Apply S52-36 and S52-35, returning GateDecision with explanatory fields {ECE, cov_hat, OOD(x)}.

VI. Metrology Flows & Run Diagram

• Mx-62 Calibration & confidence construction
  Freeze Calibration.card; learn T* or the monotone map; emit CalibReport and reliability.csv.
• Mx-63 Uncertainty estimation & conformal coverage
  Choose method and instantiate Uncertainty.card; run I50-16 to obtain q_{1 - delta_cov} and Pi.sig; publish pi_coverage.csv.
• Mx-64 Online gating integration
  Periodically compute {ECE, cov_hat, OOD}; update {tau_hold, tau_block} and alpha_spend(t) per Gate.policy; on anomalies, trigger degradation and rollback.

VII. Verification & Test Matrix

1. Calibration effectiveness
   • ECE ≤ ECE_target, NLL ≤ NLL_target.
   • reliability.csv shows no systematic bias (with MCE within threshold).
2. Coverage & robustness
   • Estimate cov_hat and Wilson intervals on the validation set and OOD subsets; require
     cov_hat ≥ 1 - delta_cov - tau_cov.
   • Under broken exchangeability (shuffling or drift), report coverage degradation and confidence-correction strategy.
3. Decision correctness
   Replay GateDecision logs; mis-block and miss-block rates satisfy budget.power and rho_budget.
4. Drift & consistency
   When delta_offon breaches threshold, verify that changes in ECE/cov_hat promptly trigger hold/block, with acceptable false-alarm rates.

VIII. Cross-References & Dependencies

• Depends on: Core.Metrology (metric definitions & intervals), Core.Errors (error types & thresholds), Core.DataSpec (splits & lineage).
• Coordinates with: Chapter 7 (shares alpha_spend & gating policy), EFT.WP.Methods.Inference Chapter 7 (shares ECE/MCE/NLL & reliability plots), Chapter 9 (online gating interfaces).

IX. Risks, Limitations & Open Questions

• Risks & limitations
  Exchangeability breaks under drift ⇒ conformal guarantees fail; temperature scaling under-covers tails and multimodal uncertainty; OOD × confidence coupling may over-block; ensemble costs may conflict with budget.cpu/gpu/mem.
• Open questions
  Unified scheduling of online FDR with a coverage budget; cross-domain calibration transfer of Pi.sig; power analysis and adaptive thresholds for ΔECE/ΔNLL.

X. Deliverables & Versioning

1. Deliverables
   Calibration.card, Uncertainty.card, Conformal.card, CalibReport, UncertaintyReport, reliability.csv, pi_coverage.csv, ood_thresholds.yaml, Gate.policy, Evidence.bundle (with hash(•) and fingerprint).
2. Versioning policy
   • Adjusting delta_cov / ECE_target → minor bump; changing the calibration or conformal strategy → major bump.
   • Any change to gating thresholds or budgets requires updated signatures and Appendix C registry.