16-EFT.WP.Methods.Cleaning v1.0 | Appendix E Errors and Uncertainty Propagation

Home ／ Docs-Technical WhitePaper ／ 16-EFT.WP.Methods.Cleaning v1.0

Appendix E Errors and Uncertainty Propagation

One-Sentence Goal
Provide error models and uncertainty-propagation conventions for the entire lifecycle—from acquisition and calibration through cleansing to publication—so that auditable error budgets and coverage intervals U = k * u_c are produced and written into the manifest and contracts.

I. Scope & Targets

• Applicable objects
  Field-level metrics, derived quantities, and composite indicators in batch and streaming settings (including q_score, T_arr, delta_form, timing.*, etc.).
• Inputs & boundaries
  Nominal values x, standard uncertainties u(x), covariances cov(x_i, x_j), units and dimensions unit(x), dim(x), and time-base synchronization parameters offset/skew/J for tau_mono → ts.
• Outputs
  Combined uncertainty u_c(y), coverage uncertainty U(y) = k * u_c(y), degrees-of-freedom or confidence conventions, and a decomposed error budget with dominant contributors ranked.

II. Terms & Symbols

• Standard uncertainty: u(x); combined uncertainty: u_c(y); coverage uncertainty: U(y) = k * u_c(y).
• Covariance matrix: Cov(x) = [ cov(x_i, x_j) ]; correlation coefficient: rho_ij = cov(x_i, x_j) / ( u(x_i) * u(x_j) ).
• Jacobian: J = ∂f/∂x, where y = f(x).
• Types: u_A (statistical, from samples), u_B (non-statistical, from references and bounds).
• Drift and systematic components: u^2(x) = u_rand^2(x) + u_sys^2(x).
• Units/dimensions: unit(•), dim(•); check via check_dim( y - f(x) ) = true/false.

III. Minimal List of Error Sources

• Instrument resolution & repeatability: u_res, u_rep.
• Calibration chain residuals: u_cal.
• Environmental-correction uncertainty: u_env (introduced by corr_env(x; RefCond)).
• Time synchronization & jitter: u_offset, u_skew, u_J.
• Numerical processing: interpolation, filtering, resampling u_num.
• Model-convention gap: u(delta_form) for the two-form arrival difference.
• Correlations: covariance terms arising from shared channels, shared calibrations, or shared filtering windows.

IV. Linearized Propagation Rules (General Convention)

1. Vector form
   • y = f(x), x ∈ R^n, first-order approximation: Cov(y) = J * Cov(x) * J^T, with J_ij = ∂y_i/∂x_j.
   • Scalar output: u_c^2(y) = grad_x(f)^T * Cov(x) * grad_x(f).
2. Weighted sum
   y = Σ_i a_i x_i: u_c^2(y) = Σ_i a_i^2 u^2(x_i) + 2 Σ_{i<j} a_i a_j cov(x_i, x_j).
3. Products & ratios (relative form)
   • y = x * z: ( u(y) / |y| ) ≈ sqrt( ( u(x)/x )^2 + ( u(z)/z )^2 + 2 rho_xz ( u(x)/x ) ( u(z)/z ) ).
   • y = x / z: ( u(y) / |y| ) ≈ sqrt( ( u(x)/x )^2 + ( u(z)/z )^2 - 2 rho_xz ( u(x)/x ) ( u(z)/z ) ).
4. Powers, logs, exponentials
   • y = x^a: ( u(y) / |y| ) ≈ |a| * ( u(x) / |x| ).
   • y = ln(x): u(y) ≈ u(x) / |x|.
   • y = exp(x): u(y) ≈ |exp(x)| * u(x).
5. Convolution / moving average
   y_t = Σ_i w_i x_{t-i}: u_c^2(y_t) = Σ_i Σ_j w_i w_j cov( x_{t-i}, x_{t-j} ) (reduces to Σ_i w_i^2 u^2(x_{t-i}) under independence).

V. Numerical Integration & Path Quantities

1. Line integral via discrete sum
   • I = ( ∫ g(ell) d ell ) ≈ Σ_i w_i g_i, then
   • u_c^2(I) = Σ_i w_i^2 u^2(g_i) + 2 Σ_{i<j} w_i w_j cov(g_i, g_j).
2. Arrival time, two forms
   • T1 = ( 1 / c_ref ) * ( ∫ n_eff d ell ); T2 = ( ∫ ( n_eff / c_ref ) d ell ).
   • General case (allowing c_ref = c_ref(ell)):
     1. u_c^2(T1) = ( ∂T1/∂c_ref0 )^2 u^2(c_ref0) + ∬ ( 1 / c_ref0^2 ) cov( n_eff(ell), n_eff(ell') ) d ell d ell' + cross.
     2. u_c^2(T2) = ∬ cov( n_eff(ell)/c_ref(ell), n_eff(ell')/c_ref(ell') ) d ell d ell'.
   • If c_ref is constant and independent of n_eff:
     1. u_c^2(T1) ≈ ( I_n / c_ref^2 )^2 u^2(c_ref) + ( 1 / c_ref^2 ) u_c^2( I_n ), where I_n = ( ∫ n_eff d ell ).
     2. u_c^2(T2) ≈ ( 1 / c_ref^2 ) u_c^2( I_n ).
   • Two-form difference
     delta_form = | T1 - T2 |, with unsigned difference D = T1 - T2:
     u_c(delta_form) = sqrt( u_c^2(T1) + u_c^2(T2) - 2 cov(T1, T2) ).

VI. Time Mapping & Synchronization Terms

1. Affine mapping
   • ts = a * tau_mono + b, a = 1 + skew, b = offset.
   • u_c^2(ts) = ( tau_mono^2 ) u^2(a) + u^2(b) + a^2 u^2(tau_mono) + 2 tau_mono a cov(a, tau_mono).
2. Jitter & bucketing
   When ts is bucketed/windowed by Δt, report J_ms_p99 and approximate u(J) ≈ J_ms_p99 / z_p (e.g., z_p ≈ 2.33 for p = 0.99).

VII. Uncertainty from Environmental Correction & Imputation

1. Environmental correction
   x' = corr_env(x; RefCond, θ), first-order propagation:
   u_c^2(x') = ( ∂x'/∂x )^2 u^2(x) + Σ_j ( ∂x'/∂θ_j )^2 u^2(θ_j) + 2 Σ_j ( ∂x'/∂x )( ∂x'/∂θ_j ) cov( x, θ_j ).
2. Imputation
   • Linear interpolation x'(t) = α x(t0) + (1-α) x(t1):
     u_c^2 = α^2 u^2( x(t0) ) + (1-α)^2 u^2( x(t1) ) + 2 α(1-α) cov( x(t0), x(t1) ).
   • Model-based imputation (e.g., regression): add model variance u_model^2 and parameter-covariance terms.

VIII. Uncertainty for Aggregate Indicators

• Quality score
  q_score = clamp01( Σ_i w_i Q_i ):
  u_c^2(q_score) = Σ_i w_i^2 u^2(Q_i) + 2 Σ_{i<j} w_i w_j cov(Q_i, Q_j).
• Drift score
  drift = clamp01( 1 - exp( - beta * d_raw ) ), with d_raw = Σ_k alpha_k d_k:
  u(drift) ≈ | beta * exp( - beta * d_raw ) | * u( d_raw ).

IX. Degrees of Freedom & Coverage Factor

1. Welch–Satterthwaite approximation
   ν_eff = ( u_c^4 ) / ( Σ_i ( c_i^4 u^4(x_i) / ν_i ) ), where c_i = ∂f/∂x_i.
2. Choosing k
   • Normal approximation: k_0.95 ≈ 1.96, k_0.99 ≈ 2.58; for finite samples, use t(ν_eff) to select k_p.
   • Publish U = k * u_c, and record k, p, ν_eff, assumption in the manifest.

X. Nonlinear and Non-Gaussian Cases

• Monte Carlo propagation
  Sample x ~ P_x, compute y = f(x), and take empirical quantile intervals as coverage intervals.
• Bootstrap (data-driven)
  Resample observations to estimate u(y) and bias correction.
• Non-smooth functions (min/max/clip/threshold)
  Prefer MC or piecewise-linear approximation with upper-bound correction—avoid pure linearization.
• Robust conventions
  Use MAD or inter-quantile range IQR for u(x): u(x) ≈ 1.4826 * MAD.

XI. Unit & Dimension Consistency

1. Rules
   • Run repair_units before propagation; validate check_dim( y - f(x) ) for f and x.
   • If dim(u(x)) ≠ dim(x) (they should match), reject publication and mark a contract violation.
2. Integrals / convolutions
   dim( ∫ g d ell ) = dim(g) * [L]; dim(W1) = unit(x); probability measures such as PSI/JS are dimensionless.

XII. Audit, Reporting, and Manifest Keys

1. Suggested minimal manifest keys
   • manifest.uncertainty.fields[k] = { u, U, k_factor, nu_eff, method, unit }.
   • manifest.uncertainty.derived[y] = { jacobian_hash, contributors:[(name, share)], U }.
   • manifest.timing.uncertainty = { u_offset, u_skew, u_J }.
   • manifest.arrival.uncertainty = { u_T1, u_T2, u_delta_form }.
2. Contract assertions
   Add to contracts.tests: [ "U(y) ≤ U_max(y)", "u(delta_form) ≤ tolU_Tarr", "missing U for all publish keys == false" ].

XIII. Implementation Bindings I10-E (Reference Interfaces)

• estimate_cov(ds, fields, policy) -> Sigma: estimate Cov(x) by window or model.
• propagate_uncertainty(ds, expr_graph, Sigma, policy) -> u_report: compute u_c, U via Jacobians and numerical integration.
• mc_propagate(ds, expr_graph, prior, N) -> samples, intervals: Monte Carlo propagation.
• arrival_uncertainty(ds) -> { u_T1, u_T2, u_delta_form }: uncertainties and covariance for the two arrival forms.
• sync_uncertainty(timing) -> { u_offset, u_skew, u_J }: evaluate time-sync terms.
• emit_uncertainty_manifest(u_report) -> manifest.patch: generate a signed manifest fragment.
• assert_uncertainty(u_report, thresholds) -> report: convert to contracts, producing pass/fail and root-cause analysis.

XIV. Usage Notes & Suggested Thresholds

1. Batch: fix Sigma on W_ref, propagate u_c on W_now, and bound the change rate |U_now - U_ref| ≤ tol_Udiff.
2. Streaming: estimate Sigma_t on sliding windows; smooth U exponentially to avoid alert jitter.
3. Suggested limits
   • Arrival two-form coverage: U(delta_form) ≤ 0.2 * tol_Tarr.
   • Time-base coverage: U(offset) ≤ 0.1 * Δt_publish; U(J) ≤ J_max / 2.
   • Key published quantities: U(y)/|y| ≤ r_max (e.g., r_max = 2%); exceeding this triggers down-weighting or rollback.

Summary
This appendix specifies a two-tier propagation toolbox—from Jacobian linearization to Monte Carlo—covering path integrals, time synchronization, environmental corrections, and aggregate indicators. Uncertainties are unified as u_c and published as U = k * u_c, with manifest.*.uncertainty and contracts enabling a traceable, auditable, and revertible uncertainty-governance loop.