16-EFT.WP.Methods.Cleaning v1.0 | Appendix D Quality and Drift Metrics

Home ／ Docs-Technical WhitePaper ／ 16-EFT.WP.Methods.Cleaning v1.0

Appendix D Quality and Drift Metrics

One-Sentence Goal
Define end-to-end conventions, formulas, and alert thresholds for quality and drift metrics so that q_score, drift, PSI, KS, and related indicators are comparable, auditable, and revertible across batch jobs, streaming windows, and online snapshots.

I. Scope & Targets

1. Applicable artifacts
   Cleansed release outputs and their manifests for batch snapshots, streaming sliding windows, and online snapshots.
2. Inputs & windows
   Reference window W_ref = [ts_ref0, ts_ref1), current window W_now = [ts_now0, ts_now1), both evaluated on tau_mono and published on ts.
3. Outputs & constraints
   • Metric families Q.* and DR.*, with aggregate scores q_score ∈ [0,1] and drift ∈ [0,1], and contract thresholds asserted under contracts.
   • When time-base is implicated, report offset/skew/J together with their bounds.

II. Metric Hierarchy & Naming

1. Hierarchy
   • Field level: Q.field.*(k), DR.field.*(k), where k is the field name.
   • Dataset level: Q.ds.*, DR.ds.*, obtained via weighting or envelope aggregation over field-level metrics.
2. Prefixes
   • Quality metrics use the Q.* prefix; drift metrics use DR.*.
   • Time-base–related indicators are carried under timing.*; arrival-time consistency under arrival.*.

III. Statistical Windows & Sampling

1. Window definitions
   • N_ref = count(W_ref), N_now = count(W_now), Delta_t = ts_now1 - ts_now0.
   • For event streams, use parallel sliding windows: W_ref fixed or exponentially decayed; W_now is the most recent window.
2. Sampling & regularization
   • If full data are unavailable, use reservoir sampling with size n = min(N_now, n_max).
   • For histograms and distribution estimates, apply empty-bin smoothing with epsilon > 0, default epsilon = 1e-6.

IV. Quality Metrics (Field-Level and Dataset-Level)

1. Completeness & missingness
   • Q.field.comp(k) = 1 - missing_ratio(k) = 1 - ( Σ m_k == 0 ) / N_now.
   • Q.ds.comp = ( 1 / |K| ) * Σ_k Q.field.comp(k).
2. Validity & constraints
   • Q.field.valid(k) = ( count(valid(k)) / N_now ).
   • Q.ds.valid = ( 1 / |K| ) * Σ_k Q.field.valid(k).
3. Uniqueness & dedup outcome
   dup_ratio = ( dup_count / N_now ), Q.ds.unique = 1 - dup_ratio.
4. Referential integrity
   orphan_ratio = ( orphan_fk / N_now ), Q.ds.fk = 1 - orphan_ratio.
5. Dimensions & units
   Q.ds.dim = 1 if all check_dim(expr) pass; otherwise Q.ds.dim = 0 (or use a continuous form 1 - ( violations / checks )).
6. Freshness & timeliness
   fresh_age_s = ( ts_now1 - max(ts) ), Q.ds.fresh = max( 0, 1 - fresh_age_s / T_fresh_max ).
7. Time-base quality
   • timing.offset = offset, timing.skew = skew_ppm, timing.J = J_ms_p99;
   • Q.ds.timing = g_t( offset, skew_ppm, J_ms_p99 ), e.g., g_t = exp( - a1*|offset| - a2*|skew_ppm| - a3*|J_ms_p99| ).
8. Arrival-time two-form consistency
   • arrival.delta_form = | ( 1 / c_ref ) * ( ∫ n_eff d ell ) - ( ∫ ( n_eff / c_ref ) d ell ) |;
   • Q.ds.arrival = max( 0, 1 - p99_delta_form_s / tolP99_Tarr_s ).
9. Aggregate quality score
   q_score = clamp01( Σ_i w_i * Q_i ), with Σ_i w_i = 1, Q_i ∈ { Q.ds.comp, Q.ds.valid, Q.ds.unique, Q.ds.fk, Q.ds.dim, Q.ds.fresh, Q.ds.timing, Q.ds.arrival }.

V. Drift Metrics & Statistical Tests (W_ref vs W_now)

1. Histogram-based (discrete or binned)
   • Let p_i = P_ref(B_i), q_i = P_now(B_i), with Σ_i p_i = Σ_i q_i = 1.
   • DR.field.PSI(k) = Σ_i ( q_i - p_i ) * ln( ( q_i + epsilon ) / ( p_i + epsilon ) ).
   • DR.field.KL(k) = Σ_i p_i * ln( ( p_i + epsilon ) / ( q_i + epsilon ) ).
   • DR.field.JS(k) = ( 1 / 2 ) * KL( p || m ) + ( 1 / 2 ) * KL( q || m ), with m = ( p + q ) / 2.
2. Continuous distribution distances
   • DR.field.W1(k) = ( ∫ | F_ref(x) - F_now(x) | dx ) (Wasserstein-1).
   • DR.field.KS(k) = sup_x | F_ref(x) - F_now(x) |.
3. Kernel two-sample distance
   DR.field.MMD2(k) = ( 1 / n^2 ) Σ k(x_i, x_j) + ( 1 / m^2 ) Σ k(y_i, y_j) - ( 2 / (nm) ) Σ k(x_i, y_j ).
4. Categorical consistency
   DR.field.chi2(k) = Σ_i ( ( c_now_i - E_now_i )^2 / ( E_now_i + epsilon ) ), where E_now_i = N_now * p_i.
5. Multivariate drift (optional)
   DR.ds.mmd2 = MMD2( X_ref, X_now ) on a chosen field subset.
6. Concept-drift proxies (when labels/models exist)
   • Predicted-mean drift: DR.pred.mean = | E_ref[f(x)] - E_now[f(x)] |.
   • Performance drift: DR.perf = | metric_ref - metric_now |.
7. Change points & online tests
   • CUSUM: S_t = max( 0, S_{t-1} + ( x_t - mu_0 - k ) ), alarm if S_t > h.
   • Page–Hinkley: m_t = min( m_{t-1}, x_t ), PH_t = x_t - m_t - lambda, alarm if PH_t > h.
   • ADWIN: adaptive window split; alarm when | mean(W_L) - mean(W_R) | > eps.

VI. Drift Aggregation & Normalization

1. Field contributions & aggregation
   • Given field-level drift scores d_k (e.g., JS or PSI) and weights alpha_k ≥ 0 with Σ alpha_k = 1,
     DR.ds.raw = Σ_k alpha_k * d_k.
   • Normalize to [0,1]: drift = clamp01( 1 - exp( - beta * DR.ds.raw ) ), beta > 0.
2. Explainability
   Field contribution share share_k = ( alpha_k * d_k ) / ( Σ_j alpha_j * d_j + epsilon ); report top-K_top fields and top-bins_top bin contributions in descending order.

VII. SLO Settings & Alert Levels

1. Suggested indicators & thresholds (to be codified in contracts)
   • SLO.q_score.p99 ≥ q_min; SLO.PSI ≤ psi_warn (warn), ≤ psi_fail (trip).
   • SLO.KS ≤ ks_max; SLO.JS ≤ js_max; SLO.W1 ≤ w1_max.
   • SLO.timing.skew_ppm ≤ skew_max, SLO.timing.J_ms_p99 ≤ J_max.
   • SLO.arrival.p99_delta_form_s ≤ tolP99_Tarr_s.
   • SLO.fresh_age_s ≤ T_fresh_max.
2. Multi-level alerts
   • warn: near-threshold—down-weight or rate-limit traffic.
   • critical: rollback or quarantine; block release at freeze_release.
   • For streaming, add a suppression window to avoid alert storms due to jitter.

VIII. Uncertainty & Dimensional Checks (Embedded)

1. Units & dimensions
   • Where metrics carry dimensions (time, distance, probability, etc.), run repair_units and validate with check_dim(expr) first:
   • Examples: dim(T_arr) = [T], dim(W1) = unit(x), probability metrics like PSI are dimensionless ([]).
2. Uncertainty propagation
   • If x has uncertainty u(x), a linearized approximation for a drift score is
     u(d_k) ≈ sqrt( Σ ( ∂d_k/∂x_i )^2 * u(x_i)^2 ).
   • At publication, provide a coverage-factor estimate U = k * u_c, recorded under manifest.env_correction.uncertainty_U or extended keys under quality.*.

IX. Implementation Bindings & Interfaces (Aligned with I10-*)

1. Compute & persist
   • compute_quality(ds, policy) -> metrics_Q: returns Q.ds.*, field-level Q.field.*, and q_score.
   • compute_drift(ds_ref, ds_now, policy) -> metrics_DR: returns field- and dataset-level drift, drift, and explainability breakdown.
   • emit_contracts(metrics, thresholds) -> tests[]: generate contracts.tests and evaluate pass/fail.
   • freeze_release(ds, tag, metrics) -> manifest: persist under manifest.quality, manifest.outlier_drift, and contracts.
2. Online / streaming
   In the Threads execution graph, attach a sliding-window instance of compute_drift upstream of detect_outlier; aggregate TS.sli.* by window.

Summary

• This appendix specifies standard, tau_mono-aligned conventions for quality (missingness, validity, uniqueness, referential integrity, dimensions, freshness, time-base quality, and arrival two-form consistency) and drift (PSI/KL/JS/KS/W1/MMD, change-point detection, etc.).
• With
  q_score = clamp01( Σ w_i * Q_i )
  and
  drift = clamp01( 1 - exp( - beta * Σ alpha_k * d_k ) ),
  the aggregate scores become comparable across modalities; thresholds enter contracts, results land in the manifest, and the audit chain plus rollback anchors close the loop.