16-EFT.WP.Methods.Cleaning v1.0 | Appendix B Contract Library and Policy Cards

Home ／ Docs-Technical WhitePaper ／ 16-EFT.WP.Methods.Cleaning v1.0

Appendix B Contract Library and Policy Cards

One-Sentence Goal
Provide ready-to-assemble standard contract templates and scenario-specific policy cards for assert_contract, unifying thresholds, windows, remediation actions, and audit conventions.

I. General Rules & Scope

1. Applicable objects
   • Batch datasets ds, online traffic req/resp, and event-stream messages msg.
   • Coverage spans schema, metrology, time base, path & arrival time, missingness & imputation, anomalies & drift, dedup & referential integrity, and release & audit.
2. Basic notation & windows
   • Internal calculations run on tau_mono, publication on ts.
   • Statistical window Delta_t and quantile function Q_p(x) (e.g., Q_0.99(x) = P99).
   • Mask m ∈ {0,1} is policy-defined; by default m=1 means “missing.”
3. Contract execution & persistence
   • Contracts are passed as named assertion sets tests.* to assert_contract(ds, tests).
   • Failure severity sev ∈ {S0,S1,S2,S3} triggers mapped actions and audit-chain updates.
   • Any contract involving T_arr must compute both forms in parallel and persist delta_form.

II. Contract Namespaces & Levels

• tests.schema.* — schema & keys
• tests.units.* — units & dimensions
• tests.time.* — time axis & synchronization
• tests.path.* — path & arrival time
• tests.missing.* — missingness & imputation
• tests.range.* — ranges & value domains
• tests.outlier.* — anomalies & drift
• tests.relate.* — dedup & referential integrity
• tests.release.* — release & audit

III. Contract Templates (Assertion Specs)

1. tests.schema.unique(keys)
   • Goal: unique(keys).
   • Fail: conflict groups found. sev = S2 (default).
2. tests.schema.foreign_key(child.key -> parent.key)
   • Goal: foreign keys resolvable.
   • Fail: orphans exceed orphan_max. sev = S2 | S3.
3. tests.units.dim(expr, target_dim)
   • Goal: check_dim( expr - as_dim(target_dim) ) = 0.
   • Fail: dimensional inconsistency. sev = S3.
4. tests.units.unit_system(system)
   • Goal: field units belong to system, convertible when needed.
   • Fail: non-convertible fields exceed threshold. sev = S2.
5. tests.time.monotone(field)
   • Goal: non_decreasing(field).
   • Fail: reversal or rollback. sev = S2.
6. tests.time.sync_bounds(offset_max, skew_max, J_max)
   • Goal: |offset| ≤ offset_max, |skew| ≤ skew_max, J ≤ J_max.
   • Fail: out of bounds. sev = S1 | S2.
7. tests.path.monotone(ell)
   • Goal: non_decreasing(ell).
   • Fail: path backtracking. sev = S2.
8. tests.path.length(L_min)
   • Goal: L_gamma = ( ∫_gamma 1 d ell ) ≥ L_min.
   • Fail: insufficient path length. sev = S1.
9. tests.path.arrival_forms(c_ref, tol_Tarr, tolP99_Tarr)
   • Goal:
     1. T_arr_1 = ( 1 / c_ref ) * ( ∫_{gamma(ell)} n_eff d ell )
     2. T_arr_2 = ( ∫_{gamma(ell)} ( n_eff / c_ref ) d ell )
     3. delta_form = | T_arr_1 - T_arr_2 |
     4. Assert: delta_form ≤ tol_Tarr and Q_0.99(delta_form) ≤ tolP99_Tarr.
   • Fail: sev = S2 | S3.
10. tests.missing.coverage(m, max_ratio)
    • Goal: mean(m) ≤ max_ratio (default m=1 means missing).
    • Fail: insufficient coverage. sev = S1.
11. tests.missing.impute_tagged(fields)
    • Goal: imputed fields carry provenance and RefCond.
    • Fail: untagged imputation. sev = S2.
12. tests.range.within(field, lo, hi, inclusive)
    • Goal: field ∈ [lo,hi] or (lo,hi).
    • Fail: out-of-range share exceeds threshold. sev = S1 | S2.
13. tests.outlier.rate(method, fields, rate_max)
    • Goal: outlier rate ≤ rate_max.
    • Fail: sev = S1.
14. tests.outlier.drift(method, ref, psi_max | ks_max)
    • Goal: drift metric ≤ threshold.
    • Fail: sev = S1 | S2.
15. tests.relate.dedup_resolved(keys, conflicts_max)
    • Goal: post-dedup residual conflicts ≤ conflicts_max.
    • Fail: sev = S2.
16. tests.release.manifest_signed()
    • Goal: manifest exists pre-release with hash_sha256(blob) and signature.
    • Fail: sev = S3.

IV. Policy Card Templates (Structure & Fields)

1. Fields
   • name: policy card name
   • scene: applicable scenario
   • Delta_t: statistical window
   • thresholds: threshold set (incl. tol_Tarr, tolP99_Tarr, offset/skew/J, etc.)
   • actions: severity-to-remediation mapping
   • audit: audit & alert channels
   • notes: additional constraints
2. Default severity mapping
   • S0: record in audit only, raise alert
   • S1: down-weight or local quarantine
   • S2: quarantine to topic_quarantine or ds_quarantine, require human review
   • S3: rollback / block release, trigger emergency process

V. Sample Policy Cards (Recommended Defaults)

1. SC-Offline-Daily
   • scene: T+1 batch publication
   • Delta_t = 24 h
   • thresholds
     1. tests.schema.unique(pk)
     2. tests.units.unit_system("SI")
     3. tests.units.dim( t_arr, "[T]" )
     4. tests.time.monotone(ts)
     5. tests.path.arrival_forms(c_ref=const, tol_Tarr=5e-6 s, tolP99_Tarr=1e-5 s)
     6. tests.missing.coverage(m, max_ratio=0.05)
     7. tests.range.within(q_score, 0, 1, inclusive=true)
   • actions: S1 → warn, S2 → quarantine_batch, S3 → rollback_release
   • audit: append to audit chain and daily report
2. SC-Online-API
   • scene: online read service
   • Delta_t = 5 min
   • thresholds
     1. tests.time.sync_bounds(offset_max=5 ms, skew_max=50 ppm, J_max=3 ms)
     2. tests.path.arrival_forms(c_ref=const, tol_Tarr=2e-6 s, tolP99_Tarr=5e-6 s)
     3. tests.outlier.rate(method="MAD", fields=[x], rate_max=0.01)
   • actions: S1 → degrade_feature, S2 → shadow_read+quarantine, S3 → circuit_break
   • audit: real-time alerts and dashboards
3. SC-Stream-Topic
   • scene: event-stream cleansing
   • Delta_t = 15 min rolling
   • thresholds
     1. tests.time.monotone(tau_mono)
     2. tests.missing.coverage(m, max_ratio=0.02)
     3. tests.outlier.drift(method="PSI", ref="last_7d", psi_max=0.1)
   • actions: S1 → tag, S2 → route_to(topic_quarantine), S3 → halt_partition
   • audit: persist offsets and replay commands

VI. Computation & Metric Conventions

1. Quantiles & windows
   • Estimate Q_p(x) within Delta_t via bounded-memory algorithms; record u(Q_p).
   • For arrival time, emit both Q_0.50(delta_form) and Q_0.99(delta_form).
2. Error & budgets
   Align with TS.sli.*; policy cards may reference err_rate, lat_ms, fresh_age, etc., to couple actions.

VII. Contract DSL (Suggested Grammar)

1. Primitives
   • UNIQUE(pk)
   • FK(child.key -> parent.key)
   • DIM("t_arr","[T]")
   • UNIT_SYSTEM("SI")
   • MONOTONE("ts")
   • SYNC_BOUNDS(offset<=5 ms, skew<=50 ppm, J<=3 ms)
   • ARRIVAL_FORMS(c_ref=const, tol=5e-6 s, tolP99=1e-5 s)
   • COVERAGE("m", <=0.05)
   • WITHIN("q_score", 0, 1, inclusive=true)
   • OUTLIER_RATE(method="MAD", fields=["x"], <=0.01)
   • DRIFT(method="PSI", ref="last_7d", <=0.1)
   • DEDUP_RESOLVED(keys=["pk"], conflicts_max=0)
   • MANIFEST_SIGNED()
2. Composition & conditionals
   ALL( ... ), ANY( ... ), IF tag("lab") THEN RELAX(OUTLIER_RATE, x2).

VIII. Severity → Action Mapping (Unified Rules)

• S0 (Informational): audit only and panel display.
• S1 (Minor): tag & down-weight; allow pipeline to proceed.
• S2 (Major): quarantine object to ds_quarantine or topic_quarantine; block external release.
• S3 (Critical): rollback or freeze the current release; trigger emergency & change review.

IX. Integrated Examples (Test Sets for assert_contract)

1. Batch minimal contract set (example)
   • UNIQUE(pk)
   • FK(child.pid -> parent.pid)
   • UNIT_SYSTEM("SI")
   • DIM("t_arr","[T]")
   • MONOTONE("ts")
   • ARRIVAL_FORMS(c_ref=c_ref_2024Q4, tol=5e-6 s, tolP99=1e-5 s)
   • COVERAGE("m", <=0.05)
   • WITHIN("q_score", 0, 1, inclusive=true)
   • MANIFEST_SIGNED()
2. Event-stream minimal contract set (example)
   • MONOTONE("tau_mono")
   • SYNC_BOUNDS(offset<=5 ms, skew<=50 ppm, J<=3 ms)
   • OUTLIER_RATE(method="MAD", fields=["Xi"], <=0.01)

X. Audit & Rollback Coordination

1. Audit chain
   • Every contract execution emits an audit_event containing who, when, tests, pass, hash_prev, and computes hash_sha256(blob).
   • Before release freeze, MANIFEST_SIGNED() must pass and the signature be recorded.
2. Rollback policy
   S3 triggers freeze_release rollback to the last signed version; for streaming, pause affected partitions and hold replay.

Summary
This appendix defines a reusable contract library—from assertion semantics to scenario-tailored thresholds and actions—and unifies windows, thresholds, and audit coupling via policy cards. By assembling tests.* in assert_contract, you can enforce declarable, auditable, and revertible quality gates across batch, online, and streaming deployments.