16-EFT.WP.Methods.Cleaning v1.0 | Appendix A Interface Reference (I10 Full Set)

Home ／ Docs-Technical WhitePaper ／ 16-EFT.WP.Methods.Cleaning v1.0

Appendix A Interface Reference (I10 Full Set)

One-Sentence Goal
Summarize the I10-* interfaces in this volume—their signatures, parameters, returns, and invariants—and define mappings and call-sequence conventions relative to the Core volumes.

I. Unified Conventions & Naming

1. Notation & types
   • ds: dataset (sequence of records)
   • rec: a single record
   • SRef: standard schema registry & constraints
   • policy.*: policy bundle (units, time, anomalies, SLOs, etc.)
   • tests.*: contract test suites
   • tags: set of labels and alerts
   • TS.sli.*: service-layer indicators
   • manifest: manifest and signature artifacts
   • TraceID: end-to-end trace identifier
2. Preset invariants (apply to all interfaces)
   • Units & dimensions: every field entering an expression carries unit(x) and dim(x); check_dim(expr) = 0.
   • Time base: compute on tau_mono, publish on ts; sync metadata includes offset/skew/J.
   • Arrival-time two forms: whenever T_arr is computed, produce both forms in parallel and satisfy delta_form ≤ tol_Tarr.
   • Path: non_decreasing(ell); L_gamma = ( ∫_gamma 1 d ell ) is computable.
   • Traceability: outputs admit hash_sha256(blob) and are verifiable by signature.

II. Core Data Structures (Summary)

• registry: { alias_map, required_fields, pk, idx_k, foreign_key, unit_rules }
• sync_ref: { anchor, clock_model, max_skew, max_J }
• policy.units: { system, conversions, tolerance }
• policy.slo: { targets, EB, burn_thresholds }
• strategy.missing: { mark_mask, drop_rules, impute_rules, env_ref = RefCond }
• tests.*: { dim, unique, monotone, foreign_key, range, arrival_forms }

III. I10-3 Standard Inputs & Schema Binding

1. register_schema(SRef) -> registry_id
   • Effect: register standard schema with aliases, keys, and a contract subset.
   • Invariant: unique(pk); no conflicts in alias_map.
2. standardize_names(ds, registry) -> ds', report
   • Role: unify field names per alias_map and complete the minimal term set.
   • Report keys: added, renamed, dropped, missing_required.
3. validate_dataset(ds, SRef, strict) -> assert_report
   • Checks: field presence, types, pk uniqueness, resolvable foreign_key.
   • Pass condition: assert_report.fail = 0.

IV. I10-4 Units, Dimensions & Metrological Harmonization

1. repair_units(ds, policy.units) -> ds', report
   • Role: unit normalization, dimensional checks, numeric conversions.
   • Invariant: for any expression y - f(x), check_dim( y - f(x) ) = 0.
   • Report keys: converted, coerced, rejected.
2. check_dim_expr(ds, exprs[]) -> report
   Use: batch-validate dimensional consistency of key expressions.

V. I10-5 Time Axis & Synchronization Cleansing

1. align_timebase(ds, sync_ref) -> ds', timing_report
   • Role: establish tau_mono ↔ ts mapping; estimate and record offset/skew/J.
   • Invariant: non_decreasing(tau_mono).
   • Report keys: offset, skew, J, u(offset), dropped_out_of_window.
2. resample_window(ds, Delta_t, mode) -> ds'
   Use: windowed alignment and aggregation while preserving time semantics.

VI. I10-6 Path & Arrival-Time Cleansing

1. enforce_arrival_time_convention(ds, c_ref, tol_Tarr) -> ds', delta_report
   • Computations:
     1. T_arr_1 = ( 1 / c_ref ) * ( ∫_{gamma(ell)} n_eff d ell )
     2. T_arr_2 = ( ∫_{gamma(ell)} ( n_eff / c_ref ) d ell )
     3. delta_form = | T_arr_1 - T_arr_2 |
   • Assertion: delta_form ≤ tol_Tarr; persist both forms and delta_form.
   • Report keys: count, violations, P99(delta_form).
2. check_path_monotonicity(ds, ell_field) -> report
   Checks: non_decreasing(ell) and computability of L_gamma.

VII. I10-7 Missingness, Mask & Imputation Governance

1. handle_missing(ds, strategy.missing) -> ds', manifest_missing
   • Behavior: generate mask m ∈ {0,1}; drop/impute per rules; record RefCond and uncertainty.
   • Invariant: imputed fields carry explicit provenance and method.
   • Manifest keys: mask_coverage, impute_method, u(imputed).
2. mark_quality(ds, rules) -> ds'
   Use: create or update the components that form q_score ∈ [0,1] (coverage, integrity, timeliness).

VIII. I10-8 Anomalies, Drift & Outlier Governance

1. detect_outlier(ds, method, fields, Delta_t, params) -> tags, report
   • Method examples: zscore, MAD, IQR, robust_lof.
   • Report keys: rate, by_field, suppressed.
2. monitor_drift(ds, ref, method, Delta_t) -> drift, report
   • Use: distribution drift detection (e.g., KS, PSI, ADWIN, CUSUM).
   • Invariant: on threshold breach, do not modify source data; only label and alert.

IX. I10-9 Deduplication, Association & Referential Integrity

1. deduplicate(ds, keys, semantics, tiebreaker) -> ds', dup_report
   • Semantics examples: exact, time_window, fuzzy.
   • Invariant: unique(keys) holds post-op.
   • Report keys: groups, resolved, conflicts.
2. drop_orphan(ds, foreign_key) -> ds', fk_report
   • Role: remove foreign_key orphans and count them.
   • Invariant: foreign_key relations are fully resolvable.

X. I10-10 Compliance, Contracts & Release Freeze

1. assert_contract(ds, tests.*) -> assert_report, pass
   • Coverage: unique, monotone, range, foreign_key, dim, arrival_forms.
   • Invariant: pass = ( violations = 0 ).
2. export_manifest(ds, context) -> manifest
   Contents: version, SRef, timing, units, arrival_forms, asserts, hash, signature.
3. freeze_release(ds, tag) -> manifest
   Role: freeze artifacts, sign, tag, and produce an auditable snapshot.
4. emit_audit(event) -> audit_head
   Use: extend the audit hash chain; event includes who/when/what/hash_prev.

XI. I10-11 Streaming Cleansing & Backpressure Nodes

• thr_node_align_timebase(msg, sync_ref) -> msg'
  Streaming time-alignment node; preserves each message’s TraceID.
• thr_node_detect_outlier(msg, state) -> msg', tags, state'
  Online outlier detection with state updates.
• thr_backpressure_controller(metrics, policy) -> throttle_cmd
  Rule: when rho = lambda / mu > rho_max or burn ≥ burn_hi, emit throttle/degrade commands.

XII. I10-12 Environmental Correction & Arrival-Time Harmonization

• Role: apply corr_env(x; RefCond); recompute both T_arr forms for affected fields.
• Invariant: P99(delta_form) does not increase; report includes before/after.

XIII. I10-13 Density, Probability & Normalization Cleansing

• normalize_density(ds, p_field, domain, measure) -> ds', norm_report
  Assertion: ( ∫_domain p_field d measure ) = 1; persist eps_norm.
• check_mass_conservation(ds, rho_field, V_domain) -> res_mass, report
  Assertion: res_mass = ( ∫_V rho dV ) - M_ref within threshold.
• spectral_energy_check(ds, S_field, band, measure) -> res_energy
  Use: consistency check between spectral density and energy.

XIV. I10-14 Quality Scoring, SLO & Audit

• compute_sli(metrics_source, policy.sli, Delta_t) -> TS.sli.*
  Outputs: lat_ms, err_rate, fresh_age, delta_form, drift quantiles, etc.
• score_quality(TS.sli.*, weights) -> q_score, breakdown
  Formula: q_score = clip( ( ∑ w_i * s_i ) / ( ∑ w_i ), 0, 1 ) or geometric form.
• manage_error_budget(TS.sli.*, policy.slo) -> EB, burn, action
  Behavior: propose hold/limit/rollback actions based on burn.

XV. I10-15 Composite Use-Case Interfaces

• batch_clean_and_freeze(input_glob, tag, policy) -> ds_clean, manifest, audit_head
  Flow: I10-3 → I10-4 → I10-5 → I10-6 → I10-7 → I10-8 → I10-9 → I10-10.
• serve_clean_payload(req, policy, tol_Tarr) -> resp, tags
  In-request cleansing and two-form judgment.
• stream_clean_aggregate(topic_in, Delta_t, rho_max) -> topic_out, topic_quarantine
  Windowed cleansing, assertions, and backpressure loop for streams.
• recompute_arrival_on_drift(ds, threshold) -> ds', delta_report
  Recompute T_arr when drift > threshold.
• publish_quality_panel(q_score, TS.sli.*, EB, burn) -> panel
  Emit quality dashboard and alerts.

XVI. Cross-Interface Invariants & Assertion Checklist

1. Keys & path
   • unique(pk); resolvable foreign_key; non_decreasing(ts|ell).
   • L_gamma = ( ∫_gamma 1 d ell ) is computable.
2. Units & dimensions
   unit(t_arr) = "s", dim(t_arr) = "[T]"; check_dim( y - f(x) ) = 0.
3. Arrival-time two forms
   delta_form = | ( 1 / c_ref ) * ( ∫ n_eff d ell ) - ( ∫ ( n_eff / c_ref ) d ell ) |; delta_form ≤ tol_Tarr.
4. Quality & SLO
   P99(TS.sli.err_rate) ≤ E_target; P99(TS.sli.lat_ms) ≤ L_target; error-budget burn is traceable.
5. Audit & signatures
   Every release carries hash_sha256(blob) and signature, chain-verifiable.

XVII. Mapping to Core Volumes

• EFT.WP.Core.DataSpec v1.0
  Maps I10-3.* to SRef/registry, field lexicon, key constraints, and unit rules.
• EFT.WP.Core.Sea v1.0
  Maps I10-6.* to gamma(ell), n_eff, c_ref, path acquisition and arrival-time semantics.
• EFT.WP.Core.Threads v1.0
  Maps I10-11.* to streaming nodes, thr/chan, backpressure protocol, and throttling commands.
• EFT.WP.Core.Density v1.0
  Maps I10-13.* to measures, domains, and normalization formulas.

XVIII. Versioning & Compatibility Strategy

• Semantic versioning: I10-x.y.z; x denotes breaking change.
• Compatibility window: minor versions are backward-compatible; deprecations observe a two-minor transition.
• manifest.interface_versions must include the set of I10-* versions in use.

XIX. Typical Call Sequences (Summary)

• Batch: standardize_names → repair_units → align_timebase → enforce_arrival_time_convention → handle_missing → detect_outlier → deduplicate → assert_contract → freeze_release.
• Online: standardize_names → repair_units → align_timebase → enforce_arrival_time_convention (micro-batch) → assert_contract (necessary subset).
• Streaming: thr_node_align_timebase → thr_node_detect_outlier → stream_clean_aggregate → assert_contract → backpressure_controller.

Summary
This appendix provides unified signatures, I/O, report keys, and cross-interface invariants for I10-*, and marks each mapping to the Core volumes. With these, practitioners can reuse interfaces across batch, online, and streaming scenarios—ensuring the end-to-end path from schema binding to release freeze remains computable, auditable, and revertible.