14-EFT.WP.Methods.Inference v1.0 | Chapter 4: Data & Feature Interfaces

Home ／ Docs-Technical WhitePaper ／ 14-EFT.WP.Methods.Inference v1.0

Chapter 4: Data & Feature Interfaces

I. Scope & Objectives

1. Establish a unified convention for inference data and feature interfaces, covering input modalities, window alignment, feature processing, dimensional checks, lineage, and drift monitoring. The goal is training–inference consistency and cross-domain portability.
2. Target outputs
   • Field and version requirements for FeatureCard and InferPipelineCard.
   • Minimal equations S42-* and postulates P41-* that govern feature processing (additions introduced in this chapter).
   • Offline/online feature-parity metric delta_fp and the gate gate.inf.feature.
3. Intended readers: feature engineering, data platform, model, and runtime teams. Pass criteria chain to the audit trail with Chapter 3 & Chapter 6 via delta_offon, ECE/NLL.

II. Terms & Symbols

1. Data & time
   • x_raw (raw input stream), x_feat (processed feature vector), y (label), id (entity/session id), ts_event (event time), ts_proc (processing time), window = [t0, t1], lookback, stride, lag_k.
   • tau_mono, ts, alpha, beta (time-base mapping).
2. Processing & checks
   z (standardized value), mu, sigma, epsilon, winsor(a,b), clip(a,b), impute, onehot, emb(W), norm(•), check_dim(expr).
3. Lineage & signatures
   hash(•), fingerprint, EnvLock, anchor, FeatureCard, Lineage.
4. Parity & drift
   delta_fp = ( norm( x_feat_off - x_feat_on ) / norm( x_feat_off ) ), R_infer = 1 - delta_offon, D_KL(p || q) = Σ p_i * ln( p_i / q_i ), W1 (first-order Wasserstein).

III. Postulates & Minimal Equations

1. P41-4 Reproducible features (locked environment & spec)
   With fixed FeatureCard, Graph(theta), and EnvLock, given the same anchor and input x_raw, offline and online must satisfy x_feat_off = x_feat_on. If randomized operators exist, nondet_guard = true still yields equality.
2. P41-5 No-leakage postulate (lookahead = 0)
   Any feature may depend only on information available within window = [t0, t1] with ts_event <= t1. Disallow lead_k (k > 0) and any use of y or its proxies as inputs.
3. S42-5 Standardization & robustness
   z = ( x - mu ) / ( sigma + epsilon ); for the robust form, mu = median(x), sigma = mad(x). Optional winsor(a,b) or clip(a,b) may limit outlier influence.
4. S42-6 Window aggregation & measures
   • Mean: agg_mean = ( 1 / |W| ) * Σ_{t ∈ W} x(t).
   • Exponential moving average: agg_ema = ( Σ w_t * x(t) ) / ( Σ w_t ), with w_t = exp( - lambda * ( t1 - t ) ).
   • Frequency-domain energy (aligned with spectral convention): var( x ) ≈ ( ∫ S_xx(f) df ). Always declare the window and ENBW.
5. S42-7 Feature parity & drift metrics
   • Sample-level parity: delta_fp = ( norm( x_feat_off - x_feat_on ) / norm( x_feat_off ) ).
   • Distribution drift: D_KL( p_off || p_on ), W1( p_off, p_on ), and moment gaps | mu_off - mu_on |, | sigma_off - sigma_on |.
   • Example gates: delta_fp ≤ tau_fp, D_KL ≤ tau_kl, W1 ≤ tau_w1.
6. S42-8 Processing signature & traceability
   fingerprint = hash( FeatureCard || code_rev || anchor || {alpha,beta} || schema ). Every feature release publishes fingerprint and Lineage.

IV. Data & Manifest Conventions

• Per-record minimal field set
  id, ts_event, ts_proc, source, x_raw, x_feat, y (if offline supervised), mask_missing, quality_flag, window_id, partition.
• FeatureCard minimal keys
  name, version, owner, inputs (with source and units), ops (serialized processing operators), params (e.g., mu,sigma,lambda,epsilon), window_spec (lookback,stride), timebase (alpha,beta), impute_policy, checklist (check_dim, range_check, etc.), hash(•), fingerprint.
• Time & window conventions
  All aggregations are measured by ts_event. For cross-device/node comparisons, align time using ts = alpha + beta * tau_mono. Rules for late data inclusion and re-computation windows must be explicit in FeatureCard.window_spec.
• Units & dimensions
  Run check_dim(expr) before publishing every feature. Arguments to log/exp/ln must be dimensionless. For derived quantities (e.g., speed, power), provide the unit-conversion path.

V. Algorithms & Implementation Bindings

1. New prototypes
   • I40-11 build_features(stream:any, card:dict) -> {x_feat:any, qc:dict}
   • I40-12 validate_features(x_feat:any, card:dict) -> {pass:bool, report:dict}
   • I40-13 align_windows(records:any, alpha:float, beta:float, spec:dict) -> records
   • I40-14 compare_feature_parity(off:any, on:any, policy:dict) -> {delta_fp:float, pass:bool}
   • I40-15 monitor_feature_drift(dist_off:any, dist_on:any, metrics:list) -> DriftReport
2. Pseudocode (abridged)
   • Align: records ← align_timebase(records, {alpha,beta}).
   • Window: slice by window_spec and construct context/history.
   • Process: execute impute → transform → aggregate → standardize in the ops order.
   • Validate: validate_features runs check_dim/range/missing and leakage scans.
   • Trace: generate fingerprint and Lineage.

VI. Metrology Flows & Run Diagram

• Mx-45 Feature-spec freeze: produce FeatureCard v*, alpha,beta, impute_policy, and unit conventions; sign fingerprint.
• Mx-46 Offline materialization & report: materialize x_feat_off for train/eval partitions; output {mu,sigma}, a quality report, and baseline histograms.
• Mx-47 Online parity patrol: sample x_feat_on, compute delta_fp and {D_KL, W1}; on breaches, raise TS.error and trigger rollback (see Chapter 10).
• Mx-48 Drift & re-calibration entry: when DriftReport fires, enter Chapter 7 calibration and Chapter 12 re-acceptance.

VII. Verification & Test Matrix

1. Minimum required cases
   • Unit consistency: random sampling through check_dim(expr); assert all pass.
   • Leakage guard: for tasks with y, run feature scans; assert no lead_k (k>0) and no functional dependence on y.
   • Robust standardization: inject outliers; compare z from mean-variance vs. median-MAD.
   • Window endpoints: at t1 boundaries, verify include/exclude policy, timezone, and DST consistency.
   • Offline/online parity: compute delta_fp; assert delta_fp ≤ tau_fp.
   • Spectral convention: for time-series features, verify var( x ) ≈ ( ∫ S_xx(f) df ) and that ENBW configuration matches.
2. Boundary & extreme scenarios
   Late/out-of-order events, batch replays, empty windows, full missingness, ultra-low cardinality classes, cold-start for emb(W), cross-device time stretching with beta ≠ 1.

VIII. Cross-References & Dependencies

• Depend on Core.DataSpec for schema, units, and partitioning; on Core.Metrology for S_xx(f), U_w, ENBW, and error aggregation; on Core.Threads for TS.* metrics and pipeline scheduling semantics.
• Share hash(•), fingerprint, lineage, and de-identification conventions with EFT.WP.Methods.Repro Chapter 4. Share ts alignment and consistency equations with Chapters 3 and 6.

IX. Risks, Limitations & Open Questions

• Recomputations from late/backfilled data pollute historical windows; prefer watermarks with two-phase confirmation windows and record re-materialized window_id.
• Category explosion and rare-class jitter destabilize onehot/emb; configure min_freq and other_bucket, and define a robust cold-start policy.
• Implicit unit changes and timezone drift in upstream sources; lock unit/timezone in FeatureCard and subscribe to upstream change alerts.
• Quantization and device-specific operators subtly affect feature paths (e.g., float16 rounding); tighten tau_fp in parity patrols.

X. Deliverables & Versioning

1. Deliverables
   • FeatureCard/*.json (containing window_spec/ops/params/unit/timebase).
   • FeatureLineage.md (sources, lineage, conventions, fingerprint).
   • FeatureParityReport (delta_fp, D_KL, W1, mu/sigma).
   • QCReport (missingness rates, range violations, dimensional audits).
2. Versioning policy
   • Any change to ops, window_spec, impute_policy, {mu,sigma}, {alpha,beta}, or external schema bumps the minor version and triggers full Mx-47 parity regression.
   • Documentation/visual updates alone do not trigger regression, but must roll the fingerprint and append change records (see Appendix C).