44-EFT.WP.Data.ModelCards v1.0 | Chapter 9 Preprocessing & Feature Engineering

Home ／ Docs-Technical WhitePaper ／ 44-EFT.WP.Data.ModelCards v1.0

Chapter 9 Preprocessing & Feature Engineering

I. Chapter Purpose & Scope

II. Terminology & Dependencies

• Terminology source: Comprehensive Template v0.1; this chapter only adds fields for preprocessing and feature engineering.
• Dependent volumes: Data contract/export Core.DataSpec v1.0; units & dimensional checks Core.Metrology v1.0; path-dependent expressions (if any) Core.Equations v1.1; training data & splits DatasetCards v1.0.
• Math & symbols: Always wrap inline symbols in backticks (e.g., x, z, μ, σ, f_θ(x)); any expression with division/integral/composite operators must use parentheses; no Chinese in formulas/symbols/definitions.

III. Fields & Structure (Normative)

preprocess:

pipeline_id: "<string>" # semantic pipeline identifier

steps: # ordered, idempotent steps

- name: "<clean|filter|normalize|standardize|resample|impute|encode|tokenize|stft|specaugment|feature_map|pca|custom>"

enabled: true

idempotent: true

params: { ... } # explicit; include units/dimensions

inputs: ["<field>"]

outputs: ["<field>"]

notes: "<non-normative>"

feature_space: # feature space (train/infer consistent)

type: "<dense|sparse|sequence|image|audio_spec|tabular|embedding>"

shape: "<(…)>"

dtype: "<float32|int32|...>"

normalization: "<zscore|minmax|robust|unit-norm|none>"

dictionary?: "<path-or-ref>" # tokenizer/subword/category vocab

parameter_lock: true # freeze parameters (incl. statistics)

randomness:

seed: 1701

libraries: {numpy:"1.26.4", torch:"2.3.1"}

environment:

os: "ubuntu22.04"

toolchain: ["python3.11","fftw3"]

containers: ["ghcr.io/eift/model-prep:1.0.2"]

audits: ["nan-check","range-check","leakage","class-imbalance","drift"]

artifacts:

- {path:"preprocess/logs/step-01.jsonl", sha256:"..."}

- {path:"preprocess/configs/lock.yaml", sha256:"..."}

IV. Train/Infer Consistency & Leakage Prevention

• Consistency: Training and inference must use the same pipeline_id/steps and feature_space; statistics (e.g., μ/σ, quantiles, idf) are computed train-only and stored in the lock file.
• Leakage prevention: Do not use validation/test statistics or derived artifacts (including normalization params, vocab/encoding tables, PCA loadings, etc.); record checks and dispositions under audits.
• Frozen-split binding: Use the same splits_ref as in Training Data & Sampling Binding to guarantee replay comparability.

V. Canonical Postures for Common Operations

• Cleaning clean / Filtering filter: Declare policy, thresholds/windows, boundary handling; numeric units pass check_dim.
• Normalization normalize / Standardization standardize: Specify basis (zscore/minmax/robust/unit-norm) and source of statistics (train-only).
• Resampling resample: Declare f_samp and anti-alias filtering; for audio/time series, provide target rate and interpolation method.
• Missing-value handling impute: Mean/median/KNN/model; record impacts on uncertainty (feed the Model Card uncertainty).
• Encoding/Tokenization encode|tokenize: Vocab version, unk/pad policy, max length, truncation/sliding policy.
• Spectrogram & augmentation stft|specaugment: Window/stride, freq/time masks, and RNG seed.
• Feature construction feature_map / Dimensionality reduction pca: Function/kernel, hyperparameters, applicable dims; for PCA, record loading matrix and explained variance ratio.
• Custom custom: Provide script/container reference and parameter hash; list in artifacts and in the export manifest.

VI. Feature Space & I/O Alignment

• Shape/semantics: feature_space.shape/dtype/normalization must match the io_schema inputs from Chapter 6.
• Embeddings/vocab: If inputs are embeddings or sparse encodings, record dimensionality, vocab size, and OOV strategy in feature_space.
• Multimodal: Provide separate sub-spaces per modality and report per-modality impacts in evaluation.

VII. Metrology & Units

1. All parameters with physical/time/frequency quantities declare units in params and pass check_dim via Metrology v1.0.
2. If features or targets involve path quantities (e.g., T_arr), register delta_form, path gamma(ell), and measure d ell, and use one of the two equivalent expressions for consistency checks:
   • T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell )
   • T_arr = ( ∫ ( n_eff / c_ref ) d ell ).

VIII. Machine-Readable Fragment (Drop-in)

preprocess:

pipeline_id: "img-prep-v1"

steps:

- name: "clean"

enabled: true

idempotent: true

params: {policy:"drop-out-of-range", lo:0, hi:255}

inputs: ["raw_image"]

outputs: ["cln_image"]

- name: "standardize"

enabled: true

idempotent: true

params: {type:"zscore", mean:[0.485,0.456,0.406], std:[0.229,0.224,0.225], stats_from:"train-only"}

inputs: ["cln_image"]

outputs: ["std_image"]

- name: "feature_map"

enabled: true

idempotent: true

params: {type:"hog", cell:8, block:2, bin:9}

inputs: ["std_image"]

outputs: ["feat_hog"]

feature_space:

type: "dense"

shape: "(H', W', C')"

dtype: "float32"

normalization: "zscore"

parameter_lock: true

randomness: {seed:1701, libraries:{numpy:"1.26.4"}}

environment: {os:"ubuntu22.04", containers:["ghcr.io/eift/model-prep:1.0.2"]}

audits: ["nan-check","range-check","leakage","drift"]

artifacts:

- {path:"preprocess/configs/lock.yaml", sha256:"..."}

IX. Consistency with Evaluation, Optimization & Hyperparameters

• Evaluation uses the same preprocess and feature_space as training.
• If inference adds lightweight transforms (e.g., pre-quantization normalization), reflect their cost in deployment/resources and explain metric impact & significance in evaluation.

X. Export Manifest & Audit Trail

export_manifest:

artifacts:

- {path:"preprocess/logs/step-*.jsonl", sha256:"..."}

- {path:"preprocess/configs/lock.yaml", sha256:"..."}

- {path:"features/spec.yaml", sha256:"..."}

references:

- "EFT.WP.Core.DataSpec v1.0:EXPORT"

- "EFT.WP.Core.Metrology v1.0:check_dim"

XI. Chapter Compliance Checklist

• pipeline_id/steps, feature_space, parameter locking, and environment fully recorded; train/infer consistent; statistics from train-only.
• Leakage-prevention and audit outcomes recorded and passed; splits_ref matches Training Data & Sampling Binding.
• All parameters with units pass check_dim; for path quantities, delta_form/path/measure are complete and the two T_arr expressions are consistent.
• Export manifest includes logs, lock configs, and feature spec file with sha256; references[] use “Volume vX.Y:Anchor”.