45-EFT.WP.Data.Pipeline v1.0 | Chapter 1 Overview & Scope

Home ／ Docs-Technical WhitePaper ／ 45-EFT.WP.Data.Pipeline v1.0

Chapter 1 Overview & Scope

I. Chapter Purpose & Audience

• Purpose: Establish the role of data pipelines in the EFT data stack, the minimum compliance requirements and applicability boundaries; define how pipelines interface with Dataset Cards, Model Cards, metrology, and citation anchors, and fix machine-readable deliverables and release gates.
• Audience: Data/platform engineers, feature & modeling engineers, MLOps/DevOps, quality & compliance owners, audit and reproducibility operators.

II. Terminology & Citation Posture

• Terminology source: Follow EFT Technical Whitepaper & Notes — Comprehensive Template v0.1; this volume only adds pipeline-specific increments (e.g., stage/operator, contract/schema, DQ gate, lineage, SLA/SLO, orchestrator).
• Citation format: Cross-volume citations must carry “Volume vX.Y: Chapter/Anchor”, preferably clause-level P/S/M/I anchors.
• Math & symbols: Wrap all inline symbols with backticks (e.g., QPS, T_inf, f_samp, T_arr); any expression with division/integral/composite operators must use parentheses and explicitly declare path gamma(ell) and measure d ell; no Chinese in formulas/symbols/definitions.

III. In Scope

1. Objects: Normative requirements and engineering practices for end-to-end pipelines from source → validate → transform → feature → distribute → monitor, including:
   • Layering & topology (layers[]/edges[]) and contracts (Σ_in/Σ_out);
   • Data sources & ingest, schema/contract management, data validation & DQ gates;
   • Transform & preprocessing, feature pipelines & reuse;
   • Sampling, splits & distribution; orchestration & scheduling, resources & SLAs;
   • Versioning & lineage, monitoring & observability, performance & cost, privacy/security/compliance;
   • Machine-readable Schema & Lint, implementation binding & execution APIs, templates & examples.
2. Relation to Dataset/Model Cards: The pipeline is the normative production process; data facts and splits reference EFT.WP.Data.DatasetCards v1.0, feature/I-O assumptions reference EFT.WP.Data.ModelCards v1.0.

IV. Out of Scope

V. Deliverables & Compliance Gate

1. Deliverables:
   • pipeline.yaml (or JSON) — the complete pipeline specification;
   • pipeline.schema.json and lint_rules.yaml — machine-readable validation and blocking rules;
   • export_manifest — includes version, references[], and artifact sha256;
   • Audit artifacts: DQ reports, lineage graph, runtime metrics, and replay logs.
2. Minimum compliance (must pass before release):
   • Required fields complete; type/regex/dependency checks pass;
   • Metrology check units="SI" & check_dim=true;
   • Frozen splits and leakage guardrails in place;
   • Citations use “Volume+Version+Anchor”, no shortcodes/aliases;
   • DQ, privacy, and regional-compliance checks pass.

VI. Document Structure & Cross-Volume Dependency Map

1. Structure map:
   • Ch.3–Ch.5: layering & overview; sources & ingest; schema/contracts;
   • Ch.6–Ch.8: data validation & DQ; transform & preprocessing; feature pipelines;
   • Ch.9–Ch.10: sampling/splits/distribution; orchestration/scheduling/resources;
   • Ch.11–Ch.13: versioning & lineage; monitoring/logging/observability; performance/cost/scale;
   • Ch.14–Ch.15: privacy/security/compliance; fault-tolerance/recovery/DR;
   • Ch.16–Ch.18: machine-readable Schema/Lint; execution APIs; templates.
2. Dependency constraints:
   • Data contracts & exports: Core.DataSpec v1.0;
   • Units/dimensions: Core.Metrology v1.0;
   • Data/Model Cards: DatasetCards v1.0, ModelCards v1.0 respectively.

VII. Naming & Field Style

• Naming: Keys use snake_case. Reserved pipeline keys: pipeline.id/version/layers/edges/resources/scheduling/quality_gates/export_manifest/metrology, etc.; their semantics must not be repurposed.
• Conflict enforcement: T_fil vs. T_trans must not be mixed; n vs. n_eff strictly distinguished; no Chinese in formulas/symbols/definitions.

VIII. Machine-Readable & Validation Interfaces (Overview)

• Schema & Lint: Provide pipeline.schema.json and lint_rules.yaml (blockers: anchor regex, frozen splits, metrology checks, idempotency/retry/timeout, leakage guardrails, etc.).
• Execution & validation APIs: /pipelines/validate|plan|run|metrics|lineage; fixed auth, idempotency and rate limits; unified request/response envelope; return shape {"ok":bool,"errors":[],"warnings":[],"metrics":{...}}.

IX. Quality, Reproducibility & Audit

• Quality gates: schema compliance, DQ pass, coverage with CIs, privacy compliance & access control, SLAs & alerts.
• Reproducibility: version-lock sources/config/containers/environment and artifact hashes; replay logs with shadow comparisons.
• Audit trail: versioning & citation anchors, lineage graphs, DQ reports & runtime metrics, artifact hashes are all traceable in exports.

X. Usage & Maintenance

• Usage: For internal and external purposes, the pipeline specification is the single source of truth; cross-volume citations reference only stabilized lines (e.g., v1.*).
• Maintenance: When layers/operators/contracts or dependencies change, release a new version per this volume’s “Versioning & Lineage”, and reflect citation deltas & notices in the export_manifest.