06-EFT.WP.Core.DataSpec v1.0 | Preface

Home ／ Docs-Technical WhitePaper ／ 06-EFT.WP.Core.DataSpec v1.0

Preface

I. Scope and Objectives

1. Define a minimal closed loop from D (dataset) to S (schema) to contract, standardizing field semantics, units and dimensions, time and path annotations, traceability and versioning—so that cross-system and cross-volume work is consistent and reproducible.
2. Objectives:
   • Map data elements to verifiable metrological semantics—unit(•), dim(•), check_dim(•)—to eliminate same-name/different-meaning ambiguity.
   • Constrain producer/consumer boundaries with schemas and contracts to yield a data pipeline that is failable, diagnosable, and traceable.
   • Provide unified fields and a consistent wrapper for path-dependent quantities (e.g., T_arr) with both canonical formulations.
3. Applicability: experimental / simulation / operational data; row- and column-oriented storage; offline and streaming settings.

II. Readers and Reading Path

1. Roles:
   • Producers (acquisition, simulation, services): focus on field registration, contract validation, and trace anchoring.
   • Consumers (modeling, analysis, visualization): focus on unit/dimension consistency, resampling and windows, missingness and drift.
   • Platform & governance: focus on version compatibility, release freezes, privacy retention, and quality gates.
2. Suggested path:
   • Chapters 1–2: establish naming and schemas.
   • Chapter 4: operationalize contracts and quality gates.
   • Chapters 7–8: version/change management and drift.
   • Chapter 10: cross-volume use cases (datasets for T_arr and gamma(ell)).

III. Design Principles and Non-Negotiables

1. Semantics first: any field that participates in equations must declare unit(field_i) and dim(field_i) and pass check_dim( y - f(x; theta) ).
2. Environment explicit: any quantity requiring correction is written corr_env(x; RefCond) and must record RefCond.
3. Time ordered: time series satisfy non-decreasing ts; resampling must declare method and Delta_t.
4. Path consistent: datasets for arrival-time must include pid, non-decreasing ell, CRS, and L_gamma = ( ∫_gamma 1 d ell ).
5. Missing explicit: denote missing with m = 0; do not use dummy values.
6. Version closure: schema_version follows semantic versioning; breaking changes require major+1 and a published diff plus compatibility layer.
7. Two T_arr forms coexist and must reconcile:
   • Constant-factored: T_arr = ( 1 / c_ref ) * ( ∫_gamma n_eff d ell )
   • General: T_arr = ( ∫_gamma ( n_eff / c_ref ) d ell )
   • Datasets must expose delta_form:
     delta_form = | ( 1 / c_ref ) * ( ∫_gamma n_eff d ell ) - ( ∫_gamma ( n_eff / c_ref ) d ell ) |.

IV. Relation to the Companion White Papers

• Core.Metrology: reuse L0, t0, c_ref conventions and unit(•), dim(•), check_dim(•); keep arrival-time two-form and the gamma(ell) measure d ell identical in expression.
• Core.Equations: bind fields to Sxx-* via bind_to_equations(•) to ensure dimensional closure and symbol alignment.
• Core.Parameters: include parameter sets in the dataset manifest and support bind_to_parameters(•) to create “data + parameters” snapshots.
• Core.Errors: share error domains and coding for contract failures, drift, and quality alerts; residuals and weight functions may be reused in data-quality scoring.
• Cross-volume citation uses the fixed form: “see companion white paper Energy Threads, Chapter x, S/P/M/I…”.

V. Numbering System and Compliance Levels

1. Numbering: postulates P6x-*; minimal equations S6x-* (data mappings); data workflows M6-*; implementation bindings I60-*.
2. Compliance levels:
   • Level-1 (required): unique pk, complete unit/dim, time/path rules satisfied, manifest and Trace reproducible.
   • Level-2 (recommended): contracts cover ≥ 90% of fields; quality metrics and drift monitoring in place; release freezes are traceable.
   • Level-3 (preferred): dual-form T_arr co-reported, delta_form controlled, automated cross-volume checks pass.

VI. Core Concepts and the Data-Contract Triplet

1. Triplet <schema S, contract C, manifest M>
   • S: names, types, unit(•), dim(•), nullability, and indices.
   • C: executable assertions (unique / range / regex / cross-field) and failure handling.
   • M: production context (RefCond, CRS, versions, provenance, fingerprints, and signatures).
2. Trace chain: Trace = [source -> method -> artifact] together with hash_sha256(blob) and signature composes the evidence chain.

VII. Minimal Compliance Checklist (Extract)

• P61-1 Unique primary key: ∀ r_i ≠ r_j : pk(r_i) ≠ pk(r_j).
• P61-2 Dimensional closure: fields in equations must pass check_dim.
• P61-3 Ordered time: non-decreasing ts; resampling declares Delta_t and method.
• P61-4 Path consistency: pid, non-decreasing ell, CRS, and complete L_gamma.
• P61-5 Missing explicit: mark with m ∈ {0,1}.
• P61-6 Version closure: semantic schema_version with a published diff.

VIII. Shared Terms and Symbol Anchors (Cross-Volume)

• Data & schema: D, S, R, N = |R|, field_i, type(field_i), unit(field_i), dim(field_i), nullable(field_i), manifest, schema_version.
• Keys & indices: rid, uid, sid, tid, pid, pk, idx_k.
• Time–space & windows: ts, t0, t1, Delta_t, fs, x,y,z, lon,lat,alt, CRS.
• Path & arrival time: gamma(ell), ell, L_gamma = ( ∫_gamma 1 d ell ), T_arr, n_eff(x,t), c_ref, d ell.
• Metrology & uncertainty: unit(•), dim(•), check_dim(•), u(x), U = k * u_c.
• Quality & missingness: m ∈ {0,1}, q_score ∈ [0,1], drift.
• Trace & formats: hash_sha256(blob), signature, Trace, fmt ∈ {"jsonl","csv","parquet","nc","tfrecord"}.

IX. Out of Scope

X. Implementation Bindings — Preview

• I60 1–10 provide unified prototypes for schema registration; contract validation; serialization and I/O; traceability and fingerprints; partitioning and indexing; quality and drift; versioning and change; privacy and governance; cross-volume interop; manifest import/export.
• In Chapter 10, readers can use enforce_arrival_time_convention(ds) to verify T_arr two-form consistency and bind end-to-end via bind_to_equations(ds, eqn_refs) and bind_to_parameters(ds, params).