43-EFT.WP.Data.DatasetCards v1.0 | Chapter 11 Splits & Distribution

Home ／ Docs-Technical WhitePaper ／ 43-EFT.WP.Data.DatasetCards v1.0

Chapter 11 Splits & Distribution

I. Chapter Purpose & Scope

II. Terminology & Dependencies

• Terminology source: EFT.WP.Core.Terms v1.0; this chapter only adds split/distribution-specific fields.
• Dependent volumes: Data contract/export & artifact organization: Core.DataSpec v1.0; metrology/units & dimensional checks: Core.Metrology v1.0; cleaning & sampling constraints see Chapters 6–7; citation anchors/version carrying follow Citation & Cross-Reference Spec v0.1.

III. Fields & Structure (Normative)

splits:

train: {count: <int>, ratio: <0..1>}

validation: {count: <int>, ratio: <0..1>}

test: {count: <int>, ratio: <0..1>}

policy:

leakage_guard: ["per-object","per-timewindow"] # leakage-prevention granularity

stratify_by: ["class","region","snr_bin"] # align with sampling.strata

freeze_indices: true # freeze indices for reproducibility

audit:

coverage: {by:"class", report:true}

leakage: {cross_split:"forbid"}

imbalance: {metric:"gini", threshold: 0.2}

distribution:

packaging:

format: "tgz" # tgz | zip | parquet | zarr | other

shard_bytes: 134217728 # example 128 MiB

layout: ["train","validation","test"]

mirrors: ["https://mirror-a.example/foo/","s3://bucket/foo/"]

rate_limit: {mbps: 50}

regional_compliance: ["EU-GDPR","CN-DSR"] # example only

checksums:

package: {sha256: "<hex>"} # top-level package integrity

shards:

- {path:"train-000.tgz", sha256:"<hex>"}

- {path:"train-001.tgz", sha256:"<hex>"}

see:

- "EFT.WP.Core.DataSpec v1.0:EXPORT"

- "EFT.WP.Core.Metrology v1.0:check_dim"

(Exported artifacts and anchors are recorded and verifiable in the export_manifest.)

IV. Split Definitions & Consistency Constraints

• Ratio sum: splits.train/validation/test.ratio must sum to 1 ± 1e-6; sampling.rates must exactly match these ratios.
• Leakage granularity: The same object, time window, or sequence must not appear across split sets; declare in policy.leakage_guard[].
• Stratification alignment: If stratified sampling is used, stratify_by matches Chapter 6 sampling.strata; record realized per-stratum share and deviation.
• Frozen indices: Before release, freeze and export split indices (file/row/object IDs) to guarantee reproducibility and comparable baselines.

V. Distribution & Artifact Organization

• Packaging: Prefer stream/parallel-friendly formats; if archival (tgz/zip), provide sharding and checksum tables.
• Mirrors & rate: Provide at least two mirror endpoints; publish recommended concurrency and rate limits.
• Regional compliance: For privacy/geosensitive data, state allowed regions and legal bases; align with privacy/ethics extensions.
• Integrity: Provide sha256 for top-level and each shard; consider adding SIZE and LASTMOD.

VI. Linkage to Quality & Baselines

• Set split quality gates in quality.gates (coverage thresholds, leakage = 0, class-imbalance ≤ threshold).
• Report per-class/region/modality coverage and CIs in quality.coverage.
• Baseline evaluation scripts must consume frozen splits indices to ensure comparability.

VII. Metrology & Units (when splitting by time/frequency/space)

• Time windows use UTC and ISO 8601 (e.g., [t0, t1)), with metrology.time_standard fixed.
• Bands/sample-rate splits use Hz; bw_hz and f_samp units must match.
• Spatial splits declare coordinate system and angle unit (default deg, epoch e.g., J2000).

VIII. Export Manifest & References (Normative)

export_manifest:

version: "v1.0"

artifacts:

- {path:"splits/train.index", sha256:"..."}

- {path:"splits/validation.index", sha256:"..."}

- {path:"splits/test.index", sha256:"..."}

- {path:"packages/train-000.tgz", sha256:"..."}

- {path:"packages/train-001.tgz", sha256:"..."}

references:

- "EFT.WP.Core.DataSpec v1.0:EXPORT"

- "EFT.WP.Core.Metrology v1.0:check_dim"

(All artifacts must be listed and verifiable; references carry Volume+Version+Anchor.)

IX. Example Fragment (drop-in)

splits:

train: {count: 12000, ratio: 0.8}

validation: {count: 1500, ratio: 0.1}

test: {count: 1500, ratio: 0.1}

policy:

leakage_guard: ["per-object","per-timewindow"]

stratify_by: ["class","snr_bin"]

freeze_indices: true

distribution:

packaging: {format:"tgz", shard_bytes:134217728, layout:["train","validation","test"]}

mirrors: ["https://mirror-a.example/datasets/foo/","s3://bucket/foo/"]

rate_limit: {mbps: 50}

checksums:

package: {sha256: "…"}

shards:

- {path:"train-000.tgz", sha256:"…"}

- {path:"train-001.tgz", sha256:"…"}

X. Chapter Compliance Checklist

• splits ratios and counts are consistent; sampling.rates aligned; leakage audit passes; stratification deviation within thresholds.
• distribution provides mirrors, sharding, and sha256 integrity; regional compliance matches privacy/ethics.
• export_manifest lists split indices and packages/shards with anchors; no shortcodes/aliases; version and anchor required.