46-EFT.WP.Data.Benchmarks v1.0 | Chapter 5 Data Sources, Sampling & Frozen Splits

Home ／ Docs-Technical WhitePaper ／ 46-EFT.WP.Data.Benchmarks v1.0

Chapter 5 Data Sources, Sampling & Frozen Splits

I. Chapter Purpose & Scope

II. Terminology & Dependencies

• Terms: dataset_ref, sampling.strategy/strata/weights, splits.train/val/test, freeze_indices, leakage_guard, items_ref, index.
• Dependencies: data contracts & exports (Core.DataSpec v1.0); units & dimensions (Core.Metrology v1.0); splits/coverage & DQ (DatasetCards v1.0); evaluation protocol (ModelCards v1.0); distribution & mirrors (Pipeline v1.0).
• Math & symbols: wrap inline symbols (e.g., QPS, T_inf, ρ, ψ) with backticks; any division/integral/composite operator must use parentheses; if path quantity T_arr is involved, declare gamma(ell) and d ell; no Chinese in formulas/symbols/definitions.

III. Fields & Structure (Normative)

data:

dataset_ref: "datasets/<name>@vX.Y" # reference, do not copy

sources: ["<uri-or-citation>", "..."] # data sources & citations

licensing: "CC-BY-4.0|ODC-BY|custom"

provenance:

collection_window: "<YYYY-MM-DD..YYYY-MM-DD>"

geography: ["<region>"]

permits: ["<ethics/permit-ref>"]

sampling:

strategy: "random|stratified|time-based|spatial-tiles|systematic"

strata: [{by:"<label|locale|domain|difficulty|snr_bin>", buckets: {"A":100,"B":200}}]

weights: {class:"inverse_freq|none"} # training re-weighting statement

seed: 1701

splits:

train: {frozen:true, index:"splits/train.index", sha256:"<hex>"}

val: {frozen:true, index:"splits/val.index", sha256:"<hex>"}

test: {frozen:true, index:"splits/test.index", sha256:"<hex>"}

ratio: {train:0.8, val:0.1, test:0.1}

freeze_indices: true

leakage_guard:

policy: ["per-object","per-timewindow","per-scene"]

audits:

report: "splits/leakage_report.csv"

sha256: "<hex>"

IV. Source Compliance & Citation Posture

• Reference only, no copying: all data facts are cited via dataset_ref and sources[]; record license and ethics-permit anchors in the export manifest.
• Window & geography: specify collection_window/geography for risk and shift assessment; reconcile with the coverage matrix.
• License consistency: align with distribution/mirroring strategy; if restricted fields exist, synchronize constraints in tasks.

V. Sampling Strategy & Stratification

• Strategies: random (global uniform), stratified (by class/region/locale/difficulty/SNR), time-based (sliding/rolling), spatial-tiles (spatial tiling), systematic (fixed-step).
• Strata: strata[] must enumerate buckets and quotas; report realized counts and deviation (%) after sampling, and provide seed.
• Re-sampling/Re-weighting: when training uses weights, record the policy and justify its impact/significance in the Model Card evaluation section.

VI. Frozen Splits & Consistency

• Ratio sum: ratio.train + ratio.val + ratio.test = 1±1e-6.
• Frozen indices: set freeze_indices=true; fix train/val/test by index files (row/file/object IDs); all coverage statistics and evaluations must use frozen indices.
• Cross-volume consistency: Model Card evaluations must use these frozen indices; Pipeline distribution/mirrors consume the same indices; Dataset Card frozen splits align with them.

VII. Leakage Prevention & Audit Exports

• Granularity: per-object|per-timewindow|per-scene; the same object/adjacent windows/same scene must not span split sets.
• Blocking: any detected leakage is blocking.
• Audit: produce splits/leakage_report.csv with summary metrics (overlap counts/ratios, per-granularity breakdown) and register sha256.

VIII. Metrology & Units (SI)

1. Performance & volume: QPS(1/s), T_inf(ms), ρ(—), net_mbps, size_bytes.
2. Mandatory: metrology:{units:"SI", check_dim:true}; normalize units first before composing derived quantities.
3. Path quantities (T_arr): if sampling/splits couple to path-dependent quantities, register delta_form, path="gamma(ell)", measure="d ell", and use
   • T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell ), or
   • T_arr = ( ∫ ( n_eff / c_ref ) d ell ),
     with check_dim.

IX. Machine-Readable Fragment (Drop-in)

data:

dataset_ref: "datasets/core_cls@v1.0"

sources: ["doi:10.1234/core-ds", "arXiv:2501.01234"]

licensing: "CC-BY-4.0"

provenance: {collection_window:"2024-01-01..2025-06-30", geography:["EU","US"], permits:["ethics-IRB-2024-09"]}

sampling:

strategy: "stratified"

strata: [{by:"label", buckets:{"A":520,"B":2100,"C":12380}}]

weights: {class:"inverse_freq"}

seed: 1701

splits:

train: {frozen:true, index:"splits/train.index", sha256:"..."}

val: {frozen:true, index:"splits/val.index", sha256:"..."}

test: {frozen:true, index:"splits/test.index", sha256:"..."}

ratio: {train:0.8, val:0.1, test:0.1}

freeze_indices: true

leakage_guard:

policy: ["per-object","per-timewindow"]

audits: {report:"splits/leakage_report.csv", sha256:"..."}

metrology: {units:"SI", check_dim:true}

X. Lint Rules (Excerpt, Normative)

lint_rules:

- id: DATA.REF_FORMAT

when: "$.data.dataset_ref"

assert: "matches('^datasets/[a-z0-9_\\-]+@v\\d+\\.\\d+$')"

level: error

- id: SAMPLE.STRATEGY_ALLOWED

when: "$.sampling.strategy"

assert: "value in ['random','stratified','time-based','spatial-tiles','systematic']"

level: error

- id: SPLITS.RATIO_SUM

when: "$.splits.ratio"

assert: "abs(value.train + value.val + value.test - 1) <= 1e-6"

level: error

- id: SPLITS.FROZEN_REQUIRED

when: "$.splits"

assert: "splits.train.frozen and splits.val.frozen and splits.test.frozen and splits.freeze_indices == true"

level: error

- id: LEAKAGE.GUARD_PRESENT

when: "$.leakage_guard.policy"

assert: "contains_any(['per-object','per-timewindow','per-scene'])"

level: error

- id: AUDIT.REPORT_HASH

when: "$.leakage_guard.audits"

assert: "has_keys(report, sha256)"

level: error

- id: METROLOGY.SI_AND_CHECKDIM

when: "$.metrology"

assert: "units == 'SI' and check_dim == true"

level: error

XI. Cross-Reference Anchors

• Data sources & contracts: EFT.WP.Core.DataSpec v1.0:EXPORT.
• Frozen splits & distribution: EFT.WP.Data.DatasetCards v1.0, Ch.11; EFT.WP.Data.Pipeline v1.0, Ch.9.
• Evaluation protocol & metrics: EFT.WP.Data.ModelCards v1.0, Ch.11.
• Unit & dimension checks: EFT.WP.Core.Metrology v1.0:check_dim.

XII. Chapter Compliance Checklist

• dataset_ref/sources/licensing/provenance are explicit and reference-only; no duplication of data facts.
• sampling.strategy/strata/weights/seed present; sampling deviations reported.
• splits ratios sum to 1±1e-6; freeze_indices=true; index files carry sha256.
• Leakage guardrails enabled and audit exported; any leakage is blocking.
• SI metrology with check_dim=true; if T_arr appears, delta_form/path/measure registered and validated.
• Export manifest lists index and leakage-report artifacts and anchors with sha256, satisfying release gates.