46-EFT.WP.Data.Benchmarks v1.0 | Chapter 10 Runtime Environment & Metrological Load

Home ／ Docs-Technical WhitePaper ／ 46-EFT.WP.Data.Benchmarks v1.0

Chapter 10 Runtime Environment & Metrological Load

I. Chapter Purpose & Scope

II. Terminology & Dependencies

1. Terms: deps_lock, container@digest, power_meter, P_idle/P_load, QPS, T_inf, ρ, p50/p95/p99, pin_core/numa_aware, throttle/limit, cooldown.
2. Dependencies: units & dimensional checks (Core.Metrology v1.0:check_dim); orchestration/scheduling/resources (Pipeline v1.0, Ch.10/12/13); evaluation protocol (ModelCards v1.0, Ch.11); metric system (this volume, Ch.6).
3. Math & symbols: wrap inline symbols; any division/integral/composite operator must use parentheses; for T_arr use
   • T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell ), or
   • T_arr = ( ∫ ( n_eff / c_ref ) d ell ),
     declaring gamma(ell) and d ell. No Chinese in formulas/symbols/definitions.

III. Fields & Structure (Normative)

env:

hardware:

cpu: "16c@3.0GHz | model-id"

mem_gb: 64

gpu: "0|A100-40GB:x1"

storage: {type:"ssd|nvme|hdd", iops:">=50k", size_gb: 512}

network: {nic_gbps: 10}

power_meter: {make:"yokogawa|on-chip", sample_hz: 1}

os: "ubuntu-22.04"

kernel: "linux-6.8"

containers: ["ghcr.io/eift/runner@sha256:<hex>"]

deps_lock: "env.lock"

process:

pin_core: true

numa_aware: true

threads: {workers: 8, threads_per_worker: 2}

limits:

concurrency: 64

rate_limit_qps: 500

memory_gb: 32

caching:

enabled: false

policy: "none|warm|full"

power_policy:

idle_w: null

cap_w: null

IV. Metrological Load & Power Posture

• Performance: report in SI—QPS(1/s), latency_ms.{p50,p95,p99}, utilization ρ; match offline/online windows (1m|5m|15m).
• Power/Energy: specify sample_hz; report P_idle/P_load, average power P_avg(W), and energy per request E_per_req(Wh/1k req); if using on-board meters, note calibration and error bounds.
• Environment deltas: differences across hardware/container/driver versions must be expressed as Δ±CI_95 and included in reports.

V. Concurrency, Quotas & Throttling

• Concurrency/rate: concurrency/rate_limit_qps must be explicit; if they conflict with the evaluation protocol, the protocol prevails and the environment section must restate them.
• Quotas: lock memory/VRAM/storage and network bandwidth limits; record throttle triggers and rejection policies.
• Cooldown: use cooldown for load/scale hysteresis; record enter/exit conditions.

VI. Load Testing & Profiling

• Load testing: step-ramp (ramp) to produce QPS–latency curves; detect bottlenecks via saturation_criteria.
• Profiling: use perf/py-spy/jfr/flamegraph to attribute CPU/memory/IO/network; fix sampling period and rate; export flamegraphs and hotspot lists.
• Repro experiments: provide scripts and params; results must reproduce under the same env.lock and container digest.

VII. Reproducibility & Sensitivity Analysis

• Locks: pin containers@digest, deps_lock, kernel & driver versions.
• Randomness: fix seed and record framework (e.g., MKL/OMP) thread affinity policies.
• Sensitivity: change one factor (CPU SKU/power cap/cache policy), measure Δ, report with CI_95, and annotate on the leaderboard.

VIII. Metrology & Units (SI)

• Mandatory: metrology:{units:"SI", check_dim:true}; use SI for performance/power/energy/volume/network; normalize units first before composing.
• Path quantities: if environment or metrology involves T_arr, register delta_form/path/measure and validate using either equivalence.

IX. Machine-Readable Fragment (Drop-in)

env:

hardware:

cpu: "AmdEPYC-32c@2.8GHz"

mem_gb: 128

gpu: "A100-40GB:x1"

storage: {type:"nvme", iops:">=200k", size_gb:1000}

network: {nic_gbps: 25}

power_meter: {make:"yokogawa", sample_hz:1}

os: "ubuntu-22.04"

kernel: "linux-6.8"

containers: ["ghcr.io/eift/runner@sha256:abcdef..."]

deps_lock: "env.lock"

process: {pin_core:true, numa_aware:true, threads:{workers:16, threads_per_worker:2}}

limits: {concurrency:128, rate_limit_qps:1000, memory_gb:64}

caching: {enabled:true, policy:"warm"}

power_policy: {idle_w:80, cap_w:300}

reporting:

perf: ["QPS","latency_ms.p99","ρ"]

power: ["P_avg","E_per_req"]

window: "1m"

metrology: {units:"SI", check_dim:true}

X. Lint Rules (Excerpt, Normative)

lint_rules:

- id: ENV.CONTAINER_DIGEST

when: "$.env.containers[*]"

assert: "matches('^.+@sha256:[0-9a-f]{64}$')"

level: error

- id: ENV.DEPS_LOCK_PRESENT

when: "$.env.deps_lock"

assert: "value != null"

level: error

- id: ENV.LIMITS_DEFINED

when: "$.env.limits"

assert: "has_keys(concurrency, rate_limit_qps, memory_gb)"

level: error

- id: POWER.METER_DEFINED

when: "$.env.hardware.power_meter"

assert: "has_keys(make, sample_hz)"

level: error

- id: REPORT.WINDOW_ALLOWED

when: "$.reporting.window"

assert: "value in ['1m','5m','15m']"

level: error

- id: METROLOGY.SI_AND_CHECKDIM

when: "$.metrology"

assert: "units == 'SI' and check_dim == true"

level: error

XI. Cross-Reference Anchors

• Scheduling & resources: EFT.WP.Data.Pipeline v1.0, Ch.10/13.
• Online windows & monitoring: EFT.WP.Data.Pipeline v1.0, Ch.12.
• Evaluation protocol: EFT.WP.Data.ModelCards v1.0, Ch.11.
• Metric system & units: this volume, Ch.6; unit & dimension checks: EFT.WP.Core.Metrology v1.0:check_dim.

XII. Chapter Compliance Checklist

• Environment is locked: container @digest, deps_lock, kernel/driver versions, threads & affinity are explicit.
• Concurrency/rate/quotas consistent and non-conflicting with protocol; throttling and cooldown documented.
• Performance & power metrology use SI; QPS/T_inf/ρ/P_avg/E_per_req are complete; composites pass check_dim.
• Load/profiling artifacts (curves, flamegraphs, hotspot lists) exported; sensitivity analyses report Δ±CI_95.
• If T_arr appears, delta_form/path/measure registered and validated.
• Export manifest lists environment & metrology reports with sha256, satisfying release & reproducibility gates.