46-EFT.WP.Data.Benchmarks v1.0 | Chapter 7 Evaluation Protocol (Offline/Online/Streaming/Interactive)

Home ／ Docs-Technical WhitePaper ／ 46-EFT.WP.Data.Benchmarks v1.0

Chapter 7 Evaluation Protocol (Offline/Online/Streaming/Interactive)

I. Chapter Purpose & Scope

II. Terminology & Dependencies

1. Terms: mode (offline/online/stream/interactive), seed, repeats, temperature, context_length, rounds, canary/shadow, traffic_allocation, caching, tools_allowed, retrieval/open_book, runtime_limits, concurrency, rate_limit.
2. Dependencies: frozen splits & distribution (DatasetCards v1.0, Ch.11), evaluation & aggregation (ModelCards v1.0, Ch.11), monitoring & online windows (Pipeline v1.0, Ch.12), units & dimensions (Core.Metrology v1.0:check_dim).
3. Math & symbols: wrap inline symbols; any division/integral/composite operator must use parentheses; for path quantity T_arr use
   • T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell ), or
   • T_arr = ( ∫ ( n_eff / c_ref ) d ell ),
     declaring gamma(ell) and d ell. No Chinese in formulas/symbols/definitions.

III. Fields & Structure (Normative)

protocol:

mode: "offline|online|stream|interactive"

seed: 1701

repeats: 5

temperature: 0.0

max_tokens: 0

context:

length: 4096

template_ref: "prompts/<id>@vX.Y"

tools:

allowed: false

retrieval: false

open_book: false

registry_ref: null

runtime_limits:

timeout_s: 3600

memory_gb: 16

execution:

concurrency: 8

rate_limit_qps: 50

batching: {enabled: true, max_batch: 32}

caching: {enabled: false, policy: "none|warm|full"}

stream:

window_ms: 1000

hop_ms: 250

max_latency_ms: 200

watermark: "event_time|processing_time"

interactive:

rounds: 3

turn_timeout_s: 30

max_context_turns: 8

online:

traffic_allocation: {control: 0.5, treatment: 0.5}

exposure: {shadow: true, canary: 0.05}

guardrails: ["latency_ms.p99<=200","error_rate<=0.01"]

logging:

format: "jsonl"

fields: ["ts","task_id","item_id","run_id","trace_id","input_hash","output_hash","latency_ms","error_code"]

retention: "P30D"

reporting:

metrics: ["F1_macro","ECE","latency_ms.p99","QPS"]

target_ci: {method: "bootstrap", level: 0.95}

see:

- "EFT.WP.Data.ModelCards v1.0:Ch.11"

- "EFT.WP.Core.Metrology v1.0:check_dim"

IV. Protocol Modes

• Offline: fix seed/repeats; use frozen S_test only; external knowledge and tools are forbidden unless the track declares otherwise; logs and results must be fully replayable.
• Online: declare traffic_allocation, shadow/canary, and guardrails; online window window ∈ {1m,5m,15m} aligns with Pipeline Ch.12; guardrail breach is blocking and must rollback.
• Streaming: fix window_ms/hop_ms/watermark and end-to-end max_latency_ms; stream metrics aggregate per window.
• Interactive: define rounds/turn_timeout_s/max_context_turns; fix prompt templates and state-update policy; record latency and results per round.

V. Tracks & Resource Constraints

• Track switching: closed-book|open_book|tools; switching must update tools/retrieval/open_book, execution.caching, and runtime_limits; open-book must declare knowledge sources and interface latency budgets.
• Resources: SI-only—QPS(1/s), latency_ms.{p50,p95,p99}, memory_gb; normalize units first; check_dim=true.

VI. Statistics & Significance

• Repeats & intervals: repeats ≥ 1; report CI_95 and method; for online A/B, provide power or minimum detectable effect (Δ).
• Multiple comparisons: apply corrections (e.g., Holm–Bonferroni) at suite-aggregation level and record them.

VII. Machine-Readable Fragments (Drop-in)

# Offline protocol example

protocol:

mode: "offline"

seed: 1701

repeats: 5

temperature: 0.0

context: {length: 4096, template_ref: "prompts/qa_v1@v1.0"}

tools: {allowed: false, retrieval: false, open_book: false}

runtime_limits: {timeout_s: 3600, memory_gb: 16}

execution: {concurrency: 8, rate_limit_qps: 50, batching:{enabled:true, max_batch:32}}

logging: {format:"jsonl", fields:["ts","task_id","item_id","run_id","latency_ms"], retention:"P30D"}

reporting: {metrics:["F1_macro","ECE"], target_ci:{method:"bootstrap", level:0.95}}

see: ["EFT.WP.Data.ModelCards v1.0:Ch.11","EFT.WP.Core.Metrology v1.0:check_dim"]

# Online protocol example

protocol:

mode: "online"

seed: 1701

repeats: 1

online:

traffic_allocation: {control: 0.5, treatment: 0.5}

exposure: {shadow: true, canary: 0.05}

guardrails: ["latency_ms.p99<=200","error_rate<=0.01"]

execution: {concurrency: 64, rate_limit_qps: 500, batching:{enabled:false}}

logging: {format:"jsonl", fields:["ts","trace_id","latency_ms","error_code"], retention:"P30D"}

reporting: {metrics:["QPS","latency_ms.p99"], target_ci:{method:"t", level:0.95}}

see: ["EFT.WP.Data.Pipeline v1.0:Ch.12","EFT.WP.Core.Metrology v1.0:check_dim"]

VIII. Lint Rules (Excerpt, Normative)

lint_rules:

- id: PROTOCOL.MODE_ALLOWED

when: "$.protocol.mode"

assert: "value in ['offline','online','stream','interactive']"

level: error

- id: PROTOCOL.SEED_REPEATS

when: "$.protocol"

assert: "has_key(seed) and (mode != 'online' -> has_key(repeats))"

level: error

- id: PROTOCOL.FROZEN_SPLITS_REQUIRED

when: "$.splits"

assert: "splits.train.frozen and splits.val.frozen and splits.test.frozen"

level: error

- id: PROTOCOL.TOOLS_TRACK_CONSISTENCY

when: "$.protocol.tools"

assert: "value.allowed == false or has_key($.tracks)"

level: error

- id: ONLINE.TRAFFIC_SUM

when: "$.protocol.online.traffic_allocation"

assert: "abs(value.control + value.treatment - 1) <= 1e-6"

level: error

- id: STREAM.WINDOW_PARAMS

when: "$.protocol.mode == 'stream'"

assert: "has_keys($.protocol.stream.window_ms, $.protocol.stream.hop_ms, $.protocol.stream.max_latency_ms)"

level: error

- id: INTERACTIVE.ROUNDS_DEFINED

when: "$.protocol.mode == 'interactive'"

assert: "has_keys($.protocol.interactive.rounds, $.protocol.interactive.turn_timeout_s)"

level: error

- id: METROLOGY.SI_AND_CHECKDIM

when: "$.metrology"

assert: "units == 'SI' and check_dim == true"

level: error

IX. Cross-Reference Anchors

• Tasks & I/O: EFT.WP.Data.ModelCards v1.0, Ch.6/Ch.11.
• Metrics & windows: EFT.WP.Data.Benchmarks v1.0, Ch.6; EFT.WP.Data.Pipeline v1.0, Ch.12.
• Frozen splits & leakage: EFT.WP.Data.DatasetCards v1.0, Ch.11.
• Units & dimensions: EFT.WP.Core.Metrology v1.0:check_dim.

X. Chapter Compliance Checklist

• Fields mode/seed/repeats/temperature/context/tools/runtime_limits/execution complete and consistent with task/track constraints.
• Offline protocol uses frozen splits only; online protocol traffic shares sum to 1 and guardrails active with rollback on breach.
• Streaming protocol fixes window/hop/watermark; interactive protocol fixes rounds/timeout; logging fields and retention explicit.
• Metrics and target_ci/significance methods explicit; consistent with Chapter 6 units & aggregation posture.
• SI metrology with check_dim=true; if T_arr appears, delta_form/path/measure registered and validated.
• Machine-readable fragments are drop-in and lint-clean; export_manifest.references[] use “Volume vX.Y:Anchor.”