13-EFT.WP.Methods.SimStack v1.0 | Chapter 9: Benchmark Suites & Performance Calibration

Home ／ Docs-Technical WhitePaper ／ 13-EFT.WP.Methods.SimStack v1.0

Chapter 9: Benchmark Suites & Performance Calibration

I. Scope & Objectives

• Define a reproducible family of benchmarks with a scoring system that covers micro-kernels, system-level, and cross-scale scenarios, unifying input conditions, metrics, and reporting conventions.
• Objective: evaluate correctness, stability, SLO, and throughput with manifest-backed evidence, then use results for parameter calibration and controlled A/B studies. All temporal quantities are recorded on tau_mono and published as ts. The two T_arr formulations are always reported in parallel.

II. Terms & Symbols

1. Benchmarks & conditions
   B.id (benchmark identifier), cfg (configuration), seed (random seed), cap (resource ceiling), quota(k) (tenant quota).
2. Scores & metrics
   • S_total (overall score), S_cor (correctness), S_stb (stability), S_slo (SLO), S_thr (throughput).
   • TS.latency.p99, TS.throughput.rps, TS.util.*, TS.hb.violations.
3. Conservation & two-form
   • eps_mass, eps_norm, eps_psd, T_arr.general, T_arr.factorized, delta_form.
   • Paths & measures: gamma(ell), d ell.
4. Time base & windows
   warmup (preheating duration), W (statistics window), U_w, ENBW.

III. Postulates & Minimal Equations (P61-/S62-)

• P61-18 (Reproducibility & environment locking)
  Any publication for a given B.id must persist cfg, seed, container image, and dependency fingerprints; hidden mutables are disallowed.
• P61-19 (Fair comparison)
  Comparisons are valid only under identical cap, quota(k), placement/policy, and gamma(ell) conventions. Any change must be a single-factor experiment.
• P61-20 (Conservation gates first)
  If any of eps_mass, eps_norm, eps_psd, or delta_form exceeds its gate, the run is unscored.
• S62-60 (Score aggregation: weighted geometric mean)
  S_total = ( S_cor^w1 * S_stb^w2 * S_slo^w3 * S_thr^w4 )^( 1 / ( w1 + w2 + w3 + w4 ) ), with w1 ≥ w2 ≥ w3 ≥ w4 recommended.
• S62-61 (Correctness score)
  For any QoI: e_rms = sqrt( mean( ( Q - Q_ref )^2 ) ),
  S_cor = max( 0 , 1 - e_rms / gate(Q) ).
• S62-62 (Stability score)
  cv = std(Q) / max( ε , mean(Q) ) over window W,
  S_stb = 1 / ( 1 + cv / target_cv ).
• S62-63 (SLO score)
  S_slo = min( 1 , target_p99 / TS.latency.p99 ) * I( TS.hb.violations = 0 ), with I an indicator.
• S62-64 (Throughput score)
  Resource-normalized throughput: thr_norm = TS.throughput.rps / cost(res),
  S_thr = min( 1 , thr_norm / target_thr ).
• S62-65 (Warmup and truncation)
  Statistics use interval [warmup , T_end - cooldown], with spectral quantities corrected by U_w and ENBW.

IV. Benchmark Manifest & Input Conditions

1. Micro-kernels (analytic or quasi-analytic references)
   • B-μ1: Path-integral kernel — compute T_arr.general and T_arr.factorized, report delta_form and the path manifest gamma(ell).
   • B-μ2: Conservation kernel — in volume V, evaluate M(t) = ( ∫_V rho dV ) and flux closure; report eps_mass.
   • B-μ3: Spectral consistency kernel — generate time series from S_xx(f); validate eps_psd and U_w/ENBW conventions.
2. System-level (end-to-end thread network)
   • B-S1: Bounded-backpressure pipeline — fixed G=(V,E) and req(v); measure TS.latency.p99, TS.queue.backlog.
   • B-S2: Network-aware placement — switch placement policy; compare CommCost and S_total.
   • B-S3: Elastic scaling — trigger autoscale; evaluate oscillation and TS.hb.violations.
3. Cross-scale (continuous × discrete coupling)
   • B-X1: CFL adaptive loop — I60-1 couple_fields + I60-3 advance_dt; evaluate stability and correctness.
   • B-X2: Arrival-time calibration closed loop — I60-4 calibrate_arrival_time with observation alignment; assess delta_form and σ_ts propagation.
   • B-X3: Multi-path gamma(ell) composition — compare equivalence handling vs. explicit enumeration.

V. Metric Family & Scoring Conventions

• Correctness
  Micro-kernels use analytic/high-fidelity Q_ref; system-level uses baseline versions as Q_ref; cross-scale always reports both T_arr forms plus conservation evidence.
• Stability
  Compute cv over window W and publish both ts and tau_mono variants; spectral metrics apply U_w/ENBW corrections.
• SLO
  At minimum include TS.latency.p99, TS.sli.success_rate, TS.hb.violations.
• Throughput
  Report TS.throughput.rps and thr_norm, together with the resource bill cost(res).

VI. Calibration Flow & A/B Procedure (Mx-64 benchmark-report)

1. Entrance conditions
   Pass Mx-61 (time calibration) and Mx-62 (conservation checks). Lock cfg/seed/cap/quota and placement policy.
2. Flow
   • Warmup & truncation: run to warmup, then collect window W in steady state.
   • Micro-kernel correctness: compute S_cor and eps_*.
   • System-level SLO: measure TS.latency.p99, TS.queue.backlog, compute S_slo.
   • Throughput & cost: collect TS.throughput.rps, TS.util.*, and cost(res); compute S_thr.
   • Stability: compute S_stb and eps_psd.
   • Aggregate score: output S_total per S62-60.
   • Controlled A/B: change exactly one factor, repeat steps 1–6, and produce Δmetrics with confidence intervals.
3. Artifact
   BenchmarkReport = { B.id , cfg , cap , quota , warmup , W , S_* , eps_* , TS.* , audit.trail }.

VII. Operating Rules & Fairness

• Resources & quotas
  Fix cap and quota(k); for GPUs, record MIG or exclusive mode; persist NUMA affinity.
• Placement & policy
  Keep placement/policy fixed; when comparing placement, vary only that factor.
• Time base & randomness
  Fix seed. Collect all temporal quantities on tau_mono and publish ts with alpha/beta mapping.
• Observability & overhead
  Observation channels must not exceed the defined cap; include obs_overhead in cost(res).

VIII. Report & Publication Conventions

1. Report structure
   • Summary: B.id, version, and environment fingerprints.
   • Inputs: cfg/seed/cap/quota/placement.
   • Results: S_cor/S_stb/S_slo/S_thr/S_total, TS.*, eps_*, delta_form.
   • Evidence: gamma(ell) summary, U_w/ENBW, audit.trail.
   • Conclusion: pass/fail with recommended actions.
2. Publication conventions
   All magnitudes and units pass check_dim(expr); publish all plots with consistent axes and windowing.

IX. Cross-References & Dependencies

• With the Continuous Kernel (Chapter 2)
  Units of n_eff(x,t), rho(x,t), and c_ref determine comparability for T_arr and eps_mass.
• With Coupled Advancement (Chapter 4)
  advance_dt and sync strategy affect stability and SLO; record strategy flags in reports.
• With Time Calibration (Chapter 5)
  alpha/beta/r_rms feed σ_ts and two-form alignment; always report delta_form in parallel.
• With Data Persistence (Chapter 6)
  The manifest includes benchmark lists, metrics, windows, paths, and evidence; A/B differences are persisted as Δmetrics.
• With Parallelization (Chapter 7)
  Placement, quotas, and autoscale effects on throughput and tail latency must be annotated; TS.hb.violations is mandatory.
• With Error Budget (Chapter 8)
  Conservation gates must pass before scoring; N_min and power planning are provided by I60-19.

X. Risks, Limitations & Open Questions

• Risks
  Hidden resource contention or measurement overhead inflates throughput; poor window/ENBW choice biases results; sparse path sampling underestimates delta_form.
• Limitations
  Quasi-analytic references may be unavailable in complex media; scores are sensitive to weights w*.
• Open questions
  Joint benchmarks for step-size × placement × synchronization; cross-tenant fairness scoring under dom_k controls.

XI. Implementation Bindings & Function Prototypes (I60-*)

• I60-21 run_benchmark(B_id:str, cfg:dict, cap:dict, quota:dict) -> RunRef
  Launch a run, locking cfg/seed/placement and the tau_mono time base.
• I60-22 collect_bench_metrics(run:RunRef, W:float) -> dict
  Return TS.*, eps_*, T_arr.*, cost(res), and window/spectral correction params.
• I60-23 score_benchmark(metrics:dict, weights:dict) -> Scorecard
  Implement S62-60..64, output S_* and S_total.
• I60-24 compare_runs(base:RunRef, cand:RunRef, alpha:float) -> DiffReport
  Single-factor comparison with significance decision and Δmetrics.
• I60-25 publish_report(run:RunRef, score:Scorecard, evidence:any) -> BenchmarkReport
  Persist artifacts, generate the publication, and update audit.trail.

XII. Verification & Test Matrix

• Micro-kernel required
  B-μ1/μ2/μ3: eps_* and delta_form within gates; S_cor ≥ target_cor.
• System-level required
  B-S1/S2/S3: TS.hb.violations = 0, TS.latency.p99 ≤ target_p99, thr_norm ≥ target_thr.
• Cross-scale required
  B-X1/X2/X3: two-form consistency and stability pass; σ_ts propagation and reporting complete.
• Regression gates
  If ΔS_total < -tol or ΔTS.latency.p99 > tol_p99, declare regression and block publication.

XIII. Deliverables & Versioning

• Deliverables
  Benchmark suite & condition library, scorer, A/B scripts, report templates, dashboards, and export tools.
• Versioning
  From v1.0, freeze B.id, field names, and scoring conventions. Introduce new benchmarks as nominal extensions with migration guidance.

XIV. New Terms & Symbols (to memorize)

• Benchmarks & conditions: B.id, cfg, seed, cap, quota(k), placement/policy, warmup, W.
• Metrics & scoring: S_total, S_cor, S_stb, S_slo, S_thr, thr_norm, cost(res), target_*, Δmetrics.
• Conservation & two-form: eps_mass, eps_norm, eps_psd, T_arr.general, T_arr.factorized, delta_form, gamma(ell), d ell.
• Implementation bindings: RunRef, Scorecard, BenchmarkReport, DiffReport.