13-EFT.WP.Methods.SimStack v1.0 | Chapter 11: Reproducibility Experiments & Pipeline Compliance

Home ／ Docs-Technical WhitePaper ／ 13-EFT.WP.Methods.SimStack v1.0

Chapter 11: Reproducibility Experiments & Pipeline Compliance

I. Scope & Objectives

• Define compliance requirements and artifact standards for a fully reproducible simulation-to-observation fitting pipeline, covering environment locking, data and convention freezing, run attestation, and publishing gates.
• Objective: use the minimal reproducibility bundle — PipeCard + ParamCard + EnvLock + Attest — to guarantee statistically equivalent results under declared resources and time-base conventions. If gates are not met, provide falsification and remediation paths.

II. Terms & Symbols

1. Artifacts & cards
   PipeCard (pipeline card), ParamCard (parameter card), EnvLock (environment lock), Attest (run attestation), FitArtifact (fit publication).
2. Reproducibility levels
   CL0 (runnable), CL1 (parameter reproducible), CL2 (statistically reproducible), CL3 (bit-wise reproducible).
3. Result equivalence metrics
   • Numerical fields: eps_rel = ||y' - y||_2 / ||y||_2, eps_abs = ||y' - y||_∞.
   • Spectral quantity: Delta_S(f) = | S'_xx(f) - S_xx(f) | with windows aligned by U_w and ENBW.
   • Arrival time: Delta_T = | T'_arr - T_arr |, always publishing delta_form.
4. Time & conventions
   • Time-base mapping: ts = alpha + beta * tau_mono.
   • Two T_arr formulations: T_arr.general, T_arr.factorized, discrepancy delta_form.
   • Path & measure: gamma(ell) with d ell.

III. Postulates & Minimal Equations (P61-/S62-)

1. P61-30 (Minimal reproducibility bundle)
   Any external publication must include {PipeCard, ParamCard, EnvLock, Attest} plus an alignment manifest (units, time base, path conventions).
2. P61-31 (Statistics first)
   Prioritize meeting CL2 (statistical reproducibility). CL3 (bit-wise) requires declaring fp_mode, parallel reductions, and random sources.
3. P61-32 (Convention freeze)
   Do not modify ParamCard until units, H(*) mapping, alpha/beta, c_ref, and gamma(ell) are frozen.
4. P61-33 (Falsification first)
   If eps_rel or Delta_T exceeds gates, check delta_form and time-base alignment before retuning parameters.
5. S62-80 (Equivalence criteria)
   • Numerical equivalence: ( eps_rel ≤ gate_rel ) AND ( eps_abs ≤ gate_abs ).
   • Spectral equivalence: max_f Delta_S(f) ≤ gate_spec.
   • Arrival-time equivalence: Delta_T ≤ gate_T and |delta_form| ≤ gate_form.
6. S62-81 (Randomness & determinism)
   Declare seed, rng_family, parallel ordering, and reduction strategy. Bit-wise repro requires rng_stream partitioning and fixed reduce(order="pairwise") or reduce(kahan).
7. S62-82 (Environment lock hash)
   Environment hash: H_env = sha256( oci_digest || deps.lock || sys.meta ). Bit-wise repro requires H'_env = H_env.
8. S62-83 (Convention immutability)
   Changes to H, (alpha,beta), or the two T_arr.* forms constitute a new convention; do not mix old and new results.
9. S62-84 (Compliance score)
   Score = w1*(1 - eps_rel) + w2*(1 - Delta_S_norm) + w3*(1 - Delta_T_norm), with w1 + w2 + w3 = 1, for benchmarking and gating.

IV. Data & Manifest Conventions

1. PipeCard (required fields)
   • graph = G = (V,E), stage.io, H.id, timebase.{alpha,beta}, paths.gamma.list, c_ref, units.*, TS.* observation points.
   • Resource policy: cap, quota, placement.policy, and hb/bp semantics aligned with Chapters 3 and 7.
2. ParamCard
   theta.init, theta.map, pri(theta), bnds, controller (step size, CFL, error gates), fp_mode, reduction strategy reduce.
3. EnvLock
   • oci_digest, deps.lock, sys.meta (kernel, ISA, BLAS/FFT family & version), compiler.flags.
   • Optional driver/runtime digests (e.g., CUDA/ROCm versions). Use English, plain text.
4. Attest
   run.id, H_env, sha256(artifacts), provenance (call chain, timestamps ts, tau_mono reference), r.stats, J, delta_form.
5. Publication requirements
   All fields must pass check_dim(expr); spectral publications include U_w, ENBW, and the window function.

V. Algorithms & Implementation Bindings (I60-*)

• I60-6 freeze_env(env_spec:dict) -> EnvLock
  Produce EnvLock and H_env. If required hardware traits are missing, return error E_MISSING_HW_CAP.
• I60-7 generate_pipeline_card(sim_state:any, fit:FitArtifact, opts:dict) -> PipeCard
  Derive PipeCard from current topology and conventions; audit units and time base.
• I60-8 materialize_pipeline(PipeCard, ParamCard, EnvLock) -> PipeRef
  Instantiate a runnable pipeline and return PipeRef plus placement plan. Idempotent.
• I60-9 run_with_attestation(PipeRef, seeds:list) -> Attest
  Execute and produce Attest, including sha256, J, r.stats, delta_form, and TS.*.
• I60-10 verify_repro(artifact:any, gates:dict) -> Verdict
  Judge equivalence per S62-80 and return CL0..CL3 and Score.
• I60-11 export_repro_pack(PipeCard, ParamCard, EnvLock, Attest) -> ReproPack
  Package the minimal reproducibility bundle and generate publication artifacts plus an audit-trail index.

VI. Metrology Flow & Run Diagram (Mx-66 repro-and-compliance)

• Prepare & freeze
  Harmonize units and conventions; call I60-6 to produce EnvLock and H_env; lock versions and hashes of PipeCard/ParamCard.
• Instantiate & dry-run
  Use I60-8 to obtain PipeRef; dry-run to verify TS.* observation points and pass check_dim(expr).
• Main run & attestation
  Run with declared seed/rng_stream; persist Attest and all sha256(artifacts).
• Result equivalence check
  Against baseline, call I60-10 to evaluate eps_rel/eps_abs/Delta_S/Delta_T, then compute Score and CL*.
• Publish & register
  Use I60-11 to build ReproPack; in audit.trail, register run.id → H_env → PipeCard@hash → ParamCard@hash.
• Refute & remediate
  If gates fail, check in order: delta_form → time base (alpha,beta) → H(*) → noise convention S_eta(f) → environment difference H'_env. Fix and return to step 1 or 3.

VII. Verification & Test Matrix

1. Minimum required
   • EnvLock complete and H'_env = H_env (if CL3 claimed). PipeCard/ParamCard hashes verifiable. TS.* points present.
   • Equivalence gates: eps_rel ≤ gate_rel, eps_abs ≤ gate_abs, max_f Delta_S(f) ≤ gate_spec, Delta_T ≤ gate_T, |delta_form| ≤ gate_form.
2. Boundary scenarios
   Non-deterministic parallel reductions (amplified FP accumulation error); cross-ISA differences; sparse gamma(ell) with weight concentration; very low SNR.
3. Degradation & fallback
   If CL3 unattainable, degrade to CL2 and widen statistical CIs; publish reasons and remediation plan.
4. Prediction & cross-repro
   Cross-node, cross-container, cross-timezone reproducibility. For the Mx-65 fitting loop, perform leave-one-out checks and record RMSE_cv/MAE_cv.

VIII. Cross-References & Dependencies

• Time and arrival-time conventions align with Chapters 5 and 2; two-form reporting with delta_form is mandatory.
• Data and manifest fields inherit strictly from Chapter 6; spectral outputs must include U_w/ENBW.
• Concurrency and resource policies reference TS.* metrics and placement rules from Chapter 7.
• Benchmarks and scoring follow Chapter 9; publish BenchmarkReport alongside ReproPack.
• Observation mapping and fitting evidence attach to Chapter 10’s FitArtifact and H(*).

IX. Risks, Limitations & Open Questions

• Risks
  Unfrozen H(*) or (alpha,beta) yields pseudo-repro; heterogeneous BLAS/FFT backends induce hidden non-determinism; path-weight bias w_k corrupts T_arr.
• Limitations
  For strongly nonlinear or distribution-shifted cases, CL3 cost may be prohibitive; statistical reproducibility depends on sample size and reliability of noise conventions.
• Open questions
  Online convention attestation for multi-path media; standardized deterministic reductions across heterogeneous hardware; joint optimization of Score and SLO.

X. Deliverables & Versioning

1. Deliverables
   • ReproPack = {PipeCard@hash, ParamCard@hash, EnvLock@hash, Attest, FitArtifact, BenchmarkReport}.
   • Audit index: audit.trail mapping run.id → hashes → H_env → gates → CL* → Score.
2. Versioning
   • Semantic versioning: major for convention changes (units, H(*), alpha/beta, T_arr.*); minor for parameter or implementation tweaks; patch for environment updates.
   • Any cross-version comparison must declare the alignment strategy and republishing gates.

XI. New Terms & Symbols (to memorize)

• Artifacts & locks: PipeCard, ParamCard, EnvLock, ReproPack, Attest, H_env.
• Repro levels & scoring: CL0..CL3, Score, gate_rel/gate_abs/gate_spec/gate_T/gate_form.
• Equivalence metrics & errors: eps_rel, eps_abs, Delta_S(f), Delta_T, delta_form.
• Environment & numeric policy: fp_mode, reduce(order="pairwise" | "kahan"), rng_family, seed.
• Process & interfaces: Mx-66, interfaces I60-6..I60-11.