12-EFT.WP.Methods.Repro v1.0 | Chapter 12 Reproducibility Acceptance and Score Publication

Home ／ Docs-Technical WhitePaper ／ 12-EFT.WP.Methods.Repro v1.0

Chapter 12 Reproducibility Acceptance and Score Publication

I. Scope and Objectives

• Establish a unified acceptance workflow and publication gauge for reproducibility so that, under fixed EnvLock, seed, and time-base mapping ts = alpha + beta * tau_mono, candidate implementations are judged automatically against the benchmark suite and gates, then promoted in staged release.
• Expected outputs: RepReport, BenchReport, the aggregate score with confidence interval, audit signatures, and the release bundle. Differences from the dual arrival-time gauges T_arr (captured as delta_form) and spectral consistency are included in gate decisions.

II. Terms and Symbols

• score: aggregate reproducibility score in [0,1] (higher is better).
• tau_score: score gate; CI_score: confidence interval for score; z: quantile for the desired confidence.
• r_tb: residual for time-base alignment; tau_tb: its gate; delta_psd: normalized spectral discrepancy; tau_psd: its gate.
• gate.rep: hard gate on result differences; eps_mass, eps_norm: conservation & normalization residuals; tau_mass: conservation gate.
• case_weight: per-case weight in the benchmark; w_rep, w_psd, w_tb, w_perf, w_cons: dimension weights with w_rep + w_psd + w_tb + w_perf + w_cons = 1.
• Channels: canary, stable, LTS; release bundles are identified by hash(•) and fingerprint.

III. Postulates and Minimal Equations

1. P31-30 Acceptance stability postulate
   With EnvLock, seed, alpha, beta, and data fingerprint fixed, acceptance decisions are consistent within gate tolerances across repeated runs, i.e.,
   Pr( decision_run1 = decision_run2 ) → 1 as sample size and observation window grow.
2. P31-31 Non-worsening scoring postulate
   If changes are limited to bug fixes that do not alter gauges or data distributions, then score_new >= score_old - tau_reg, where tau_reg is the regression tolerance.
3. S32-39 Per-case scoring
   score_case = w_rep*( 1 - delta_rep ) + w_psd*( 1 - delta_psd ) + w_tb*( 1 - clip( r_tb / tau_tb , 0 , 1 ) ) + w_perf*perf_norm + w_cons*( 1 - clip( eps_mass / tau_mass , 0 , 1 ) ).
4. S32-40 Suite aggregation
   score = ( Σ ( case_weight * score_case ) ) / ( Σ case_weight ); report R_coef = 1 - delta_rep alongside.
5. S32-41 Confidence interval and pass gate
   • sigma_hat = std( { score_case } );
     CI_score = [ score - z * ( sigma_hat / sqrt(N) ) , score + z * ( sigma_hat / sqrt(N) ) ];
   • Acceptance condition:
     delta_rep <= gate.rep ∧ delta_psd <= tau_psd ∧ r_tb <= tau_tb ∧ lower( CI_score ) >= tau_score.
6. S32-42 Dual arrival-time consistency
   Compute both T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell ) and T_arr = ( ∫ ( n_eff / c_ref ) d ell ); publish
   delta_form = | T_arr(form1) - T_arr(form2) | / max( |T_arr(form1)| , |T_arr(form2)| ) and fold it into score_case via the w_cons term.

IV. Data and Manifest Gauges

• Acceptance bundle — minimal fields
  EnvLock, PipelineCard, ParamCard, data fingerprint, artifact hash(•), benchmark-suite version and case list, U_w, ENBW, gamma(ell) and d ell (for path integrals), alpha, beta with fit diagnostics, seed and RNG family.
• Gauge consistency
  Declare all units and run check_dim(expr); when T_arr, S_xx(f) are involved, publish window function and T_obs, and fix the applicability domain of c_ref, n_eff.
• Audit and signatures
  Per Chapter 9, link events with H_k = hash( H_{k-1} || event ) and attach sig_k; the acceptance bundle includes a public verifier pointer verifier.pk_ref.

V. Algorithms and Implementation Bindings

• I30-3 run_benchmark_suite(card:dict) -> BenchReport
  Inputs PipelineCard / ParamCard and the benchmark suite; outputs per-case delta_rep, delta_psd, r_tb, eps_mass, perf_norm, R_coef, and summary observables.
• I30-4 verify_reproduction(golden:any, candidate:any, metrics:dict) -> RepReport
  With fixed EnvLock, seed, alpha, beta, align then test each case against gates and produce a delta decomposition.
• I30-22 compose_score(bench:BenchReport, weights:dict) -> { score:float, CI_score:list, details:any }
  Implements S32-39/40/41; returns score, CI_score, and per-dimension contributions.
• I30-23 prepare_release_bundle(rep:RepReport, score:any, audit:any) -> ReleaseBundle
  Aggregates acceptance bundle, audit trail, and public summary; computes fingerprint and verifiable metadata.
• I30-24 sign_and_publish(bundle:any, channel:str) -> PublishReport
  Signs and pushes the bundle to canary/stable/LTS; records submit fingerprints and rollback anchors.

VI. Metrology Flows and Run Graph

1. Mx-31 precheck-and-freeze
   • Validate EnvLock and data fingerprint.
   • Refit time-base alpha, beta; confirm fit quality.
   • Fix seed, lock T_obs and U_w / ENBW.
2. Mx-32 run-and-measure
   • Use I30-3 and I30-4 to produce BenchReport / RepReport.
   • Compute score and CI_score (I30-22).
   • Emit per-case and aggregate gate decisions.
3. Mx-33 decide-and-route
   • If S32-41 holds, issue Accept.
   • If lower( CI_score ) < tau_score but hard gates pass, mark Conditional and request more samples or longer T_obs.
   • Else Reject, attach rollback guidance and route to Chapter 11 playbooks.
4. Mx-34 sign-and-release
   • Assemble & sign the release bundle (I30-23/24), choose canary/stable/LTS.
   • Register run-time SLOs and reproducibility-gate alerts on the Chapter 9 monitoring stream.
   • Archive the acceptance bundle to the long-term forensic channel.

VII. Verification and Test Matrix

1. Minimum required
   • Repeated acceptances under identical distributions validate P31-30: identical decisions and var(score) → 0.
   • Regression-fix scenario validates P31-31: score_new - score_old >= -tau_reg.
   • Spectral cases: enforce var( x ) ≈ ( ∫ S_xx(f) d f ) and delta_psd <= tau_psd.
   • Dual T_arr: publish delta_form and verify its contribution to score_case remains under the configured cap.
2. Boundary & extreme
   • Small-sample uncertainty: shrink T_obs or case count; inspect CI_score width and decision stability.
   • High-concurrency jitter: when hb drops and bp rises, confirm weights on r_tb and perf_norm do not cause false negatives for reproducibility.
3. Gates and SLOs
   • Hard gates: delta_rep <= gate.rep, r_tb <= tau_tb, delta_psd <= tau_psd, eps_mass <= tau_mass.
   • Soft gate: lower( CI_score ) >= tau_score, R_coef >= 1 - gate.rep.

VIII. Cross-References and Dependencies

IX. Risks, Limits, and Open Questions

• Risks
  Sensitivity of delta_psd to U_w / ENBW; cross-site errors in c_ref and n_eff can inflate delta_form; weight misconfiguration can induce Simpson’s paradox.
• Limits
  Linear score synthesis may underperform on hard nonlinearities and threshold effects; normality for CI_score can be poor with small N.
• Open questions
  Adaptive weighting and risk-aware scoring; non-linear distribution-matching synthesis; dynamic quota allocation for dual T_arr gauges.

X. Deliverables and Versioning

1. Deliverables
   • RepReport and BenchReport (per-case metrics and delta decomposition).
   • ScoreSummary (score, CI_score, weights, gates, contribution breakdown).
   • ReleaseBundle (EnvLock, cards, fingerprints, signatures, audit chain).
   • Public announcement abstract and third-party verification guide.
2. Version policy
   • Changes to tau_score / gate.rep / tau_psd / tau_tb are policy upgrades requiring a minor version bump and historical benchmark replays.
   • Expanding the benchmark suite or retuning weights requires baseline score updates and an impact bulletin.
   • All acceptance and release bundles must be archived with hash(•) and fingerprint for long-term forensic viability.