56-Report-Level Methods Appendix Template v1.0 | Chapter 10 Ablation & Sensitivity Analysis

Home ／ Docs-Technical WhitePaper ／ 56-Report-Level Methods Appendix Template v1.0

Chapter 10 Ablation & Sensitivity Analysis

I. Chapter Goals & Scope (Mandatory)

• Establish a unified evaluation framework for Ablation—Sensitivity—Robustness, quantifying each component/variable’s marginal contribution and weak spots, with full replayability and auditability.
• Aligned with Ch.5 (Data & Experimental Design), Ch.6 (Math & Pseudocode), Ch.8 (Evaluation & Metrics), Ch.9 (Uncertainty & Tests), and Ch.11 (Implementation Binding).

II. Definitions & Targets (Mandatory)

• Ablation: remove/freeze a component or constraint while holding other factors fixed to measure performance change Δmetric.
• Sensitivity: local or global response to hyperparameter/path/input perturbations δx, reported as ∂metric/∂x or a distribution of Δmetric.
• Target layers: component (algo module/feature/regularizer), path (gamma(ell) sampling rules), metrology (calibration/environmental corrections), data (partitions/weights/sampling).

III. Design Principles & Controls (Mandatory)

• Baseline: declare Option Base; all comparisons reference the baseline.
• Controls: list control vs drift variables; remove confounding across batches/regions via stratified matching or weighting.
• Randomness: fix seed; for Monte Carlo perturbations, recommend sample size N≥100.

IV. Metrics & Effect Sizes (Mandatory)

• Primary metrics: gate_accuracy (↑), gate_latency (↓), compat_rate (↑), error_rate (↓); add domain metrics as needed.
• Effect size: ΔS = S_variant − S_base; continuous: Cohen’s d; proportions: OR/RR.
• Confidence & tests: report CI_95% and the relevant tests (see Ch.9); control multiplicity with Holm/BH.

V. Arrival-Time–Related Variants (Mandatory)

1. Unified forms:
   • Factored: T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell )
   • General: T_arr = ( ∫ ( n_eff / c_ref ) d ell )
2. Variant examples: change Δell/ε_int, switch general|factored, swap c_ref calibration bandwidth band; in the same paragraph state gamma(ell) and d ell, with check_dim=true.

VI. Ablation Checklist (copy-ready, Mandatory)

VII. Sensitivity Strategies (Mandatory)

• Local: one-at-a-time (OAT), steps {±1σ} or grid {−kσ,…,+kσ}; report gradient-like Δmetric/Δx.
• Global: Sobol/FAST/Latin Hypercube; output first/total indices S_i, S_Ti with CI.
• Path perturbations: systematically vary gamma(ell) sampling density/curvature radius; report ΔT_arr distribution and gate pass-rate changes.

VIII. Outputs & Decision Rules (Mandatory)

• Output for each variant: raw measures + CI + effect size + gate decision + artifact hashes + script locators.
• Decision: any hard gate failure marks the variant nonconforming; for equivalence forms (e.g., A3) provide a non-inferiority interval.

IX. Human × Machine Mapping (Mandatory)

X. Field & Constraint List (copy-ready)

XI. Machine Schema (YAML; JSON-equivalent, Mandatory)

ablation:

base: "Option Base"

variants:

- name: "A1_remove_component_X"

change: "disable module X in pipeline"

tags: ["component"]

- name: "A2_freeze_param_Y"

change: "set param Y constant at baseline value"

tags: ["param"]

- name: "A3_general_to_factored"

change: "switch arrival-time from general to factored when τ_c satisfied"

tags: ["path","metrology"]

controls:

fixed: ["dataset@v3","seed=20250927","band=L2","pipeline_version=1.4.0"]

allowed_drift: ["weather","load"]

evaluation:

metrics:

- { name: "gate_accuracy", direction: "↑" }

- { name: "gate_latency", direction: "↓" }

- { name: "compat_rate", direction: "↑" }

gates:

hard: ["gate_accuracy>=0.99@7d","compat_rate>=0.995@replay"]

soft: ["unit_cost<=1.0x@30d"]

arrival_time:

delta_form: "general|factored"

path: "gamma(ell)"

measure: "d ell"

check_dim: true

effects:

- { variant: "A1_remove_component_X", type: "d", value: -0.28, ci: [-0.40, -0.16] }

- { variant: "A3_general_to_factored", type: "RD", value: -0.003, ci: [-0.009, 0.002] }

tests:

- { name: "welch", H0: "ΔS=0", side: "two", stat: -4.1, p: 3e-5, p_adj: 9e-5 }

reproducibility:

container: "registry/replay:2025.09@sha256:…"

scripts:

- "ablate_component.py@a1b2c3"

- "switch_arrival_form.py@9f8e7d"

repro_cmd: "docker run … ablate_component.py --variant A1 --seed 20250927"

artifacts: ["yaml","json","pdf"]

XII. Minimal Sample (human abstract × machine snippet, Mandatory)

• Human abstract: A1 removing module X yields ΔS=-0.28 (d=-0.28, CI_95% [-0.40,-0.16]); A3 switches general→factored within τ_c, RD=-0.003 [−0.009, 0.002], passes non-inferiority interval; all hard gates pass.
• Machine snippet:

effects: [{ variant:"A1_remove_component_X", type:"d", value:-0.28, ci:[-0.40,-0.16] }]

tests: [{ name:"welch", H0:"ΔS=0", side:"two", stat:-4.1, p:3e-5, p_adj:9e-5 }]

XIII. Validation Rules (regex/consistency, Mandatory)

• Variant naming: ^[A-Z][A-Za-z0-9_]+$; unique.
• Gate expressions: ^gate_[a-z0-9_]+(>=|<=|==)[^@\\s]+@[^\\s]+$.
• Arrival-time: if T_arr appears, require arrival_time.path/measure and math.check_dim=true.
• Reproducibility: reproducibility.container/scripts/repro_cmd must be non-empty, and scripts must carry @commit.

XIV. Citation & Cross-Reference Style (Mandatory)