44-EFT.WP.Data.ModelCards v1.0 | Chapter 13 Robustness, Shift & Adversarial

Home ／ Docs-Technical WhitePaper ／ 44-EFT.WP.Data.ModelCards v1.0

Chapter 13 Robustness, Shift & Adversarial

I. Chapter Purpose & Scope

II. Fields & Structure (Normative)

robustness:

shift_tests: # synthetic shifts & perturbations

- {name:"snr_drop", severity:[3,6,9], policy:"additive-noise"}

- {name:"time_jitter", ms:[5,10,20], policy:"shuffle-window"}

- {name:"spec_notch", bands:[["0.3","0.5"],["0.6","0.7"]], unit:"fraction"}

natural_shifts: # in-the-wild shifts (device/region/season/domain)

axes: ["device","region","season"]

splits: ["val","test"]

adversarial: # adversarial evaluation (if enabled)

enabled: false

threat_model: "whitebox|blackbox|transfer"

norm: "Linf|L2|L1"

epsilon: 0.01

steps: 10

restarts: 1

targeted: false

metrics: # robustness metrics

primary: ["Δ_rel","acc_robust","auc_robust"]

curves: ["acc-vs-ε","acc-vs-SNR","acc-vs-mask"]

thresholds: # blocking & warning thresholds

drop_rel_max: 0.10 # max allowed relative drop

acc_robust_min: 0.80 # min robust accuracy under specified shift

ece_max_under_shift: 0.05 # calibration drift ceiling

online_consistency: # prod-facing posture (shadow/canary)

shadow_mode: true

window: "7d"

drift_monitors: ["drift_kl","psi"]

alert_rules:

- {name:"robust_drop", rule:"Δ_rel>0.10 for 60m", severity:"high"}

reporting:

table_axes: ["shift","severity","metric"]

include_ci: true # pair metrics with 95% CIs

significance: {test:"bootstrap", alpha:0.05}

notes?: "<non-normative>"

III. Synthetic Shifts (Definitions & Controls)

• snr_drop: Additive noise with SNR levels in dB; declare noise type (Gaussian/colored), seed policy, and whether applied pre/post-normalization.
• time_jitter: Temporal jitter/reshuffling; specify jitter window in milliseconds and boundary handling.
• spec_notch: Frequency-band masking; declare normalized band ranges, mask value (zero/median), and whether contiguous or random bands per sample.

IV. Natural Shifts (In-the-Wild)

• Enumerate axes (e.g., device models, geographies, seasons, domains).
• Report per-axis coverage and sample counts; ensure mapping aligns with Dataset Card coverage.
• For each axis, provide stratified metrics and confidence intervals; flag gaps exceeding fairness thresholds (see Chapter 11).

V. Adversarial Evaluation (If Enabled)

• Threat model: whitebox (e.g., PGD), blackbox (score/decision-based), or transfer.
• Norm & magnitude: enforce ‖δ‖_p ≤ ε; list step count/restarts and whether targeted is used.
• Safety guardrails: deny-list unsafe transforms for production; adversarial testing remains offline unless explicitly canaried.

VI. Metrics & Thresholds

• Relative degradation: Δ_rel = ( baseline - under_shift ) / max( baseline, ε ).
• Robust accuracy: acc_robust at given shift level or worst-case over a set.
• Area metrics: auc_robust over ε/SNR/mask spans.
• Calibration under shift: report ECE/Brier; ensure ece_max_under_shift ≤ threshold.
• Blocking policy: Fails release if any of: Δ_rel > drop_rel_max, acc_robust < acc_robust_min, or calibration exceeds ceiling.

VII. Online Robustness & Replay Consistency

• Shadow/Canary: run shadow inference for window and track drift monitors; raise alerts on sustained breaches.
• Replay: de-identified log replay must reproduce offline trends within tolerance; document deviations and root causes.

VIII. Metrology & Units

1. Declare units for time/frequency/energy/performance metrics; validate via check_dim.
2. When robustness pertains to path-dependent quantities, state delta_form, path gamma(ell), and measure d ell; use one of the two equivalences for T_arr:
   • T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell )
   • T_arr = ( ∫ ( n_eff / c_ref ) d ell ).

IX. Machine-Readable Fragment (Drop-in)

robustness:

shift_tests:

- {name:"snr_drop", severity:[3,6,9], policy:"additive-noise"}

- {name:"time_jitter", ms:[5,10,20], policy:"shuffle-window"}

- {name:"spec_notch", bands:[["0.3","0.5"],["0.6","0.7"]], unit:"fraction"}

natural_shifts: {axes:["device","region"], splits:["val","test"]}

adversarial: {enabled:false, threat_model:"whitebox", norm:"Linf", epsilon:0.01, steps:10, restarts:1, targeted:false}

metrics: {primary:["Δ_rel","acc_robust"], curves:["acc-vs-ε","acc-vs-SNR"]}

thresholds: {drop_rel_max:0.10, acc_robust_min:0.80, ece_max_under_shift:0.05}

online_consistency:

shadow_mode: true

window: "7d"

drift_monitors: ["drift_kl","psi"]

alert_rules: [{name:"robust_drop", rule:"Δ_rel>0.10 for 60m", severity:"high"}]

reporting: {table_axes:["shift","severity","metric"], include_ci:true, significance:{test:"bootstrap", alpha:0.05}}

X. Export Manifest & Audit Trail

export_manifest:

artifacts:

- {path:"robustness/summary.csv", sha256:"..."}

- {path:"robustness/acc_vs_eps.csv", sha256:"..."}

- {path:"robustness/acc_vs_snr.csv", sha256:"..."}

- {path:"robustness/calibration_under_shift.csv", sha256:"..."}

- {path:"robustness/alert_rules.yaml", sha256:"..."}

references:

- "EFT.WP.Core.DataSpec v1.0:EXPORT"

- "EFT.WP.Core.Metrology v1.0:check_dim"

XI. Chapter Compliance Checklist

• Synthetic and natural shifts are explicitly defined with parameters; metrics paired with 95% CIs and significance tests.
• Blocking thresholds (drop_rel_max, acc_robust_min, ece_max_under_shift) are set and met; adversarial settings (if enabled) specify norm/ε/steps/restarts.
• Units for time/frequency/performance pass check_dim; path-dependent metrics register delta_form/path/measure when applicable.
• Shadow/canary posture and replay consistency are documented; drift alerts configured.
• Export manifest lists robustness artifacts with sha256; references use “Volume vX.Y:Anchor.”