46-EFT.WP.Data.Benchmarks v1.0 | Chapter 13 Fairness, Ethics & Safety Stress

Home ／ Docs-Technical WhitePaper ／ 46-EFT.WP.Data.Benchmarks v1.0

Chapter 13 Fairness, Ethics & Safety Stress

I. Chapter Purpose & Scope

II. Terminology & Dependencies

1. Terms: slices, gap_metric (abs_diff|ratio|stat_parity|eq_opp), harm_suite (harm sample set), policy (allow/ban/restrict), gating (release gates), red_team (stress testing), incident, appeal.
2. Dependencies: privacy, security & compliance (Pipeline v1.0, Ch.14); evaluation protocol (ModelCards v1.0, Ch.11); metrics & units (this volume, Ch.6); robustness & adversarial (this volume, Ch.12); unit & dimension checks (Core.Metrology v1.0:check_dim).
3. Math & symbols: wrap inline symbols; any division/integral/composite operator must use parentheses; for T_arr use
   • T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell ), or
   • T_arr = ( ∫ ( n_eff / c_ref ) d ell ),
     declaring gamma(ell) and d ell. No Chinese in formulas/symbols/definitions.

III. Fields & Structure (Normative)

fairness_ethics:

slices:

- {axis:"locale", buckets:["en","zh","es"]}

- {axis:"gender*", buckets:["f","m","other"], note:"if legally permissible and de-identified"}

- {axis:"device", buckets:["mobile","desktop"]}

gap_metric: "abs_diff|ratio|stat_parity|eq_opp"

thresholds:

fairness_warn: 0.03

fairness_block: 0.05

harms:

harm_suite_ref: "safety/harm_suite@vX.Y"

categories: ["toxicity","self-harm","privacy","misinfo","bias"]

scoring: ["toxicity@prob","privacy_leak@binary","prompt_injection@binary"]

policies:

allowed_use: ["academic","benchmark"]

prohibited_use:["surveillance","unlawful_discrimination"]

restricted_use:["medical_advice","financial_advice"]

red_team:

enabled: true

playbooks: ["redteam/prompt_injection.md","redteam/toxicity.md"]

exposure: {shadow:true, canary:0.02}

reporting:

table_axes: ["axis","bucket","metric"]

include_ci: true

significance: {method:"bootstrap", B:10000, alpha:0.05, correction:"Holm-Bonferroni"}

disclosures:

sensitive_attributes: "de-identified|N/A"

human_in_the_loop: true

governance:

gating: {require_ci:true, min_runs:3}

incident:

notify: "security@org.example"

sla_hours: 72

appeal_window_days: 14

IV. Fairness Slices & Disparity Measures

1. Slices: choose task-relevant and legally compliant axes (e.g., locale/device/region); for potentially sensitive axes, ensure de-identification and legality review.
2. Disparities:
   • abs_diff = ( metric_ref - metric_grp );
   • ratio = ( metric_grp / metric_ref );
   • stat_parity/eq_opp follow positive-class/threshold definitions in the protocol.
     Report Δ and CI_95, apply multiple-comparison correction.

V. Harm Datasets & Safety Stress

• Harm suite: harm_suite_ref points to reproducible items and labeling rules; cover toxicity/self-harm/privacy/misinfo/bias categories.
• Stress/red team: shadow or canary exposure; declare pass/block criteria (e.g., toxicity@prob<=τ, prompt_injection@binary==0); failures are blocking.

VI. Gates & Governance

• Gates: if gap_metric exceeds fairness_block or harm scores violate limits, block release; over fairness_warn requires remediation and re-evaluation.
• Governance: record incident, response SLA sla_hours, and appeal_window_days; disclose policy and changes externally (not output here).

VII. Statistics & Reporting

• Significance: default bootstrap (B≥10k, α=0.05); apply Holm–Bonferroni across buckets/multi-dimensional slices.
• Reports: tables over axis/bucket/metric with CI_95; include fairness heatmaps and harm pass-rate curves; disclose sensitive_attributes/human_in_the_loop.

VIII. Metrology & Units (SI)

• Performance & ratios: QPS(1/s), latency_ms.{p50,p95,p99}, ρ(—); proportions/probabilities use — (dimensionless) or %.
• Mandatory: metrology:{units:"SI", check_dim:true}; normalize units first before composing/comparing.
• Path quantities: if fairness/stress experiments involve T_arr, register delta_form/path/measure and validate using the equivalences.

IX. Machine-Readable Fragment (Drop-in)

fairness_ethics:

slices:

- {axis:"locale", buckets:["en","zh","es"]}

- {axis:"device", buckets:["mobile","desktop"]}

gap_metric: "abs_diff"

thresholds: {fairness_warn:0.03, fairness_block:0.05}

harms:

harm_suite_ref: "safety/harm_suite@v1.1"

categories: ["toxicity","privacy","prompt_injection"]

scoring: ["toxicity@prob","privacy_leak@binary","prompt_injection@binary"]

policies:

allowed_use: ["academic","benchmark"]

prohibited_use: ["surveillance"]

red_team:

enabled: true

playbooks: ["redteam/prompt_injection.md"]

exposure: {shadow:true, canary:0.02}

reporting:

table_axes: ["axis","bucket","metric"]

include_ci: true

significance: {method:"bootstrap", B:10000, alpha:0.05, correction:"Holm-Bonferroni"}

disclosures: {sensitive_attributes:"de-identified", human_in_the_loop:true}

governance:

gating: {require_ci:true, min_runs:3}

incident: {notify:"security@org.example", sla_hours:72, appeal_window_days:14}

metrology: {units:"SI", check_dim:true}

X. Lint Rules (Excerpt, Normative)

lint_rules:

- id: FAIR.SLICES_DEFINED

when: "$.fairness_ethics.slices"

assert: "len(value) >= 1 and all(has_keys(_, 'axis','buckets') for _ in value)"

level: error

- id: FAIR.GAP_METRIC_ALLOWED

when: "$.fairness_ethics.gap_metric"

assert: "value in ['abs_diff','ratio','stat_parity','eq_opp']"

level: error

- id: HARM.SUITE_REF_REQUIRED

when: "$.fairness_ethics.harms"

assert: "has_key(value, 'harm_suite_ref')"

level: error

- id: GOVERN.GATING_PARAMS

when: "$.fairness_ethics.governance.gating"

assert: "has_keys(require_ci, min_runs)"

level: error

- id: REPORT.SIGNIFICANCE_PARAMS

when: "$.fairness_ethics.reporting.significance"

assert: "has_keys(method, B, alpha)"

level: error

- id: METROLOGY.SI_AND_CHECKDIM

when: "$.metrology"

assert: "units == 'SI' and check_dim == true"

level: error

XI. Cross-Reference Anchors

• Metrics & units: EFT.WP.Data.Benchmarks v1.0, Ch.6.
• Scoring & gates: Ch.8.
• Evaluation protocol & runtime environment: EFT.WP.Data.ModelCards v1.0, Ch.11; this volume, Ch.10.
• Privacy, security & compliance: EFT.WP.Data.Pipeline v1.0, Ch.14.
• Unit & dimension checks: EFT.WP.Core.Metrology v1.0:check_dim.

XII. Chapter Compliance Checklist

• Slices, disparity measures, harm categories, and scoring posture complete; thresholds fairness_warn/fairness_block explicit.
• Red-team/stress exposure, criteria, and guardrails active; incidents have sla_hours and an appeal window.
• Significance with multiple-comparison correction configured; reports include slice tables, heatmaps, and harm curves.
• SI metrology with check_dim=true; if T_arr appears, delta_form/path/measure registered and validated.
• Machine-readable fragment is drop-in and lint-clean; export_manifest.references[] use “Volume vX.Y:Anchor.”