18-EFT.WP.Methods.CrossStats v1.0 | Chapter 12 — Hierarchical Models and Partial Pooling (Borrowing Strength)

Home ／ Docs-Technical WhitePaper ／ 18-EFT.WP.Methods.CrossStats v1.0

Chapter 12 — Hierarchical Models and Partial Pooling (Borrowing Strength)

One-line objective: Unify cross-group estimation and prediction via hierarchical Bayes and mixed-effects frameworks so that, in small-sample and heterogeneous settings, partial pooling yields robust, auditable, and dimensionally consistent statistical outputs.

I. Scope & Targets

1. Scope
   • Grouped and small-area estimation: g ∈ {1..G}, within-group observations y_{g,j}, cross-group sharing through priors or random effects.
   • Linear/Generalized Linear Mixed Models (LMM/GLMM), Fay–Herriot small-area models, hierarchical normal means, hierarchical Poisson/Binomial rates.
   • Batch and streaming inference; window Delta_t with time-base tau_mono → ts mapping.
2. Targets
   • Inputs: D = { (y_{g,j}, x_{g,j}, g, ts_{g,j}, m_{g,j}) }, with units & dimensions, within-group variance estimates sigma_g^2 or exposure E_g, and synchronization metadata offset/skew/J.
   • Outputs: group-level posterior p(theta_g|D), BLUP/EBLUP, predictions hat{y}_{g,*} with intervals, and manifest.stats.hier.*.

II. Terms & Symbols

• Groups & sample size: g ∈ {1..G}, n_g, ybar_g = ( 1 / n_g ) * ∑_j y_{g,j}.
• Hierarchical parameters: theta_g (true group-level quantity), population mu, variance tau^2.
• Noise & variances: within-group sigma_g^2, measurement error epsilon_{g,j}.
• Mixed effects: y = X beta + Z b + e; random effects b ~ N(0, G), residuals e ~ N(0, R).
• GLMM link: g( E[y|x] ) = X beta + Z b.
• Shrinkage factors & weights: w_g, B_g.
• Small-area model: y_g = theta_g + e_g, theta_g = X_g' beta + u_g.

*III. Axioms P312- **

• P312-1 (Exchangeability & Partial Pooling): Within each group, conditional on theta_g, observations are exchangeable; across groups, priors or random effects share statistical strength.
• P312-2 (Explicit Priors & Positive Semidefinite Variances): Specify p(mu), p(tau^2); enforce tau^2 ≥ 0, and G, R are PSD.
• P312-3 (Identifiability & Centering): With an intercept, impose sum-to-zero on varying intercepts or apply prior centering; numerically center columns of X, Z to reduce collinearity.
• P312-4 (Time Base & Arrival Time): Estimate on tau_mono, publish on ts; if including T_arr, record both formulations and delta_form.
• P312-5 (Units & Dimensions): All estimates and predictions satisfy check_dim( y - f(x) ); for ratios/rates, introduce exposure or base counts and declare units.
• P312-6 (Protection for Extreme Groups): For very small n_g or very large sigma_g^2, apply stronger pooling and interval widening strategies.

*IV. Minimal Equations S312- **

1. S312-1 (Hierarchical Normal Means)
   • Observations: ybar_g ~ N( theta_g, sigma_g^2 ); prior: theta_g ~ N( mu, tau^2 ).
   • Posterior mean: E[ theta_g | D ] = w_g * ybar_g + ( 1 - w_g ) * mu, where w_g = tau^2 / ( tau^2 + sigma_g^2 ).
   • Posterior variance: Var( theta_g | D ) = ( sigma_g^2 * tau^2 ) / ( sigma_g^2 + tau^2 ).
   • Limits: tau^2 → 0 gives full pooling theta_g → mu; tau^2 → ∞ gives no pooling theta_g ≈ ybar_g.
2. S312-2 (BLUP for Linear Mixed Effects)
   • Model: y = X beta + Z b + e, b ~ N(0, G), e ~ N(0, R).
   • V = Z G Z' + R; hat{beta} = ( X' V^{-1} X )^{-1} X' V^{-1} y.
   • hat{b} = G Z' V^{-1} ( y - X hat{beta} ) (group-level hat{b_g} is the corresponding subvector).
3. S312-3 (REML Marginal Likelihood)
   logL_REML = -0.5 * ( log|V| + (y - X beta)' V^{-1} (y - X beta) + const ); numerically maximize over parameters of G, R.
4. S312-4 (Hierarchical GLMs)
   g( E[y_{g,j}|x] ) = X_{g,j} beta + Z_{g,j} b_g, b_g ~ N(0, G); approximate inference via Laplace/AGHQ or sampling; prediction intervals via linearization or posterior quantiles.
5. S312-5 (Fay–Herriot EBLUP for Small Areas)
   • y_g = theta_g + e_g, theta_g = X_g' beta + u_g, e_g ~ N(0, V_g), u_g ~ N(0, A).
   • hat{theta_g} = w_g * y_g + ( 1 - w_g ) * X_g' hat{beta}, w_g = A / ( A + V_g ).
6. S312-6 (Hierarchical Shrinkage for Discrete Rates)
   • Poisson: y_g ~ Poisson( E_g * lambda_g ), lambda_g ~ Gamma(a,b); E[ lambda_g | D ] = ( a + y_g ) / ( b + E_g ).
   • Binomial: k_g ~ Binom( n_g, p_g ), p_g ~ Beta(a,b); E[ p_g | D ] = ( a + k_g ) / ( a + b + n_g ).

V. Metrology Flow M30-12 (Ready → Model → Diagnose → Publish)

• Ready
  Run time_align_for_stats to align to tau_mono; declare unit(y), dim(y); compute sigma_g^2 or E_g; handle missingness m ∈ {0,1}.
• Model Selection
  Prefer S312-1/5/6 for small-sample or summary-stat settings; prefer S312-2/4 for point-level data; select identity/logit/log links by response family.
• Estimation
  For LMM/GLMM, estimate G, R and beta; or estimate (mu, tau^2), A, and hyperparameters (a, b); use REML/Laplace/sampling.
• Diagnostics
  Check monotonicity of shrinkage curves w_g vs. sigma_g^2; residuals & calibration; variance decomposition; interval widths for extreme groups.
• Prediction & Uncertainty
  Produce hat{theta_g}, hat{y}_{g,*} with intervals; report U = k * u_c or posterior quantiles.
• Publish & Manifest
  Persist emit_hier_manifest: hyperparameters, w_g summary, coverage, contract evaluation, TraceID, offset/skew/J, and (if used) delta_form.

VI. Contracts & Assertions C30-121x

• C30-1211 (PSD Variances): tau^2 ≥ 0, eigmin(G) ≥ 0, eigmin(R) ≥ 0; optimizer converges with grad_norm ≤ tol_grad.
• C30-1212 (Monotone Shrinkage): dw_g / d sigma_g^2 ≤ 0 (empirical check: corr(w_g, sigma_g^2) ≤ 0).
• C30-1213 (Coverage): group-level PI_coverage@q ∈ [q - tol_cov, q + tol_cov].
• C30-1214 (Extreme-Group Protection): if n_g < n_min or sigma_g^2 > s_max, then w_g ≤ w_cap_small and interval expansion factor ≥ f_min.
• C30-1215 (Units & Dimensions): unit(hat{theta_g}) == unit(y); for rate-type quantities, enforce check_dim( y / exposure ).
• C30-1216 (Time-Base Consistency & Arrival Time): if the model includes T_arr, assert delta_form ≤ tol_Tarr; ensure offset/skew/J ≤ policy.max.

*VII. Implementation Bindings I30- **

• I30-121 fit_hier_normal_means(groups, ybar, sigma2, prior) -> posterior
• I30-122 fit_lmm(formula, ds, method) -> {beta, G, R, BLUP}
• I30-123 fit_glmm(formula, family, link, method) -> model
• I30-124 fay_herriot(y, V, X) -> {beta, A, theta_hat, w}
• I30-125 hier_poisson_rate(y, E, a, b) -> posterior
• I30-126 shrinkage_summary(posterior) -> {w_stats, coverage}
• I30-127 predict_hier(model, newdata, level, interval) -> {pred, PI}
• I30-128 evaluate_hier_contracts(report, rules) -> contract_report
• I30-129 emit_hier_manifest(results, policy) -> manifest.stats.hier
• Invariants: len(unique(groups)) == G; G, R are PSD; ∑_g w_g / G ∈ [w_lo, w_hi]; pre-publish contract_report.pass == true.

VIII. Cross-References

• Unit & dimension harmonization: see Methods.Cleaning v1.0, Chapter 4.
• Timelines & synchronization: see Methods.Cleaning v1.0, Chapter 5.
• Multiple comparisons & family-wise budgeting (for multi-group comparisons): see this volume, Chapter 6.
• Meta-analysis & research synthesis: see this volume, Chapter 13.
• Cross-domain calibration transfer for imaging: see Methods.Imaging v1.0, Chapters 9 & 14.

IX. Quality & Risk Control

1. SLI/SLO
   PI_coverage@0.95 ≥ 0.92; converged == true; cond(Hessian) ≤ cond_max; latency_ms_p99 ≤ 800.
2. Risks & Fallbacks
   • Variance estimates at the boundary: use profile likelihood or regularizing priors; if needed, fall back to full-pooling or no-pooling baselines with alerts.
   • Extreme groups: trigger C30-1214, cap w_g, widen intervals, and schedule extra sampling or group merges.
   • Drift: monitor the distribution of w_g and group-level residual drift (see Chapter 7); if thresholds are exceeded, retrain and apply strategy cards.

Summary