09-EFT.WP.Core.Density v1.0 | Chapter 8 — Regularization and Maximum Entropy

Home ／ Docs-Technical WhitePaper ／ 09-EFT.WP.Core.Density v1.0

Chapter 8 — Regularization and Maximum Entropy

I. Purpose and Scope

• Unify the inverse-problem formulation for density fields as a weighted sum of a data fidelity term D(y, A rho) and regularizers R_k(rho), covering the two principal families: L2/L1/TV and maximum entropy (MaxEnt).
• Clarify how probability density p(x) and physical density rho(x) differ yet interact under regularization and entropy constraints; all integrals must state both measure and domain explicitly.
• Deliver an executable workflow Mx-97 (regularization parameter selection) aligned with mass conservation S92-11 (Chapter 7), Jacobian-bearing variable transforms and normalization S92-14/15 (Chapter 9), and uncertainty interfaces (Chapter 10).

II. Symbols and Assumptions

• Variables and operators: observations y, forward operator A, density field rho(x) or pdf p(x), domain X, measure dx or dV.
• Regularizer family and weights: {R_k(rho)}, weights alpha_k ≥ 0, collected as alpha.
• Constraints and feasible set: C(rho) = 0/≥0. Typical constraints: nonnegativity and conservation, e.g., rho(x) ≥ 0, M = ( ∫ rho dV ). In the probabilistic case, enforce ( ∫ p(x) dx ) = 1.
• Statistical fidelity: choose D(·,·) consistent with noise (Poisson/ Gaussian ⇒ log-likelihood / L2). If the time axis is aligned via T_arr, record delta_form in metadata.

III. Unified Inverse-Problem Objective and Fidelity Terms

• Convex master objective:
  S92-41 : J(rho) = D(y, A rho) + ∑_k alpha_k * R_k(rho) , subject to rho ∈ C.
• Gaussian error (weighted least squares):
  S92-42 : D_2(y, A rho) = (1/2) * || W * ( A rho - y ) ||_2^2.
• Poisson counts (negative log-likelihood):
  S92-43 : D_P(y, A rho) = ( ∫ ( A rho ) dV ) - ( ∫ y * log( A rho + eps ) dV ).
• Dimensional soundness: check_dim( D + ∑ alpha_k R_k ) must pass. The small eps is for numerical stability; publish its value.

IV. Regularizer Library (L2 / L1 / TV and Prior Mapping)

• Tikhonov / L2:
  S92-44 : R_L2(rho) = (1/2) * || L rho ||_2^2 where L may be identity, gradient, or Laplacian.
• Sparsity / L1:
  S92-45 : R_L1(rho) = || L rho ||_1.
• Total Variation (TV, continuous isotropic):
  S92-46 : R_TV(rho) = ( ∫ sqrt( (∂_x rho)^2 + (∂_y rho)^2 + (∂_z rho)^2 ) dV ).
• MAP view (regularizers as negative log-priors):
  S92-53 : p(rho) ∝ exp( - ∑_k alpha_k * R_k(rho) ), so maximizing the posterior equals minimizing J(rho).

V. Maximum Entropy (MaxEnt: Shannon / Relative Entropy)

• Shannon entropy (continuous):
  S92-47 : H[p] = - ( ∫ p(x) * log p(x) dx ).
• Classical constraints (moments + normalization):
  S92-48 : maximize H[p] , s.t. ( ∫ p dx = 1 ) , ( ∫ phi_k(x) * p(x) dx = m_k , k=1..K ).
• Exponential-family solution:
  S92-13 : p*(x) ∝ exp( ∑_{k=1}^K lambda_k * phi_k(x) ), with partition Z(lambda) = ( ∫ exp( ∑ lambda_k phi_k(x) ) dx ).
• Relative entropy with reference q(x):
  S92-51 : D_KL( p || q ) = ( ∫ p(x) * log( p(x) / q(x) ) dx ),
  so p*(x) ∝ q(x) * exp( ∑ lambda_k * phi_k(x) ).
• Discrete (histogram):
  S92-52 : H[p] = - ∑_i p_i * log p_i, with ∑ p_i = 1 and ∑ p_i * phi_k(i) = m_k.

VI. Entropization and Nondimensionalization for Physical Density

• Normalize the physical density prior to MaxEnt:
  S92-58 : p_rho(x) = rho(x) / M , M = ( ∫ rho dV ).
• Impose MaxEnt and moment constraints on p_rho, solve for p_rho*, and map back:
  rho*(x) = M * p_rho*(x). The mass M may be set by conservation or metrological reference (Chapter 2 and Chapter 7 via S92-11).

VII. Bias–Variance Trade-offs and Model Selection

• Structural risk: increasing alpha_k typically decreases variance and increases bias; decreasing alpha_k does the opposite.
• Information criteria / effective dof:
  S92-55 : AIC = 2 * k_eff - 2 * log L_hat, BIC = k_eff * log N - 2 * log L_hat.
• GCV (linear case):
  S92-56 : GCV = ( RSS / N ) / ( 1 - trace(H)/N )^2, where H is the hat matrix.
• MaxEnt dual optimality: use dual gap and constraint residuals to decide convergence.

VIII. Workflow Mx-97 — Regularization and MaxEnt Parameter Selection

• Problem setup. Choose D (S92-42/43), {R_k} (S92-44/45/46), and C (nonnegativity, conservation, etc.). Use p(x) for probability fields; for physical fields, use rho(x) then entropize via S92-58.
• Paths over hyperparameters. Define a log-grid or adaptive path for alpha; for MaxEnt, set moment targets m_k and bases phi_k.
• Numerics. L2: CG or normal equations; L1/TV: ADMM/FISTA; MaxEnt: Newton–dual or quasi-Newton.
• Selection criteria. Cross-validation, AIC/BIC/GCV, data fidelity, and conservation error scored jointly; optionally constrain ||∇rho||_1.
• Stopping and validation. KKT residual, dual gap, sum_pDelta = 1 or mass error S92-11 ≤ tol; if time is aligned via T_arr, record delta_form.
• Publication and provenance. Persist alpha*, scores, residuals, mass_rel_err, ts, tau_mono, fmt, q_score.

IX. Constraints and Boundary Conditions

• Nonnegativity and conservation:
  S92-57 : rho(x) ≥ 0 , ( ∫ rho dV = M ); boundaries may be Dirichlet/Neumann/Robin (see Chapter 2).
• TV boundary handling: reflect / zero-pad / periodic—document the choice.
• Variable transforms + Jacobian: follow S92-15 (Chapter 9); any coordinate map must include | det( ∂x/∂u ) |.

X. Numerical and Robustness Guidelines

• Preprocessing: de-trend and scale z = ( x - mu_x ) / sigma_x (S92-14) but report in physical units after inversion.
• Stable sparsity: for L1/TV use safeguarded step sizes, backtracking, and a small eps_tv to avoid oscillation at kinks.
• Ill-conditioned operators: precondition A (left/right) or screen with an L-curve; combine L2+TV if needed.
• Randomization: for large-scale problems, randomized solvers / subsampling are acceptable, provided fs and anti-aliasing satisfy Core.Sea constraints.

XI. MaxEnt Dual Conditions and Solvability

• Lagrangian:
  S92-50 : L[p, lambda] = -H[p] + lambda_0 * ( ( ∫ p dx ) - 1 ) + ∑_k lambda_k * ( ( ∫ phi_k p dx ) - m_k ).
• First-order optimum ⇒ exponential family S92-13; the dual minimizes log Z(lambda) - ∑_k lambda_k m_k.
• Feasibility: the moment targets {m_k} must lie in the attainable moment set; numerically verify dual convexity and positive-definite Hessian.

XII. Reporting and Compliance

• Required: fidelity type and parameters, chosen regularizers and alpha*, boundary conditions, feasible set, KKT/dual gaps, sum_pDelta or mass_rel_err, ENBW_Hz/U_w when regularizing in the spectral domain.
• Data governance: record ts and tau_mono; if cross-domain alignment uses T_arr, attach delta_form; variable names and units must follow the volume-wide conventions.

XIII. Cross-Volume and Cross-Chapter Consistency

• With Chapter 7: conservation S92-11 must hold after regularization and any regridding.
• With Chapter 4: kde_h(x) is a special case of R_L2 (kernel smoothing); bandwidth h and regularization strength play dual roles.
• With Chapter 6: spectral smoothing must co-publish ENBW_Hz and window power U_w.
• With Chapter 9: all transforms and normalization follow S92-14/15.
• With Chapter 10: use I90 7 to compute I_F/CRLB and quantify how alpha* impacts estimator variance.

XIV. Key Formula Index (for quick reference)