09-EFT.WP.Core.Density v1.0 | Chapter 3 — Probability Densities and Sampling

Home ／ Docs-Technical WhitePaper ／ 09-EFT.WP.Core.Density v1.0

Chapter 3 — Probability Densities and Sampling

I. Objectives and Scope

• Establish a unified treatment for pdf/cdf/pmf, make sampling assumptions explicit (iid / weak dependence), and normalize handling for truncation/censoring, weights / re-scaling, and consistent likelihood and information expressions.
• Release this chapter’s minimal equations: S92-3 (likelihood) and S92-4 (Fisher information); provide the estimation workflow Mx-92 and publication checklist fields, aligned with Core.Sea’s timeline and window Delta_t.

II. Basic Objects and Notation

• Random variables and samples.
  X : Omega → R^d, samples x_1,...,x_N, with wall-clock tags ts_i and monotone clock tau_mono (see Core.Sea Chapter 3).
  Support supp(p) ⊆ R^d; measures are explicit: continuous ( ∫ p(x) dx = 1 ), discrete sum_i p_i = 1.
• Distribution functions.
  pdf : p(x); cdf : F(x) = ( ∫_{(-∞,x]} p(u) du ) (in 1D); pmf : P(X=x_k) = p_k.

III. Sampling Models and Independence Assumptions

• iid baseline. {x_i} are conditionally iid given parameter theta; the window width Delta_t must make independence approximately valid (see this volume’s preface on keeping discrete/continuous concepts separate and the approx independence convention).
• Weak dependence and block independence.
  For short-range dependence, define block indices B_j; treat {x_i}_{i∈B_j} as within-block dependent and blockwise nearly independent; form the likelihood as a product over blocks.
• Weighted sampling (correcting selection bias).
  With observation weights w_i > 0, normalize w_i' = w_i / ( (1/N) * ∑ w_i ); use linear weights w_i' in the log-likelihood.

IV. Unified Description of Densities and Mass Functions

• Continuous. p(x) ≥ 0 and ( ∫_{R^d} p(x) dx = 1 ); expectation E[g(X)] = ( ∫ g(x) p(x) dx ).
• Discrete. p_k ≥ 0 and sum_k p_k = 1; expectation E[g(X)] = sum_k g(x_k) p_k.
• Truncation and censoring.
  Truncation on domain A: conditional density p_A(x) = p(x) / P(X∈A), with P(X∈A) = ( ∫_A p(x) dx ).
  Right-censor time c_i: observed y_i = min(x_i, c_i), indicator delta_i = 1_{x_i ≤ c_i}; the per-sample likelihood term is p(y_i) when delta_i=1, or survival S(c_i) = 1 - F(c_i) when delta_i=0.

V. Likelihood, Score, and Information (Minimal Equations)

• Minimal equation S92-3 (likelihood).
  S92-3 : L(theta) = ∏_{i=1}^N p(x_i | theta); log-likelihood ell(theta) = ∑_{i=1}^N log p(x_i | theta).
  With weights: ell_w(theta) = ∑ w_i' * log p(x_i | theta).
• Minimal equation S92-4 (Fisher information).
  S92-4 : I_F(theta) = E[ ( ∂_theta log p(X|theta) ) ( ∂_theta log p(X|theta) )^T ].
  Sample approximation: I_F_hat(theta) = (1/N) * ∑ s_i s_i^T, with s_i = ∂_theta log p(x_i | theta).
• MLE and MAP.
  theta_MLE = argmax_theta ell(theta); with prior pi(theta), theta_MAP = argmax_theta ( ell(theta) + log pi(theta) ).

VI. Change of Variables and Standardization

• Invertible transform y = g(x). p_Y(y) = p_X( g^{-1}(y) ) * | det( ∂x/∂y ) | (the Jacobian is mandatory per this volume’s rule).
• Standardization (aligned with Chapter 9).
  z = ( x - mu_x ) / sigma_x; if modeling and publishing p_Z(z), then provide
  p_X(x) = p_Z( (x - mu_x)/sigma_x ) * ( 1 / |sigma_x| ).

VII. Parametric Families and Mixtures

• Exponential family.
  p(x|theta) = h(x) * exp( eta(theta) • T(x) - A(theta) ); convexity supports well-behaved ell(theta) optimization.
• Mixtures.
  p(x | {pi_k, theta_k}) = ∑_{k=1}^K pi_k * p_k(x | theta_k), with pi_k ≥ 0 and sum pi_k = 1. Estimation typically uses EM: E-step responsibilities, M-step maximization of expected complete-data likelihood.

VIII. Sampling and Generation

• Rejection sampling.
  With proposal q(x) and c ≥ 1 such that p(x) ≤ c q(x): acceptance rate is 1/c. Record the reject fraction for quality audit.
• Importance sampling.
  Estimate E_p[g(X)] = ( ∫ g(x) p(x) dx ) = ( ∫ g(x) w(x) q(x) dx ) with weights w(x) = p(x)/q(x); use normalized weights to reduce variance.
• MCMC notes.
  Publish chain autocorrelation time and effective sample size (ESS), and include convergence diagnostics (e.g., R_hat).

IX. Unified Likelihood for Truncation/Censoring/Missingness

X. Estimation & Publication Workflow Mx-92 (Samples → Parameters → Density)

• Data intake. Validate ts/tau_mono, window Delta_t, and near-independence; record unit(•) and dim(•) (per P91-2).
• Model selection. Choose parametric / mixture / semi-parametric (KDE in Chapter 4); declare supp(p), truncation/censoring policies.
• Likelihood construction. Build ell(theta) or ell_w(theta) from type_i and normalized weights w_i'; add regularization or priors if needed.
• Solve and diagnose. Report theta_hat, gradient norm, Hessian/information I_F_hat, condition number, and optimizer exit status.
• Uncertainty. Approximate covariance cov(theta_hat) ≈ I_F_hat^{-1}; intervals via theta_hat ± z * sqrt(diag(cov)) or Delta-method (consistent with Chapter 10).
• Goodness-of-fit. KS/AD tests, log score -ell(theta_hat)/N, AIC/BIC, and posterior predictive checks if priors are used.
• Publish & ingest. Serialize pdf parameters, supp, method, diagnostics, ess/accept_rate, and tag the manifest Mx-92-pass|warn|fail.

XI. Quality Control and Common Failure Modes

• Normalization errors. Check ( ∫ p(x) dx ) ≈ 1 or sum p_k = 1; exceeding tolerance is a hard fail.
• Model misspecification. Strong residual structure or ill-conditioned information (large condition number) triggers “family expansion” recommendations (e.g., from unimodal to mixture).
• Selection bias. Large variance in w_i' lowers ESS; report ESS = ( ∑ w_i' )^2 / ∑ (w_i')^2.
• Timeline misalignment. If density depends on arrival time T_arr, record both formulations and delta_form (see Core.Sea Chapter 8).

XII. Implementation Bindings (I90)

• pdf_fit(data:any, family:str, init:dict|None=None) -> PdfRef — embeds S92-3, S92-4, diagnostic matrices, and publication checklist.
• change_of_variables(pdf:PdfRef, mapping:any, jacobian:any) -> PdfRef — the Jacobian is mandatory.
• kde_build(...) and kde_eval(...) — see Chapter 4; connect to Mx-92 as the nonparametric branch.
• bind_to_parameters(ds:any, params:list[str]) -> bool — bind theta_hat, cov(theta_hat) to data assets.

XIII. Minimal Manifest Fields

• sampling = {"model":"iid|block|mcmc", "Delta_t":..., "approx_indep":true|false}
• likelihood = {"family":"...", "weighted":true|false, "trunc":"A|none", "censor":"right|left|interval|none"}
• fit = {"theta_hat":..., "I_F_hat":..., "cov_hat":..., "status":"converged|...", "crit":{"AIC":...,"BIC":...,"logscore":...}}
• diag = {"ESS":..., "accept_rate":..., "cond(H)":..., "KS_p":..., "AD_p":...}
• timing = {"ts":"UTC", "tau_mono":"...", "T_arr":{"gamma":"...","d ell":"...","c_ref":...,"n_eff":"...","delta_form":...}}

XIV. Chapter Highlights

• Solidified S92-3, S92-4 and the Mx-92 workflow; provided unified likelihoods for truncation/censoring/missingness, weighting, and change-of-variable rules.
• Cross-references: Chapter 4 provides kde_h(x) and MISE(h); Chapter 9 supplies publication conventions for cross-domain normalization and transforms; Chapter 10 gives the CRLB and principled interval construction.