19-EFT.WP.Methods.SynthData v1.0 | Chapter 1 — Definition and Scope of the Synthetic Data Domain

Home ／ Docs-Technical WhitePaper ／ 19-EFT.WP.Methods.SynthData v1.0

Chapter 1 — Definition and Scope of the Synthetic Data Domain

One-line objective: Delimit the objects, inputs/outputs, constraints, and boundaries of synthetic data within the EFT framework, establishing a minimal loop that is traceable, auditable, and reproducible.

I. Scope & Objects

1. Input artifacts
   • D_real (real data validated against schema & contracts; see Methods.Cleaning v1.0).
   • SRef (standard schema/field lexicon/primary–foreign keys/units & dimensions).
   • Constraint set Rules (uniqueness, referential integrity, physical/geometry/energy conservation, time/path consistency).
   • Target distribution & utility Goals = {fidelity, utility}.
   • Privacy budget & policy Privacy = {eps, delta, budget, accounting}.
   • Runtime constraints Runtime = {SLO, cap, chan, retry} (see EFT.WP.Core.Threads v1.0).
2. Output artifacts
   • D_syn (single-modal synthetic data) or Bundle = {Tab, TS, Image, Text, Audio, Graph} (multimodal).
   • Evaluation reports {report.fidelity, report.privacy, report.contracts}.
   • Publication manifest manifest.synth with signature.
3. Applicable modalities
   Tabular, time series & event streams, graphs, imaging, and multimodal compositions; both offline batch generation and on-demand streaming.
4. Boundaries & non-goals
   • No adjudication of legal texts; provide compliance interfaces and evidentiary logging only.
   • No prescription of specific training tricks or engineering minutiae; external interfaces unified as I40-*.

II. Terms & Variables

• Data & distributions: D_real ~ p_data(x), D_syn ~ p_model(x; theta), theta, z ~ p(z).
• Conditioning & control: c (condition/prompt), SCM, G=(V,E) (scene graph).
• Time & paths: tau_mono, ts, gamma(ell), T_arr, delta_form, Delta_t.
• Metrology & units: unit(x), dim(x), check_dim(expr).
• Weights & representativeness: w(i), pi(i), n_eff = ( (∑ w)^2 ) / ( ∑ w^2 ).
• Distances & metrics: KL, JS, W1, MMD, FID, KID, covg.
• Privacy & risk: eps, delta, eps_total, MI (membership inference risk).
• Quality & operations: q_score, psi (population stability index), rho (load), P99.

*III. Axioms P401- **

• P401-1 (Traceability): Every generation and evaluation step must persist TraceID, seed, rng, and provenance.
• P401-2 (Schema-first): Bind SRef and Rules before choosing engines and sampling; schema-less generation is prohibited.
• P401-3 (Dimensional conservation): Fields entering equations must carry unit/dim; run check_dim(expr) prior to publication.
• P401-4 (Unified time base): Use tau_mono internally and ts externally; record offset/skew/J at publication.
• P401-5 (Dual arrival-time forms): Whenever paths/propagation are involved, compute both
  T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell ) and T_arr = ( ∫ ( n_eff / c_ref ) d ell ), and evaluate delta_form.
• P401-6 (Explicit privacy accounting): Apply DP(eps, delta) accounting; cumulative eps_total must respect policy budgets.
• P401-7 (Publication gate): Release only when pass = fidelity_ok ∧ privacy_ok ∧ contract_ok ∧ manifest_signed.
• P401-8 (Reproducibility): reproducible(seed); identical SynthSpec, seed/rng, and version yield equivalent samples.
• P401-9 (Minimal leakage): Disclose upper bounds and lower confidence bounds for MI/linkability/attribute risks.
• P401-10 (Streaming stability): On the execution graph G=(V,E), satisfy queue-stability and backpressure constraints (see Core.Threads v1.0).

*IV. Minimal Equations S401- **

• S401-1 (Representativeness & weights)
  n_eff = ( (∑ w_i)^2 ) / ( ∑ w_i^2 ); W_norm = ( ∑ w_i ) / N.
• S401-2 (Arrival-form discrepancy)
  delta_form = | ( 1 / c_ref ) * ( ∫ n_eff d ell ) - ( ∫ ( n_eff / c_ref ) d ell ) |.
• S401-3 (Time mapping)
  ts = map_tau_to_ts( tau_mono; offset, skew ), with jitter bound J.
• S401-4 (Privacy accounting, simple composition)
  eps_total = ( ∑ eps_k ), delta_total = ( ∑ delta_k ).
• S401-5 (Coverage, discrete case)
  covg = | supp(D_syn) ∩ supp(D_real) | / | supp(D_real) |.
• S401-6 (Publication criterion)
  pass = fidelity_ok ∧ privacy_ok ∧ contract_ok ∧ manifest_signed.

V. Synthesis Flow M40-1 (End-to-End)

1. Readiness
   • register_schema(SRef); validate_dataset(D_real, Rules); pass repair_units and check_dim.
   • Fix Goals, Privacy, Runtime, and a first draft of SynthSpec.
2. Design & Modeling
   • Choose engine engine ∈ {copula, VAE, GAN, flow, diffusion, SCM}; set priors/regularizers and controls c.
   • If physical/geometry constraints apply, build SCM or scene graph G=(V,E).
3. Training & Calibration
   fit_engine(D_real, SynthSpec); track convergence & early stopping; lock theta_ref and version.
4. Sampling & Assembly
   sample(engine, n, condition=c, seed); for multimodal, compose_multimodal to assemble Bundle.
5. Rule Execution
   enforce_constraints(D_syn, Rules); deduplicate/foreign_key/unique; align_timepath and write both T_arr forms and delta_form.
6. Evaluation & Thresholds
   • measure_fidelity(D_real, D_syn, metrics={W1,MMD,FID,KID,covg});
   • measure_privacy(D_real, D_syn, attacks, eps_delta);
   • Produce uncertainty & coverage, compute n_eff.
7. Contracts & Release
   assert_synth_contract(contracts); if pass, run freeze_release_synth(D_syn, tag) and sign; otherwise rollback(tag_prev).

VI. Contracts & Assertions (Examples C40-*)

• C40-101 (Schema & referential): unique(pk) = true, foreign_key = true.
• C40-102 (Dimensions): check_dim( y - f(x) ) = true.
• C40-103 (Arrival time): delta_form ≤ tol_Tarr.
• C40-104 (Fidelity): W1 ≤ tol_W1, MMD ≤ tol_MMD, FID ≤ tol_FID, covg ≥ covg_min.
• C40-105 (Privacy): eps_total ≤ bud_eps, MI ≤ tol_MI (with compliant lower CI).
• C40-106 (Scale & coverage): |D_syn| ≥ N_min, and covg_group ≥ covg_min for key enumerations/strata.
• C40-107 (Operations): latency_p99 ≤ SLO.latency, error_rate ≤ SLO.error, streaming rho < 1.

VII. Implementation Bindings I40- (Chapter Anchors)*

• I40-11 design_synth_spec(schema, goals, constraints) -> SynthSpec
• I40-12 fit_engine(ds, model, priors|arch, privacy) -> engine
• I40-13 sample(engine, n, condition, seed) -> ds_syn
• I40-14 enforce_constraints(ds_syn, rules) -> ds_syn'
• I40-15 align_timepath(ds_syn, ref) -> ds_syn' (records both T_arr forms and delta_form)
• I40-16 measure_fidelity(real, syn, metrics) -> report
• I40-17 measure_privacy(real, syn, attacks, eps_delta) -> report
• I40-18 assert_synth_contract(ds_syn, contracts) -> report
• I40-19 freeze_release_synth(ds_syn, tag) -> manifest.synth
• Invariants: reproducible(seed); sum(weights)/N ≈ 1; eps_total under budget; delta_form ≤ tol_Tarr; unit/dim checks passed.

VIII. Cross-References

• Cleaning: units/dimensions, dual formulations, time base & paths, contracts & release — EFT.WP.Methods.Cleaning v1.0, Chapters 4/5/6/10.
• Imaging: physically constrained generation, PSF/noise, time gating — EFT.WP.Methods.Imaging v1.0, Chapters 5/7/13.
• Cross-Statistics: coverage/power/drift & multiplicity — EFT.WP.Methods.CrossStats v1.0, Chapters 4/6/7/14.
• Core: patterns, acquisition, execution graphs, and measures in EFT.WP.Core.DataSpec/Sea/Threads/Density.

IX. Quality Metrics & Risk Control

• SLI/SLO: latency_ms_p99, throughput, error_rate, drift_alert_rate, retrain_mttr.
• Drift & baseline updates: periodic psi/W1/KL monitoring; on threshold crossing, run recalibrate(engine) or rollback.
• Audit & replay: persist TraceID, seed, rng, theta_ref; support “same-manifest replay”.

Summary