51-Pipeline Card Template v1.0 | Chapter 3 — Architecture & Computation Graph (DAG/State Machine)

Home ／ Docs-Technical WhitePaper ／ 51-Pipeline Card Template v1.0

Chapter 3 — Architecture & Computation Graph (DAG/State Machine)

I. Purpose & Scope

• Provide unified conventions for pipeline architecture and computation graph (DAG(V,E)) with its state machine, covering topological constraints, stage boundaries, transactions & idempotency, checkpoints & replay, Exactly-Once/At-Least-Once semantics, and compliance modules.
• For stages involving path quantities (arrival time/phase), the text must explicitly show gamma(ell) and measure d ell; the data side records delta_form ∈ {general, factored}; publication requires p_dim = 1.0.

II. Definitions & Objects

• Pipeline: processing flow composed of stages and directed edges.
• DAG(V,E): directed acyclic graph with stage set V and dependency edge set E.
• Stage: minimal executable unit with input/output contracts and quality gates.
• Operator: parallelizable operator inside a stage.
• Artifact: stage outputs (tables/files/reports/figures).
• ctx: execution context (parameters, environment, idempotency_key, trace IDs).
• State Machine: state ∈ {pending, running, succeeded, failed, rolled_back} with transition set T.

III. DAG Modeling

• Acyclicity: DAG must be cycle-free; implicit dependencies are forbidden (all edges declared explicitly).
• Edge semantics: (u → v) ∈ E means artifacts for v are complete and pre-gates satisfied; for bypass/optional inputs, use conditional edges cond(u→v).
• Parallelism & branches: branch nodes propagate artifacts with copy semantics; join nodes aggregate with alignment semantics (same partition/window).
• Idempotency & domains: each path deduplicates by idempotency_key = f(run_id, partition, window, …) to keep outputs invariant under retries.
• Checkpoints & replay: a stage v may declare checkpoints; replay must follow reverse-topology order and idempotency, without breaking upstream artifact consistency.
• Contract alignment: schemas (fields/units/dimensions) at both ends of an edge must match or be explicitly mapped; implicit unit conversion is not allowed.

IV. State Machine & Transitions

1. Basic transitions: pending → running → {succeeded | failed}; succeeded/failed → rolled_back only if checkpoint/compensation exists.
2. Guards:
   • pending→running: all deps succeeded and gates G1–G3 pass.
   • running→succeeded: stage gates G1–G8 all pass.
   • running→failed: any of S1–S5 (dimensional/freshness/path/covariance/citation failure).
   • failed→rolled_back: compensation/rollback defined and audit ready.
3. Retry policy: retry_policy = {max_retries, backoff, deadline}; retries must not change idempotency_key nor the input snapshot.
4. Exactly-Once/At-Least-Once: default At-Least-Once; declaring Exactly-Once requires dedupe and output idempotency evidence (primary keys/idempotent writes).

V. Path-Aware Stage Requirements

• Inputs must explicitly include: gamma(ell), d ell, n_eff(ell), c_ref, and (for phase) λ_ref; record delta_form on data side.
• Unified forms:
  T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell ); T_arr = ( ∫ ( n_eff / c_ref ) d ell );
  Phase: Phi = ( 2π / λ_ref ) * ( ∫ n_eff d ell ).
• Sampling consistency: len(gamma_ell)=len(d_ell)=len(n_eff)≥2; Δell ≤ ( c_ref / f_s ) / max(n_eff).

VI. Consistency Invariants

• Schema invariant: artifacts along edges satisfy target input contracts; unit/dimension closure (p_dim = 1.0).
• Side-effect isolation: external side effects must be compensable/idempotent; no irrecoverable state after failure.
• Resources & timebase: clock_state="locked", |ts_start − calib.timestamp| ≤ τ_calib; jitter & drift within thresholds (σ_y(τ) targets).
• Covariance consistency: cov_group and kernel params aligned with the Error Budget; Σ positive definite.

VII. Machine-Readable Artifacts
A. pipeline_graph.json

{

"version": "1.0.0",

"nodes": [

{"id":"ingest"},

{"id":"calibrate"},

{"id":"arrival","checkpoint":true,"retry_policy":{"max_retries":3,"backoff":"exp","deadline":"10m"}}

],

"edges": [

{"from":"ingest","to":"calibrate"},

{"from":"calibrate","to":"arrival","cond":"schema_ok && gates_ok"}

],

"idempotency_key":"run_id+partition"

}

B. state_machine.yaml

version: "1.0.0"

states: [pending, running, succeeded, failed, rolled_back]

transitions:

- { from: pending, to: running, when: "deps_succeeded && G1..G3" }

- { from: running, to: succeeded, when: "G1..G8 && !S1..S5" }

- { from: running, to: failed, when: "any(S1..S5)" }

- { from: failed, to: rolled_back, when: "has_checkpoint && do_compensate" }

gates: ["G1","G2","G3","G4","G5","G6","G7","G8"]

stops: ["S1","S2","S3","S4","S5"]

C. stage.yaml (excerpt)

version: "1.0.0"

stage:

id: "stage-30-arrival"

inputs: ["path.gamma_ell","path.d_ell","medium.n_eff_profile","ref.c_ref"]

outputs: ["obs.T_arr"]

control:

form: "T_arr = ∫ ( n_eff / c_ref ) d ell"

delta_form: "general"

quality_gates: ["G1","G3","G4","G6"]

see:

- "EFT.WP.Core.Equations v1.1:S20-1"

- "EFT.WP.Core.Metrology v1.0:check_dim"

VIII. Validation & Gates

• Pipeline-level G1–G8: schema completeness, citation compliance, path conventions, dimensional closure, freshness, coverage conventions, covariance consistency, uniqueness; triggering S1–S5 blocks execution and release.
• /validate must return per-node and global gate pass status and stop reasons; dimensional closure follows check_dim_report.json.

IX. Monitoring & Alerts

• KPIs: Latency_P50/P95, Throughput, ρ, Q_res, p_dim, σ_y(τ), ε_flux.
• Alerts: transition failures, PD failures, gate breaches, frequent replays, idempotency conflicts; support silence windows and suppression.
• Observability: each stage emits audit.jsonl events with idempotency_key, input hashes, retry counts, and final state.

X. Release & References

• Artifacts: pipeline_graph.json, state_machine.yaml, per-stage stage.yaml, check_dim_report.json, audit.jsonl, figures, and report_manifest.yaml; dual exports (PDF/SVG and PNG).
• Citation format: See "<Volume> vX.Y" <Chapter> <Anchor>, prioritize P/S/M/I; external links/aliases are forbidden.