07-EFT.WP.Core.Threads v1.0 | Chapter 5 — Timeouts, Retries, and Idempotency

Home ／ Docs-Technical WhitePaper ／ 07-EFT.WP.Core.Threads v1.0

Chapter 5 — Timeouts, Retries, and Idempotency

I. Scope and Objectives

• Establish a unified family of strategies for timeout, retry, and idempotency so that latency is predictable on the tau_mono time base and external side-effects remain safe.
• Provide upper-bound equations, composition rules, parameterization guidance, and quality gates, yielding postulates P75-, minimal equations S75-, and the operational flow Mx-4.
• Remain consistent with interfaces I70-4 (timeouts/retries), observability I70-7, contracts I70-8, and cross-volume arrival-time T_arr calibration.

II. Terms and Variables

• Time budget: timeout, deadline, J (single jitter), J_total = ∑ J_i。
• Attempt counts: retries (additional tries, not total), attempts = retries + 1。
• Failure/retry probabilities: p_drop, p_retry。
• Idempotency: idemp_key, Delta_t_dedup, dedup_table。
• Clocks: tau_mono, ts (external audit); mapping is defined in Chapter 4.

III. Postulates P75

• P75-1 (Baseline consistency): all strategy timers are measured on tau_mono; only audits are recorded in ts.
• P75-2 (Budget closure): any external call must satisfy deadline >= W_retry + margin.
• P75-3 (Bounded failure): once attempts, timeout, and J_total are declared, the latency upper bound is governed by S75-1.
• P75-4 (Idempotency first): any path with sem="at_least_once" or sem="exactly_once*" must declare idemp_key and Delta_t_dedup.
• P75-5 (Window dominance): Delta_t_dedup >= span_reorder, and it must be consistent with the storage ttl.
• P75-6 (Cancellation propagation): a cancel_token that crosses hb edges takes precedence over subsequent retries.
• P75-7 (SLO coupling): the choice of timeout and retries is constrained by SLA_window and the P99 budget.

IV. Minimal Equations S75 (Upper Bounds and Success Rate)

• S75-1 (Latency upper bound): W_retry <= timeout * ( retries + 1 ) + J_total。
• S75-2 (Success rate approximation, independent drops): p_ok = 1 - ( p_drop )^( retries + 1 )。
• S75-3 (Expected attempts, geometric approximation): E[attempts_used] approx ∑_{k=1}^{retries+1} ( p_drop )^(k-1 )。
• S75-4 (Split budget): budget_total = W_q_in + w(proc) + W_q_out + W_net <= timeout。
• S75-5 (Composite deadline): deadline = max_i( W_retry_i ) (parallel branches take the max), or deadline = ∑_i W_retry_i (serial branches sum).

V. Timeout Strategy Families (on tau_mono)

1. Fixed timeout: timeout = const for low-variance RPCs.
2. Dynamic timeout: timeout = Pq(latency_hist, q) with q ∈ {0.95, 0.99} and a safety factor multiplier.
3. Phased timeout: timeout_phase_i for connection / first-byte / full-body phases.
4. Composition guidance
   • Control plane favors short timeout; data plane favors phased or dynamic timeouts.
   • Nodes on crit(G) should receive tighter timeout and explicit alerting.

VI. Retry Strategy Families and Jitter

1. Fixed interval: backoff_k = base。
2. Exponential backoff: backoff_k = base * factor^k。
3. Capped backoff: backoff_k = min( cap, base * factor^k )。
4. Jitter injection (recommended): backoff_k = min( cap, base * factor^k ) + U( -jitter, +jitter )。
5. Deadline conservation: ∑ backoff_k + attempts * timeout <= deadline - margin。
6. Trigger conditions
   • Retry for confirmable failures (5xx, timeout, p_drop); do not retry business errors (4xx, semantic denial).
   • Respect retry-after when present; it overrides local backoff_k.
7. Bypass strategies
   • Hedging (mirrored requests): bound N_hedge, bind idemp_key; converge on the fastest success and cancel the rest.
   • Circuit breaker: trip on error-rate/latency thresholds, half-open probes to restore.

VII. Idempotency and Deduplication

1. Definition: f(x; idemp_key) = f(x; idemp_key)。
2. Key design
   • Natural key: business primary key or (resource_id, op_type, ts_bucket)。
   • Proxy key: hash_sha256( payload || ts_bucket || uid )。
3. Dedup table
   • Structure: dedup_table = { idemp_key -> ts_last }。
   • Rule: if tau_mono_now - ts_last <= Delta_t_dedup then drop else accept & update。
4. Approximate once semantics (exactly_once*)
   • Outbox/Inbox: co-transactional message write or consumer-first idempotent write.
   • Atomic tag: atomic_write( idemp_key, state ) plus consumer offset checks.
   • Engineering conditions: atomic ∧ dedup(Delta_t_dedup) ∧ ordered_ack。

VIII. Policy Composition and Compatibility

1. Composition sequence
   Choose sem → generate idemp_key → set Delta_t_dedup → choose timeout → configure retries/backoff → align deadline → bind alerts and fallback.
2. Compatibility matrix
   • at_most_once ↔ short timeout, zero/rare retries, no dedup_table.
   • at_least_once ↔ normal retries, must be idempotent with dedup.
   • exactly_once* ↔ idempotent store, dedup, ordered ACK, potential throughput trade-offs.
3. Interaction with backpressure/limiting
   • When rho >= 1 or q_len exceeds thresholds, first tighten retries and lengthen backoff_k to avoid amplifying congestion.
   • Combine with rate_limiter on the send side to preserve stability.

IX. Interface Bindings and Policy Prototypes (I70-4 / I70-7 / I70-8)

• with_timeout(timeout:float) -> ctx
  Establishes remaining time on tau_mono upon entering the context; on expiry, raises a retry-eligible error type.
• retry(policy:dict) -> callable
  Suggested fields: {"attempts":int,"base":ms,"factor":float,"cap":ms,"jitter":ms,"retry_on":[codes],"respect_retry_after":true}。
  Compliance assertion: assert ∑ backoff_k + attempts * timeout <= deadline - margin (pre-run check).
• ensure_idempotent(fn, key_fn, window:float)
  Requires window == Delta_t_dedup; key_fn must be a pure function and a stable mapping to the business key.
• Observability (metric_emit)
  Emit Threads.retry.rate, Threads.timeout.rate, Threads.dedup.drop_rate, Threads.hedge.count, Threads.circuit.open_ratio.

X. Quality Indicators and Thresholds (SLI/SLO)

1. Indicators
   • SLI.retry_rate = retries_total / requests_total
   • SLI.timeout_rate = timeouts / requests_total
   • SLI.dup_drop_rate = dedup_drops / consumed_total
   • SLI.p99_latency_ms, SLI.p50_latency_ms, SLI.error_ratio
2. Suggested gates
   • Control plane: SLI.timeout_rate <= 0.5%, p99_latency_ms <= SLO_control
   • Data plane: dup_drop_rate <= 1% (may relax in reordering peaks, but must regress afterward)
3. Self-healing
   When retry_rate or timeout_rate breaches thresholds, automatically lengthen backoff_k or reduce QPS, and record eid_policy_change.

XI. Contracts and Test Matrix (with I70-8)

1. Assertion templates
   • {"type":"timeout_upper_bound","W_retry_le":budget}
   • {"type":"retry_success_prob","p_ok_ge":target}
   • {"type":"idempotency","key_stability":true,"window":Delta_t_dedup}
   • {"type":"dedup_window","span_reorder_le":Delta_t_dedup}
   • {"type":"semantics","sem":"at_least_once","no_side_effect_dup":true}
2. Reordering/duplication cases
   • Construct span_reorder near the window bound; verify no misses and no false drops.
   • Inject jitter J and network partitions; verify W_retry and p_ok track S75-* predictions.

XII. Operational Flow Mx-4 (Deployment Steps)

• Select delivery semantics sem; produce the idemp_key design and Delta_t_dedup.
• Set timeout and backoff prototypes from historical latency, satisfying S75-1 and S75-5.
• Configure retry(policy) and with_timeout(timeout); set deadline and alerts for critical calls.
• Bind ensure_idempotent(fn,key_fn,window); persist or cache the dedup_table.
• Execute the contract test matrix; verify p_ok and W_retry; record baseline indicators.
• Post-launch, continuously collect SLI.*; upon threshold breaches, trigger policy self-healing and audit replay.

XIII. Exit Criteria and Deliverables

• Full pass on P75-, S75-, Mx-4; SLIs achieve targets; assert_thread_contract report passes.
• Deliverables include: the policy catalogue and parameters (attempts/base/factor/cap/jitter/timeout/window), the idemp_key specification, Delta_t_dedup derivation, thresholds and self-healing rules, audit samples, and replay scripts.