07-EFT.WP.Core.Threads v1.0 | Chapter 6 — Resource Quotas and Isolation

Home ／ Docs-Technical WhitePaper ／ 07-EFT.WP.Core.Threads v1.0

Chapter 6 — Resource Quotas and Isolation

I. Scope and Objectives

• Define the budgeting model, isolation domains, and enforcement mechanisms for R_cpu, R_mem, and R_io, so multi-tenant, multi-graph concurrency remains predictable and faults are contained.
• Deliver postulates P76-, minimal equations S76-, and the operational flow Mx-5, aligned with I70-5, I70-1, I70-2, I70-6, I70-7, I70-8.
• Provide deterministic resource lanes on crit(G) and other high-priority paths to curb cascading degradation and priority inversion.

II. Terms and Variables

• Budgets and load: R_cpu (vCPUs or shares), R_mem (bytes), R_io (MB/s or IOPS).
• Utilization and occupancy: u_cpu, u_mem, u_io, rho_cpu = ∑ u_cpu / capacity_cpu, rho_io = lambda_io / mu_io.
• Isolation: iso_dom (cgroup/namespace/cpuset), affinity (CPU/NUMA), quota (hard), burst (short over-commit).
• Threads/graphs: thr, pid_thr, gid, prio, K_thr, w(v), c(e), crit(G).
• Safety margin: headroom ∈ (0,1), reserve_mem, reserve_io.

III. Postulates P76 (Resources and Isolation)

• P76-1 (Quota-as-contract): every gid must declare {R_cpu,R_mem,R_io} and headroom; without it, admission to the run queue is denied.
• P76-2 (Hard bound first): quota is a hard ceiling; burst is permitted only when resources are slack and rho_* < 1 - headroom.
• P76-3 (Stability threshold): any domain must sustain rho_cpu < 1 - headroom and rho_io < 1 - headroom, else trigger degradation and rate limiting.
• P76-4 (Accounting consistency): all metering is sampled on tau_mono; settlement windows align to SLA_window; audits are persisted in ts.
• P76-5 (Layered isolation): isolate by tenant → graph (gid) → thread (thr); escalation across layers only reduces, never increases, allocations.
• P76-6 (Affinity preserves order): nodes on crit(G) keep fixed affinity; lower prio threads must not preempt critical cores.
• P76-7 (Memory first): R_mem is sized to the working set WSS; enforce R_mem >= WSS * safety_factor; otherwise reject early or shard.
• P76-8 (Reverse pressure): when q_len crosses thresholds or u_* approaches limits, shrink burst first, then lower K_thr, then engage rate_limiter.

IV. Minimal Equations S76 (Capacity and Bounds)

• S76-1 (CPU headroom): rho_cpu = ( ∑_i u_cpu_i ) / capacity_cpu <= 1 - headroom。
• S76-2 (IO steady state): rho_io = lambda_io / mu_io <= 1 - headroom。
• S76-3 (No memory thrash): ∑_i WSS_i + cache_working <= R_mem_total - reserve_mem。
• S76-4 (Node runtime lower bound on shared cores):
  T_node(v) >= cycles(v) / ( share(v) * freq_eff )，where share(v) = R_cpu(v) / ∑_co_scheduled R_cpu。
• S76-5 (Graph makespan vs. resources):
  T_make(G) >= ∑_{v ∈ crit(G)} T_node(v) + ∑_{e ∈ crit(G)} c(e)。
• S76-6 (NUMA cross-domain penalty, approximate):
  lat_remote approx lat_local + delta_numa，with delta_numa given by the QPI/UPI topology.

V. Resource Domain Model and Parameters

1. CPU
   • Express R_cpu in shares or vCPUs; prio orders allocation within the same domain.
   • Recommend headroom_cpu ∈ [0.1, 0.3]; for control plane and crit(G), use dedicated cores or high shares.
   • Coalesce tiny nodes: batch minuscule w(v) to cut context-switch overhead.
2. Memory
   • Declare WSS: WSS = P95(working_set_samples); typical safety_factor ∈ [1.2, 1.5].
   • High-water and OOM policy: R_mem_high < R_mem_max; upon breach, throttle → degrade → reclaim caches → terminate edge-most threads.
3. IO
   • Split R_io into throughput and IOPS: R_io = {MBps, IOPS}; cap IOPS to tame small-block jitter.
   • Control write amplification: sequentialize flushes, coalesce small writes, enable batched ACK; isolate the logging path.

VI. Isolation Domains and Topology

1. Logical layering
   • iso_dom(tenant) → iso_dom(gid) → iso_dom(thr); bind respectively to cgroup, cpuset, and io.max.
   • Each layer maintains its own quota and burst; cumulative usage must not exceed the parent.
2. Affinity and NUMA
   • Pin to one NUMA node; reserve R_mem on the same node; cross-node access is fallback only.
   • For latency-sensitive chains, set affinity = dedicated_cores and avoid frequent migrations.

VII. Admission and Queuing (with I70-2)

• Admission rule
  admit(gid) iff rho_cpu_pred < 1 - headroom_cpu and rho_io_pred < 1 - headroom_io and R_mem_free >= R_mem_req.
• Queue shaping
  Weighted fair sharing by prio and share; K_thr caps are jointly decided by R_cpu and rho_*.
• Critical-path preference
  Boost nodes on crit(G) via w_crit_boost so the lower bound in S76-5 is achievable.

VIII. Quota Enforcement Algorithms and Policies

• CPU shares + cap: allocate time slices by shares; enforce quota_period and quota_limit to bound bursts.
• IO tokens: token bucket for R_io; bucket_rate = MBps, bucket_burst = burst_window * MBps.
• Memory watermarks: low/high/max thresholds; above high, start reclaim and limit_acquire; above max, terminate the smallest-impact set.
• Degradation order
  Revoke burst → lower K_thr → drop low-prio non-critical work → trip ingress (paired with Chapter 7 rate limiting).

IX. Observability and Alerting (I70-7/I70-8)

1. Key metrics
   • Threads.cpu.rho, Threads.io.rho, Threads.mem.wss, Threads.mem.oom_events
   • Threads.quota.throttle_ms, Threads.burst.active, Threads.numa.remote_ratio
2. Thresholds
   • If rho_cpu > 1 - headroom for SLA_window/10, alert and auto-remediate.
   • If remote_ratio > 5% on crit(G), trigger affinity re-layout.
3. Contract assertions
   • {"type":"capacity_headroom","rho_cpu_le":1-headroom}
   • {"type":"wss_fit","WSS_le":R_mem/safety_factor}
   • {"type":"isolation","no_cross_tenant_contention":true}

X. Interface Bindings (I70-5 / I70-1 / I70-2 / I70-6)

• set_quota(scope:str, R:dict) -> None
  e.g., {"cpu":2,"mem":"8Gi","io":{"MBps":50,"IOPS":2000}}; success creates/updates iso_dom(scope).
• reserve(scope:str, R:dict) -> Ticket / release(ticket) -> None
  Runtime temporary reservation subject to burst and headroom.
• set_affinity(thr, affinity:list[int]) and set_priority(thr, prio:int)
  Bind critical cores and threads; within a gid, enforce a priority ceiling to avoid inversion.
• Rate-limiter coupling
  When throttle_ms rises, automatically reduce send-side QPS to stabilize rho_*.

XI. NUMA Best Practices

• Co-locate w(v) with its data source; pin(thr) -> numa_node(data).
• Use huge pages for read-only models or long-lived caches; avoid frequent map/split churn.
• If lat_remote inflates P99, prefer migrating the thread rather than the data path.

XII. Fault Containment and Recovery

• Soft containment
  On quota breach, throttle and degrade first; preserve crit(G) resources; pause new admissions.
• Hard containment
  At the max threshold, forcibly terminate the smallest-impact set (low prio, no hb dependents) and record eid_killset.
• Recovery
  Gradually restore burst and K_thr; verify rho_* retreat and P99 recovery within SLA_window.

XIII. Operational Flow Mx-5 (Go-Live Steps)

• Estimate WSS, lambda_io, cycles; set {R_cpu,R_mem,R_io} and headroom; carve iso_dom.
• Assign affinity and prio for crit(G); dedicate cores or high shares on its channels.
• Bind interfaces: set_quota → reserve (optional) → set_affinity; install IO token buckets and memory watermarks.
• Configure alerts and contract assertions; run compliance drills to validate S76-* and P76-*.
• Warm up and observe rho_*, P99, remote_ratio; on threshold breach, execute: revoke burst → reduce K_thr → rate-limit.
• Factory acceptance: assert_thread_contract passes; SLOs met; retain audit and replay artifacts.

XIV. Deliverables and Acceptance Criteria

• Resource baselines at three tiers (tenant / graph / thread) and the iso_dom topology.
• Quota and policy configs (CPU shares, IO tokens, memory watermarks, headroom, burst).
• Affinity and priority lists, dedicated-core mapping for crit(G).
• Observability dashboards and alert thresholds, contract assertion report.
• Fault-containment & recovery playbooks, example eid_killset, and replay logs.