22-EFT.WP.Metrology.Instrument v1.0 | Chapter 5 — Linearity and Nonlinear Calibration (INL/DNL/Polynomial/LUT)

Home ／ Docs-Technical WhitePaper ／ 22-EFT.WP.Metrology.Instrument v1.0

Chapter 5 — Linearity and Nonlinear Calibration (INL/DNL/Polynomial/LUT)

One-Sentence Objective
Provide measurable conventions and uncertainties for INL/DNL, and define polynomial- and table-based (LUT) nonlinear calibration workflows and contracts to enable consistent cross-instrument publication.

I. Scope and Objects

1. Scope
   • Static linearity evaluation and calibration for ADC and DAC; dynamic nonlinearity is treated at the level of residuals and cross-validation hints.
   • Supports two common quantization tests: ramp/triangle (code-density method) and sine-density method.
2. Objects
   Default input-referred convention: for ADC, use input physical quantity x as the independent variable; for DAC, use code as the independent variable and convert to input-equivalent.
3. Outputs
   DNL[k], INL[k], missing_codes, monotone, epsilon_nl(x), poly_coeff, LUT, residual_nl_rms, and manifest.instrument.linearity.*.

II. Terms and Variables

• Signals & mappings: T_true(x), T_ideal(x) = G * x + O, T_meas(x) = T_true(x) + n, nonlinearity term epsilon_nl(x) = T_true(x) - T_ideal(x).
• Codes & boundaries: code ∈ {0..2^N-1}, code boundary b_k, code width w_k = b_{k+1}-b_k, ideal width W_ideal = LSB.
• Linearity metrics: DNL[k], INL[k], peak and rms inl_pk, dnl_pk, inl_rms.
• Quantization & counts: LSB, histogram H[k], total samples N_total, expected probability p_k.
• Calibration params: polynomial coefficients a_m, lookup table LUT[k], interpolator interp(•).
• Timebase & units: tau_mono, ts, offset/skew/J; unit(x), dim(x), check_dim(expr).

III. Postulates P705-*

• P705-1 (Fixed convention): publish INL/DNL using either best-fit line (BFL) or end-point (EP) only; record ref_line ∈ {BFL, EP} in the manifest, and keep that convention consistent across batches.
• P705-2 (Density assumptions): for code-density tests, the input must be a slow, monotonic ramp/triangle, or a sine with known distribution; publish input_profile and how p_k was obtained.
• P705-3 (Monotonicity & missing codes): w_k > 0 is sufficient for monotonicity; H[k] = 0 is treated as a missing code.
• P705-4 (Calibratability & invertibility): the calibration map C(•) must be monotone, invertible, and Lipschitz-bounded within the working range; no extrapolated publication beyond boundaries.
• P705-5 (Dimensional conservation): calibration changes the mapping only, not the physical unit; check_dim(all)=true before publication.
• P705-6 (Integrated timebase): perform sample counting and trigger statistics on tau_mono; publish on ts with offset/skew/J recorded.

IV. Minimal Equations S705-*

1. S705-1 (DNL by code-width convention)
   DNL[k] = ( w_k / W_ideal ) - 1.
   For ramp/triangle code-density (where w_k ∝ H[k]):
   DNL[k] = ( H[k] / H_mean ) - 1, with H_mean = ( N_total / 2^N ).
2. S705-2 (DNL by sine-density convention)
   DNL[k] = ( H[k] / ( N_total * p_k ) ) - 1, where
   p_k = ( ∫_{b_k}^{b_{k+1}} p_sine(x) dx ), p_sine(x) = 1 / ( pi * sqrt(1 - x^2) ) (normalized to the full-scale interval).
3. S705-3 (INL accumulation & de-biasing)
   INL[k] = ( ∑_{i=0}^{k-1} DNL[i] ) - bias, with bias per convention:
   • BFL: remove gain/offset via least-squares line;
   • EP: enforce INL[0] = INL[2^N] = 0.
4. S705-4 (Missing codes & monotonicity)
   missing_codes = { k | H[k] = 0 }; monotone ⇔ ∀k, w_k > 0.
5. S705-5 (Polynomial calibration in input domain)
   x_corr = poly(x_raw) = ∑_{m=0}^M a_m * x_raw^m; residual epsilon_res(x) = T_meas(x_corr) - T_ideal(x).
   Prefer orthogonal bases (e.g., Chebyshev) to reduce ill-conditioning: poly(x) = ∑ a_m * T_m( x_norm ).
6. S705-6 (LUT calibration in code domain)
   code_corr = interp( LUT, code_raw ) or x_corr = interp( LUT_x, code_raw ).
   Interpolation must be monotone; use piecewise-linear or no-overshoot splines.
7. S705-7 (Uncertainty approximations)
   For histogram methods: Var( DNL[k] ) ≈ ( 1 - p_k ) / ( N_total * p_k ), u(DNL[k]) = sqrt( Var );
   u(INL[k]) follows from cumulative covariance propagation or bootstrap estimates.
8. S705-8 (Residuals & publication)
   residual_nl_rms = sqrt( E[ epsilon_res(x)^2 ] ); publish inl_pk = max |INL[k]|, dnl_pk = max |DNL[k]|.

V. Metrology Procedure M70-5 (Measure H → DNL/INL → Fit → Calibrate → Accept → Persist)

1. Readiness & timebase
   Fix RefCond (temperature, supply, load); perform calibration triggers on tau_mono; record offset/skew/J.
2. Acquisition & histogramming
   Ramp/triangle: cover all codes with N_total ≥ N_min; or sine: record input_profile and compute p_k.
3. Linearization baseline
   Estimate G, O; choose ref_line ∈ {BFL, EP}; compute DNL[k] (S705-1/2) and INL[k] (S705-3); detect missing_codes and monotone.
4. Calibration modeling
   • Plan A (polynomial): fit poly in the input domain, controlling order M and regularization.
   • Plan B (LUT): build LUT and choose interpolation; constrain each segment’s slope to be positive.
5. Application & closed loop
   After applying C(•), re-measure to obtain INL', DNL', residual_nl_rms; if contracts fail, adjust order or refine segmentation.
6. Uncertainty & regression tests
   Evaluate u(DNL[k]), u(INL[k]), and bootstrap intervals for residual_nl_rms; perform cross-temperature regression over RefCond.
7. Persistence & signing
   Write manifest.instrument.linearity.*, version and validity of poly_coeff/LUT, plus TraceID and signature.

VI. Contracts & Assertions C70-5*

• C70-51 dnl.peak: dnl_pk ≤ dnl_max.
• C70-52 inl.peak: inl_pk ≤ inl_max (report ref_line).
• C70-53 monotone: monotone = true, otherwise fail.
• C70-54 missing.codes: |missing_codes| = 0.
• C70-55 residual.bound: residual_nl_rms ≤ nl_rms_max.
• C70-56 calib.stability: cross-RefCond drift Δinl_pk ≤ inl_temp_max.
• C70-57 dim.check: check_dim(all)=true; mapping is monotone and invertible.
• C70-58 versioning: poly|LUT carries version and validity; expiry triggers alert and rollback.

VII. Implementation Bindings I70-5* (Interface Prototypes)

• measure_code_density(ds, mode) -> H, meta (with mode ∈ {ramp, sine}, returning p_k or its generation method)
• compute_inl_dnl(H, pk_profile, ref_line) -> DNL, INL, stats
• fit_poly_calibrator(data, basis, M, reg) -> poly_coeff (with basis ∈ {power, chebyshev})
• build_lut_calibrator(breakpoints, values, monotone) -> LUT
• apply_calibration(stream, C) -> stream' (supports real-time and offline)
• verify_linearity(stream', ref_line) -> inl_pk, dnl_pk, residual_nl_rms
• bootstrap_uncertainty(H, B) -> u_DNL, u_INL
• emit_linearity_manifest(results, policy) -> manifest.instrument.linearity
  Invariants: C(•) monotone & invertible; interfaces idempotent; unit/dim checks pass; record RefCond and ref_line.

VIII. Cross-References

• Range/LSB/ENOB: see Chapter 4 of this volume.
• Timebase-jitter impacts on linearity measurement: see EFT.WP.Metrology.TimeBase v1.0, Chapters 6–7.
• Cleaning & release freeze: see EFT.WP.Methods.Cleaning v1.0, Chapter 10.
• Imaging-chain nonlinearity & gamma: see EFT.WP.Methods.Imaging v1.0, Chapter 10.

IX. Quality & Risk Control

• SLI/SLO: inl_pk_p95, dnl_pk_p95, residual_nl_rms_p95, missing_codes_rate, calibration_age_days.
• Drift & alerts: thresholds on inl_pk or dnl_pk trigger re-calibration; rising residual_nl_rms correlated with temperature prompts RefCond adaptive switching; provide safe rollback thresholds to prior poly|LUT.

Summary
Using P705-* / S705-* / M70-5 / C70-5* / I70-5*, this chapter establishes consistent measurement and uncertainty conventions for INL/DNL, and provides two traceable nonlinear-calibration paths—polynomial and LUT—to ensure cross-instrument, cross-batch linearity metrics that are comparable, controllable, and publishable.