ergon-tune

ergon-tune checks a Linux host’s low-latency settings. Each finding says what is wrong, the exact fix, and its evidence: an archived A/B where one exists (evidence), otherwise the kernel’s, Kubernetes’, Aeron’s or a vendor’s documentation and “not measured here”. It changes nothing. ergon-io and the ergon-md-dpdk media driver run the checks for you (where).

AI assistance. Like the rest of ergon, this crate was written with heavy AI assistance (disclosure). It is not production-ready.

[dependencies]
ergon-tune = { version = "0.1", features = ["nic"] }   # `nic` adds N1–N4

Glossary

In brackets: the check (T1–N4). Times are in µs.

Why check the host

• A busy-spinning thread loops on one CPU and never sleeps. It loses time only when something takes that CPU: a timer tick, an interrupt (IRQ), a kernel thread, a busy hyperthread on its core.
• Those pauses set the tail: p99.99 (the slowest 1 in 10,000) and the max.
• Each comes from a host setting, and a reboot, a new image or a NIC reset can undo it. Check at start-up and again later.

What it checks

Not checked: CPU idle states and frequency.

Symptom, check, fix

Each finding carries the fix for that host, with its real CPU lists and IRQ numbers. T1 with the loop on CPU 2 of 8: isolcpus=nohz,domain,managed_irq,2 nohz_full=2 rcu_nocbs=2 irqaffinity=0-1,3-7.

Evidence per check

Each finding’s evidence field names its source. The A/B tables are in git history: git show 128747b9^:io/benches/results/<run>/tables.md (or tables-local.md). All are AWS c7i in Tokyo; each figure in a finding keeps its max.

Where the checks run

Every printed warning line ends with (ERGON_HOST_CHECKS=off silences host-check warnings). The K checks run only inside a pod (the kubepods cgroup path, the service-account mount, or KUBERNETES_SERVICE_HOST) and each says whether its fix is in the pod spec or on the node.

require is a readiness gate, not a blanket refusal. It refuses on a warning (a check fired) and on a check that could not prove a property of this pod or process — its CPUs are its own, they are isolated, there is no CPU quota, its helpers have a core — because starting on an unproven claim is the false pass those checks exist to stop. An unknown of a check whose subsystem the node may not have (PSI and memory.high, a NUMA node, a NIC knob, THP) is printed and allowed through: that is a real host, and refusing there would stop every start on it. Which checks are which is CheckId::unknown_stops_require.

Two consequences under require:

• T10 warns on every x86 host with SMT on, so require refuses it, including the SQPOLL-on-sibling layout T8 accepts.
• K2 proves exclusive CPUs only from an isolated or root cpuset partition, or a matching cpuset.cpus.exclusive. The kubelet’s static CPU manager writes neither, so a stock Guaranteed pod is K2 unknown and refused. The nri-latency-pods plugin’s isolated partition (Linux 6.7+) passes it.

Limits:

• ergon-io needs the loop’s CPU: pin the thread that calls Driver::new to one CPU, or set loop_cpu. Otherwise it skips T1, T8 and T2, and says so.
• The media driver checks the socket transport’s NICs only if RUSTERON_MEDIA_DRIVER_DATA_NICS names them. It adds its own checks, M1–M4 (ergon-md-dpdk).
• Both print each finding once on stderr, never from a busy thread.
• In a pod the NIC checks report unknown: run them on the node (Kubernetes). T9 reads the container’s own cgroup; the pod’s quota shows only in cpu.stat.local (Linux 6.6+), which ThrottleWatch reads.
• In CI, run them on a captured /sys and /proc tree: Host::at(dir).

Call them yourself

The checks are plain functions. They read files and allocate: call them before the loop or from housekeeping, never inside the loop.

• Host::system() is the running host. Host::at(dir) is a directory that stands for a host’s /, on any OS.
• HotSet lists the CPUs your threads busy-spin on, each with a Role (Loop, Sender, SqPoll, …), so a message names the thread.
• Component (Io, Md, Rt) is the caller. It sets T3’s severity: warn for Md and Rt, info for Io.

/// Every host check, with `hot` and each interface the host shows.
fn run_all(host: &Host, cpus: &HotSet, interfaces: &[String]) -> Vec<Finding> {
    let mut all = Vec::new();
    all.extend(t1_isolation(host, cpus));
    for c in [Component::Md, Component::Io, Component::Rt] {
        all.extend(t3_thp(host, c));
    }
    all.push(t4_aeron_dir(
        host,
        Path::new("/dev/shm/ergon-tune-live/aeron"),
    ));
    all.push(t5_numa_balancing(host));
    all.push(t7_sched_policy(host, Thread::Current, Role::Loop));
    all.extend(t8_smt_siblings(host, cpus));
    all.push(t9_cpu_quota(host));
    all.push(t10_smt_on(host));
    all.extend(k8s_checks(
        host,
        cpus,
        &Needs::new(Thread::Current, Role::Loop),
    ));
    for i in interfaces {
        let nic = Nic::Interface(i.clone());
        all.extend(t2_nic_irqs(host, &nic, cpus));
        all.extend(t6_nic_node(host, &nic, cpus));
    }
    all
}

The NIC checks need the nic feature. Keep one EthtoolCache::new() for the process: it asks the kernel once per interface, and again only after a reset (when carrier_changes moves). Here i is an interface name:

            let mut v = vec![
                n1_adaptive_rx(&host, &i, &mut cache),
                n3_azure_vf(&host, &i),
                n4_rx_ring(&host, &i, &mut cache),
            ];
            v.extend(n2_deferred_irqs(&host, &i, Some(false)));

(From tune/tests/live.rs, which runs every check against the host it runs on: cargo test -p ergon-tune --all-features.)

A Nic is Nic::Interface("ens5") or a DPDK port, Nic::Pci("0000:00:06.0"). The readers underneath (thread_affinity, carrier_changes, nic_irqs, busy_poll_sysctls, fs_magic, cpu_cgroup, Platform::detect) are public too.

Read a finding

Display prints one line, T4 warn: <message> (evidence: <where>) or T4 unknown: <message>: <reason>:

#[test]
fn findings_print_as_one_line_each() {
    let f = t4_from(Path::new("/var/lib/aeron"), Ok(FsMagic(0xEF53)));
    let line = f.to_string();
    assert!(line.starts_with("T4 warn: "), "{line}");
    assert!(line.contains("(evidence: "), "{line}");
    assert!(!line.contains('\n'), "{line}");
    let u = t4_from(Path::new("/aeron"), Err(ReadError::NotLinux("statfs")));
    assert!(u.to_string().starts_with("T4 unknown: "), "{u}");
}

• Warn: fix it.
• Info: real but minor, such as THP for ergon-io or an IRQ whose new affinity takes effect at its next interrupt.
• Unknown is never a warning: a pod’s network namespace, hidepid, an interface with no device, not Linux. Log it as info.
• To make warnings fatal, use the media driver’s require, or is_warning() in your own start-up.

Alternatives

Best practices and pitfalls

• Pin first, then check. ergon-io takes the loop’s CPU from the calling thread’s affinity.
• Re-check after every NIC reset. Whenever ENA brings an interface up (a device reset, ethtool -G or -L, a NIC handed back from DPDK), it puts queue i’s interrupt on CPU i, isolated CPUs included, and turns adaptive moderation back on. Re-pin from a link-event hook.
• N1, busy polling: adaptive-rx off rx-usecs 0. Against adaptive-rx off alone (one run, two reps): WebSocket p99.99 78–80 against 72–76 µs (max 233–344 against 354–970), HTTP p99.99 74–81 against 83–116 µs (max 202–227 against 202–981), market-data p99 7.9–8.6 against 9.3–12.9 µs (max 34–41 against 25–29). The interrupt-driven path is not A/B’d here.
• Use THP never, not madvise. khugepaged still runs under madvise (kernel transhuge.rst); the cost is not measured here.
• No CPU quota on a busy-spinning container. /sys/fs/cgroup/cpu.max should read max 100000, and nr_throttled should not grow while it runs (T9, ThrottleWatch). Set-up: Kubernetes.
• Graviton has no SMT, so T8 fires there only when two busy threads share one CPU. N2 does not suggest busy polling there.
• Test your host image. Build a tree with the files your provisioning writes and assert on the findings, as tune/tests/common does for five hosts.
• Measure your own host. The measured effects come from AWS c7i runs.

How to apply each setting on AWS and Azure, x86 and Graviton: Hosts.