Kubernetes

Where ergon-io’s helper threads run inside a pod. On a host they run on the housekeeping CPUs. In a pod there are none. This page says what to do instead, for every pod size.

Terms: io glossary, deployment glossary.

Why a pod has no housekeeping CPUs

• On a host, the housekeeping CPUs are the ones the kernel leaves out of isolcpus: everything except the isolated cores.
• Those files describe the node, not the pod. /sys/devices/system/cpu/{online,isolated} are host-wide. A pod runs only on the CPUs the kubelet gave it: its cpuset (cpuset.cpus.effective). A Guaranteed pod with the static CPU manager gets exactly its whole-number cpu request. The node’s housekeeping CPUs are outside that set.
• A thread outside the cpuset never starts. The kernel refuses the mask (EINVAL), and io_uring_setup refuses an SQPOLL CPU outside it the same way.
• So the helper threads run on the pod’s own CPUs. Driver::new reads the pod’s cpuset once and chooses from it (placement): helper_cpus / ERGON_HELPER_CPUS first, then the loop CPU’s SMT sibling, then the pod’s other CPUs, preferring ones the host does not isolate. Never the polling CPU when it is known: set loop_cpu, or pin the loop thread before Driver::new. When it is not known the choice cannot keep off it, and the start-up line says the helpers may include the CPU the loop ends up on. The line reports the choice.

Pod layouts

Every layout: Guaranteed QoS (requests equal limits), a whole-number CPU request, and the kubelet’s cpuManagerPolicy: static (on x86 with SMT on, cpuManagerPolicyOptions: full-pcpus-only) — the node settings are under deployment: Kubernetes.

Minimal pod, cpu: 2 (change to "1" for one CPU). Requests equal limits, and the CPU request is a whole number:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    resources:
      requests: { cpu: "2", memory: 1Gi }
      limits:   { cpu: "2", memory: 1Gi }

cpu: 1

connect_background refuses at once with Error::Cpu (NoSpareCpu) rather than start a dial helper on the polling CPU, and uring-sqpoll refuses at Driver::new. The warning thread shares the core, parked until there is work. One warning line at start says all this.

/// A pod with one CPU (`cpu: 1`). `connect_background` refuses: its dial
/// helper would time-slice the polling thread. With
/// `inline_connect_fallback=true` it connects as `Driver::connect` does
/// (the TLS handshake on the loop, about 0.7 ms of CPU once per connect
/// or reconnect). Never `uring-sqpoll` here: it is refused at
/// `Driver::new`.
fn one_cpu_pod() -> ergon_io::Result<Config> {
    Config::parse("recv-busy,inline_connect_fallback=true")
}

• With inline_connect_fallback=true, connect_background connects as Driver::connect does: the TLS handshake runs on the loop, a long poll per connect or reconnect (connection set-up). A helper thread sharing the loop’s CPU was not measured.
• For production, give the pod 2 CPUs (the warning line ends with that). ERGON_HOST_CHECKS=off silences the line; the typed refusals stay.

x86 with SMT, cpu: 2: one physical core

cpuManagerPolicyOptions: full-pcpus-only: the pod holds one core’s two hyperthreads. The loop takes one; SQPOLL, the dial helper and the warning thread take the other. Nothing to configure.

/// x86 with SMT, `cpu: 2` and the kubelet's `full-pcpus-only`: one
/// physical core, its two hyperthreads. The loop takes one; `SQPOLL`, the
/// dial helper and the warning thread take the other (the loop's SMT
/// sibling). Nothing to configure.
fn x86_two_cpu_pod() -> ergon_io::Result<Config> {
    Config::parse("uring-sqpoll")
}

What the SQPOLL thread on the sibling costs against a core of its own: SMT sibling cost.

• It needs both hyperthreads and nothing else on the sibling.
• For a core of its own instead: a cpu: 4 pod (two physical cores) with sqpoll_cpu on the second core, or SMT off on the node.

Graviton, cpu: 2: two cores

Graviton has no SMT: two vCPUs are two cores. The loop takes one; the helpers take the other. Nothing to configure.

/// Graviton (no SMT), `cpu: 2`: two cores. The loop takes one; the
/// helpers take the other. Nothing to configure.
fn graviton_two_cpu_pod() -> ergon_io::Result<Config> {
    Config::parse("recv-busy")
}

Shared CPUs from the node

A node-level mechanism can add shared CPUs to the pod’s cpuset for the helper threads (dial, warnings), and keep the loop on its exclusive CPU: deployment: shared CPUs. Name those CPUs in Config::helper_cpus, or let the mechanism set ERGON_HELPER_CPUS (same value: ERGON_SHARED_CPUS).

/// Shared CPUs a node-level mechanism added to the pod's cpuset: name
/// them for the helper threads. A CPU outside the process's CPUs is
/// refused (`Error::Cpu`), never used.
fn shared_cpu_pod() -> ergon_io::Result<Config> {
    Config::parse("recv-busy,helper_cpus=3-4")
}

/// The same from the pod's environment: `ERGON_HELPER_CPUS` is a kernel
/// CPU list (`3`, `3,7`, `2-3`). `Config::helper_cpus` wins when both are
/// set. `Driver::new` reads the variable; the application does not.
const fn helper_cpus_env_name() -> &'static str {
    HELPER_CPUS_ENV
}

env:
- name: ERGON_HELPER_CPUS
  value: "3-4"          # a kernel CPU list: 3, 3,7, 2-3

• A CPU outside the pod’s cpuset is refused (Error::Cpu, Outside), never used. Config::helper_cpus wins over the variable.
• Keep uring-sqpoll off shared CPUs: its SQPOLL thread spins (io_uring_setup(2), IORING_SETUP_SQPOLL). The default never puts it there: with no sqpoll_cpu it takes the lowest CPU of the pod’s own outside that list, and refuses (Error::Cpu, NoExclusiveCpu) when the list holds every CPU but the loop’s. Then set sqpoll_cpu to an exclusive CPU of the pod, or pick a single-CPU backend (recv-busy, per_queue). The dial helper and the warning thread are the ones that belong on the shared CPUs.

Pin the loop thread inside the pod

The kubelet gives the pod its CPUs; it never pins threads to them. Each busy thread pins itself to one CPU of the pod’s cpuset before Driver::new, so the host checks and the helper choice know the polling CPU. Node settings: deployment: Kubernetes.

/// Pin the loop thread to one of the pod's CPUs, before `Driver::new`.
/// The kubelet gives the pod its CPUs (its cpuset); it never pins threads
/// to them. `Driver::new` then knows the polling CPU, for its host checks
/// and for the helper-thread choice.
fn pin_loop() -> R {
    let (cpus, source) = placement::process_cpus(&Host::system());
    let cpu = cpus
        .iter()
        .next()
        .ok_or_else(|| format!("no CPU to pin to: {source}"))?;
    ergon_io::clock::pin_current_thread(usize::try_from(cpu)?)?;
    Ok(())
}

• The pod’s CPUs: cpuset.cpus.effective (what placement::process_cpus reads), or Cpus_allowed_list in /proc/self/status.
• Pinned by other means? Name the CPU as loop_cpu in the backend configuration (backends).

What the driver reports

• One start-up line with the helper CPUs and why (HostReport::helper_line), or, in a pod with one CPU, the warning above.
• Driver::host_report() carries the same (HostReport::helpers).
• ERGON_HOST_CHECKS=off silences the warning lines. The typed refusals (Error::Cpu) are never silenced.