Latency Pods

Five things a normal Kubernetes pod cannot do, and what to use instead. They live in k8s/nri-latency-pods (node setup and the admission policy) and k8s/nri-shared-cpus (the NRI plugin). This page is the contract.

In short: five opt-in grants: a sized hugetlbfs for Aeron’s directory, an isolated cpuset partition, an io_uring seccomp profile, an admission policy against exec probes and sidecars, and device grants (the NIC’s PHC). Each is a mechanism; none has a measured latency effect. SCHED_FIFO is built in the shared-cpus plugin and off by default (evidence: mixed).

Status: the grants ship in the public image ghcr.io/mimran1980/ergon-shared-cpus:0.1.0. Install it with the chart oci://ghcr.io/mimran1980/charts/ergon-shared-cpus version 0.1.0 (Shared CPUs). This page’s admission YAML and node units are not an image. Chart 0.1.0 leaves seccomp.ioUring and ptp off, because its ConfigMap template does not write those keys. k8s/nri-shared-cpus/examples/try installs that chart and shows two things on kind: the shared-CPU grant, and this page’s admission policy denying an exec probe. It does not exercise hugepages, the cpuset partition, io_uring seccomp, or the PHC. The five-grant kind e2e (test/e2e.sh) builds the plugin from source. Not verified on EKS. No latency figure is measured.

The five

Every grant is opt-in by pod annotation. A grant that cannot be given is refused: the kubelet shows the text in the pod’s events as CreateContainerError, and the text names the fix. A pod author names kinds and modes only; values come from the plugin config or the node’s facts.

Ride-along: rlimits.ergon.io/<ctr> grants MEMLOCK, NOFILE and RTPRIO (a limit value only — see below). The values are ceilings the pod may ask for, set in the plugin config.

Also numa.ergon.io/<ctr>: "0", "0,1" or "" binds the container’s memory to those NUMA nodes (cpuset.mems and MPOL_BIND). "" means the data NIC’s node, which needs that node recorded; on a one-node machine the only node is the answer. A request the node cannot answer is refused and says why — an unread topology, a node the machine does not have, or "" with several nodes and no NIC node. A binding is never dropped in silence.

The five grants on one pod. Everything stays inside the pod’s own cpuset: exclusive CPUs from the kubelet’s static CPU manager, shared CPUs from ergon.io/shared-millicpus. Helpers go on the shared CPUs; SQPOLL takes an exclusive CPU of the pod’s own (SMT sibling).

flowchart TD
    POD["latency pod: exclusive CPU(s) and shared CPU(s) in its cpuset"]
    POD --> A["1 hugepages.ergon.io"]
    POD --> B["2 cpu-partition.ergon.io"]
    POD --> C["3 seccomp.ergon.io/io-uring"]
    POD --> D["4 ergon.io/latency hot"]
    POD --> E["5 ptp. and cpuqos.ergon.io"]
    A --> A1["sized hugetlbfs + hugepage limit"]
    B --> B1["child cgroup, isolated partition"]
    C --> C1["seccomp + io_uring_* rules"]
    D --> D1["no exec probes, no Always sidecars"]
    E --> E1["PHC, cpu_dma_latency, node facts"]
    SQ["helpers on shared CPUs"] -.-> POD

1. The app owns AERON_DIR

The plugin mounts a sized hugetlbfs at /dev/hugepages and exports ERGON_HUGEPAGES_DIR and ERGON_HUGEPAGE_SIZE. It does not set AERON_DIR. The app creates $ERGON_HUGEPAGES_DIR/aeron, points AERON_DIR there, and, if it runs the media driver, sets AERON_FILE_PAGE_SIZE=$ERGON_HUGEPAGE_SIZE.

Why: an unsized hugetlbfs is what an emptyDir medium: HugePages gives you, and Aeron refuses it. A 4 KiB AERON_DIR takes a page fault on each first touch of a page. Latency effect not measured here.

2. Isolated partition

cpuset.cpus.partition=isolated makes the kernel treat the set as isolated: no load balancing into it (cgroup-v2). The grant is applied to a child cgroup so the pod’s own cgroup keeps its shared CPUs. Latency effect not measured here.

The child is threaded, and a threaded child is a legal partition root (k8s/nri-shared-cpus/internal/cpuset/partition.go). So:

A program that does not make these moves itself, such as Aeron’s C media driver, keeps the boot line’s isolation and sched_setaffinity: see the worked example.

cpuset.cpus.exclusive is a replace, not a union, so every directory the new partition shares with an already-running pod has to be updated with the union of both reservations. The hook does that under a lock and reads every write back; a write that does not stick fails the create.

3. io_uring

RuntimeDefault blocks io_uring (containerd issue 9320). Aeron’s C media driver does not use io_uring; the grant is for applications that do, such as ergon-io’s uring-spin and uring-sqpoll backends. It is a clone of the pod’s own seccomp profile with the io_uring_* calls allowed — never Unconfined, which the plugin refuses.

4. Admission

An exec probe starts a process inside the container on any CPU of its cpuset, the loop’s exclusive CPU included (k8s/nri-shared-cpus/README.md, Limits). A container with restartPolicy: Always is an injected sidecar: it shares the pod’s CPU budget and can run on the hot core.

The policy refuses both on pods labelled ergon.io/latency: "hot". It also requires a priorityClassName (eviction and preemption stop a hot pod mid-trade), the ergon.io/shared-millicpus resource, and the required-plugins.noderesource.dev annotation naming the plugin.

httpGet, tcpSocket and grpc readiness probes are fine: they run from outside the container.

5. Devices

The NIC’s PHC (/dev/ptp*) for clock work, and on x86 /dev/cpu_dma_latency so a pod can cap C-states. The cap’s measured effect is mixed (C-states), and a spinning core never idles, so treat the grant as access, not a latency win. On arm64 the plugin ignores the cpuqos annotation (k8s/nri-latency-pods/README.md, item 5).

Real-time scheduling: built, default off

SCHED_FIFO is not part of the five grants. It is in k8s/nri-shared-cpus, off unless a node turns it on:

Why it is off: its one committed A/B is mixed, and under it the loop’s nohz_full CPU took timer interrupts again (SCHED_FIFO). RT throttling (kernel.sched_rt_runtime_us, 950,000 of 1,000,000 µs by default, sched-rt-group) can take the CPU away. ergon-tune T7 warns on a real-time policy.

Deliberately not built

RTPRIO is granted as a limit only on the latency-pods path. That plugin never changes a scheduler policy, and the e2e asserts PID 1 stays SCHED_OTHER and ERGON_SCHED_FIFO is empty.

Node setup

The full check list is k8s/nri-latency-pods/node-checks.md. The short version:

Why not AKS: it has no supported way to change containerd (so an NRI plugin cannot be installed) and no reservedSystemCPUs list. Everything here needs one of those. The target is EKS on AL2023.

The two labels

They are different on purpose.

Related

• Shared CPUs for Helper Threads — the plugin that grants the shared housekeeping CPUs
• Kubernetes — deployment
• Kubernetes HFT tuning — every setting and its evidence