Kubernetes

A busy-spinning pod needs what a busy-spinning process on a VM needs: a whole core per busy thread, interrupts elsewhere, no page faults. Kubelet has to be told to give it that. The worked example is the EKS deployment of the DPDK media driver, md-dpdk/deploy/aws-eks. Its node and pod settings apply to any low-latency pod; its DPDK parts are a proof of concept, not for production.

Terms: deployment glossary, network and DPDK glossary.

The kubelet and pod settings are not x86-specific, but no Graviton node has been built with them. For Graviton’s kernel arguments see hosts.

Latency checklist

Use a dedicated node pool, tainted so only latency pods land there (keep other pods in check).

Node (kubelet and host):

1. SMT off on x86, so one Kubernetes CPU is one physical core, the same as on Graviton (SMT off).
2. cpuManagerPolicy: static with full-pcpus-only: "true": Guaranteed pods get whole cores of their own.
3. reservedSystemCPUs: the housekeeping cores, the same ones the kernel’s boot line leaves out of isolation.
4. topologyManagerPolicy: single-numa-node: a pod’s CPUs and devices on one NUMA node.
5. No CPU quota on the latency containers: kubelet 1.33 or later (the labs ran 1.36), cpuCFSQuota left at its default true. Do not set cpuCFSQuota: false: it removes every pod’s CPU limit on the node (why).
6. The host settings under the kubelet: isolated cores, interrupts on housekeeping, transparent hugepages off, hugepages (hosts).

Pod:

7. Guaranteed QoS: requests equal limits for every resource, in every container.
8. A whole-number CPU request on every busy-spinning container. With SMT off, one CPU is one core.
9. Each busy thread pinned to its own core from the pod’s cpuset, under SCHED_OTHER (SCHED_FIFO was mixed: evidence).
10. On CRI-O only: the annotations in No CPU quota. containerd (EKS, AKS, k3s) ignores them.

Check:

11. At start: ergon-io and the media driver run ergon-tune, and T9 warns about a CPU quota. In your own loop, call t9_cpu_quota before it.
12. While it runs: nr_throttled stays flat (verify). In your own loop, call ergon-tune’s ThrottleWatch::check from housekeeping.

Nodes

Kubelet gives Guaranteed pods whole cores of their own (the static CPU manager) and keeps a pod’s CPUs and devices on one NUMA node (the topology manager). node/userdata.sh writes this, with $reserved the housekeeping CPUs (0 on a c7i.4xlarge: one CPU is one physical core, because SMT is off):

  kubelet:
    config:
      cpuManagerPolicy: static
      cpuManagerPolicyOptions:
        full-pcpus-only: "true"
      reservedSystemCPUs: "$reserved"
      kubeReservedCgroup: ""
      systemReservedCgroup: ""
      topologyManagerPolicy: single-numa-node
      topologyManagerScope: pod
      memoryManagerPolicy: None
    flags:
      - --node-labels=rusteron-dpdk/ena=true
      - --register-with-taints=rusteron-dpdk=true:NoSchedule

• full-pcpus-only: whole physical cores, never one hyperthread shared with another pod. With SMT off it is a safety net.
• kubeReservedCgroup and systemReservedCgroup are emptied because kubelet rejects reservedSystemCPUs while they are set.
• memoryManagerPolicy: None: the deployed c7i.4xlarge has one NUMA node (node/userdata.sh). On a multi-socket type, revisit it.
• cpuCFSQuota is not set: it stays true, so every pod keeps its CPU limit. The kubelet drops the quota itself on a container with exclusive CPUs (No CPU quota).

The node scripts (tested against fixtures by just test-dpdk-bootstrap):

Clocksource is left to the kernel. A forced clocksource=tsc tsc=reliable turns the kernel’s TSC checks off, so bare metal takes it only with RUSTERON_FORCE_TSC=1, after the TSC has been measured as the same across cores against a PTP/PHC reference. A VM never gets it.

Kubelet’s reserved CPUs and the kernel’s housekeeping CPUs are the same set: interrupts and system daemons stay off the pods’ cores. Dedicated nodes are labelled rusteron-dpdk/ena=true and tainted rusteron-dpdk:NoSchedule.

SMT off

SMT (hyper-threading) runs two logical CPUs on one physical core. They share the core’s execution resources: a pod with cpu: 1 shares a core with its twin, and its latency follows what the twin does (SMT sibling). Turn SMT off on x86 latency nodes, so one Kubernetes CPU is one physical core, as on Graviton, which has no SMT.

• With ThreadsPerCore: 1 the instance runs half its usual vCPU count (a c7i.4xlarge: 8 cores, 8 vCPUs, not 16). Size pods and quotas for cores.
• thread_siblings_list still lists the offline twin. Name one CPU per core (the lowest id) in reservedSystemCPUs and in kernel arguments. The node scripts do.
• The launch template that sets it: node/launch-template.py. The live lab fails a node whose /sys/devices/system/cpu/smt/active is 1 (md-dpdk/benches/cloud/aws/eks.sh).

Pods

• Guaranteed QoS with whole cores: requests equal to limits on every resource, and a whole-number CPU request.
• Pin each busy thread to its own core from the pod’s cpuset. Read it from Cpus_allowed_list in /proc/self/status; in a privileged container /sys/fs/cgroup/cpuset.cpus.effective lists every host CPU. One logical CPU per physical core, and Aeron’s client conductor off the busy cores.
• Hugepages as a resource (hugepages-2Mi), Aeron’s directory from the node’s hugetlbfs as a hostPath, and AERON_CLIENT_PRE_TOUCH_MAPPED_MEMORY=true on clients.
• Handle SIGTERM. Kubelet stops a pod with it; a PID 1 with no handler ignores it until SIGKILL, closing nothing.
• Privileged only where needed: the DPDK driver pod (it maps the NIC and locks hugepages). A client that only reaches the driver through Aeron’s directory is not.
• hostNetwork: true takes a kernel-UDP pod off the pod network’s virtual interface. Not measured here.

Keep other pods in check

A dedicated node pool only works if nothing else lands there.

• Taint the latency nodes (NoSchedule) and let only latency pods and the DaemonSets that must run there tolerate it: the CNI, kube-proxy, the node agent, a log collector, the media driver. Everything else stays off.
• A LimitRange per namespace with a default CPU request and limit for containers, and a ResourceQuota for the namespace. An unlabelled pod then cannot ask for half a node.
• Small CPU requests and limits on the DaemonSets that stay (log collectors, a metrics agent): a limit keeps a burst off the latency cores. Give both; a limit with no request still lets them burst up to it.
• PriorityClass: one high class for latency pods, a low one for the rest. A latency pod then preempts, not the other way round.
• Kubelet 1.33 or later with the static CPU manager: only a Guaranteed whole-CPU container loses its quota; everything else keeps its CPU limit (No CPU quota).

strict-cpu-reservation (kubelet policy option, GA in 1.35): by default the shared CPU pool includes reservedSystemCPUs, so a Burstable pod or a fractional sidecar can run on the reserved CPUs. strict-cpu-reservation: "true" keeps everything that was not given those CPUs off them. Use it only when nothing but the system (and threads you pin there yourself) may use them: it breaks anything that needs the shared pool by default, such as a fractional sidecar in a latency pod or ergon-stack’s host-stats thread. The shared-CPU NRI plugin (ghcr.io/mimran1980/ergon-shared-cpus:0.1.0; chart oci://ghcr.io/mimran1980/charts/ergon-shared-cpus version 0.1.0) lends shared CPUs from reservedSystemCPUs to a pod’s helper threads and expects the default, so leave strict-cpu-reservation off while using it.

No CPU quota

• A CPU quota (Linux’s CFS bandwidth control) lets a container’s threads run for at most the quota in each 100 ms period. Kubernetes sets it from the container’s CPU limit.
• A busy-spinning thread uses its core all the time, so it runs at the cap. When the container’s other threads push it over, the kernel stops every thread in it until the period ends: a stall of milliseconds.
• A pod has a quota of its own too: the sum of its containers’ limits. From inside a container, only cpu.stat.local shows it.

What the kubelet writes:

• Exclusive CPUs need all three: cpuManagerPolicy: static, a Guaranteed pod, and a whole-number CPU request on the container. Kubelet 1.33 is the first that then drops the quota.
• Do not set cpuCFSQuota: false. It is node-wide: it removes the CPU limit of every pod there, log collectors and system pods too, and they then run unlimited on the shared CPUs. The kubelet’s own behaviour above is enough, and it only touches the containers with exclusive CPUs.
• Kubelet 1.31 and 1.32 (EKS extended support, AKS) always set the quota. Use 1.33 or later (the lab’s k3s and the EKS deployment run 1.36).
• Changing cpuManagerPolicy: drain the node, stop the kubelet, delete /var/lib/kubelet/cpu_manager_state, start the kubelet. Without the delete, the kubelet crash-loops.

The kubelet config file:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  full-pcpus-only: "true"
reservedSystemCPUs: "0"     # the housekeeping cores (one CPU is one core)
topologyManagerPolicy: single-numa-node
# cpuCFSQuota is not set: it stays true, so every pod keeps its CPU limit.

EKS, Amazon Linux 2023: nodeadm merges spec.kubelet.config into its defaults. Put the NodeConfig in the launch template’s user data, as a MIME part with Content-Type: application/node.eks.aws. The full one, with the cluster fields, is in Nodes.

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  kubelet:
    config:
      cpuManagerPolicy: static
      cpuManagerPolicyOptions:
        full-pcpus-only: "true"
      reservedSystemCPUs: "0"

EKS, Bottlerocket: in the user data (TOML). Reboot after changing the policy. It has reserved-cpus and cpu-manager-policy-options too, but no feature gates. Leave cpu-cfs-quota-enforced at its default true.

[settings.kubernetes]
cpu-manager-policy = "static"

AKS: per node pool. AKS has no reservedSystemCPUs field.

echo '{"cpuManagerPolicy": "static"}' > kubelet.json
az aks nodepool add --resource-group <group> --cluster-name <cluster> \
  --name latency --kubelet-config kubelet.json

k3s: in /etc/rancher/k3s/config.yaml, or as a KubeletConfiguration drop-in under /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/.

kubelet-arg:
  - "cpu-manager-policy=static"
  - "reserved-cpus=0"

CRI-O (OpenShift): the kubelet settings above are enough. CRI-O can also drop the quota per pod, but only with the high-performance runtime class (OpenShift’s Node Tuning Operator makes performance-<profile>), a Guaranteed pod and whole CPUs. containerd ignores these annotations, so they are safe in a shared manifest.

metadata:
  annotations:
    cpu-quota.crio.io: "disable"
    cpu-load-balancing.crio.io: "disable"
    irq-load-balancing.crio.io: "disable"
spec:
  runtimeClassName: performance-<profile>

Verify

On the node:

kubectl get node <node> -o jsonpath='{.status.nodeInfo.kubeletVersion}'   # 1.33 or later
kubectl get --raw /api/v1/nodes/<node>/proxy/configz \
  | jq '.kubeletconfig | {cpuCFSQuota, cpuManagerPolicy, cpuManagerPolicyOptions, reservedSystemCPUs}'

cpuCFSQuota reads true (or is absent: that is the default).

In the pod:

cat /sys/fs/cgroup/cpu.max            # max 100000
grep throttled /sys/fs/cgroup/cpu.stat
cat /sys/fs/cgroup/cpu.stat.local     # Linux 6.6+: counts the pod's quota too

• nr_throttled and throttled_usec stay flat while the pod runs. Read them twice, a minute apart.
• In Prometheus, cAdvisor’s container_cpu_cfs_throttled_periods_total stays flat.
• ergon-tune: T9 runs at start in ergon-io and the media driver. ThrottleWatch is for your own loop’s housekeeping; nothing in this repository calls it yet.

The media driver DaemonSet

One driver pod per DPDK node: one ENA pair from the device plugin, six CPUs (one CPU is one core), memory and hugepages, requests equal to limits. The entrypoint uses three (housekeeping, sender, receiver) and leaves the rest for a client in the same pod (entrypoint.sh:13-14). The EKS lab ran the pod on four.

# Media Driver DaemonSet. One driver per DPDK node, each pod requesting one
# rusteron.io/dpdk-ena-pair, six CPUs, memory, and 2 MiB hugepages under
# Guaranteed QoS (requests == limits on every resource, so the kubelet's CPU
# Manager static policy hands it a dedicated cpuset).
#
# SMT (hyper-threading) is off on every latency node (the launch template's
# CpuOptions ThreadsPerCore=1; nosmt and provision-node.sh on bare metal), so
# one CPU is one physical core and six CPUs are six cores: housekeeping (the
# conductor, which does not busy-spin, plus the EAL and transport housekeeping
# threads), sender and receiver (the last two busy-spinning) and three spare.
# The entrypoint (entrypoint.sh) assigns them from the pod's cpuset and fails
# fast if fewer than three physical cores remain after SMT-sibling exclusion
# (a safety net: with SMT off every CPU is already its own core). A node
# therefore needs at least 5 physical cores, which provision-node.sh checks:
# one reserved for the system and kubelet, three for the driver (housekeeping,
# sender, receiver) and one for an application. The smallest c7i with them is
# c7i.4xlarge (8 cores).
#
# The secondary ENAs are bound on the host to igb_uio loaded with
# wc_activate=1, the only binding on virtualised EC2 that maps the ENA's
# low-latency-queue BAR write-combining (there is no IOMMU, and mainline
# vfio-pci maps no write-combining). On *.metal, which has an IOMMU, the same
# binding works in passthrough (iommu=pt, which provision-node.sh sets); it is
# the configuration the metal lab measured (BENCHMARKS.md, AWS round 4). With
# igb_uio the DPDK EAL runs in IOVA-as-PA mode (it logs
# `EAL: Selected IOVA mode 'PA'`), so the container must be privileged:
# translating hugepage addresses through /proc/self/pagemap needs
# CAP_SYS_ADMIN and CAP_DAC_READ_SEARCH, locking hugepages needs CAP_IPC_LOCK
# (https://doc.dpdk.org/guides/linux_gsg/enable_func.html), and the PMD maps
# the device's BARs through its sysfs resource files, which need a read-write
# /sys that only a privileged container gets.
#
# The device plugin injects the pair's /dev/uioN nodes + RUSTERON_DPDK_* env
# at Allocate time, so no device hostPath is declared here.
#
# Client pods on the node (the README's "Client pods") reach the driver
# through its Aeron directory: the /mnt/huge-aeron hostPath (type Directory)
# mounted at /mnt/huge-aeron, with AERON_DIR=/mnt/huge-aeron/aeron,
# AERON_TERM_BUFFER_SPARSE_FILE=false and
# AERON_CLIENT_PRE_TOUCH_MAPPED_MEMORY=true. Give them Guaranteed QoS with
# whole cores (requests equal to limits) and pin each busy-spinning thread to
# a core of its own in the pod's cpuset.
#
# kubelet stops the pod with SIGTERM, on which the driver stops and then
# releases the DPDK transport, as on SIGINT; the grace period bounds that.
#
# Nodes are picked by the rusteron-dpdk/ena=true label and taint; the VPC CNI
# keeps the primary ENA for pod networking, while the two secondary ENAs stay
# bound to igb_uio on the host (never claimed by the CNI: they are tagged
# node.k8s.amazonaws.com/no_manage=true when created).
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: rusteron-media-driver
spec:
  selector:
    matchLabels:
      app: rusteron-media-driver
  template:
    metadata:
      labels:
        app: rusteron-media-driver
    spec:
      nodeSelector:
        rusteron-dpdk/ena: "true"
      tolerations:
        - key: rusteron-dpdk
          operator: Exists
          effect: NoSchedule
      # SIGTERM first, SIGKILL once this has passed.
      terminationGracePeriodSeconds: 15
      containers:
        - name: media-driver
          image: rusteron-media-driver:dpdk
          envFrom:
            - configMapRef:
                name: rusteron-media-driver
          securityContext:
            # igb_uio + IOVA-as-PA: see the header comment.
            privileged: true
          resources:
            requests:
              rusteron.io/dpdk-ena-pair: "1"
              cpu: "6"
              memory: 2Gi
              hugepages-2Mi: 2Gi
            limits:
              rusteron.io/dpdk-ena-pair: "1"
              cpu: "6"
              memory: 2Gi
              hugepages-2Mi: 2Gi
          volumeMounts:
            - name: hugepages
              mountPath: /dev/hugepages
            # Aeron's directory: the node's sized hugetlbfs (bootstrap-dpdk-ena.sh)
            - name: aeron-huge
              mountPath: /mnt/huge-aeron
      volumes:
        - name: hugepages
          hostPath:
            path: /dev/hugepages
        - name: aeron-huge
          hostPath:
            path: /mnt/huge-aeron
            type: Directory

Its ConfigMap, with the reason for each value:

# Media Driver tunables shared by every DPDK pod. Per-pair values (PCI, CIDR,
# gateway) are injected by the device plugin at Allocate time, and the
# entrypoint (entrypoint.sh) derives the CPU roles, idle strategies, wildcard
# port ranges and EAL lcore from the pod's cpuset. Pinned here: the settings
# that are the same on every node and either differ from rusteron-media-driver's
# built-ins or are values every measurement used (so a change of default does
# not silently change production). They match the VM lab's driver
# (benches/cloud/vm.sh).
apiVersion: v1
kind: ConfigMap
metadata:
  name: rusteron-media-driver
data:
  # 2 MiB hugepages mounted by the DaemonSet; the DPDK EAL mempools live here.
  RUSTERON_DPDK_HUGE_DIR: "/dev/hugepages"
  # RUSTERON_DPDK_RX_DESCRIPTORS is left unset: each port takes min(4096, the
  # NIC's maximum), 4096 on an ENA: room for a burst. Against 1024 it did not
  # separate at idle (BENCHMARKS.md, "Measured, but no setting recommended").
  # The pool below holds that ring.
  # ENA queues take at most 512 TX descriptors (the device reports
  # tx_desc_lim.nb_max = 512); a larger ring fails port init.
  RUSTERON_DPDK_TX_DESCRIPTORS: "512"
  # The VM lab's count, which every DPDK measurement used (benches/cloud/vm.sh).
  # Each mbuf is about 2.3 KiB of 2 MiB hugepages (a 2 KiB data room, the
  # headroom and its headers), so two ports take about 75 MiB of the pod's
  # hugepages-2Mi; see the README's hugepage budget before raising it.
  RUSTERON_DPDK_MBUFS_PER_PORT: "16384"
  # A vfio-pci node's write-combining, as the node's preflight confirmed it.
  # vfio-pci maps an ENA's low-latency-queue BAR through its device file, so
  # /proc/self/maps cannot show it write-combined and the transport's LLQ
  # check would refuse the port; it accepts one bound to vfio-pci when this
  # is 1. The bootstrap binds the ENAs to vfio-pci (RUSTERON_DPDK_DRIVER) only
  # after preflight.sh passed, which for vfio-pci requires the operator's
  # RUSTERON_DPDK_VFIO_WC=1 on the node (a write-combining vfio-pci, Amazon's
  # patch); a failed run leaves no inventory, so the device plugin hands out
  # no pair. Inert for igb_uio ports, which must still show their BAR mapped
  # write-combined (resource2_wc).
  RUSTERON_DPDK_VFIO_WC: "1"
  RUSTERON_DPDK_MEMPOOL_CACHE: "256"
  RUSTERON_DPDK_BURST_SIZE: "32"
  # The Aeron directory on the node's own 1 GiB hugetlbfs for it (the
  # DaemonSet's /mnt/huge-aeron hostPath, mounted by the node bootstrap), so
  # clients on the node can map it and the term buffers sit on hugepages, as in
  # the VM lab. It needs a sized mount: an unsized hugetlbfs reports no free
  # space and Aeron's storage check refuses every log buffer. Hugetlbfs files
  # must be whole 2 MiB pages and cannot be sparse. A restarted driver replaces
  # its old directory.
  AERON_DIR: "/mnt/huge-aeron/aeron"
  RUSTERON_MEDIA_DRIVER_DIR: "/mnt/huge-aeron/aeron"
  AERON_DIR_DELETE_ON_START: "true"
  AERON_FILE_PAGE_SIZE: "2097152"
  AERON_TERM_BUFFER_SPARSE_FILE: "false"
  # The receive window needs a socket buffer at least as large (Aeron refuses
  # to start otherwise); the node bootstrap raises rmem_max/wmem_max to allow it.
  AERON_RCV_INITIAL_WINDOW_LENGTH: "2097152"
  AERON_SOCKET_SO_RCVBUF: "2097152"
  AERON_SOCKET_SO_SNDBUF: "2097152"
  # Aeron UDP MTU that fits the DPDK max_aeron_mtu default.
  AERON_MTU_LENGTH: "1408"
  # Aeron's maximum (its default is 4): with DPDK at 1,344 B 600 k msgs/s no
  # NAK against 16,247 at the default (BENCHMARKS.md, one run). The DPDK
  # receive poll drains whole bursts whatever this is; it still sizes each
  # receive from a host-local socket. Pinned although the media driver also
  # defaults to 16 under dpdk-ena, so a rollback edits one visible value. A
  # rollback to the socket transport sets it back to 4 (the runbook's
  # rollback): kernel busy polling with 16 dropped frames at the NIC.
  AERON_RECEIVER_IO_VECTOR_CAPACITY: "16"
  # With the entrypoint's Aeron wildcard ranges set, Aeron itself picks the
  # source port of a publication without a fixed port (a plain endpoint=
  # channel) from 20000-20999, and receiver-side wildcard binds from
  # 21000-21999; peers' status messages and NAKs come back to those ports, so
  # the security group allows UDP 20000-21999 between the data subnets. This
  # range is the transport's own fallback for a wildcard bind Aeron leaves at
  # port 0: disjoint from the kernel's 32768-60999, so it never collides with
  # kernel sockets.
  RUSTERON_DPDK_EPHEMERAL_PORT_RANGE: "61000 63999"

(md-dpdk/deploy/aws-eks/; the node scripts and the entrypoint are tested against fixtures by just test-dpdk-bootstrap.)

The entrypoint reads the pod’s cpuset and hands out cores in order: housekeeping, sender, receiver, then the rest for a client. Node requirements, the hugepage budget and the device plugin: the deployment’s README.

Checks inside a pod

A pod sees the host’s CPU, interrupt and memory files but its own network namespace. ergon-tune reports the network settings it cannot read there as unknown, never as a warning. Run the NIC checks on the node, or in a pod on the host’s network. T9 reads the container’s own cgroup; the pod’s quota shows only in cpu.stat.local (verify).

Under ERGON_HOST_CHECKS=require, ergon-io refuses a stock static-CPU-manager pod: K2 cannot prove exclusive CPUs from the kubelet’s cpuset, so it reports unknown. An isolated cpuset partition, as the latency pods plugin grants, passes it (ergon-tune).