Kubernetes HFT tuning

Every latency setting in this guide, where it goes, and its evidence. Measured links to a committed run below. Not measured here means the setting rests on kernel, vendor or Kubernetes documentation only. Node needs node-level access (boot line, kubelet, root on the host); a pod cannot set it. Pod works from the pod spec or the app.

Terms: deployment glossary.

This chapter is the list. It is not an image and it is not a second chart. The shared-CPU grant is Shared CPUs: public image ghcr.io/mimran1980/ergon-shared-cpus:0.1.0 and chart oci://ghcr.io/mimran1980/charts/ergon-shared-cpus version 0.1.0. The refusal of an exec probe on a pod labelled ergon.io/latency: hot is the admission policy in Latency Pods. Hugepages, the isolated cpuset partition, io_uring seccomp and the NIC PHC are opt-in grants in that same image. Chart 0.1.0 does not write seccomp, ptp or schedFifo, so those stay off. The kind sample k8s/nri-shared-cpus/examples/try installs the public chart, prints the shared-CPU grant, and shows the admission denial. It does not measure the rows below.

Settings

Evidence

Every row: one run, arms compared only with the same run’s baseline. µs. Raw tables: git show <commit>:<path> as named. Hosts: AWS c7i.2xlarge receiver (SMT on), c7i.xlarge sender, kernel 7.0; Azure Standard_D8ds_v6, kernel 6.17; Graviton c8g.2xlarge (io/benches/BENCHMARK.md). Boot lines and NIC settings per run are in the committed env-rcv*.txt next to each run id under io/benches/results/.

Isolation

A loop that only reads the CPU counter, on the CPU the busy loop uses, records every gap. One sample per boot line; the loop sends nothing. Sources: git show 2517309d:io/benches/results/flags-aws-20261014T163114Z/tables.md and …/flags-azure-20261014T144146Z/tables.md, “Platform noise”.

isolcpus cut gaps over 20 µs. Adding nohz_full cut gaps over 5 µs (about 1,000 a second before it, the tick rate it stops: NO_HZ) by 97.7 % (Azure) and 99.9 % (AWS). The longest gap is one event per sample and does not rank the two isolated lines.

Interrupt placement

Graviton (fair run, io/benches/results/flags-aws-20261019T175248Z/fair.md, committed): WebSocket round trip, interrupt-driven backends (epoll-spin, mio, uring-spin), moderation off everywhere, two rounds. Six points’ senders fell behind; the file’s top note names them.

Same file, own-core IRQs, p50: at 1 connection the interrupt backends read 38.78–39.33 against 36.01–37.04 for NIC polling; at 4 connections the two families overlap. From 16 connections recv-busy (48.98–49.63 at 16, 78.14–78.29 at 64, from-due p99 9.7 s) and recv-busy-pq (44.26–44.73) sit above the interrupt backends (39.27–40.42); uring-sqpoll stays level.

x86 (git show 2517309d:io/benches/results/flags-aws-20261015T062428Z/tables.md, irqlocal arm: every queue IRQ on the loop’s CPU), recv-busy market data, kernel arrival → decoded:

The WebSocket round trip in that run showed no consistent change.

Interrupt moderation

git show 2517309d:io/benches/results/flags-aws-20261015T070339Z/tables.md, AWS c7i, recv-busy, 10,000 round trips a second. Baseline: adaptive-rx on (io/benches/scripts/vm-setup.sh nodim). Rounds 1 / 2.

WebSocket p99.99 fell 36–47 µs in both rounds. HTTP p99.99 moved no more than the 42 µs gap between its own two baselines. One run. The interrupt path with moderation on is not measured.

SMT sibling

git show 2517309d:io/benches/results/flags-aws-20261015T122127Z/tables.md, AWS c7i, uring-sqpoll market data, kernel arrival → decoded, three rounds: the SQPOLL kernel thread on its own core against the loop’s SMT sibling.

At 1 connection the own core is lower at p50 and the sibling at p99. At 10 and 100 connections the own core is lower at p50 and p99 in every round. This is one busy sibling (SQPOLL); other sibling loads are not measured.

C-states

…/flags-aws-20261015T062428Z/tables.md, cstate arm: every idle state deeper than C1 off, AWS c7i VM. Rounds 1 / 2.

WebSocket worse at p50 and p99.99, HTTP better at p99.99: no consistent gain.

SCHED_FIFO

…/flags-aws-20261015T070339Z/tables.md, rt arm: the loop under chrt -f 50, kernel.sched_rt_runtime_us=-1. Rounds 1 / 2.

Mixed. Under the policy the loop’s nohz_full CPU took 533–947 timer interrupts a second, against 11–23 without it.

Copy-paste

Kubelet (EKS NodeConfig; k3s and AKS: Kubernetes):

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  full-pcpus-only: "true"
reservedSystemCPUs: "0"          # the housekeeping cores
topologyManagerPolicy: single-numa-node
# cpuCFSQuota stays true: the kubelet drops it for exclusive CPUs (1.33+)

App pod: Guaranteed, whole cores, Aeron’s directory on the node’s hugetlbfs. AERON_FILE_PAGE_SIZE and AERON_TERM_BUFFER_SPARSE_FILE are read only by the media driver: set them on the driver’s pod.

apiVersion: v1
kind: Pod
spec:
  priorityClassName: <your latency class>
  containers:
  - name: app
    resources:
      requests: { cpu: "2", memory: 1Gi, hugepages-2Mi: 1Gi }
      limits:   { cpu: "2", memory: 1Gi, hugepages-2Mi: 1Gi }
    env:
    - { name: AERON_CLIENT_PRE_TOUCH_MAPPED_MEMORY, value: "true" }
    volumeMounts:
    - { name: huge, mountPath: /mnt/huge-aeron }
  volumes:
  - name: huge
    hostPath: { path: /mnt/huge-aeron, type: Directory }

Pin each busy thread to one CPU of the pod’s cpuset before the loop starts (io: Kubernetes).

Check it

# 1. no CPU quota on the busy container
cat /sys/fs/cgroup/cpu.max               # expect: max 100000
grep throttled /sys/fs/cgroup/cpu.stat   # nr_throttled stays flat

# 2. the pod got whole cores of its own
grep Cpus_allowed_list /proc/self/status

# 3. the host: ergon-tune names each missing setting and its fix
#    (runs at start-up in ergon-io's Driver::new and the media driver)

ERGON_HOST_CHECKS=require (media driver alias RUSTERON_MEDIA_DRIVER_HOST_CHECK) refuses to start while any check warns or a proof check stays unproven (ergon-tune). In a stock static-CPU-manager pod ergon-io’s K2 stays unproven, so require refuses it (checks inside a pod).

Open

• No committed run measures THP, hugepages, the RX ring, NUMA settings or the CPU quota on latency.
• Most maxima above are not traced to a cause.

More