Shared CPUs for Helper Threads (k8s/nri-shared-cpus)

How to install and use k8s/nri-shared-cpus on a Kubernetes node.

A busy-spinning pod gets whole cores of its own from the kubelet. Its helpers (dial, TLS, warnings, Aeron conductors) do not need a whole core, and running them on the loop’s core is what this layout forbids (its latency cost is not measured). This is a containerd NRI plugin and a kubelet device plugin that give the pod the node’s shared housekeeping CPUs as well, and start every thread there.

The busy-spin loop keeps its exclusive core. Only sleepy helpers go to the shared CPUs. Never a spinner or SQPOLL there.

Source: k8s/nri-shared-cpus/.

Status: published as tag 0.1.0. Anyone can pull the image and the chart. Tested on kind; not yet on a real EKS node (see Limits). The latency numbers are not measured. AKS cannot run it. Prior art: k8s/nri-shared-cpus/README.md.

Use the public chart

Do these in order. The chart installs the DaemonSet. It does not prepare the node, and it does not install the hot-pod admission policy.

1. Prepare the node. Node setup below. Label it ergon.io/shared-cpus=true.
2. Install. config.sharedCpus is the housekeeping CPU, inside reservedSystemCPUs, and not the IRQ CPU. For the 8-CPU split in that section the value is 0. One Helm release per shared-CPU layout.

helm install ergon-shared-cpus oci://ghcr.io/mimran1980/charts/ergon-shared-cpus \
  --version 0.1.0 \
  --namespace ergon-system --create-namespace \
  --set config.sharedCpus=0 \
  --set 'config.namespaces={trading}'

Check the pulled image. This prints the index digest sha256:d28fea2f95ebce3e53d33f8770f3ba69be75c76ec7437ea2b92c1329759e07e9:

docker buildx imagetools inspect ghcr.io/mimran1980/ergon-shared-cpus:0.1.0

The chart’s image helper is repository:tag. image.tag cannot be @sha256:…. Chart digest sha256:7347131d7d87caee5cf6de11b2a11e2b62454da59c41092b3805bc2834795dbd.

helm install of this 0.1.0 package prints “This chart is NOT released”. That note was packed before the push. The tag is the public chart. The note in the repo is corrected for a later tag. This tag is not republished to change the note.

Chart 0.1.0 writes sharedCpus, oversubscription, namespaces, defaultAffinity, hookPath, annotationOptIn, metricsAddress and devRoot. It does not write seccomp, ptp or schedFifo. --set of those keys does nothing on this package. The plugin leaves them off. A later edit of the ConfigMap can turn a key on, and the pod still has to opt in. Aeron’s C media driver does not use io_uring. SCHED_FIFO stays off.

3. Apply the hot-pod policy. These two files only, from k8s/nri-latency-pods/admission/:

kubectl apply -f priority-class.yaml -f validating-admission-policy.yaml

Leave examples/ and mutating-admission-policy.yaml unapplied. A pod labelled ergon.io/latency: hot is refused when it has an exec probe, a container with restartPolicy: Always, no ergon.io/shared-millicpus request, no required-plugins.noderesource.dev annotation naming ergon-shared-cpus, or no priorityClassName.

4. Run a Guaranteed pod in an allowlisted namespace. Whole-number cpu, requests equal limits, ergon.io/shared-millicpus. The pod spec is under Pod. Inside it, ERGON_SHARED_CPUS is the housekeeping list, ERGON_ISOLATED_CPUS is the exclusive list, PID 1 Cpus_allowed_list is the shared CPUs only, and cpuset.cpus.effective is the union.

Flux: a HelmRepository of type oci at oci://ghcr.io/mimran1980/charts, chart ergon-shared-cpus version 0.1.0, targetNamespace: ergon-system, and the same config values. Copy the two admission files into the Git repo Flux already applies. Do not point Flux at the whole admission/ directory: that directory also holds pods the policy is meant to refuse.

See it on kind

k8s/nri-shared-cpus/examples/try/run.sh creates a throwaway kind cluster (kindest/node:v1.33.0, static CPU manager, reservedSystemCPUs: "0,1"), installs this public chart with sharedCpus: "0", applies the two admission files, and runs one hot pod. The pod’s log must show ERGON_SHARED_CPUS=0, a non-empty ERGON_ISOLATED_CPUS, PID 1 Cpus_allowed_list of 0, and nproc=1. A second pod with an exec liveness probe must be denied. The script deletes the cluster on the way out and uses a kubeconfig under /tmp.

That run shows the shared-CPU grant and the HFT admission check. It does not measure latency. Kind on Docker Desktop has no hugepage pool, no NIC PHC and no cpuset.cpus.exclusive partition to demonstrate, and this sample does not ask for those grants.

One run of that script on 2026-10-22, chart 0.1.0, kindest/node:v1.33.0, Docker Desktop linux aarch64 with 10 CPUs, printed ERGON_ISOLATED_CPUS=2,3, ERGON_SHARED_CPUS=0, cpuset.cpus.effective=0,2-3, PID 1 Cpus_allowed_list 0, nproc=1. Node allocatable ergon.io/shared-millicpus was 1k. The exec-probe pod was denied by ValidatingAdmissionPolicy ergon-latency-pods. The cluster was deleted afterwards. The log lines are in k8s/nri-shared-cpus/examples/try/README.md.

The 18-step test/e2e.sh is a different run. It builds ergon-shared-cpus:dev from this tree. It does not pull the public chart.

What it gives

M is millicpus because extended resources are whole numbers and requests must equal limits. 250 is a quarter core of scheduler accounting only: the plugin still adds one whole shared CPU, and there is no hard cap.

Timer slack

The kernel’s default timer slack is 50,000 ns. It is added to every sleep and to every epoll/io_uring timeout. The hook sets 1 ns, the kernel’s floor, on every thread the container has when it starts; the kernel copies it to every thread and child after that. A container that cannot have it does not start. The OCI hook runs as root, so it can write /proc/<tid>/timerslack_ns; a program can always set its own slack with prctl(PR_SET_TIMERSLACK).

What the 1 ns slack is worth in latency is not measured. The loop spins and never sleeps; the slack applies to sleeping helpers such as Aeron’s conductor.

Node setup

Do this before installing the plugin. Everything here is per node. The rest of the node checklist is Latency Pods: node setup.

CPU split. Pick three CPU sets on the node and write them down; every later step names one of them.

Kernel boot line. Isolate the exclusive CPUs only. Never the shared one.

nohz_full=2-5 rcu_nocbs=2-5 isolcpus=managed_irq,domain,2-5

kubelet — reservedSystemCPUs: "0,1" (shared + IRQ) with the policy above. Restart kubelet and check it took.

containerd — NRI on (disable = false) and the default validator on. The validator is what makes an unmet required-plugins annotation fail the create instead of starting the pod half-granted. Leave required_plugins empty: a node-wide list would also block kube-system pods and the plugin’s own pod while it is down.

Install on Kubernetes

The plugin is public. No image pull secret.

helm install ergon-shared-cpus oci://ghcr.io/mimran1980/charts/ergon-shared-cpus \
  --version 0.1.0 \
  --namespace ergon-system --create-namespace \
  --set config.sharedCpus=0 \
  --set 'config.namespaces={trading}'

config.sharedCpus is this node’s shared CPU. For the 8-CPU split above that is 0: inside reservedSystemCPUs, and not the IRQ CPU. The chart creates ergon-system with the privileged Pod Security labels and selects nodes labelled ergon.io/shared-cpus=true. It does not label nodes, and it does not set the boot line, the kubelet, or containerd.

Do the node setup above first, then the steps below. The public install is Use the public chart. Chart 0.1.0 does not emit seccomp, ptp or schedFifo. Step 4 below is the local image ergon-shared-cpus:dev. The kind sample that pulls the public chart is See it on kind.

1. Label the nodes that take the plugin

kubectl label node <node> ergon.io/shared-cpus=true

Only labelled nodes run the DaemonSet. A pod that asks for the device on an unlabelled node stays pending.

2. Edit deploy/configmap.yaml

This is the step people get wrong. The file is k8s/nri-shared-cpus/deploy/configmap.yaml, and it becomes /etc/ergon-shared-cpus/config.yaml in the plugin’s container.

sharedCpus: "0"          # the node's shared CPUs, kernel cpulist
oversubscription: 1
namespaces: ["trading"]  # who may opt in. Empty refuses every pod
defaultAffinity: shared
annotationOptIn: false
devRoot: /host/dev       # where the DaemonSet mounts the host's /dev; without
                         # it the plugin finds no PHC and refuses ptp grants

# These three are "off" by default. A latency node turns the first two on.
# Each one is a node-level permission: the pod still opts in by annotation,
# and a pod that asks for a grant this field does not allow is refused.
seccomp:
  ioUring: "allow"       # default "off". Without it, a pod asking for
                         # seccomp.ergon.io/<ctr>: "io-uring" is refused.
                         # For apps that use io_uring (ergon-io's uring
                         # backends). Aeron's C media driver does not use it
ptp: "auto"              # default "off". Without it, a pod asking for
                         # ptp.ergon.io/<ctr>: "auto" is refused. "auto" is
                         # the NIC's PHC, or give a /dev/ptp* path the node has
schedFifo: "off"         # off by default; its one A/B was mixed (see
                         # Kubernetes HFT tuning, SCHED_FIFO)

Rules:

• sharedCpus must sit inside the kubelet’s reservedSystemCPUs, minus the IRQ CPU(s). In the table above that is 0.
• namespaces is an allowlist. Create those namespaces before applying.
• Quote off, on, yes, no. YAML 1.1 turns unquoted ones into booleans.
• An unknown field or an out-of-range value stops the plugin at start, and the message names the field.
• off is not harmless. It is a refusal. A pod whose annotation asks for a grant the ConfigMap does not allow does not start: the kubelet shows CreateContainerError and the text names the field to change.

3. Install the public chart

Use the helm install at the top of this section. One Helm release per shared-CPU layout: the chart has one ConfigMap.

The plugin is root on the node and installs an executable at /opt/ergon/bin/ergon-cpus. Only this repository’s publish workflow pushes.

Kind, or a node with no registry: docker build -t ergon-shared-cpus:dev . from k8s/nri-shared-cpus, load that image, and apply the three manifests in the next step. deploy/daemonset.yaml ships image: ergon-shared-cpus:dev with imagePullPolicy: IfNotPresent.

4. Apply the three manifests

Skip this when the Helm release above is installed. It is the kind path.

kubectl apply -f deploy/namespace.yaml \
              -f deploy/configmap.yaml \
              -f deploy/daemonset.yaml

The DaemonSet needs five host paths: /var/run/nri, /var/lib/kubelet/device-plugins and /opt/ergon/bin read-write, plus /dev at /host/dev and the host cgroup v2 tree at /sys/fs/cgroup, both read-only. The namespace is labelled Pod Security privileged for that reason. The plugin is root on the node: anything on the NRI socket can change any container.

5. Verify the plugin before any workload

The pod is Ready on every labelled node, allocatable shows ergon.io/shared-millicpus greater than zero, /readyz is healthy, and the logs show no config error (strict YAML: a bad field stops it). If allocatable is empty the device plugin did not register: check the logs and that /var/lib/kubelet/device-plugins is mounted.

6. Run a pod and check the grants landed

Use the pod under Pod. Inside it:

kubectl exec trader -- sh -c 'echo isolated=$ERGON_ISOLATED_CPUS shared=$ERGON_SHARED_CPUS'
kubectl exec trader -- cat /proc/1/status | grep Cpus_allowed_list

Cpus_allowed_list must print the shared CPUs only. That is the point of the plugin.

Config

/etc/ergon-shared-cpus/config.yaml, from the ConfigMap. Strict YAML: an unknown field is an error at start. No hot reload — edit the ConfigMap and restart the DaemonSet.

Quote off, on, yes and no in this file. YAML 1.1 turns unquoted ones into booleans.

Unknown fields and out-of-range values are refused at start, each naming its field: oversubscription outside 1–100, a namespaces entry that is not a namespace name, hookPath not ending in /ergon-cpus, an rlimits value that is neither inf nor a positive count, schedFifo / ptp / seccomp.ioUring outside their sets.

Pod

apiVersion: v1
kind: Pod
metadata:
  name: trader
  namespace: trading                       # must be in the plugin's allowlist
  annotations:
    required-plugins.noderesource.dev: '["ergon-shared-cpus"]'
spec:
  nodeSelector: { ergon.io/shared-cpus: "true" }
  containers:
  - name: app
    image: my-app
    resources:
      requests: { cpu: "2", memory: 1Gi, ergon.io/shared-millicpus: "250" }
      limits:   { cpu: "2", memory: 1Gi, ergon.io/shared-millicpus: "250" }

Inside the container:

Env vars

Read the CPUs from these variables or from cpuset.cpus.effective, never from the process’s own affinity: the start mask is the shared CPUs.

Pinning by index

Pin each hot thread to isolated index i. The app never needs the real CPU numbers.

/* SPDX-License-Identifier: Apache-2.0 */
/*
 * Pin a hot thread by index into ERGON_ISOLATED_CPUS. Leave helpers alone:
 * every thread already starts on the shared CPUs.
 *
 * Build:  cc -O2 -o pin pin.c
 * Run:    in a container that asked for ergon.io/shared-millicpus.
 */
#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

/* The i-th number of a list like "1,5"; -1 if there is none. */
static int nth_cpu(const char *list, int i)
{
    if (list == NULL)
        return -1;
    for (const char *p = list; *p != '\0'; i--) {
        char *end;
        long cpu = strtol(p, &end, 10);
        if (end == p)
            return -1;
        if (i == 0)
            return (int)cpu;
        p = (*end == ',') ? end + 1 : "";
    }
    return -1;
}

/* Pin the calling thread to isolated CPU index i. Returns the CPU, or -1. */
int pin_isolated(int i)
{
    int cpu = nth_cpu(getenv("ERGON_ISOLATED_CPUS"), i);
    if (cpu < 0)
        return -1;
    cpu_set_t set;
    CPU_ZERO(&set);
    CPU_SET(cpu, &set);
    return sched_setaffinity(0, sizeof set, &set) == 0 ? cpu : -1;
}

/* Report which CPU the calling thread is on and where its mask is. */
static void report(const char *who)
{
    int now = sched_getcpu();
    cpu_set_t set;
    CPU_ZERO(&set);
    if (sched_getaffinity(0, sizeof set, &set) != 0) {
        perror("sched_getaffinity");
        return;
    }
    printf("%s: running on cpu %d, mask", who, now);
    for (int c = 0; c < CPU_SETSIZE; c++)
        if (CPU_ISSET(c, &set))
            printf(" %d", c);
    printf("\n");
}

int main(void)
{
    const char *iso = getenv("ERGON_ISOLATED_CPUS");
    const char *sh = getenv("ERGON_SHARED_CPUS");
    printf("ERGON_ISOLATED_CPUS=%s\nERGON_SHARED_CPUS=%s\n",
           iso ? iso : "(unset)", sh ? sh : "(unset)");

    /* Every thread starts on the shared CPUs. */
    report("helper (default)");

    /* Pin the hot thread to isolated index 0. */
    int cpu = pin_isolated(0);
    if (cpu < 0) {
        fprintf(stderr, "pin_isolated(0) failed\n");
        return 1;
    }
    printf("pinned the hot thread to isolated index 0 = cpu %d\n", cpu);
    report("hot (pinned)");

    /* Index 1, when the pod asked for more than one CPU. */
    int cpu1 = pin_isolated(1);
    if (cpu1 >= 0) {
        printf("isolated index 1 = cpu %d\n", cpu1);
        report("hot2 (pinned)");
    }
    return 0;
}

Rules:

• leave helpers alone: they start on the shared CPUs;
• create helpers from a thread that is still on the shared set (a new thread copies its creator’s mask);
• each sleepy helper checks sched_getcpu() when it wakes and alarms if it is not on ERGON_SHARED_CPUS;
• never a spinner or SQPOLL on a shared CPU. ergon-io keeps this: with no sqpoll_cpu, uring-sqpoll takes a CPU of the pod’s own outside ERGON_HELPER_CPUS and refuses (Error::Cpu) when there is none (io: Kubernetes);
• two indices are two physical cores only when they are two groups in ERGON_ISOLATED_CORES. On SMT, 1,5 is one core and its two threads share every execution unit: two “hot cores” there is one hot core.

A core is never split between the exclusive and the shared CPUs. On SMT the hot thread and the helpers would time-share one physical core, which is the contention this whole setup exists to remove. The plugin refuses such a container rather than start it claiming an isolation it does not have. On x86 keep SMT off, or use full-pcpus-only and put whole cores in reservedSystemCPUs.

Worked example: an Aeron media driver

Two pinned cores (sender and receiver) plus the conductor on the shared CPU, with huge pages. This is the layout a latency node wants.

Node boot line (from the table above), for exclusive CPUs 2 and 3, shared 0, IRQ 1:

nohz_full=2-3 rcu_nocbs=2-3 isolcpus=managed_irq,domain,2-3 hugepagesz=2M hugepages=512

ConfigMap (sharedCpus: "0"), kubelet (reservedSystemCPUs: "0,1"), and a namespace in the allowlist.

Pod. Three grants on top of the shared CPU: a sized hugetlbfs for AERON_DIR, io_uring seccomp, and the NIC PHC. The isolation of CPUs 2-3 comes from the boot line, not from a grant: cpu-partition needs the program to move its own threads between cgroups (Latency Pods), and aeronmd does not.

apiVersion: v1
kind: Pod
metadata:
  name: aeron-driver
  namespace: trading
  labels: { ergon.io/latency: "hot" }
  annotations:
    required-plugins.noderesource.dev: '["ergon-shared-cpus"]'
    hugepages.ergon.io/app: "2M"          # sized hugetlbfs for AERON_DIR
    seccomp.ergon.io/app: "io-uring"      # for apps that use io_uring; Aeron does not
    ptp.ergon.io/app: "auto"              # the NIC's PHC
spec:
  nodeSelector: { ergon.io/shared-cpus: "true" }
  priorityClassName: ergon-hot              # k8s/nri-latency-pods/admission/priority-class.yaml
  containers:
  - name: app
    image: ergon-media-driver:dev
    command: ["ergon-cpus",
              "--env", "AERON_SENDER_CPU_AFFINITY=isolated:0",
              "--env", "AERON_RECEIVER_CPU_AFFINITY=isolated:1",
              "--", "aeronmd"]
    env:
    - { name: AERON_THREADING_MODE, value: DEDICATED }
    - { name: AERON_DIR, value: /dev/hugepages/aeron }   # set by the app, see below
    resources:
      requests: { cpu: "2", memory: 2Gi, ergon.io/shared-millicpus: "250", hugepages-2Mi: 256Mi }
      limits:   { cpu: "2", memory: 2Gi, ergon.io/shared-millicpus: "250", hugepages-2Mi: 256Mi }

What each piece does:

The app sets AERON_DIR itself. The plugin exports ERGON_HUGEPAGES_DIR and ERGON_HUGEPAGE_SIZE and does not set AERON_DIR, because the app must create the subdirectory:

mkdir -p "$ERGON_HUGEPAGES_DIR/aeron"
export AERON_DIR="$ERGON_HUGEPAGES_DIR/aeron"
export AERON_FILE_PAGE_SIZE="$ERGON_HUGEPAGE_SIZE"

An unsized hugetlbfs — what emptyDir: {medium: HugePages} gives you — is rejected by Aeron. That is why the size comes from the annotation.

Affinity variables:

ergon-cpus turns isolated:<i> into a real CPU number before exec. It also sets its own affinity to the shared CPUs, so the driver starts there.

Check it worked:

kubectl exec aeron-driver -- sh -c 'echo isolated=$ERGON_ISOLATED_CPUS shared=$ERGON_SHARED_CPUS'
kubectl exec aeron-driver -- sh -c 'echo hugepages=$ERGON_HUGEPAGES_DIR size=$ERGON_HUGEPAGE_SIZE'
kubectl exec aeron-driver -- sh -c 'ps -L -o pid,tid,psr,comm | head'

psr must show the sender and receiver on 2 and 3 and every other thread on 0. mount | grep hugepages must show a size, not just hugetlbfs.

Fractional shared CPUs

• Extended resources are whole numbers and requests must equal limits, so the unit is millicpus: 250 = a quarter.
• The plugin adds ceil(M/1000) shared CPUs, at least one.
• The fraction is scheduler accounting only. With 2 shared CPUs and oversubscription 2, a node offers 4000; sixteen pods at 250 fit.
• There is no hard cap. Helpers may use the whole shared CPU, shared with the kubelet, containerd, system daemons and other pods.
• A helper wake-up on a shared CPU waits on the node’s other load; the size of that wait is not measured. It slows connects and the conductor, not the loop.

Fail closed

Limits and what is not verified

• The plugin is root on the node. It mounts five host paths: /var/run/nri, /var/lib/kubelet/device-plugins and /opt/ergon/bin read-write, the host’s /dev at /host/dev and /sys/fs/cgroup read-only. No hostPID, no hostNetwork, not privileged, capabilities.drop: [ALL], readOnlyRootFilesystem: true, seccomp RuntimeDefault, no API access. Only namespaces in the allowlist may opt in.
• AKS has no supported way to enable containerd NRI plugins and no reservedSystemCPUs. Not supported.
• CRI-O users use its shared_cpuset instead.
• Bottlerocket: whether runc may exec a hook binary that a pod wrote to a host path is not verified. Use defaultAffinity: all and the wrapper until checked.
• kubectl exec and exec probes start on any CPU of the cpuset, exclusive ones included. No exec probes on hot containers; run debug commands as ergon-cpus -- <cmd>.
• In-place CPU resize leaves the CPU env stale. Use resizePolicy: RestartContainer for cpu.
• Not measured / not verified: the latency cost of helpers on shared CPUs, and a real EKS node (the NRI socket, nodeadm TOML merge, /opt writable).

Decision. Use it on kind or a self-managed node where you can set reservedSystemCPUs and enable NRI. That the exclusive cores stay quiet with it is a design claim, not a measurement. Not for AKS or CRI-O.