Backends and Configuration

A backend is the kernel interface the Driver waits and reads with. It is configuration, so one binary can run with each. Which to pick: Choosing a backend.

• epoll_wait(0) never blocks, but never polls the NIC either. mio, and so tokio, is this path too.
• Plain recv-busy reads every socket on every poll, so each socket adds a little to every poll. per_queue reads one socket per NIC queue instead: see cost per poll.
• uring-sqpoll costs a spinning kernel thread per ring. Rings in one process can share one; other processes cannot. Its kernel thread polls each NIC queue once per pass, so it has no per-socket scan.

Configuration strings

backend[,key=value...], parsed and checked by Config::parse:

#[test]
fn keys_override_defaults() -> R {
    let c =
        Config::parse("uring-sqpoll, sqpoll_cpu=3, busy_poll_us=2, rcvbuf=0, rx_timestamps=false")?;
    assert_eq!(c.backend, Backend::UringSqpoll);
    assert_eq!(c.sqpoll_cpu, Some(3));
    assert_eq!(c.busy_poll_us, 2);
    assert_eq!(c.rcvbuf, None);
    assert!(!c.rx_timestamps);
    Ok(())
}

(From io/tests/backend_config.rs; cargo test -p ergon-io runs it on every platform.)

An unknown key, a bad value or a setting the backend cannot use is an error (Config::validate), never ignored.

The four one-core backends

recv-busy, recv-busy,per_queue, epoll-spin and uring-spin run on the loop’s CPU alone: no SQPOLL kernel thread. uring-sqpoll is the only backend that needs another CPU; it is not in this section. The interrupt backends (epoll-spin, uring-spin) were measured with the NIC’s interrupts on a core of their own; on the loop’s CPU their p99 rose from 4 sockets (fair run).

/// The four backends that cost one core each. `uring-sqpoll` is not here:
/// its SQPOLL kernel thread spins on a CPU of its own.
#[test]
fn the_one_core_backends() -> R {
    // recv-busy: one recvmsg per socket per poll. Best at 1-4 sockets.
    let plain = Config::parse("recv-busy")?;
    assert_eq!(plain.backend, Backend::RecvBusy);
    assert!(!plain.per_queue);

    // per_queue: one busy-polled read per NIC queue, then only the sockets
    // epoll_wait(0) reports ready. The same latency as plain at 1-16 sockets,
    // and the only one-core choice still flat at 256.
    let queued = Config::parse("recv-busy,per_queue=true")?;
    assert_eq!(queued.backend, Backend::RecvBusy);
    assert!(queued.per_queue);
    assert_eq!(queued.busy_poll_us, 1);

    // epoll-spin: epoll_wait(0), then recvmsg on the ready sockets. Cheapest
    // per socket, but it waits for the NIC interrupt.
    let epoll = Config::parse("epoll-spin")?;
    assert_eq!(epoll.backend, Backend::EpollSpin);
    assert!(!epoll.per_queue);

    // uring-spin: IORING_OP_RECVMSG with DEFER_TASKRUN. The fewest syscalls of
    // any spin loop; also interrupt-driven.
    let uring = Config::parse("uring-spin")?;
    assert_eq!(uring.backend, Backend::UringSpin);
    assert!(!uring.per_queue);

    Ok(())
}

(From io/tests/backend_config.rs; cargo test -p ergon-io runs it on every platform.)

What one poll does:

Which one to take: choosing a backend.

per_queue: one read per NIC queue

Plain recv-busy calls recvmsg on every socket every poll. Each call runs one NAPI busy-poll pass even when the socket is idle, so almost all of those reads return nothing and much of the core sits in the kernel. A message then waits for its socket’s turn, half a poll on average and up to a whole one.

per_queue changes the scan. It reads one socket per NIC queue (that read polls the queue), then calls epoll_wait(0) and reads only the sockets that are ready. The cost stops growing at the queue count.

As struct fields, for when more than one setting changes:

/// The same choice as struct fields, for when more than one setting changes.
#[test]
fn per_queue_as_struct_fields() {
    let mut cfg = Config::new(Backend::RecvBusy);
    cfg.per_queue = true; // one read per NIC queue, then epoll_wait(0)
    cfg.busy_poll_us = 1; // one NAPI pass per empty read
    cfg.rcvbuf = Some(4 << 20);

    assert!(cfg.per_queue);
    assert_eq!(cfg.busy_poll_us, 1);
    assert_eq!(cfg.rcvbuf, Some(4 << 20));
}

It is recv-busy and nothing else. On any other backend it is a parse error, never ignored:

    for bad in [
        "epoll-spin,per_queue=true",   // error
        "uring-spin,per_queue=true",   // error
        "uring-sqpoll,per_queue=true", // error
        "recv-busy,per_queue=1",       // error: not true/false
    ] {
        assert!(Config::parse(bad).is_err(), "{bad:?} accepted");
    }

On the lab command line:

# plain: one recvmsg per socket per poll
io_bench lab --variants recv-busy --cpus 2 --busy-poll 1 --wait-us 0 \
  --server SENDER:9447 --conns 100 --duration 60 --out res/c100-recv-busy

# per_queue: one read per NIC queue, then the ready sockets
io_bench lab --variants recv-busy --per-queue --cpus 2 --busy-poll 1 --wait-us 0 \
  --server SENDER:9447 --conns 100 --duration 60 --out res/c100-per-queue

$B/analyse --warmup-s 2 res/c100-*

Measured against plain recv-busy and uring-sqpoll at 1–256 connections: polling backends.

Kernel

An older kernel gives Error::Unsupported from Driver::new.

Host settings

The loop is only as quiet as its core. What mattered on AWS and Azure, for x86 and Graviton: Deployment: AWS and Azure hosts. ergon-io’s own checks (ergon-tune) report what is missing at start-up.

Cost per poll

One poll is 1 ÷ the loop’s polls per second. Measured poll times: polling backends.

One poll per backend. Where the NIC’s NAPI pass runs is what separates them:

flowchart TD
    P["one Driver::poll"]
    P --> RB["recv-busy"]
    P --> ES["epoll-spin"]
    P --> US["uring-spin"]
    P --> SQ["uring-sqpoll"]

    RB --> RB1["recvmsg on every socket"]
    RB1 --> RB2["NAPI pass per socket, even idle"]
    RB2 --> RB3["cost grows with sockets"]

    ES --> ES1["epoll_wait(0)"]
    ES1 --> ES2["recvmsg on ready only"]
    ES2 --> ES3["interrupt path, no NIC poll"]

    US --> US1["submit RECVMSG"]
    US1 --> US2["DEFER_TASKRUN complete"]
    US2 --> US3["interrupt path, no NIC poll"]

    SQ --> SQ1["SQPOLL kernel thread"]
    SQ1 --> SQ2["polls each NIC queue"]
    SQ2 --> SQ3["NAPI pass per queue"]

recv-busy,per_queue sits between the first two: one busy-polled read per NIC queue (so one NAPI pass per queue), then epoll_wait(0) and only the ready sockets. Its cost stops at the queue count.

• Plain recv-busy grows with sockets: each non-blocking recvmsg runs one NIC busy-poll pass even on an idle socket, so most reads return nothing and the core sits in the kernel.
• What it costs in latency: a message waits for its socket’s turn, half a poll on average and up to a whole poll.
• per_queue reads one socket per NIC queue (which polls that queue), then only the ready sockets. Its poll stops growing at the queue count.
• epoll-spin and uring-spin do not scan idle sockets; they wait for the NIC interrupt instead. Against the polling backends: fair run.
• Where each polling backend stops winning: choosing a backend.

Find the crossover on your own host

The crossover depends on the NIC, its queue count, the interrupt path and the traffic. Measure it with the lab tools in io/benches, on two hosts: the many-connection market-data test the tables above come from, once per backend, at your socket count (here 20).

cargo build --release --manifest-path io/benches/Cargo.toml --bin replay_server --bin io_bench --bin analyse
B=io/benches/target/release

# sender: one stream per connection; restart it before each run, copy cert-many.pem to the receiver
sudo $B/replay_server --listen 0.0.0.0:9447 --cert-out cert-many.pem --clients 20 --mode mixed --cpu 1 --duration 120 \
  --trickle-min-us 1000 --trickle-max-us 10000 --burst-every-us 2000 --burst-slots 10 --burst 10 --seed 42

# receiver, loop on isolated CPU 2
sudo $B/io_bench lab --variants recv-busy --cpus 2 --busy-poll 1 --wait-us 0 --server SENDER:9447 \
  --ca cert-many.pem --sni lab.local --conns 20 --capacity 8000000 --duration 60 --out res/c20-recv-busy
# then the same with --variants recv-busy --per-queue (out res/c20-per-queue),
# --variants epoll-spin (res/c20-epoll-spin), --variants uring-sqpoll --sqpoll-cpu 3 (res/c20-sqpoll)

$B/analyse --warmup-s 2 res/c20-*

• Compare the backends’ sender → decoded p50, p99 and max within the same session, and repeat in reversed order: the two clocks drift.
• On AWS, set the NIC on the node before each run (node-level): ethtool -C <nic> adaptive-rx off rx-usecs 0, with the NIC’s interrupts on a core of their own, as in the fair run. Otherwise the interrupt backends wait for moderation.
• HTTP with a pool: HTTP with many connections.
• Everything about the method: io/benches/BENCHMARK.md.