Configuring the DPDK Transport

Read the notice first: this transport is a proof of concept, not recommended for production. For kernel bypass with Aeron, use Aeron Premium. Terms: glossary.

Build

• dpdk feature: Linux x86-64 or aarch64, libdpdk >= 23.11 found by pkg-config (libdpdk-dev on Ubuntu 24.04 or Debian 13; from source on Amazon Linux 2023). Without it the crate still builds everywhere, and selecting DPDK returns DpdkError::FeatureDisabled.
• static: links Aeron’s C driver statically.
• Graviton: build for the core, or Ubuntu’s and Debian’s gcc turn every atomic add on the hot path into a library call (GCC’s -moutline-atomics): CFLAGS=-mcpu=native (as measured) or the core (neoverse-v2 Graviton4, neoverse-v3 Graviton5), and the same in RUSTFLAGS=-Ctarget-cpu=. rusteron builds Aeron’s own C for the build host’s CPU unless RUSTERON_C_MARCH names another.

cargo test -p ergon-md-dpdk -- --test-threads=1   # every platform, without the feature
just check-dpdk                                   # Linux with libdpdk
just verify-dpdk-docker --build --all             # the same in Docker (amd64)
just verify-dpdk-docker --arm64 --build --all     # and arm64

The tests run the C against DPDK-free fakes of the EAL and the NICs. Only the cloud labs touch a real NIC.

Run it

The bundled media_driver binary picks the transport from RUSTERON_MEDIA_DRIVER_TRANSPORT:

RUSTERON_MEDIA_DRIVER_TRANSPORT=dpdk-ena cargo run -p ergon-md-dpdk --features dpdk --bin media_driver

The whole binary, which is also the shape of a driver embedded in your own process:

//! Standalone Aeron media driver with transport selection.
//!
//! `RUSTERON_MEDIA_DRIVER_TRANSPORT=dpdk-ena` installs the DPDK ENA transport
//! (configured from the `RUSTERON_DPDK_*` environment) before the driver
//! starts; absent or `default` keeps the socket transport. A selected DPDK
//! transport that fails exits non-zero; it never falls back to the socket
//! driver. Under `dpdk-ena` the receiver IO vector capacity is 16 unless
//! `AERON_RECEIVER_IO_VECTOR_CAPACITY` says otherwise.
//! `RUSTERON_MEDIA_DRIVER_DIR` pins the Aeron directory, and
//! `RUSTERON_MEDIA_DRIVER_MLOCKALL=1` locks the process's memory, present and
//! future, before the driver starts. SIGINT, SIGTERM and SIGHUP stop it
//! cleanly. The configuration is the library's
//! ([`ergon_md_dpdk::driver::ConfiguredDriver`]), which a process embedding
//! the driver shares, host checks included: one stderr line for each
//! low-latency setting that is not fine, and under
//! `RUSTERON_MEDIA_DRIVER_HOST_CHECK=require` a non-zero exit before any
//! driver starts while one warns ([`ergon_md_dpdk::checks`]).

use ergon_md_dpdk::driver::ConfiguredDriver;
use log::info;
use rusteron_media_driver::AeronDriverContext;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Flag to indicate when the application should stop
    let running = Arc::new(AtomicBool::new(true));
    let running_clone = Arc::clone(&running);

    // SIGINT, and with ctrlc's `termination` feature SIGTERM and SIGHUP:
    // kubelet stops a pod with SIGTERM, and as the container's PID 1 (the
    // entrypoint execs it) the driver would otherwise ignore it until the
    // grace period's SIGKILL, closing neither the driver nor the transport.
    ctrlc::set_handler(move || {
        running_clone.store(false, Ordering::SeqCst);
    })?;

    // The directory, the agents' pins and the transport, from the environment
    // after the context exists and before the driver is created. A selected
    // DPDK failure propagates as a nonzero exit; it never falls back to the
    // default socket driver.
    let configured = ConfiguredDriver::from_env(AeronDriverContext::new()?)?;
    println!("transport backend: {}", configured.backend());
    println!("aeron dir: {}", configured.dir());
    configured.print_configuration();

    // RUSTERON_MEDIA_DRIVER_MLOCKALL=1 (in `start`, before the driver is
    // created): every mapping faulted in by mmap, so the agents never
    // page-fault on a log buffer
    let driver = configured.start()?;
    if driver.memory_locked() {
        println!("memory locked (mlockall current and future)");
    }
    println!("media driver started");
    info!("Aeron media driver started successfully. Stop it with SIGINT or SIGTERM.");

    // Poll for work until a stop signal arrives
    while running.load(Ordering::Acquire) {
        driver.idle(driver.do_work()?);
    }
    info!("Received signal to stop the media driver.");
    // The driver, then its context, then the transport its bindings point into.
    drop(driver);
    info!("Aeron media driver stopped successfully.");
    Ok(())
}

(md-dpdk/src/bin/media_driver.rs: built by cargo test -p ergon-md-dpdk on every platform; tests/media_driver_transport_selection.rs runs it with each value, checks it never falls back, and stops it with SIGTERM.)

Entry points

Each is called once, at start-up, and adds no work to the send or receive path. ConfiguredDriver::from_env also starts the md-host-checks thread for the socket transport (host checks). Each returns Result<_, DpdkError>, except clock().

Dropping a DpdkTransport only leaks it (md-dpdk/src/ffi.rs, no Drop). Freeing it is DpdkTransport::close, which is unsafe and sound only after every driver started from the context has stopped. EnvMediaDriver frees it in that order for you.

Environment

The names are rusteron’s, so deployments written for rusteron’s DPDK branch work unchanged. Aeron itself is set with the usual AERON_* variables. The EAL is DPDK’s start-up layer: it maps the NICs and hugepages and starts DPDK’s own threads.

Every other knob (rate caps, host-local channels, ENA device arguments, receive drain, statistics) is in md-dpdk/README.md.

Threads and cores

Aeron’s driver has three agents (threads of work): the conductor sets up channels and handles control; the sender and the receiver move the data. The thread that runs the sender or receiver must busy-spin (AERON_*_IDLE_STRATEGY noop or spin) on a CPU of its own: a thread that sleeps leaves a NIC queue unread. The install refuses a layout that breaks this.

One physical core per role, skipping cores the pre-flight spin test flags (hosts). In a pod, all four come from the pod’s cpuset (static CPU manager, whole cores):

• The agents are pinned for you, in every mode, each to the CPU its AERON_*_CPU_AFFINITY names. Running unpinned is not measured here.
• SHARED_NETWORK saves a core and costs latency. Graviton G1, one A/B: 1,344 B bursts p50 522 against 234 µs, max 886 against 745 µs (BENCHMARKS.md).
• Start the driver confined to the housekeeping CPU (taskset -c <cpu>, a CPU from the pod’s cpuset that no busy thread uses), so the threads it does not pin stay off the busy cores. Name the same CPU in RUSTERON_DPDK_HOUSEKEEPING_CPU and RUSTERON_DPDK_EAL_ARGS='-l <cpu>'.

Where the transport sits, and the same core placement on both hosts:

flowchart LR
    NIC["NIC"] --> T{"transport"}
    T -->|"kernel UDP"| MD["media driver"]
    T -->|"dpdk-ena"| MD
    MD --> HK["conductor on housekeeping"]
    MD --> TX["sender on own core"]
    MD --> RX["receiver on own core"]
    APP["app on own core"] --> MD

Same layout on host A and host B: housekeeping, app, sender, receiver. Host-local channels still use kernel sockets, so on one host use IPC or kernel UDP.

Aeron settings

What the lab used, and the evidence for each. Figures: md-dpdk/BENCHMARKS.md. Where: pod is the pod spec or the channel URI; node needs node-level access.

One AWS flow topped out at 7.75–7.77 Gbit/s of Aeron frames on c7i.4xlarge, kernel UDP and DPDK alike; on c9g.2xlarge 1,344 B messages stopped at 685,574 msgs/s (BENCHMARKS.md). AWS documents a single-flow limit of 10 Gbit/s inside a cluster placement group (EC2 network bandwidth). An Aeron channel is one flow: spread a heavier stream over several channels on different ports.

Clocks and timestamps

• Clock: the transport times intervals in nanoseconds since boot. By default (RUSTERON_DPDK_CLOCK=counter) it reads the CPU counter inline at the rate DPDK gives it (rte_get_tsc_hz()), or at the rate it measures against CLOCK_MONOTONIC_RAW at start where DPDK’s is more than 100 ppm off: no system call, no call into Rust. It falls back to CLOCK_MONOTONIC when the kernel’s clock source is not the counter or a hypervisor’s view of it, the TSC is not invariant (x86_64), or DPDK’s rate is not CNTFRQ_EL0 (aarch64). RUSTERON_DPDK_CLOCK=monotonic forces it. Counter type 80 and DpdkTransport::clock() say which, and why.
• NIC receive timestamps: RUSTERON_DPDK_RX_HW_TIMESTAMP=1. Only AWS ENA offers them (Gen7+ instances in a precision-time placement group, DPDK 25.11+); Azure MANA does not. They mean something only while chrony keeps the host on the NIC’s clock (time sync).
• Media receive timestamps (media-rcv-ts-offset=reserved on a subscription). One run (Graviton G3, DPDK): p50 within 0.07 µs of without at idle and every steady rate; idle max 153 against 79 µs; at 1,344 B bursts 1 of 5 reps lossy (short sends) against 0. Not separated (BENCHMARKS.md). Repeat before relying on it.
• Your own code: quanta for durations, CLOCK_REALTIME for stamps (clocks).

Best practices

• Measure the kernel path beside DPDK in every run, on the same pair, in random order. Compare numbers from one pair only.
• Count loss, not only latency: NAKs, retransmits, NIC drops and short sends before and after each run. A flat-out rate that loses is a ceiling, not an operating point.
• Fail closed. A DPDK transport that cannot start exits; keep it that way, so a run can never measure kernel UDP by mistake.
• Set ERGON_HOST_CHECKS=require once the host is tuned: the driver then refuses to start while a check warns, so a reboot that undid a setting stops the next start. A NIC reset while it runs is only reported, and under dpdk-ena nothing is re-checked.

Pitfalls