Java for Aeron Archive and Cluster

ergo-aeron-cluster is only a client. The cluster (consensus module, clustered services, Archive) runs on Aeron’s Java, as does an Archive beside a C media driver. This page: which Java, and the JVM and Aeron settings for latency, for Aeron 1.53.3 (the release ergon pins). Each setting comes from Aeron’s own scripts, benchmarks or wiki (sources); none has a latency A/B here.

Terms: deployment glossary, network and DPDK glossary.

Which Java

The eclipse-temurin:25-jre image has neither wget nor curl. Fetch the jar with a checksummed ADD:

FROM eclipse-temurin:25-jre
ADD --checksum=sha256:4c6c8a5014a2b67424051170392a005daa9a998ac963302d2819d0f3db749ef2 --chmod=644 \
    https://repo1.maven.org/maven2/io/aeron/aeron-all/1.53.3/aeron-all-1.53.3.jar /aeron-all.jar

Ready-made settings

Two settings files, both tested by cluster-test-harness/tests/jvm_settings.rs: the JDK takes every option without a warning, every property is one Aeron’s sources name (Aeron ignores a misspelt one), and a cluster node started with both serves a client.

java @aeron.jvm -cp aeron-all-1.53.3.jar \
    io.aeron.cluster.ClusteredMediaDriver low-latency.properties cluster.properties

@aeron.jvm is a java argument file. Archive, ArchivingMediaDriver, ClusteredMediaDriver and ClusteredServiceContainer load every properties file named on their command line, so low-latency.properties and your cluster settings go there (or pass each as -D<name>=<value>).

Adapt two settings in cluster-test-harness/jvm/aeron.jvm: the heap (-Xms2g -Xmx2g) is a placeholder, so size it for your clustered service’s state (Archive alone needs far less); the log goes to /dev/shm, which only Linux has.

# JVM options for Aeron Archive, Cluster and the Java media driver, as a java
# argument file: `java @aeron.jvm -cp aeron-all.jar <main class>`. For Java 17
# and newer; run it on Java 25. cluster-test-harness/tests/jvm_settings.rs
# starts a cluster node with it and round-trips a message through it.
# Sources: Aeron's aeron-samples/scripts/java-common and cluster scripts, and
# the aeron-io/benchmarks scripts/run-java.

# Agrona reaches jdk.internal.misc.Unsafe: without this, IllegalAccessError.
--add-opens java.base/jdk.internal.misc=ALL-UNNAMED
# Archive CRC32 checksums reach into java.util.zip.
--add-opens java.base/java.util.zip=ALL-UNNAMED

# Archive, Cluster and the driver allocate nothing in steady state: the
# collector with the cheapest barriers.
-XX:+UseParallelGC
# A fixed heap committed at start. Size it for the clustered service's state.
-Xms2g
-Xmx2g
-XX:+AlwaysPreTouch
# Room enough that code-cache pressure never deoptimises hot code.
-XX:MaxMetaspaceSize=1g
-XX:ReservedCodeCacheSize=1g

# Every collection and safepoint with its cause, written by a background
# thread to tmpfs. %p is the process id.
-Xlog:async
-Xlog:gc*,safepoint=info,arguments=info:file=/dev/shm/aeron-jvm-%p.log:time,uptime,level,tags:filecount=4,filesize=16m

-Djava.net.preferIPv4Stack=true
# No bounds checks in Agrona's buffers. Read once, when the class loads: it
# must be set here, not by the application.
-Dagrona.disable.bounds.checks=true

cluster-test-harness/jvm/low-latency.properties:

# Aeron Archive and Cluster properties for latency, Aeron 1.53.3. Every spin
# or noop thread keeps a core at 100%: give each one an isolated core, or keep
# Aeron's default backoff. Pass the file as an argument to Archive's,
# ArchivingMediaDriver's or ClusteredMediaDriver's main (they load properties
# files named there), or each line as -D<name>=<value>.
# cluster-test-harness/tests/jvm_settings.rs runs a cluster node with them.

# Archive: a thread each for the conductor, recorder and replayer (the default
# DEDICATED mode), none of which sleeps.
aeron.archive.threading.mode=DEDICATED
aeron.archive.idle.strategy=spin
aeron.archive.recorder.idle.strategy=noop
aeron.archive.replayer.idle.strategy=noop

# Consensus module and clustered service: never sleep.
aeron.cluster.idle.strategy=noop

# Log buffers written out when created and faulted in when mapped, so the
# first write to a page is not a page fault. Costs start-up time and every
# term's memory up front. Aeron's benchmarks keep the defaults (sparse, no
# pre-touch) instead.
aeron.term.buffer.sparse.file=false
aeron.pre.touch.mapped.memory=true

# Thread names short enough for Linux's 15 characters (aeron-ar-rec,
# aeron-ar-rep, aeron-ar-cnd, aeron-cl-cm, aeron-cl-cs-0), so each hot thread
# can be found and pinned by name. The classic default names
# (archive-recorder, consensus-module-0-0, ...) are cut short.
aeron.thread.naming=new

# Count every duty cycle over 1 ms (default 100 ms), to see stalls.
aeron.archive.conductor.cycle.threshold=1ms
aeron.archive.recorder.cycle.threshold=1ms
aeron.archive.replayer.cycle.threshold=1ms
aeron.cluster.cycle.threshold=1ms
aeron.cluster.service.cycle.threshold=1ms

The JVM options

The Aeron properties

These settings are left at their defaults:

• aeron.archive.file.sync.level and aeron.archive.catalog.file.sync.level stay at 0: writes go to the OS page cache only. 1 adds fdatasync and 2 adds fsync. Keep the catalog’s level at least the archive’s. Level 0 survives a process dying and loses whatever is still in the page cache if the machine does. In a 3- or 5-member cluster another member’s log covers that loss, so 0 is the usual choice. A single-member cluster’s quorum is one: its local log is the only copy, so raise both sync levels together there and pay the fsync.
• aeron.archive.file.io.max.length stays at 1m.
• aeron.archive.control.term.buffer.sparse stays at true.
• aeron.spies.simulate.connection stays false. It is a media driver setting, which Aeron’s benchmarks set to true on their driver. It matters when an Archive records a network publication by spying and nothing else subscribes: with false the publication waits for a network subscriber before it is connected.

Pinning the hot threads

Java has no CPU-affinity setting. Aeron’s benchmarks start the JVM on the housekeeping cores with numactl --membind --cpunodebind --physcpubind. They then move each hot thread to an isolated core by name with taskset -p. With aeron.thread.naming=new the names are aeron-ar-cnd, aeron-ar-rec, aeron-ar-rep, aeron-cl-cm and aeron-cl-cs-<service id>:

pid=$(pgrep -f io.aeron.cluster.ClusteredMediaDriver)
for task in /proc/"$pid"/task/*; do
    case $(cat "$task/comm") in
        aeron-cl-cm) taskset -pc 4 "${task##*/}" ;;
        aeron-ar-rec) taskset -pc 5 "${task##*/}" ;;
    esac
done

Every spin or noop thread needs a whole isolated core of its own (see Hosts). On shared cores, keep Aeron’s default backoff.

Small or shared hosts

When the JVM shares its cores, or runs in a container with a few hundred MiB:

• Keep the backoff idle strategies. A spinning thread on a shared core takes the core from everything else.
• Serial GC (-XX:+UseSerialGC) is the one-core collector; its cost against ParallelGC here is not measured.
• Set -Xms equal to -Xmx, and size -XX:MaxDirectMemorySize, -XX:MaxMetaspaceSize and -XX:ReservedCodeCacheSize. The container’s limit must cover all of them, plus thread stacks and native memory.
• Leave aeron.pre.touch.mapped.memory off. Every term a pre-touching client maps counts against the container’s memory limit.

Options to leave out of older guides

• -XX:+UseParallelOldGC: Java 17 and 25 refuse to start with it. Aeron’s wiki still lists it.
• -XX:-UseBiasedLocking: deprecated with a warning on 17. Java 25 refuses to start with it.
• -XX:+ZGenerational: Java 25 ignores it with a warning, and 27 refuses to start. ZGC is now generational only.
• The sun.misc.Unsafe deprecation (JEP 471, JEP 498) and --sun-misc-unsafe-memory-access do not apply. Agrona uses jdk.internal.misc.Unsafe, which is why the first --add-opens is needed.

Optional: measure first

These options are not in Aeron’s sources:

• -XX:+UseCompactObjectHeaders (a product option since Java 25, JEP 519) makes objects smaller.
• -XX:+PerfDisableSharedMem stops the JVM writing its hsperfdata file under /tmp, at the cost that jps and jstat no longer see it.
• Generational ZGC (-XX:+UseZGC) is only worth it for a clustered service that allocates, needs sub-millisecond pauses and has cores to spare.
• Huge pages through the JVM (-XX:+UseTransparentHugePages, Linux only) cover only the heap, which is cold in steady state. Aeron’s hot memory is its mapped log buffers; size those with aeron.file.page.size on hugetlbfs, as Hosts describes.

In ergon’s tests

The cluster harness (ergo-aeron-cluster-test-harness) runs $JAVA_HOME/bin/java, else java on PATH. A java older than 17, or one that will not run, fails the test with its path and the fix. Every Java process it starts reads cluster-test-harness/jvm/test.jvm, which follows Aeron’s own test settings: checks on, nothing spins.

# JVM options of every Java process cluster-test-harness starts, as a java
# argument file (java @test.jvm). After Aeron's own test settings in its
# build.gradle: the checks stay on and nothing spins. aeron.jvm holds the
# production options, which tests/jvm_settings.rs tries.

# Agrona reaches jdk.internal.misc.Unsafe: without this, IllegalAccessError.
--add-opens java.base/jdk.internal.misc=ALL-UNNAMED
# Archive CRC32 checksums reach into java.util.zip.
--add-opens java.base/java.util.zip=ALL-UNNAMED

# A small fixed heap: Aeron keeps its buffers off the heap, and several JVMs
# share the machine.
-Xms256m
-Xmx256m
-XX:+UseParallelGC

# Agrona's bounds checks stay on, and a misaligned atomic access fails rather
# than tearing.
-Dagrona.disable.bounds.checks=false
-Dagrona.strict.alignment.checks=true
-Djava.net.preferIPv4Stack=true

export JAVA_HOME=$(/usr/libexec/java_home -v 25)   # macOS; on Linux, the JDK's directory
just build-aeron-jars
just test-aeron-cluster-harness

Sources

• Aeron 1.53.3: aeron-samples/scripts/java-common, the cluster scripts in aeron-samples/scripts/cluster, low-latency.properties, build.gradle and its CI matrix in .github/workflows/ci.yml.
• aeron-io/benchmarks: scripts/run-java and scripts/aeron/low-latency-*.properties.
• Aeron wiki: Performance Testing.
• Gradle’s Java compatibility matrix.
• JEP 471, JEP 498 and JEP 519.