Introduction

ergo-sbe generates zero-allocation Rust codecs from Simple Binary Encoding (SBE) schemas with official-SBE wire fidelity inside the published profile (compatibility). Wire-order safety is enforced at compile time — calling methods out of schema order is a type error. Every maintained ergon/sbe-tool comparison is gated at a literal 1.00 ceiling under both LTO profiles; quote a result by run id, not by a number copied into this page. See Benchmarks.

Quick start

# Cargo.toml
[build-dependencies]
ergo-sbe = "0.1"

#![allow(unused)]
fn main() {
let mut buf = [0u8; HeartbeatEncoder::compute_length_with_header()];
// Buffer is exact size from const compute_length_with_header — no bounds check needed.
let len = HeartbeatEncoder::try_wrap_and_apply_header(&mut buf, 0)?
    .fixed(&HeartbeatFixedFields { sequence: 7, timestamp: 0 })
    .encoded_length_with_header();
let dec = HeartbeatDecoder::try_from(&buf[..len])?;
assert_eq!(dec.sequence(), 7);
}

Set up → a 3-minute path from zero to working codec. All API surfaces are covered in the Feature Tour → with compilable examples.

Feature overview

ergon is an experimental Rust workspace. ergo-sbe generates the codecs; ergo-aeron-cluster is a client-only Aeron Cluster experiment built on rusteron. Neither crate is production-ready today. APIs may change. Verify wire compatibility, failure handling, and performance for your own schemas. Exit criteria: Road to 1.0.

ergo-sbe parses Simple Binary Encoding (SBE) schemas and generates Rust codecs that match the official SBE wire layout for the features listed in the compatibility profile (header, field layout, groups, var-data, byte order — see docs/SBE_COMPATIBILITY.md).

It is not a line-for-line port of the java/rust sbe-tool stubs. The goals for the generated API are:

1. Easier to use — especially nested groups and var-data under Rust’s borrow checker
2. Safer — wire order and trust boundaries enforced by types / Result
3. Easier to read — nested structure looks like the schema, not a pile of temporary handles

Still built for low-latency official-SBE wire work. The style uses named stage structs (not Encoder<State> generics), closures + method chaining for groups, checked entry points, version-aware accessors, and optional domain/conversion helpers.

Why not parent hopping?

sbe-tool's flyweight API uses .parent() to hand ownership back up the tree. In Rust that pattern hits the borrow checker: move a group encoder in, get stuck returning the parent, lose the thread of the code.

ergo-sbe leans on scoped closures and chaining so nested schemas stay readable and you rarely pass encoder ownership field-to-field by hand:

  // Nested shape mirrors the schema — no .parent() hopscotch.
  enc.fixed(&fields)
      .bids(n, |bids| {
          bids.add(|level| {
              level.price(p).size(s);
              level.orders(m, |ords| {
                  ords.add(|o| { o.order_id(id); Ok(()) })?;
                  Ok(())
              })?;
              Ok(())
          })?;
          Ok(())
      })?
      .asks(0, |_| Ok(()))?
      .symbol(b"EURUSD")?;

Wire parity is exercised three ways: official Java .sbe fixtures, live dual-encode suites that require ergo-sbe and sbe-tool Rust bytes to be identical (sbe_tool_wire_parity_test for deep Car matrices; sbe_tool_multi_schema_wire_parity_test across example/unit schemas with checked-in sbe-tool reference crates under sbe/tests/sbe_tool_reference/), and a maintained benchmark gate versus sbe-tool-generated codecs (see BENCHMARKS.md).

Early release (0.x). Experimental APIs may change. Binary wire compatibility is covered by an automated suite (golden bytes, schema edge cases, parity benches). Pin versions and report production use — real-world feedback is how the experimental banner goes away.

Workspace

The workspace requires Rust 1.88 or newer. The sample crates are intentionally excluded from the Cargo workspace and remain publish = false.

Set up

The repository pins the upstream Aeron and SBE repositories as submodules:

git submodule update --init --recursive

Common local checks:

just policy
just check-products
just test
RUSTDOCFLAGS="-D warnings" cargo doc -p ergo-sbe --all-features --no-deps
RUSTDOCFLAGS="-D warnings" cargo doc -p ergo-aeron-cluster --no-deps

just test is intentionally not a partial/offline green path: it builds and runs the Java Cluster lifecycle/recovery lane and the HA sample. Use just test-all to add Miri and deterministic fuzz replay.

Pull-request CI also enforces the non-decreasing coverage baseline. Scheduled lanes run every fuzz target for ten minutes, Miri fixtures weekly, and critical-path mutation testing weekly. Missing or empty results fail closed.

See the samples README for standalone sample commands and Java harness requirements. Run just --list for the repository's available build, test, interoperability, and benchmark recipes.

Project boundaries

• Official SBE wire compatibility takes priority over API convenience.
• Maintained hot paths are compared with the official SBE generator output.
• Benchmark claims are parity-checked and profile-specific. The corrected suite treats a repeatable sbe-tool win as a blocking benchmark/codegen defect and publishes results with LTO both enabled and disabled.
• Codec microbenchmarking is notoriously easy to get wrong. Benchmark results are explicitly reviewable evidence, not product claims; surprising ratios should be reported and treated as suspected benchmark defects first.
• Checked entry points must report malformed input rather than manufacture default, empty, or lossy values.
• The Cluster crate implements a client, not a consensus module, service container, archive, backup node, or Cluster administration tool.
• Samples, benchmarks, Java harness code, and upstream reference sources are repository support material, not publication targets.

Getting Started

From zero to working codec in under 5 minutes. You'll add ergo-sbe as a build dependency, point it at a schema XML file, include the generated module, and encode/decode your first message.

• Depend on the Generator — one line in Cargo.toml
• Generate in build.rs — point at your schema, get a Rust module
• Include Generated Code — include! or sbe_mod! the output
• Encode and Decode — fixed and variable-length messages
• Method Chaining — write messages in one expression the way the schema reads
• Multi-Schema Patterns — share types across schemas and versions

Depend on the Generator

Minimal product path — codegen only; generated codecs embed their own sbe_rt and do not link ergo-sbe into the application. Schema parse errors render a source snippet (line + span) by default:

[build-dependencies]
ergo-sbe = "0.1"
# no [dependencies] ergo-sbe

Convenience path — also pull ergo-sbe as a normal dependency when you use sbe_mod! / include_sbe! (macros expand in the app crate):

[build-dependencies]
ergo-sbe = "0.1"

[dependencies]
ergo-sbe = "0.1"   # only needed for sbe_mod! / include_sbe!

See Samples for monorepo crates that use each pattern.

Generate in build.rs

Two APIs — pick whichever fits your project:

generate_to_dir writes generated codecs to a directory you control (prefer src/generated/ for IDE go-to-definition). The feature-tour sample uses this pattern:

#![allow(unused)]
fn main() {
    let generated_dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src/generated");
    let config = ergo_sbe::GenerationConfig::new("feature_tour")
        .with_domain_objects(ergo_sbe::DomainVarData::Strings)
        .with_domain_type(
            ergo_sbe::ConversionSelector::named_type("BooleanType"),
            "bool")
        .with_domain_type(
            ergo_sbe::ConversionSelector::semantic_type("UTCTimestamp"),
            "chrono::DateTime<chrono::Utc>")
        .with_conversion(ergo_sbe::ConversionSelector::named_type("Decimal"))
        // Same shape as Decimal, but the app supplies the impl itself — see
        // demo_domain_type_manual_impl in src/lib.rs.
        .with_manual_domain_type(
            ergo_sbe::ConversionSelector::named_type("ManualDecimal"),
            "rust_decimal::Decimal");

    ergo_sbe::generate_to_dir("schemas/feature-tour.xml", config, &generated_dir)?;
}

generate_to_out_dir writes to Cargo's $OUT_DIR and emits cargo::rerun-if-changed automatically. Use with include!(concat!(env!("OUT_DIR"), …)) — simpler, but rust-analyzer usually cannot jump into generated code.

Schema from a string / include_str!: generate_str_to_out_dir.

Need multi-schema or custom output paths? Use the lower-level parse_file + Generator API (same steps the helper runs). For shared types across schemas, see Multi-Schema Patterns below.

Include Generated Code

Prefer build-dep only for product crates (no runtime ergo-sbe link). Generated codecs embed sbe_rt; plain include! is enough:

#[path = "generated/feature_tour.rs"]
pub mod feature_tour;
pub use feature_tour::*;

(From sbe-feature-tour/src/lib.rs — the #[path] pattern used by every sample crate. The feature-tour's cargo build verifies this compiles.)

or via include!:

mod messages {
    include!(concat!(env!("OUT_DIR"), "/messages.rs"));
}
use messages::*;

Optional convenience — sbe_mod! needs ergo-sbe as a normal dependency (macro expansion only; not required for encode/decode):

// Cargo.toml: [dependencies] ergo-sbe = "0.1"
ergo_sbe::sbe_mod!(messages);
use messages::*;
// Or only the include: ergo_sbe::include_sbe!("messages");

See Samples for which crates use which pattern.

Encode and Decode

Two styles — pick whichever fits:

fixed() struct (fill every field at once — compile error if a field is missing):

#![allow(unused)]
fn main() {
pub fn demo_fixed_heartbeat() -> Result<Vec<u8>, Box<dyn std::error::Error>> {
    // Const length → stack array (no heap).
    let mut buf = [0u8; HeartbeatEncoder::compute_length_with_header()];
    let nanos: i64 = 1_720_000_000_000_000_000;
    // Buffer pre-sized via const compute_length_with_header; try_* still validates extent.
    let written = HeartbeatEncoder::try_wrap_and_apply_header(&mut buf, 0)
        .unwrap()
        .fixed(&HeartbeatFixedFields {
            sequence: 7,
            timestamp: nanos as u64,
        })
        .encoded_length_with_header();

    let dec = HeartbeatDecoder::try_decode(&buf[..written], 0)?;
    assert_eq!(dec.sequence(), 7);
    let decoded_ts: DateTime<Utc> = dec.try_timestamp()?;
    assert_eq!(decoded_ts.timestamp_nanos_opt(), Some(nanos));
    Ok(buf[..written].to_vec())
}
}

(This code comes from the sbe-feature-tour sample crate.)

{Msg}FixedFields has no Default. Every required scalar must appear in the struct literal (optional fields use Option only when schema presence="optional"). That is intentional: zero-filling required IDs hides bugs. For large messages, build the literal next to the encode call in wire order.

On a fixed-only message (no groups or var-data), as_bytes_with_header, as_body_bytes, encoded_length*, and into_remaining_mut exist only after fixed(&FixedFields). Calling them on the value returned by wrap* is a type error — that is what stops a reused buffer from publishing leftover body bytes or packing the next message over a stale body.

Individual setters stay on the unfixed encoder after wrap* and also on raw_fixed() (body-relative offsets). Prefer fixed(&FixedFields) when you have every field; use raw_fixed() when you want a dedicated writer:

#![allow(unused)]
fn main() {
// Dedicated raw writer (setters also exist on the unfixed encoder).
// `as_bytes_with_header` is only on FieldsFixed after `fixed(&FixedFields)`.
let mut buf = [0u8; HeartbeatEncoder::compute_length_with_header()];
let mut w = HeartbeatEncoder::try_wrap_and_apply_header(&mut buf, 0)?
    .raw_fixed();
w.sequence(7);
w.timestamp_wire(0);
let dec = HeartbeatDecoder::try_from(&buf[..HeartbeatEncoder::ENCODED_LENGTH])?;
assert_eq!(dec.sequence(), 7);
}

(From book/examples/heartbeat-encode.rs — compiled against the feature-tour codec.)

raw_fixed() writes into the buffer you already sized. It does not mark the message complete: omitted required setters leave stale bytes, and as_bytes_with_header / encoded_length* / into_remaining_mut stay locked. Set every required field, then call fixed(&FixedFields) for the complete-message views. Slicing &buf[..ENCODED_LENGTH] yourself is possible but is not the completeness-checked path.

Optional fields and apply_nulls

try_wrap_and_apply_header / wrap_and_apply_header write the message header only — they do not fill optional fixed fields with schema null sentinels (sbe-tool parity). Unwritten optional bytes retain whatever was already in the buffer (often zero, sometimes stale).

fixed() closes that gap for you. Optional fields are Option<T> in the generated FixedFields struct, and fixed() writes the schema null wire image for every None — including fixed arrays and nested optional composite members. Since fixed() is the only route to a message's tails and to fixed-only complete byte views, the ordinary path never leaves a stale optional behind:

// `price` is optional; None writes the schema null image, not stale bytes.
// (Illustrative — use your schema's message name and optional field.)
let len = OrderEncoder::wrap_and_apply_header(&mut buf, 0)
    .fixed(&OrderFixedFields { symbol: *b"IBM     ", price: None })
    .encoded_length_with_header();

apply_nulls() remains on the unfixed encoder after wrap*, for the case where you set individual optional fields yourself and no FixedFields value describes which optionals are unset.

See Why NullVal Instead of Option.

Character arrays: fixed-width char fields become [u8; N]. Pass a shorter &str via the _str setter — auto-padded with NULs. On decode, copy_* copies the raw bytes into your buffer, or read the slice with vehicle_code():

#![allow(unused)]
fn main() {
let complete_len = CarEncoder::compute_length()
    .fuel_figures_ragged(0, |_| Ok(()))?
    .performance_figures_ragged(0, |_| Ok(()))?
    .manufacturer(5)?
    .model(5)?
    .activation_code(3)?
    .encoded_length_with_header();
const PAD: usize = 256;
assert!(complete_len <= PAD, "car length {complete_len} exceeds pad {PAD}");
let mut storage = [0u8; PAD];
let buf = &mut storage[..complete_len];
let fields = CarFixedFields {
    serial_number: 1234,
    model_year: 2013,
    available: true.into(),
    code: Model::A,
    some_numbers: [10, 20, 30, 40],
    vehicle_code: *b"ABCDEF",
    extras: OptionalExtras::default(),
    engine: Engine::new(2000, 4, *b"123", 0i8, false.into(), Booster::new(BoostType::TURBO, 210)),
};
let n = CarEncoder::try_wrap_and_apply_header(buf, 0)?
    .fixed(&fields)
    .fuel_figures(0, |_| Ok(()))?
    .performance_figures(0, |_| Ok(()))?
    .manufacturer(b"Honda")?
    .model(b"Civic")?
    .activation_code(b"abc")?
    .encoded_length_with_header();
assert_eq!(n, complete_len);

let car = CarDecoder::try_from(&buf[..n])?;
let mut dst = [0u8; 6];
assert_eq!(car.copy_vehicle_code(&mut dst), 6);
assert_eq!(&dst, b"ABCDEF");
assert_eq!(car.vehicle_code(), *b"ABCDEF");
}

(From book/examples/fixed-char-arrays.rs — compiled against the feature-tour codec.)

The Car encoder's fixed fields include vehicle_code: [u8; 6] (schema char array) and some_numbers: [u32; 4]. See the feature tour for the complete Car example with groups and var-data.

Start here for a full runnable map of features: sbe-feature-tour (cargo run --manifest-path samples/sbe-feature-tour/Cargo.toml).
More recipes: Recipes.

Method Chaining

ergo-sbe encoders are designed so the entire encode reads as one expression, from wrap_and_apply_header through .fixed(...) and every dynamic tail, ending in .encoded_length_with_header(). Bind only the resulting length; do not retain intermediate encoder variables.

Prefer (one chain, one let):

#![allow(unused)]
fn main() {
let mut buf = [0u8; HeartbeatEncoder::compute_length_with_header()];
// Buffer is exact size from const compute_length_with_header — no bounds check needed.
let len = HeartbeatEncoder::try_wrap_and_apply_header(&mut buf, 0)?
    .fixed(&HeartbeatFixedFields { sequence: 7, timestamp: 0 })
    .encoded_length_with_header();
let dec = HeartbeatDecoder::try_from(&buf[..len])?;
assert_eq!(dec.sequence(), 7);
}

(From book/examples/heartbeat-encode.rs — compiled against the feature-tour codec.)

Staged chaining vs fixed-only

For a fixed-only message like Heartbeat, wrap* returns HeartbeatEncoder<'_, H, FieldsUnfixed>. .fixed(&HeartbeatFixedFields { … }) consumes that value and returns FieldsFixed, which is the only phase that exposes as_bytes_with_header / as_body_bytes / encoded_length* / into_remaining_mut. Individual field setters stay on the unfixed phase and on raw_fixed(); they are not on that complete view.

Avoid (interrupted chain, rebinding):

// Each `let` breaks the chain and splays the pipeline across the screen.
// The `.unwrap()` calls are a code smell — the fallible chain should use `?`.
let enc = CarEncoder::wrap_and_apply_header(&mut buf, 0).fixed(&fields);
let enc = enc.fuel_figures(2, |g| { ... }).unwrap();
let enc = enc.manufacturer(b"Honda").unwrap();
let len = enc.encoded_length_with_header();

Every encoder stage is chainable — fixed() and each tail method return the next stage (or Result<NextStage, _>) and compose with ? in the same expression. Intermediate encoder rebinding and manual .unwrap() defeat this design.

For the full Car example with groups and var-data, see the feature tour page.

Multi-Schema Patterns

SBE schemas often share types (messageHeader, groupSizeEncoding, composites, enums, sets). ergo-sbe supports two approaches:

The <include> path is what the SBE spec expects. The shared-Ir path is a convenience for tooling, build scripts, and any workflow where you already have the shared schema parsed in memory.

Shared Ir — parse, then share

// 1. Parse the shared schema once (composites, enums, sets).
let common = ergo_sbe::parse_file("schemas/common-types.xml")?;

// 2. Parse a consumer schema — no <include> needed.
let orders = ergo_sbe::parse_file_with_shared("schemas/orders.xml", &common)?;

// 3. Each schema gets its own module.
let generator = ergo_sbe::Generator::new(
    ergo_sbe::GenerationConfig::new("common_types").with_shared_module("common_types"),
);
let modules = generator.generate_multi(&[
    (&ergo_sbe::Schema::from_ir(common), "common_types"),
    (&ergo_sbe::Schema::from_ir(orders), "orders"),
])?;

With with_shared_module("common_types"), the first entry owns the shared enums/sets/composites; later entries pub use super::common_types::* and skip duplicate type generation.

Module names must be unique, non-empty Rust identifiers (no path separators or keywords). Before any file is written, shared types with the same name are compared by a canonical wire fingerprint (token order, primitive encodings, offsets, presence, null/min/max, discriminants/choices, sinceVersion, and schema byte order). A name collision with a different fingerprint fails generation with GenerateError::IncompatibleSharedType rather than silently reusing the first schema's layout.

parse_with_shared from in-memory strings

let common = ergo_sbe::parse(
    r#"<?xml version="1.0"?>
<messageSchema package="common" id="0" version="1" byteOrder="littleEndian">
  <types>
    <composite name="messageHeader">
      <type name="blockLength" primitiveType="uint16"/>
      <type name="templateId" primitiveType="uint16"/>
      <type name="schemaId" primitiveType="uint16"/>
      <type name="version" primitiveType="uint16"/>
    </composite>
    <composite name="Price">
      <type name="mantissa" primitiveType="int64"/>
      <type name="exponent" primitiveType="int8"/>
    </composite>
  </types>
</messageSchema>"#,
)?;

// No <types> / <include> — Price resolves from `common`.
let orders = ergo_sbe::parse_with_shared(
    r#"<?xml version="1.0"?>
<messageSchema package="orders" id="1" version="1" byteOrder="littleEndian"
               headerType="messageHeader">
  <message name="NewOrder" id="1">
    <field name="price" id="1" type="Price"/>
  </message>
</messageSchema>"#,
    &common,
)?;

The shared Ir path does not recover bare top-level <type> typedefs (those are inlined during parsing and dropped from the token stream). Reference them through a <composite> / <enum> / <set> in the shared schema instead.

Full build.rs — one helper call

Module names are supplied, not derived from file stems, so a hyphenated common-types.xml can emit common_types.rs. The helper parses the shared schema first, resolves consumers against it, validates the whole set, then writes. A late consumer failure leaves no partial files. Cargo watches every root and every resolved include.

// build.rs
use std::path::Path;
use ergo_sbe::{GenerationConfig, SchemaFile, generate_multi_to_out_dir};

fn main() -> ergo_sbe::miette::Result<()> {
    generate_multi_to_out_dir(
        SchemaFile::new(Path::new("schemas/common-types.xml"), "common_types"),
        &[
            SchemaFile::new(Path::new("schemas/orders.xml"), "orders"),
            SchemaFile::new(Path::new("schemas/fills.xml"), "fills"),
        ],
        GenerationConfig::new("common_types"),
    )?;
    Ok(())
}

Consumer modules import the shared module for cross-schema type resolution:

mod common_types { include!(concat!(env!("OUT_DIR"), "/common_types.rs")); }
mod orders {
    use super::common_types::*;       // shared types + header composite
    include!(concat!(env!("OUT_DIR"), "/orders.rs"));
}
mod fills {
    use super::common_types::*;
    include!(concat!(env!("OUT_DIR"), "/fills.rs"));
}

See also: sbe-codegen-examples (reusable generator setup), multi_schema_versioning_test (versioned schemas with shared types), exchange-example (multi-schema exchange feed with IPC).

Coming from sbe-tool

Side-by-side mapping for teams migrating from the official Simple Binary Encoding Rust generator (sbe-tool). Within the [published SBE profile] (../design-notes/feature-matrix.md) and docs/SBE_COMPATIBILITY.md, ergo-sbe aims for official-SBE wire fidelity with sbe-tool; the API shape is intentionally different. Do not read this as unqualified “binary compatible with every SBE feature.”

The #1 trap: wrap offset

// Frame at buf[0..]:
// sbe-tool:  enc.wrap(buf, 8)              // body starts at 8
// ergo-sbe:  Enc::wrap(buf, 0)             // message starts at 0
//            Enc::wrap_and_apply_header(buf, 0)

Passing sbe-tool’s 8 into ergo-sbe for a frame at zero mis-aligns every field. Generated rustdoc on wrap / wrap_and_apply_header / decode repeats this callout.

Header modes (fair comparison table)

Use the same logical work on both sides when comparing or porting:

• ergon wrap = message start; sbe-tool wrap = body offset.
• sbe-tool encoded_length() is body only. ergon encoded_length_with_header() includes the header. Never invent 8 + encoded_length() to “prove” a header was written if .header(0) was not called on the sbe-tool arm.

Full fairness rules for benchmarks: Benchmarks methodology.

Groups: .parent() hopscotch vs closures

#![allow(unused)]
fn main() {
pub fn encode_sample_car(buf: &mut [u8]) -> Result<usize, sbe_rt::EncodeError> {
    let mut extras = OptionalExtras::default();
    extras.cruise_control(true).sports_pack(true);

    // Buffer pre-sized from EncodedLength; try_* still validates extent.
    let len = CarEncoder::try_wrap_and_apply_header(buf, 0)
        .unwrap()
        .fixed(&CarFixedFields {
            serial_number: 1234,
            model_year: 2013,
            available: true.into(),
            code: Model::A,
            some_numbers: [10, 20, 30, 40],
            vehicle_code: [b'A', b'B', b'C', b'D', b'E', b'F'],
            extras,
            engine: Engine::new(
                2000,
                4,
                [b'1', b'2', b'3'],
                0i8,
                false.into(),
                Booster::new(BoostType::TURBO, 210),
            ),
        })
        .fuel_figures(2, |g| {
            g.add(|mut e| {
                e.speed(30).mpg(35.9);
                e.usage_description(b"Urban")
            })?;
            g.add(|mut e| {
                e.speed(60).mpg(25.0);
                e.usage_description(b"Highway")
            })?;
            Ok(())
        })?
        .performance_figures(1, |g| {
            g.add(|mut e| {
                e.octane_rating(95);
                e.acceleration(2, |a| {
                    a.add(|x| {
                        x.mph(30).seconds(4.0);
                        Ok(())
                    })?;
                    a.add(|x| {
                        x.mph(60).seconds(7.5);
                        Ok(())
                    })
                })
            })?;
            Ok(())
        })?
        .manufacturer(b"Honda")?
        .model(b"Civic VTi")?
        .activation_code(b"abcdef")?
        .encoded_length_with_header();

    Ok(len)
}
}

Compile-time order: you cannot call asks before bids — the stage type has no asks method. See Wire order via named stages.

Length APIs

Do not default to vec![0u8; 4096] or Vec::with_capacity(MAX) then truncate. See Buffer sizing and Exact sizing.

Decode entry

Both ecosystems typically wrap decoders at the body for direct field access after the header is known. ergon’s entry points take message start (not sbe-tool’s body offset):

See Trust Boundary.

Version handling

Decoders are version-aware: tail offsets use the wire acting block length, not only the compiled block length. Optional / sinceVersion fields follow schema presence rules. Prefer explicit try_* entry points when reading mixed-version streams.

What has no direct ergon equivalent (and why)

Trust boundary

try_* returns Result on short buffers (and identity mismatches). Bare wrap panics if short. Bare decode is a hybrid: panics if short, but still returns Err on wrong template/schema. decode_unchecked is unchecked extent + checked identity. After a safe constructor succeeds, fixed-field accessors are branch-free. Full detail: Trust Boundary.

Further reading

• Type-state is zero-cost
• API freeze decisions
• Feature matrix
• Sample: sbe-feature-tour

Error Diagnostics

Schema errors use miette for pinpointed diagnostics. The generator shows what went wrong, where in the XML, and a unique error code you can match on programmatically.

Invalid type reference

Referencing a type that doesn't exist:

<field name="badField" id="10" type="NonExistentType"/>

ergo_sbe::schema_parse::invalid

  × invalid type for field 'badField': NonExistentType
    ╭─[schema.xml:15:9]
 14 │         <!-- NonExistentType not defined -->
 15 │         <field name="badField" id="10" type="NonExistentType"/>
    ·         ───────────────────────────┬───────────────────────────
    ·                                    ╰── invalid here
 16 │     </message>
    ╰────

The error code ergo_sbe::schema_parse::invalid identifies the variant. The span points to the exact attribute. The source line and surrounding context are rendered automatically.

Missing required attribute

Omitting name on a <field>:

ergo_sbe::schema_parse::missing

  × missing field @name
    ╭─[schema.xml:15:9]
 14 │     <message name="TestMessage" id="1">
 15 │         <field id="10" type="uint8"/>
    ·         ──────────────┬──────────────
    ·                       ╰── missing here
 16 │     </message>
    ╰────

Duplicate template ID

Two messages sharing the same id:

ergo_sbe::schema_parse::resolve

  × resolution error: duplicate template id 1 for message
  │ AnotherMessageWithId1
  ╰─▶ duplicate template id 1 for message AnotherMessageWithId1

Invalid enum encoding type

ergo_sbe::schema_parse::invalid

  × invalid enum encodingType: NonExistentEncodingType
    ╭─[schema.xml:13:9]
 12 │             <!-- encodingType references non-existent type -->
 13 │ ╭─▶         <enum name="BadEnum" encodingType="NonExistentEncodingType">
 14 │ │               <validValue name="Value1">1</validValue>
 15 │ ├─▶         </enum>
    · ╰──── invalid here
 16 │         </types>
    ╰────

Multi-line spans show the full element, with the label pointing to the offending attribute.

Use in build scripts

ParseError implements miette::Diagnostic. Wrap it in miette::Report to render the full diagnostic with source context:

use ergo_sbe::parse_file;

match parse_file("my-schema.xml") {
    Ok(_) => { /* regenerate codec */ }
    Err(e) => {
        let report = miette::Report::new(e);
        eprintln!("{report:?}");
        std::process::exit(1);
    }
}

For programmatic handling, match on the variant directly — ParseError is a plain enum, no downcast needed. Keep a wildcard so new variants are not a compile break:

#![allow(unused)]
fn main() {
use ergo_sbe::{parse_file, ParseError};

match parse_file("my-schema.xml") {
    Ok(_) => {}
    Err(ParseError::MalformedXml { message, .. }) => {
        eprintln!("malformed XML: {message}");
    }
    Err(ParseError::Missing { what, .. }) => {
        eprintln!("missing {what}");
    }
    Err(ParseError::Invalid { what, value, .. }) => {
        eprintln!("invalid {what}: {value}");
    }
    Err(ParseError::Resolve { error, .. }) => {
        eprintln!("resolve: {error}");
    }
    Err(ParseError::Io { path, source, .. }) => {
        eprintln!("cannot read {}: {source}", path.display());
    }
    Err(ParseError::Include { href, cause, .. }) => {
        eprintln!("include {href}: {cause}");
    }
    // Forward-compatible: ParseError is #[non_exhaustive].
    Err(other) => {
        eprintln!("{other}");
    }
}
}

Out-of-range null / min / max

A present nullValue, minValue, or maxValue is parsed fail-closed against the declared primitive width. nullValue="256" on uint8 is rejected (it is not a valid one-byte sentinel). Signed types accept the type's full range (int8 -1 is fine; int8 128 is not).

ergo_sbe::schema_parse::invalid

  × invalid nullValue: '256' is out of range for UInt8

Without that check the generator used to emit 256_u64 as u8, so Some(0) and None collided on the wire.

Error variants

ParseError is #[non_exhaustive]. Match with a wildcard. Current variants:

Every diagnostic variant except Io carries source_code and an optional span. Include highlights the <include> element when the include was parsed from a document.

Migration from 0.1.x: IncludeError { message } is now Include { href, attempted, cause, .. }. Root read_to_string failures are Io, not MalformedXml.

Feature Tour

Runnable examples on these pages pull live source from the sbe-feature-tour sample crate (or a book/examples/ fragment compiled against the same codec). Those includes are compiled by docs_validation_test. Schematics that cannot run in the harness stay rust,ignore — see What Generated Code Looks Like.

• Exact Sizing — compute_length_with_header gives the byte count before you encode
• Bulk Arrays — bulk_add(&[Entry]) for fixed-stride leaf groups
• Consuming Decode Stages — walk groups and var-data in wire order
• What Generated Code Looks Like — stages, metadata, placement
• Trust Boundaries — try_from validates; wrap trusts — explicit in the types
• Domain Objects (DTOs) — owned, serialisable snapshots (never on the hot path)
• Multi-Template Dispatch — AnyMessage routes by template ID at decode time

Exact Sizing

Dynamic messages expose schema-aware size APIs. Flat shapes get a direct checked helper; nested or ragged shapes get a staged *EncodedLength builder. Allocate or claim exactly that many bytes, then write groups and var-data in wire order:

#![allow(unused)]
fn main() {
pub fn demo_car_size_and_encode() -> Result<Vec<u8>, Box<dyn std::error::Error>> {
    // Fuel: 2 entries with usage ASCII lengths 5 and 7.
    // Performance: 1 entry with 2 nested acceleration rows (fixed-only entries).
    // Message var-data: manufacturer / model / activationCode lengths.
    let complete_len = CarEncoder::compute_length()
        .fuel_figures_ragged(2, |ff| {
            ff.add()?.usage_description(5)?; // "Urban"
            ff.add()?.usage_description(7)?; // "Highway"
            Ok(())
        })?
        .performance_figures_ragged(1, |pf| {
            pf.add()?.acceleration(|acc| {
                acc.uniform(2)?;
                Ok(())
            })?;
            Ok(())
        })?
        .manufacturer(5)? // "Honda"
        .model(9)? // "Civic VTi"
        .activation_code(6)? // "abcdef"
        .encoded_length_with_header();

    // Exact size from compute_length → stack pad (this demo fits well under 512).
    const CAR_PAD: usize = 512;
    assert!(
        complete_len <= CAR_PAD,
        "sample car length {complete_len} exceeds stack pad {CAR_PAD}"
    );
    let mut storage = [0u8; CAR_PAD];
    let written = encode_sample_car(&mut storage[..complete_len])?;
    assert_eq!(
        written, complete_len,
        "CarEncodedLength must equal encoder-produced length"
    );
    Ok(storage[..written].to_vec())
}
/// Encode the canonical sample car into `buf` (must be pre-sized).
pub fn encode_sample_car(buf: &mut [u8]) -> Result<usize, sbe_rt::EncodeError> {
    let mut extras = OptionalExtras::default();
    extras.cruise_control(true).sports_pack(true);

    // Buffer pre-sized from EncodedLength; try_* still validates extent.
    let len = CarEncoder::try_wrap_and_apply_header(buf, 0)
        .unwrap()
        .fixed(&CarFixedFields {
            serial_number: 1234,
            model_year: 2013,
            available: true.into(),
            code: Model::A,
            some_numbers: [10, 20, 30, 40],
            vehicle_code: [b'A', b'B', b'C', b'D', b'E', b'F'],
            extras,
            engine: Engine::new(
                2000,
                4,
                [b'1', b'2', b'3'],
                0i8,
                false.into(),
                Booster::new(BoostType::TURBO, 210),
            ),
        })
        .fuel_figures(2, |g| {
            g.add(|mut e| {
                e.speed(30).mpg(35.9);
                e.usage_description(b"Urban")
            })?;
            g.add(|mut e| {
                e.speed(60).mpg(25.0);
                e.usage_description(b"Highway")
            })?;
            Ok(())
        })?
        .performance_figures(1, |g| {
            g.add(|mut e| {
                e.octane_rating(95);
                e.acceleration(2, |a| {
                    a.add(|x| {
                        x.mph(30).seconds(4.0);
                        Ok(())
                    })?;
                    a.add(|x| {
                        x.mph(60).seconds(7.5);
                        Ok(())
                    })
                })
            })?;
            Ok(())
        })?
        .manufacturer(b"Honda")?
        .model(b"Civic VTi")?
        .activation_code(b"abcdef")?
        .encoded_length_with_header();

    Ok(len)
}
}

(This code comes from the sbe-feature-tour sample crate.)

Bulk Arrays

For repeating groups with fixed-size entries (no var-data, no nested groups), generated encoders offer a bulk_add(&[Entry]) path that validates the destination region once and writes every entry.

Car fuelFigures has var-data (usageDescription), so it is not eligible. The nested acceleration group is — each row is mph + seconds only:

#![allow(unused)]
fn main() {
pub fn demo_bulk_add() -> Result<Vec<u8>, Box<dyn std::error::Error>> {
    let mut extras = OptionalExtras::default();
    extras.cruise_control(true);
    let rows = [
        CarPerformanceFiguresAccelerationEntry {
            mph: 30,
            seconds: 4.0,
        },
        CarPerformanceFiguresAccelerationEntry {
            mph: 60,
            seconds: 7.5,
        },
    ];
    let complete_len = CarEncoder::compute_length()
        .fuel_figures_ragged(0, |_| Ok(()))?
        .performance_figures_ragged(1, |pf| {
            pf.add()?.acceleration(|acc| {
                acc.uniform(2)?;
                Ok(())
            })?;
            Ok(())
        })?
        .manufacturer(5)?
        .model(5)?
        .activation_code(3)?
        .encoded_length_with_header();
    const PAD: usize = 256;
    assert!(
        complete_len <= PAD,
        "bulk-add car length {complete_len} exceeds pad {PAD}"
    );
    let mut storage = [0u8; PAD];
    let buf = &mut storage[..complete_len];
    let len = CarEncoder::try_wrap_and_apply_header(buf, 0)?
        .fixed(&CarFixedFields {
            serial_number: 1234,
            model_year: 2013,
            available: true.into(),
            code: Model::A,
            some_numbers: [10, 20, 30, 40],
            vehicle_code: *b"ABCDEF",
            extras,
            engine: Engine::new(
                2000,
                4,
                *b"123",
                0i8,
                false.into(),
                Booster::new(BoostType::TURBO, 210),
            ),
        })
        .fuel_figures(0, |_| Ok(()))?
        .performance_figures(1, |g| {
            g.add(|mut e| {
                e.octane_rating(95);
                e.acceleration(2, |a| {
                    a.bulk_add(&rows)?;
                    Ok(())
                })
            })?;
            Ok(())
        })?
        .manufacturer(b"Honda")?
        .model(b"Civic")?
        .activation_code(b"abc")?
        .encoded_length_with_header();
    assert_eq!(len, complete_len);
    Ok(buf[..len].to_vec())
}
}

(From samples/sbe-feature-tour — compiled and run in that crate's tests.)

bids / asks on the l3-book schema have a nested orders group, so those outer groups are not eligible either. Only a leaf group whose entries are a pure fixed block gets bulk_add.

Constants and MetaAttribute expose schema metadata on every generated type (HeartbeatDecoder::sequence_meta_attribute(MetaAttribute::Presence) and friends). See the generated module after cargo build of the feature-tour sample.

Consuming Decode Stages

Groups and var-data are consumed in schema order. finish() hands the next named stage back to you:

#![allow(unused)]
fn main() {
pub fn demo_car_decode_stages(wire: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
    let car = CarDecoder::try_decode(wire, 0)?;
    assert_eq!(car.serial_number(), 1234);
    assert_eq!(car.model_year(), 2013);
    // Domain conversion: BooleanType → bool when configured.
    let available: bool = car.try_available()?;
    assert!(available);
    assert_eq!(car.code(), Model::A);
    assert_eq!(car.discounted_model(), Model::C); // constant field
    assert_eq!(car.engine().capacity(), 2000);

    // Consuming stages enforce fuelFigures → performanceFigures → strings.
    let mut fuel = car.into_fuel_figures()?;
    let mut speeds = Vec::new();
    for entry in &mut fuel {
        speeds.push(entry?.speed());
    }
    assert_eq!(speeds, vec![30, 60]);

    let decoder = fuel.finish()?;
    let mut decoder = decoder.into_performance_figures()?;
    let mut octanes = Vec::new();
    for entry in &mut decoder {
        let e = entry?;
        octanes.push(e.octane_rating());
        let mut acc = e.into_acceleration()?;
        let mut mphs = Vec::new();
        for a in &mut acc {
            mphs.push(a.mph());
        }
        assert_eq!(mphs, vec![30, 60]);
        let _ = acc.finish()?;
    }
    assert_eq!(octanes, vec![95]);

    let decoder = decoder.finish()?;
    let (mfr, decoder) = decoder.into_manufacturer_as_str()?;
    let (model, decoder) = decoder.into_model_as_str()?;
    let (code, _decoder) = decoder.into_activation_code_as_str()?;
    // All three &str coexist — each borrows 'a from the original wire buffer.
    assert_eq!((mfr, model, code), ("Honda", "Civic VTi", "abcdef"));
    Ok(())
}
}

Each into_*_as_str() returns (&'a str, NextStage<'a>) — the &str borrows from the original wire buffer, not from the consumed stage. All three strings remain valid simultaneously while the stage chain advances.

(This code comes from the sbe-feature-tour sample crate.)

#[must_use] on stages

Consuming stages (CarDecoderAfterFuelFigures, …AfterManufacturer, CarDecoderComplete, …) are #[must_use]. Dropping a stage without into_* / finish / skip_remaining silently skips remaining wire tails (groups and var-data). That is easy to miss when a function returns early — prefer advancing until Complete or an explicit skip.

finish vs skip_remaining

Use skip_remaining when you want the intent obvious in review; both move the tail cursor in wire order.

Full-frame bytes mid-walk

See Generated code for the metadata limit vs full-frame table.

What Generated Code Looks Like

For a schema with one message (Car), ergo-sbe emits a single Rust module with:

Decoder (flyweight over &[u8])

// Each message gets a zero-allocation decoder.
pub struct CarDecoder<'a> {
    pub(crate) buf: &'a [u8],
    pub(crate) offset: usize,
    pub(crate) acting_version: u16,
    pub(crate) acting_block_length: usize,
}

impl<'a> CarDecoder<'a> {
    // Illustrative only — real values from your schema.
    pub const SCHEMA_ID: u16 = 1;
    pub const TEMPLATE_ID: u16 = 1;
    pub const BLOCK_LENGTH: usize = 45;
    pub const HEADER_LENGTH: usize = 8;

    // Checked framed entry (message start). Validates header + fixed extent.
    pub fn try_decode(buf: &'a [u8], pos: usize)
        -> Result<Self, sbe_rt::DecodeError> { ... }

    // Proves the header+fixed-body extent and panics if short.
    // Caller does not need `unsafe` — the proof is in the constructor.
    pub fn wrap(buf: &'a [u8], message_offset: usize,
               acting_block_length: usize, acting_version: u16)
        -> Self { ... }

    // Full dynamic-tail structural check (associated, not `car.verify()`).
    pub fn verify(buf: &[u8]) -> Result<(), sbe_rt::VerifyError> { ... }

    // Fixed fields are random-access — zero-copy reads after a checked wrap.
    #[inline]
    pub fn serial_number(&self) -> u64 {
        let offset = self.offset + 0;
        u64::from_le_bytes(/* private read after extent proof */)
    }
}

Encoder (type-state stages)

// Wire order is enforced by named stage types. The root encoder also carries a
// fields-state parameter: tails are reachable only after `fixed()`.
pub struct CarEncoder<
    'a,
    H: sbe_rt::HeaderState = sbe_rt::HeaderPresent,
    F: sbe_rt::FieldsState = sbe_rt::FieldsUnfixed,
> { ... }
pub struct CarAfterFuelFigures<'a, H: sbe_rt::HeaderState = sbe_rt::HeaderPresent> { ... }
pub struct CarAfterPerformanceFigures<'a, H: sbe_rt::HeaderState = sbe_rt::HeaderPresent> { ... }
pub struct CarComplete<'a, H: sbe_rt::HeaderState = sbe_rt::HeaderPresent> { ... }

// `fixed()` moves FieldsUnfixed -> FieldsFixed; the tail methods exist only on
// the fixed phase, so `wrap(...).fuel_figures(...)` is a compile error.
pub type CarUnfixedEncoder<'a, H = sbe_rt::HeaderPresent> =
    CarEncoder<'a, H, sbe_rt::FieldsUnfixed>;

impl<'a, H: sbe_rt::HeaderState> CarEncoder<'a, H, sbe_rt::FieldsUnfixed> {
    pub fn fixed(self, fields: &CarFixedFields) -> CarEncoder<'a, H, sbe_rt::FieldsFixed> { ... }
}

// Calling stages out of order is a type error — `CarEncoder` has no `asks()`.
impl<'a, H: sbe_rt::HeaderState> CarEncoder<'a, H, sbe_rt::FieldsFixed> {
    pub fn fuel_figures(self, count: u16, f: impl FnOnce(...) -> ...) -> Result<CarAfterFuelFigures> { ... }
}
impl<'a> CarAfterFuelFigures<'a> {
    pub fn performance_figures(self, ...) -> Result<CarAfterPerformanceFigures> { ... }
}
impl<'a> CarComplete<'a> {
    pub fn encoded_length_with_header(&self) -> usize { ... }
    pub fn as_bytes_with_header(&self) -> &[u8] { ... }
}

Metadata: no field-name collisions

Utility methods like remaining, buffer, as_bytes_with_header, and as_body_bytes are scoped inside a zero-copy metadata struct returned by get_metadata(). This means a schema field named remaining or buffer generates dec.remaining() / dec.buffer() as field accessors — no _field suffix needed. No generated method name can ever collide with a user's schema field name.

What does remaining() mean?

Session framing (header then app payload) must use get_metadata().remaining() for the payload bytes.

let dec = CarDecoder::try_decode(&buf, 0)?;
dec.serial_number();                              // field accessor — never collides
dec.get_metadata().remaining();                   // metadata — never collides
dec.get_metadata().buffer();                      // metadata — never collides
// Car has groups/var-data: metadata is fixed-block only (not a full frame).
dec.get_metadata().as_fixed_region_with_header()?;
// Complete frame after walking tails, or rescan without consuming:
// complete.as_bytes_with_header() / dec.as_bytes_with_header()?
dec.encoded_length_with_header()?;                // hot path stays on base struct

The metadata struct holds a reference to the parent (zero-copy):

pub struct CarDecoderMetadata<'m, 'a> {
    decoder: &'m CarDecoder<'a>,
}

Encoders have the same pattern. Fixed-only messages expose as_body_bytes / as_bytes_with_header only after fixed(&FixedFields) (on the encoder and on FieldsFixed metadata). Messages with tails use as_fixed_body_bytes / as_fixed_region_with_header until the complete stage.

let meta = enc.get_metadata();
meta.as_fixed_body_bytes();            // fixed block only when message has tails
meta.as_fixed_region_with_header();    // header + fixed block — not full frame
meta.message_offset();                 // message start in buffer

Metadata limits (tailed messages)

Do not use meta.limit() as the next-message offset in a multi-message buffer for Car-shaped schemas — that truncates at the fixed block.

acting_version / acting_block_length (dual surface)

These remain inherent on the message decoder for the hot path (dec.acting_version(), dec.acting_block_length()) and are also exposed on metadata (dec.get_metadata().acting_version()) for a uniform placement facet. Both paths return the same values. They are reserved method names: a schema field named actingVersion becomes acting_version_field on the decoder.

Exact buffer sizing

// Fixed-only messages: const length.
let mut buf = [0u8; HeartbeatEncoder::compute_length_with_header()];

// Variable-length: staged builder (zero allocation).
let len = CarEncoder::compute_length()
    .fuel_figures_ragged(2, |ff| {
        ff.add()?.usage_description(5)?;
        Ok(())
    })?
    .performance_figures_ragged(0, |_| Ok(()))?
    .manufacturer(5)?
    .model(9)?
    .activation_code(6)?
    .encoded_length_with_header();

Configuration controls output size

Every aspect of generated output is configurable:

GenerationConfig::new("msgs")
    .with_display_debug(false)   // omit Debug/Display impls
    .with_meta_attributes(false) // omit *_ENCODING_OFFSET etc.
    .with_dispatch(false)        // omit AnyMessage/FrameCursor
    .with_domain_objects(DomainVarData::Bytes) // owned DTOs

A default single-message schema with these knobs off produces minimal output.

GenerationConfig::new enables display/debug, metadata, and dispatch, and disables domain objects, deprecated attributes, blanket enum options, and automatic bool mapping. GenerationProfile::Lean turns off the three output-heavy conveniences (display/debug, metadata, dispatch). See the canonical option table.

Real example

The sbe-feature-tour sample builds and exercises a real Car schema. Run cargo test inside that directory to see the generated API in action, or open src/generated/feature_tour.rs after cargo build.

Trust Boundaries

Checked entry points validate the message header and fixed block. verify walks the complete dynamic tail before bulk access:

#![allow(unused)]
fn main() {
pub fn demo_try_vs_trusted(valid_car: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
    // `decode` validates template_id, schema_id, version, and the
    // version-aware fixed body extent at message start (offset 0).
    let _dec = CarDecoder::try_decode(valid_car, 0)?;

    // `verify` walks the complete dynamic tail (groups + var-data), not a
    // header-only peek — use it when you need full structural acceptance
    // without materialising a long-lived decoder stage chain.
    CarDecoder::verify(valid_car)?;

    // Truncated buffers fail checked entry points with Result errors.
    assert!(
        CarDecoder::try_decode(&valid_car[..8.min(valid_car.len())], 0).is_err(),
        "truncated buffer should fail try_decode"
    );
    if valid_car.len() > 16 {
        assert!(
            CarDecoder::verify(&valid_car[..16]).is_err(),
            "truncated buffer should fail verify (incomplete tail)"
        );
    }

    // `wrap` still returns Result and validates the body extent given acting
    // block_length + version (message start, not sbe-tool body offset).
    let mut hdr_bytes = [0u8; 8];
    hdr_bytes.copy_from_slice(&valid_car[..8]);
    let hdr = MessageHeader(hdr_bytes);
    let dec = CarDecoder::try_wrap(valid_car, 0, hdr.block_length() as usize, hdr.version())?;
    assert_eq!(dec.serial_number(), 1234);
    Ok(())
}
}

(This code comes from the sbe-feature-tour sample crate.)

Three-tier constructors (0.1.12+)

Bare decode looks like try_decode (Result) but short buffers still panic — prefer try_decode when every failure must be a Result. Full table: Trust Boundary (core concepts). sbe-tool offsets: Coming from sbe-tool.

Trust boundary

The constructor is the single trust checkpoint. After a safe constructor proves header + version-aware fixed extent, field accessors and setters are branch-free (unchecked loads/stores justified by that proof). Dynamic group/var-data tails still check on consume. Use *_unchecked only when you have proven the fixed extent independently (benchmarks, pre-validated buffers).

Domain Objects (DTOs)

Latency-sensitive paths: never use DTOs. Domain objects allocate (Vec, String) and copy every field out of the wire buffer. If latency matters, use the zero-copy flyweight decoder instead. DTOs are for tooling, logging, and offline processing — not the hot path.

DTO construction still owns and allocates its Vec/String fields. Re-encode does not add another allocation: wire-compatible flat groups automatically use the generated bulk_add_domain(&[EntryDomain]) path, which validates one complete output region and writes directly from the DTO slice. Groups with nested tails, var-data, optional/versioned fields, domain conversions, or bool domain remapping retain the general per-entry path.

Enable domain objects during generation when an owned application value is more convenient than a zero-copy flyweight. This fixture uses DomainVarData::Strings:

#![allow(unused)]
fn main() {
pub fn demo_car_domain_dto(wire: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
    let dec = CarDecoder::try_decode(wire, 0)?;
    // try_from_decoder (not TryFrom/From): two fallible sources — decoder vs
    // try_from_slice_with_header for framed bytes; materialisation can fail.
    let dto = CarDomain::try_from_decoder(dec)?;
    assert_eq!(dto.serial_number, 1234);
    assert!(dto.available); // bool domain field
    assert_eq!(dto.fuel_figures.len(), 2);
    assert_eq!(dto.fuel_figures[0].usage_description, "Urban");
    assert_eq!(dto.manufacturer, "Honda");

    const RE_PAD: usize = 512;
    assert!(wire.len() <= RE_PAD);
    let mut storage = [0u8; RE_PAD];
    let n = dto.encode(&mut storage[..wire.len()])?;
    assert_eq!(&storage[..n], wire, "DTO re-encode must be byte-identical");
    Ok(())
}
}

(This code comes from the sbe-feature-tour sample crate.)

Multi-Template Dispatch

AnyMessage reads the generated header layout and dispatches on the template ID. Prefer AnyMessage::try_decode at untrusted boundaries (returns Result). Bare AnyMessage::decode is the same implementation today; keep using try_* naming for consistency with the three-tier trust boundary.

#![allow(unused)]
fn main() {
pub fn demo_any_message() -> Result<(), Box<dyn std::error::Error>> {
    let mut hb = [0u8; HeartbeatEncoder::compute_length_with_header()];
    let hb_len = HeartbeatEncoder::compute_length_with_header();
    let nanos: u64 = 1_700_000_000_000_000_000;
    let _hb_len = HeartbeatEncoder::try_wrap_and_apply_header(&mut hb, 0)
        .unwrap()
        .fixed(&HeartbeatFixedFields {
            sequence: 1,
            timestamp: nanos,
        })
        .encoded_length_with_header();

    let note_body = b"hello AnyMessage";
    let note_len = NoteEncoder::compute_length_with_header(note_body.len());
    const NOTE_PAD: usize = 64;
    assert!(note_len <= NOTE_PAD);
    let mut note_storage = [0u8; NOTE_PAD];
    let note = &mut note_storage[..note_len];
    let note_written = NoteEncoder::try_wrap_and_apply_header(note, 0)?
        .fixed(&NoteFixedFields { note_id: 99 })
        .body(note_body)?
        .encoded_length_with_header();
    assert_eq!(note_written, note_len);

    // Concatenate framed messages (each includes its own SBE header).
    let mut stream = Vec::new();
    stream.extend_from_slice(&hb[..hb_len]);
    stream.extend_from_slice(&note[..note_written]);

    let mut offset = 0usize;
    let mut saw_heartbeat = false;
    let mut saw_note = false;
    while offset < stream.len() {
        match AnyMessage::try_decode(&stream, offset)? {
            AnyMessage::Heartbeat(d) => {
                assert_eq!(d.sequence(), 1);
                offset += d.encoded_length_with_header()?;
                saw_heartbeat = true;
            }
            AnyMessage::Note(d) => {
                assert_eq!(d.note_id(), 99);
                let (body, complete) = d.into_body()?;
                assert_eq!(body, note_body);
                offset += complete.encoded_length() + NoteDecoder::HEADER_LENGTH;
                saw_note = true;
            }
            AnyMessage::Car(_) => return Err("unexpected Car in this demo stream".into()),
            AnyMessage::Quote(_) => return Err("unexpected Quote in this demo stream".into()),
            AnyMessage::Unknown { .. } => {
                return Err("unexpected Unknown template".into());
            }
        }
    }
    assert!(saw_heartbeat && saw_note);
    Ok(())
}
}

(This code comes from the sbe-feature-tour sample crate.)

Core Concepts

The ideas behind ergo-sbe's API design: why wire order is enforced at compile time, how exact buffer sizing works without heap allocation, when to use flyweights vs owned structs, and how composites achieve zero-copy access.

• Trust Boundary — try_from validates, wrap trusts
• Wire Order via Named Stages — calling asks before bids is a type error
• Buffer Sizing — stack-allocate the exact byte count before writing
• Flyweight vs Whole-Struct — decode in-place or materialise an owned DTO
• Composite Layout & Endianness — wire images, packed overlays, and why BE costs ~5%

Trust Boundary

Every SBE buffer crossing a process boundary must be validated. ergo-sbe provides a three-tier constructor API, ordered from safest to fastest:

The trusted tier is safe Rust. Bare constructors run the same header + fixed-body extent proof as try_*, then panic on failure. After that proof, field accessors/setters use unchecked loads/stores (justified by the constructor). Dynamic tails still check on consume.

The unsafe fn *_unchecked variants skip the extent proof entirely — only for the case where panic machinery is measurable and the caller has proven the layout independently.

Encoder entry points

Decoder entry points

Why decode returns Result but still panics

Bare decode keeps a freeze-friendly hybrid so feed handlers can ? on WrongTemplate / WrongSchema after demux, while short buffers remain a trusted-tier panic (same extent proof as wrap). Prefer try_decode when every failure — including short buffers — must be a Result.

See Trust boundaries (feature tour) for worked examples.

Wire Order via Named Stages

Why not parent hopping?

sbe-tool's flyweight API uses .parent() to hand ownership back up the tree. In Rust that pattern hits the borrow checker: move a group encoder in, get stuck returning the parent, lose the thread of the code.

ergo-sbe leans on scoped closures and chaining so nested schemas stay readable and you rarely pass encoder ownership field-to-field by hand:

#![allow(unused)]
fn main() {
pub fn encode_sample_car(buf: &mut [u8]) -> Result<usize, sbe_rt::EncodeError> {
    let mut extras = OptionalExtras::default();
    extras.cruise_control(true).sports_pack(true);

    // Buffer pre-sized from EncodedLength; try_* still validates extent.
    let len = CarEncoder::try_wrap_and_apply_header(buf, 0)
        .unwrap()
        .fixed(&CarFixedFields {
            serial_number: 1234,
            model_year: 2013,
            available: true.into(),
            code: Model::A,
            some_numbers: [10, 20, 30, 40],
            vehicle_code: [b'A', b'B', b'C', b'D', b'E', b'F'],
            extras,
            engine: Engine::new(
                2000,
                4,
                [b'1', b'2', b'3'],
                0i8,
                false.into(),
                Booster::new(BoostType::TURBO, 210),
            ),
        })
        .fuel_figures(2, |g| {
            g.add(|mut e| {
                e.speed(30).mpg(35.9);
                e.usage_description(b"Urban")
            })?;
            g.add(|mut e| {
                e.speed(60).mpg(25.0);
                e.usage_description(b"Highway")
            })?;
            Ok(())
        })?
        .performance_figures(1, |g| {
            g.add(|mut e| {
                e.octane_rating(95);
                e.acceleration(2, |a| {
                    a.add(|x| {
                        x.mph(30).seconds(4.0);
                        Ok(())
                    })?;
                    a.add(|x| {
                        x.mph(60).seconds(7.5);
                        Ok(())
                    })
                })
            })?;
            Ok(())
        })?
        .manufacturer(b"Honda")?
        .model(b"Civic VTi")?
        .activation_code(b"abcdef")?
        .encoded_length_with_header();

    Ok(len)
}
}

(Real code from the sbe-feature-tour sample — compiles and runs in CI.)

Wire order via named stage structs

SBE is a positional wire format: groups and var-data appear in a fixed schema order with no per-field tags on the wire. That matters a lot in financial markets, where it is common to have two nearly identical repeating groups back-to-back — e.g. bids then asks (same entry layout, different meaning). If you encode or decode them in the wrong order, the bytes still look like a valid message: prices and sizes land in the opposite book side. You only discover the disaster at runtime (wrong trades, inverted books, silent corruption). Compile-time order exists so that mistake becomes a type error while you still have the schema in front of you, not a production incident.

Order is enforced with the same idea as the classic type-state pattern (Encoder<State> / PhantomData), but not that implementation.

Each wire-order transition returns a named concrete stage struct (e.g. CarAfterFuelFigures). The H: HeaderState generic on every stage is a zero-sized orthogonal marker for header-present vs body-only mode — it tracks header capability, not wire-order progression. Duplicating the stage graph for HeaderPresent and HeaderAbsent would provide no latency advantage.

All maintained SBE parity comparisons pass at or below the 1.00× ceiling under both LTO-on and LTO-off profiles. Current results are in Benchmarks; see the methodology page for reproduction.

Generated code emits separate types for each stage, same fields, different methods:

// Approximate generated shape — not Encoder<AfterBids>:
pub struct BookEncoder<'a> { /* buf, pos, … */ }
pub struct BookAfterBids<'a> { /* same layout */ }
pub struct BookAfterAsks<'a> { /* same layout */ }
// …

impl BookEncoder<'a> {
    pub fn bids(self, …) -> Result<BookAfterBids<'a>, …> { … }
    // no asks() here — bids first on the wire
}
impl BookAfterBids<'a> {
    pub fn asks(self, …) -> Result<BookAfterAsks<'a>, …> { … }
    // no bids() here — already done
}

So after fixed fields you may only call the next group/var-data in schema order. Calling asks before bids is a type error (BookEncoder has no asks method). Decoders use the same idea: consuming stages (BookDecoder → BookDecoderAfterBids → …).

Group bodies use |g| { g.add(|e| { … }) } so the outer encoder is not left half-borrowed while you fill nested levels — the closure ends, then chaining continues. That is intentional API ergonomics for Rust (avoids .parent() style ownership hand-offs that fight the borrow checker on deep books).

Buffer Sizing

Why this exists: true zero-copy publish on Aeron (and similar systems) uses try_claim / a pre-sized slot. The transport hands you a buffer of a known length; you must know the full encoded message size before you write. Guessing with an oversized scratch Vec and copying later defeats that model and is easy to get wrong for groups and var-data.

ergo-sbe therefore generates schema-aware length APIs so you describe the shape you are about to encode (counts, nested groups, var-data byte lengths) and get an exact size first — safer and easier than hand-computing header + block + Σ(groups) + Σ(var-data).

#![allow(unused)]
fn main() {
pub fn demo_car_size_and_encode() -> Result<Vec<u8>, Box<dyn std::error::Error>> {
    // Fuel: 2 entries with usage ASCII lengths 5 and 7.
    // Performance: 1 entry with 2 nested acceleration rows (fixed-only entries).
    // Message var-data: manufacturer / model / activationCode lengths.
    let complete_len = CarEncoder::compute_length()
        .fuel_figures_ragged(2, |ff| {
            ff.add()?.usage_description(5)?; // "Urban"
            ff.add()?.usage_description(7)?; // "Highway"
            Ok(())
        })?
        .performance_figures_ragged(1, |pf| {
            pf.add()?.acceleration(|acc| {
                acc.uniform(2)?;
                Ok(())
            })?;
            Ok(())
        })?
        .manufacturer(5)? // "Honda"
        .model(9)? // "Civic VTi"
        .activation_code(6)? // "abcdef"
        .encoded_length_with_header();

    // Exact size from compute_length → stack pad (this demo fits well under 512).
    const CAR_PAD: usize = 512;
    assert!(
        complete_len <= CAR_PAD,
        "sample car length {complete_len} exceeds stack pad {CAR_PAD}"
    );
    let mut storage = [0u8; CAR_PAD];
    let written = encode_sample_car(&mut storage[..complete_len])?;
    assert_eq!(
        written, complete_len,
        "CarEncodedLength must equal encoder-produced length"
    );
    Ok(storage[..written].to_vec())
}
/// Encode the canonical sample car into `buf` (must be pre-sized).
pub fn encode_sample_car(buf: &mut [u8]) -> Result<usize, sbe_rt::EncodeError> {
    let mut extras = OptionalExtras::default();
    extras.cruise_control(true).sports_pack(true);

    // Buffer pre-sized from EncodedLength; try_* still validates extent.
    let len = CarEncoder::try_wrap_and_apply_header(buf, 0)
        .unwrap()
        .fixed(&CarFixedFields {
            serial_number: 1234,
            model_year: 2013,
            available: true.into(),
            code: Model::A,
            some_numbers: [10, 20, 30, 40],
            vehicle_code: [b'A', b'B', b'C', b'D', b'E', b'F'],
            extras,
            engine: Engine::new(
                2000,
                4,
                [b'1', b'2', b'3'],
                0i8,
                false.into(),
                Booster::new(BoostType::TURBO, 210),
            ),
        })
        .fuel_figures(2, |g| {
            g.add(|mut e| {
                e.speed(30).mpg(35.9);
                e.usage_description(b"Urban")
            })?;
            g.add(|mut e| {
                e.speed(60).mpg(25.0);
                e.usage_description(b"Highway")
            })?;
            Ok(())
        })?
        .performance_figures(1, |g| {
            g.add(|mut e| {
                e.octane_rating(95);
                e.acceleration(2, |a| {
                    a.add(|x| {
                        x.mph(30).seconds(4.0);
                        Ok(())
                    })?;
                    a.add(|x| {
                        x.mph(60).seconds(7.5);
                        Ok(())
                    })
                })
            })?;
            Ok(())
        })?
        .manufacturer(b"Honda")?
        .model(b"Civic VTi")?
        .activation_code(b"abcdef")?
        .encoded_length_with_header();

    Ok(len)
}
}

(From sbe-feature-tour — EncodedLength + exact buffer encode, tested in CI.)

Nested books: book_encoded_length. API matrix: encoded_length_api_test.

Flyweight vs Whole-Struct

You can work field-by-field (classic flyweight) or fill / materialise a whole struct. Use the style that matches how much of the message you touch.

Encode — whole fixed block as a struct

When you always populate every fixed field, a struct is clearer and schema additions break at compile time:

#![allow(unused)]
fn main() {
let mut buf = [0u8; HeartbeatEncoder::compute_length_with_header()];
// Buffer is exact size from const compute_length_with_header — no bounds check needed.
let len = HeartbeatEncoder::try_wrap_and_apply_header(&mut buf, 0)?
    .fixed(&HeartbeatFixedFields { sequence: 7, timestamp: 0 })
    .encoded_length_with_header();
let dec = HeartbeatDecoder::try_from(&buf[..len])?;
assert_eq!(dec.sequence(), 7);
}

(From book/examples/heartbeat-encode.rs — compiled against the feature-tour codec.)

If the schema later adds a required field to the fixed block, this stops compiling until you add it to the struct literal — you cannot silently omit it.

Decode — individual fields (flyweight)

#![allow(unused)]
fn main() {
// Encode a sample car first (normally bytes come from the wire)
let complete_len = CarEncoder::compute_length()
    .fuel_figures_ragged(0, |_| Ok(()))?
    .performance_figures_ragged(0, |_| Ok(()))?
    .manufacturer(5)?
    .model(5)?
    .activation_code(3)?
    .encoded_length_with_header();
const PAD: usize = 256;
assert!(complete_len <= PAD, "car length {complete_len} exceeds pad {PAD}");
let mut storage = [0u8; PAD];
let buf = &mut storage[..complete_len];
let fields = CarFixedFields {
    serial_number: 1234,
    model_year: 2013,
    available: true.into(),
    code: Model::A,
    some_numbers: [10, 20, 30, 40],
    vehicle_code: *b"ABCDEF",
    extras: OptionalExtras::default(),
    engine: Engine::new(2000, 4, *b"123", 0i8, false.into(), Booster::new(BoostType::TURBO, 210)),
};
let n = CarEncoder::try_wrap_and_apply_header(buf, 0)?
    .fixed(&fields)
    .fuel_figures(0, |_| Ok(()))?
    .performance_figures(0, |_| Ok(()))?
    .manufacturer(b"Honda")?
    .model(b"Civic")?
    .activation_code(b"abc")?
    .encoded_length_with_header();
assert_eq!(n, complete_len);
// Now decode — read only the fields you need, no DTO allocation:
let car = CarDecoder::try_from(&buf[..n])?;
assert_eq!(car.serial_number(), 1234);
assert_eq!(car.model_year(), 2013);
assert_eq!(car.code(), Model::A);
assert_eq!(car.engine().capacity(), 2000);
}

(From book/examples/flyweight-access.rs — compiled against the feature-tour codec.)

Decode — whole message as a DTO

Do not use on the latency-sensitive path. DTO decode allocates Vec/String and copies every field. For the hot path, use the flyweight decoder instead.

When you always need (almost) everything, or want to pass a value across threads / into non-SBE code:

#![allow(unused)]
fn main() {
pub fn demo_car_domain_dto(wire: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
    let dec = CarDecoder::try_decode(wire, 0)?;
    // try_from_decoder (not TryFrom/From): two fallible sources — decoder vs
    // try_from_slice_with_header for framed bytes; materialisation can fail.
    let dto = CarDomain::try_from_decoder(dec)?;
    assert_eq!(dto.serial_number, 1234);
    assert!(dto.available); // bool domain field
    assert_eq!(dto.fuel_figures.len(), 2);
    assert_eq!(dto.fuel_figures[0].usage_description, "Urban");
    assert_eq!(dto.manufacturer, "Honda");

    const RE_PAD: usize = 512;
    assert!(wire.len() <= RE_PAD);
    let mut storage = [0u8; RE_PAD];
    let n = dto.encode(&mut storage[..wire.len()])?;
    assert_eq!(&storage[..n], wire, "DTO re-encode must be byte-identical");
    Ok(())
}
}

Rule of thumb: one field on the hot path → flyweight. Always fill or always consume the whole message → FixedFields / Domain for clarity and compile-time breakage on schema growth. More on DTOs in Recipes — Domain DTOs.

Composite Layout & Endianness

A common question: on a little-endian host, can a composite just be a #[repr(C)] / #[repr(C, packed)] struct overlaid on the buffer so field access is a free load?

Almost — but not via repr(C) transmute. ergo-sbe does something safer that is still effectively free on LE hosts:

On little-endian hosts, from_le_bytes lowers to a plain load (aligned or unaligned as needed) — so member access is “super fast” without casting the buffer to a padded Rust struct. The generator also emits

const _: () = assert!(core::mem::size_of::<Engine>() == 10);

so the Rust type size is locked to the wire size at compile time.

*FixedFields (e.g. CarFixedFields) is a different beast: an application struct with typed fields used to fill the fixed block in one call. It is not a zero-copy overlay of the message buffer; .fixed(&…) writes each field with endian conversion into the flyweight buffer.

Conclusion — why not repr(C, packed)?

Single-field access is already one load. Head-to-head Criterion arms on a 256-byte composite (mid-block field f15), field-only, no alloc on the timed path (layout_access_bench):

So:

1. Flyweight ≈ preheld wire-image ≈ packed for one field — all one load on LE.
2. repr(C, packed) does not unlock free access beyond what [u8; N] + from_le_bytes already gives. Hand-rolling packed overlays is extra UB/layout risk for no speed win.
3. The expensive mistake is materialising a large composite just to touch one field. Prefer flyweight when you only need a few members; use *_value() when you need the whole wire blob (or pass it around) and pay the N-byte copy once.
4. We still do not generate repr(C) / packed field structs: packing + unaligned references, big-endian schemas, enums/sets/nested composites. The transparent wire image is the portable form that already optimizes to the packed load on LE.

What about the zerocopy crate?

We evaluated using the zerocopy crate to derive FromBytes/IntoBytes on generated message structs for zero-copy buffer overlay. It was not faster.

The flyweight decoder already hits the same mov instructions without the extra dependency.

Layout contracts: composite_layout_test. Decode microbench: layout_access_bench.

Encode — FixedFields vs setters, composite write, LE vs BE

Confirmed by encode_style_bench (Apple M4, LE host; values prebuilt / seeded so LLVM cannot delete the work):

So on encode:

1. Prefer .fixed for clarity / schema completeness — not for speed.
2. Prefer a prebuilt composite wire image on the hot path when you can; for small N the win is tiny next to other field stores.
3. LE body on LE host is free endian; BE body costs a bswap per multi-byte field when building the image. Once the image exists, write cost matches LE.

Configuration

• with_conversion vs with_domain_type
• GenerationConfig Options
• Code-Generation Hooks

Wire type vs app type

  app  ──price_from / try_price──►  wire Decimal on the buffer
  buf  ──price_as / try_price──►   app value

Feature Integrations

Ergon ships optional integrations behind Cargo feature flags — add exactly what your hot path needs, keep compilation lean otherwise.

Quick reference

See Measured performance below for the full benchmark table.

Add features in your Cargo.toml:

[dependencies]
ergo-sbe = { version = "0.1", features = ["compact_str", "chrono"] }

CompactString — inline symbols (DTOs)

Primarily for domain DTOs. The codec-level accessor into_<field>_as_compact_str() exists but is secondary — the main win is replacing String with CompactString in generated *Domain structs.

compact_str::CompactString stores up to 24 bytes on the stack. Perfect for tickers ("AAPL"), currency pairs ("EUR/USD"), venue codes ("XNYS") — every string that fits in a CPU register-sized inline buffer skips the allocator entirely.

Domain DTOs

use ergo_sbe::{DomainVarData, GenerationConfig};

let config = GenerationConfig::new("msgs")
    .with_domain_objects(DomainVarData::CompactStrings);

Generated DTO:

// Type paths use ergo_sbe re-exports — no need to add compact_str directly.
pub struct QuoteDomain {
    pub symbol: ergo_sbe::compact_str::CompactString,  // was String
    pub venue: ergo_sbe::compact_str::CompactString,
    pub price: Decimal,
    // …
}

Codec accessors

When compact_str is enabled, every text var-data consuming stage gains an into_<field>_as_compact_str() method:

let stage = dec.fuel_figures()?;
let (symbol, next_stage) = stage.into_symbol_as_compact_str()?;
// symbol: CompactString — no heap allocation for ≤24B symbols

Measured performance

aarch64-apple-darwin, rustc 1.95.0 — nanoseconds per from_utf8 + conversion

Takeaway: CompactString is 46–56% faster for symbols ≤24 bytes (no heap). At larger sizes it converges with String. Bytes wins at 256 B+. SmolStr has O(1) clone regardless of length; from_utf8 cost grows with payload size.

Run: cargo bench -p ergo-sbe-benchmarks --bench var_data_types_bench --all-features

SmolStr — cheap clones (DTOs)

Primarily for domain DTOs. Like CompactString, the main use is DomainVarData::SmolStrings in generated *Domain structs.

smol_str::SmolStr clones in O(1) regardless of length. Best when DTOs are long-lived and shared across threads or cached.

let config = GenerationConfig::new("msgs")
    .with_domain_objects(DomainVarData::SmolStrings);

Codec: into_<field>_as_smol_str() — returns SmolStr.

Bytes — zero-copy relay

bytes::Bytes is a reference-counted byte buffer. Clone it without copying; slice it without allocating. Ideal for relay/forwarding pipelines where the same frame is dispatched to multiple consumers.

Domain DTOs

let config = GenerationConfig::new("relay")
    .with_domain_objects(DomainVarData::BytesCrate);

Generated DTO fields are bytes::Bytes:

let original = bytes::Bytes::copy_from_slice(b"payload");
let clone = original.clone();   // increments refcount, no copy
let sub = original.slice(4..);  // view into the same buffer

Codec accessors

into_<field>_as_bytes() returns bytes::Bytes:

let (payload, next) = stage.into_payload_as_bytes()?;
// payload: bytes::Bytes — share it, slice it, send it across threads

Chrono — typed timestamps

SBE timestamp fields are i64 on the wire (nanoseconds or microseconds since the Unix epoch). The chrono feature adds converter functions and enables with_domain_type for timestamp semantic types.

Build-time config

use ergo_sbe::{GenerationConfig, ConversionSelector};

let config = GenerationConfig::new("msgs")
    .with_domain_type(
        ConversionSelector::semantic_type("UTCTimestamp"),
        "chrono::DateTime<chrono::Utc>",
    )
    .with_domain_type(
        ConversionSelector::semantic_type("UTCTimestampMicros"),
        "chrono::NaiveDateTime",
    );

Generated API

// Decode
let created: chrono::DateTime<chrono::Utc> = dec.try_created_at()?;
let updated: chrono::NaiveDateTime = dec.try_updated_at()?;

// Encode
enc.try_created_at(chrono::Utc::now())?;
enc.try_updated_at(chrono::DateTime::from_timestamp(1_720_000_000, 0).unwrap().naive_utc())?;

Registering semantic_type("UTCTimestamp") once covers every field in the schema carrying that semanticType, not just created_at — see One selector, many fields for a worked multi-field example.

Direct converters

use ergo_sbe::chrono_converters;

// Wire nanos → DateTime
let dt = i64_nanos_to_datetime(1_720_000_000_000_000_000);

// DateTime → wire nanos
let ns = datetime_to_i64_nanos(dt);

// Wire micros → NaiveDateTime
let naive = i64_micros_to_naive(1_720_000_000_000_000);

// Roundtrip is exact
assert_eq!(naive_to_i64_micros(naive), 1_720_000_000_000_000);

Measured conversion cost

aarch64-apple-darwin, rustc 1.95.0 — ns per operation

Takeaway: Conversions add 2–6 ns — negligible next to the I/O cost of a network frame (~500 ns for 10 GbE) or the allocator cost of a var-data field (3–19 ns). The type safety is worth it.

Run: cargo bench -p ergo-sbe-benchmarks --bench chrono_converter_bench --all-features

Combining features

All four features are independent — enable any subset:

[dependencies]
ergo-sbe = { version = "0.1", features = ["compact_str", "bytes", "chrono"] }

let config = GenerationConfig::new("msgs")
    .with_domain_objects(DomainVarData::CompactStrings)
    .with_domain_type(
        ConversionSelector::semantic_type("UTCTimestamp"),
        "chrono::DateTime<chrono::Utc>",
    );

with_conversion vs with_domain_type

Do not call both for the same selector — domain type already enables conversion.

Option A — you choose the app type (Cents)

#![allow(unused)]
fn main() {
    use ergo_sbe::{ConversionSelector, GenerationConfig};
    // A — generic converter: one wire type, many app types
    let _cfg =
        GenerationConfig::new("msgs").with_conversion(ConversionSelector::named_type("Decimal"));
}

(From book/examples/conversion-config.rs — a self-contained program that compiles against ergo-sbe.)

#![allow(unused)]
fn main() {
use rust_decimal::Decimal as Rd;
// App adapter: wire Decimal ↔ rust_decimal::Decimal
struct FixedPrice { mantissa: i64, exponent: i8 }

impl TryFromSbe<Decimal> for FixedPrice {
    type Error = &'static str;
    fn try_from_sbe(wire: Decimal) -> Result<Self, Self::Error> {
        Ok(FixedPrice {
            mantissa: wire.mantissa(),
            exponent: wire.exponent(),
        })
    }
}
impl TryToSbe<Decimal> for FixedPrice {
    type Error = &'static str;
    fn try_to_sbe(&self) -> Result<Decimal, Self::Error> {
        Ok(Decimal::new(self.mantissa, self.exponent))
    }
}
}

(From book/examples/conversion-app-code.rs — app adapter pattern, compiles against tour_codec.)

// Encode using the generic conversion API:
let mut buf = [0u8; QuoteEncoder::compute_length_with_header()];
let price = Rd::new(12345, 2); // 123.45
let len = QuoteEncoder::try_wrap_and_apply_header(&mut buf, 0)?
    .price_from(&price)?
    .size_from(&Rd::new(10, 0))?
    .encoded_length_with_header();
// Decode — generic `_as::<T>()` picks your adapter:
let dec = QuoteDecoder::try_from(&buf[..len])?;
let p: Rd = dec.price_as()?;
assert_eq!(p, Rd::new(12345, 2));
// Same buffer, different app type — only possible with with_conversion:
let fixed: FixedPrice = dec.price_as()?;
assert_eq!(fixed.mantissa, 12345);
assert_eq!(fixed.exponent, -2);

(Same file — generic _from/_as encode/decode with with_conversion.)

Option B — one fixed app type

#![allow(unused)]
fn main() {
    use ergo_sbe::{ConversionSelector, GenerationConfig};
    // B — concrete mapping: one Rust type per wire type (already enables conversion)
    let _cfg = GenerationConfig::new("msgs").with_domain_type(
        ConversionSelector::named_type("Decimal"),
        "rust_decimal::Decimal",
    );
}

(Same source file — book/examples/conversion-config.rs.)

// Encode using the generic conversion API:
let mut buf = [0u8; QuoteEncoder::compute_length_with_header()];
let price = Rd::new(12345, 2); // 123.45
let len = QuoteEncoder::try_wrap_and_apply_header(&mut buf, 0)?
    .price_from(&price)?
    .size_from(&Rd::new(10, 0))?
    .encoded_length_with_header();
// Decode — generic `_as::<T>()` picks your adapter:
let dec = QuoteDecoder::try_from(&buf[..len])?;
let p: Rd = dec.price_as()?;
assert_eq!(p, Rd::new(12345, 2));
// Same buffer, different app type — only possible with with_conversion:
let fixed: FixedPrice = dec.price_as()?;
assert_eq!(fixed.mantissa, 12345);
assert_eq!(fixed.exponent, -2);

Both styles on different fields:

pub fn demo_conversion_only() -> Result<Vec<u8>, Box<dyn std::error::Error>> {
    let mut buf = [0u8; QuoteEncoder::compute_length_with_header()];

    let price = Rd::new(12345, 2); // 123.45
    let size = Rd::new(10, 0);
    let mut enc = QuoteEncoder::try_wrap_and_apply_header(&mut buf, 0)?;
    enc.price_from(&price)?;
    enc.size_from(&size)?;
    let len = QuoteEncoder::compute_length_with_header();

    let dec = QuoteDecoder::try_from(&buf[..len])?;
    let wire = dec.price_value();
    assert_eq!(wire.mantissa(), 12345);
    assert_eq!(wire.exponent(), -2);

    let price2: Rd = dec.price_as()?;
    let size2: Rd = dec.size_as()?;
    assert_eq!(price2, price);
    assert_eq!(size2, size);

    // Same buffer, different app type — only possible with with_conversion.
    let fixed: FixedPrice = dec.price_as()?;
    assert_eq!(
        fixed,
        FixedPrice {
            mantissa: 12345,
            exponent: -2
        }
    );

    let dto = QuoteDomain::try_from_decoder(dec)?;
    assert_eq!(dto.price.mantissa(), 12345);
    let mut re = [0u8; QuoteEncoder::compute_length_with_header()];
    let n = dto.encode(&mut re)?;
    assert_eq!(&re[..n], &buf[..len]);
    Ok(buf[..len].to_vec())
}

(This code comes from the sbe-feature-tour sample crate.)

Option B, manual impl — concrete signatures, your own conversion logic

with_domain_type(selector, path) is the common case: ergo-sbe also generates the TryFromSbe/TryToSbe impl for bool / rust_decimal::Decimal / chrono::DateTime<Utc>. If you need different conversion behaviour for one of those exact three types — a custom rounding rule, stricter validation, different null handling — call additive with_manual_domain_type instead: same generated try_price(...)? / try_price()? signatures, but you write the impl:

    let (_schema, src) = generate_domain_with(&path, "manual_impl_dt", |c| {
        c.with_manual_domain_type(
            ergo_sbe::ConversionSelector::named_type("Decimal"),
            "rust_decimal::Decimal",
        )
    });

        use rust_decimal::Decimal;

        // Caller-supplied impl: deliberately scales mantissa by 10 on the way
        // in/out, so a value only round-trips correctly through THIS impl —
        // proof ergo-sbe didn't quietly generate its own.
        impl TryFromSbe<self::Decimal> for rust_decimal::Decimal {
            type Error = &'static str;
            fn try_from_sbe(wire: self::Decimal) -> Result<Self, Self::Error> {
                Ok(rust_decimal::Decimal::new(wire.mantissa() / 10, (-wire.exponent()) as u32))
            }
        }
        impl TryToSbe<self::Decimal> for rust_decimal::Decimal {
            type Error = &'static str;
            fn try_to_sbe(&self) -> Result<self::Decimal, Self::Error> {
                Ok(self::Decimal::new(self.mantissa() as i64 * 10, -(self.scale() as i8)))
            }
        }

        let mut buf = [0u8; 256];
        let mut enc = OrderEncoder::wrap_and_apply_header(&mut buf, 0)
            .fixed(&OrderFixedFields { price: self::Decimal::new(0, 0), size: self::Decimal::new(0, 0) });
        enc.try_price(Decimal::new(12345, 2))?;
        enc.try_size(Decimal::new(100, 0))?;
        let encoded = enc.as_bytes_with_header().to_vec();

        let dec = OrderDecoder::try_decode(&encoded, 0)?;
        assert_eq!(dec.try_price()?, Decimal::new(12345, 2));
        assert_eq!(dec.try_size()?, Decimal::new(100, 0));

(From domain_type_manual_impl_uses_callers_own_impl in sbe/tests/baseline_test.rs — a real generated-and-compiled test. Any rust_type string that isn't one of the three built-ins never gets an auto-generated impl regardless of DomainImpl — it only matters for opting those three in or out.)

Forgot the impl? Two things soften it. First, the compile error names the missing impl directly instead of the default trait-bound message:

error[E0277]: `rust_decimal::Decimal` has no `TryFromSbe<Decimal>` impl
  |
  | missing `impl TryFromSbe<Decimal> for rust_decimal::Decimal`
  |
  = note: if this field uses DomainImpl::Manual, the generated `try_*`
          accessor's doc comment has a ready-to-paste starting point

Second, for the three built-ins, that pointer is real: the generated try_price method's own doc comment (visible on hover, or in cargo doc) carries the exact impl DomainImpl::Generated would have written — copy it out and adjust. sbe/tests/baseline_test.rs's domain_type_manual_impl_doc_comment_has_generated_snippet asserts this snippet is present in the generated source.

GenerationConfig Options

Every option except new("module_name") is a chained builder method. Boolean flags default to the value shown.

Turn off with_display_debug, with_meta_attributes, and with_dispatch to reduce generated-code size (~6,100 lines/message with all on). Text fields stay bytes unless the schema declares a character encoding (then strict UTF-8/ASCII helpers apply).

Migrating off with_error_from_impls

with_error_from_impls formats the generated error through Display and From<String>, so fields such as needed and available are lost. Implement From yourself on the generated sbe_rt types (removed in 1.0):

#![allow(unused)]
fn main() {
enum AppError {
    Encode(sbe_rt::EncodeError),
    Decode(sbe_rt::DecodeError),
}

impl From<sbe_rt::EncodeError> for AppError {
    fn from(error: sbe_rt::EncodeError) -> Self {
        match error {
            sbe_rt::EncodeError::BufferTooShort {
                field,
                needed,
                available,
            } => {
                let _ = (field, needed, available);
                Self::Encode(error)
            }
            other => Self::Encode(other),
        }
    }
}

impl From<sbe_rt::DecodeError> for AppError {
    fn from(error: sbe_rt::DecodeError) -> Self {
        Self::Decode(error)
    }
}
}

Code-Generation Hooks

Niche feature. Hooks are aimed at users who need to attach extra impl blocks to generated code — serde, custom validation, company-internal traits. Most workflows don't need them; skip this section unless you recognise your use case.

Hooks append tokens after each generated item; they cannot add a #[derive(...)] to the item itself (that attribute would have to precede the struct/enum). Emit the trait impl directly instead — that is what a derive would expand to anyway.

Hooks let you append arbitrary Rust tokens after each generated item (enum, set, composite, message decoder/encoder, domain struct). The closure receives an ItemContext with structured field/variant/choice metadata, plus a schema reference for full IR access.

Example — add serde Serialize + Deserialize to every enum and set:

// build.rs
use ergo_sbe::{GenerationConfig, ItemContext};
use quote::quote;

fn serde_hook(ctx: &ItemContext) -> Vec<proc_macro2::TokenStream> {
    match ctx {
        ItemContext::Enum { name, variants, .. } => {
            let ident = quote::format_ident!("{name}");
            let labels: Vec<_> = variants.iter().map(|v| v.label.clone()).collect();
            let names: Vec<_> = variants.iter().map(|v| quote::format_ident!("{}", v.name)).collect();
            let from_labels: Vec<_> = variants.iter().map(|v| quote::format_ident!("{}", v.name)).collect();
            vec![quote::quote! {
                impl serde::Serialize for #ident {
                    fn serialize<S: serde::Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
                        s.serialize_str(match self {
                            #(Self::#names => #labels,)*
                            _ => "NullVal",
                        })
                    }
                }
                impl<'de> serde::Deserialize<'de> for #ident {
                    fn deserialize<D: serde::Deserializer<'de>>(d: D) -> Result<Self, D::Error> {
                        let s = <&str>::deserialize(d)?;
                        match s {
                            #(#labels => Ok(Self::#from_labels),)*
                            "NullVal" => Ok(Self::NullVal),
                            // Reject unknown values — fallback is app policy,
                            // not codec behaviour.
                            other => Err(serde::de::Error::unknown_variant(
                                other, &[#(#labels,)* "NullVal"])),
                        }
                    }
                }
            }]
        }
        ItemContext::Set { name, choices, .. } => {
            let ident = quote::format_ident!("{name}");
            let is: Vec<_> = choices.iter().map(|c| quote::format_ident!("is_{}", c.snake_name)).collect();
            let labels: Vec<_> = choices.iter().map(|c| c.label.clone()).collect();
            let froms: Vec<_> = choices.iter().map(|c| quote::format_ident!("{}", c.snake_name)).collect();
            vec![quote::quote! {
                impl serde::Serialize for #ident {
                    fn serialize<S: serde::Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
                        let mut names = Vec::new();
                        #(if self.#is() { names.push(#labels); })*
                        names.serialize(s)
                    }
                }
                impl<'de> serde::Deserialize<'de> for #ident {
                    fn deserialize<D: serde::Deserializer<'de>>(d: D) -> Result<Self, D::Error> {
                        let names: Vec<&str> = <Vec<&str>>::deserialize(d)?;
                        let mut val = Self::default();
                        for n in &names {
                            match *n {
                                #(#labels => val = val.#froms(),)*
                                // Reject unknown labels — don't silently drop.
                                other => return Err(serde::de::Error::unknown_variant(
                                    other, &[#(#labels),*])),
                            }
                        }
                        Ok(val)
                    }
                }
            }]
        }
        _ => vec![],
    }
}

let config = GenerationConfig::new("msgs").with_hook(serde_hook);

Each ItemContext variant carries the fields, variants, or choices defined in the schema — use them to build custom impl blocks or trait implementations. Hooks fire in registration order; the returned tokens are appended after the generated item (so they extend it with impls, not with derives).

Full runnable example with serde + serde_json round-trip: hook_serde_test.

Recipes

Runnable, tested code for every pattern lives in sbe-feature-tour. See its src/lib.rs for the full API map.

• Aeron try_claim
• Display / Debug
• Schema Descriptions → Rustdoc
• Domain DTOs
• App Types on Composites
• Timestamp Conversions

Quick reference

Known vs unknown group count:

#![allow(unused)]
fn main() {
pub fn encode_sample_car(buf: &mut [u8]) -> Result<usize, sbe_rt::EncodeError> {
    let mut extras = OptionalExtras::default();
    extras.cruise_control(true).sports_pack(true);

    // Buffer pre-sized from EncodedLength; try_* still validates extent.
    let len = CarEncoder::try_wrap_and_apply_header(buf, 0)
        .unwrap()
        .fixed(&CarFixedFields {
            serial_number: 1234,
            model_year: 2013,
            available: true.into(),
            code: Model::A,
            some_numbers: [10, 20, 30, 40],
            vehicle_code: [b'A', b'B', b'C', b'D', b'E', b'F'],
            extras,
            engine: Engine::new(
                2000,
                4,
                [b'1', b'2', b'3'],
                0i8,
                false.into(),
                Booster::new(BoostType::TURBO, 210),
            ),
        })
        .fuel_figures(2, |g| {
            g.add(|mut e| {
                e.speed(30).mpg(35.9);
                e.usage_description(b"Urban")
            })?;
            g.add(|mut e| {
                e.speed(60).mpg(25.0);
                e.usage_description(b"Highway")
            })?;
            Ok(())
        })?
        .performance_figures(1, |g| {
            g.add(|mut e| {
                e.octane_rating(95);
                e.acceleration(2, |a| {
                    a.add(|x| {
                        x.mph(30).seconds(4.0);
                        Ok(())
                    })?;
                    a.add(|x| {
                        x.mph(60).seconds(7.5);
                        Ok(())
                    })
                })
            })?;
            Ok(())
        })?
        .manufacturer(b"Honda")?
        .model(b"Civic VTi")?
        .activation_code(b"abcdef")?
        .encoded_length_with_header();

    Ok(len)
}
}

(From sbe-feature-tour — known-count groups with chaining, tested in CI.)

Unknown size: count back-patched after the closure (streaming producers).

let unknown_len = CarEncoder::wrap_and_apply_header(&mut buf, 0)
    .fixed(&fields)
    .fuel_figures_unknown_size(|g| {
        for row in rows {
            g.add(|e| {
                e.speed(row.speed).mpg(row.mpg);
                Ok(())
            })?;
        }
        Ok(())
    })?
    .performance_figures(0, |_| Ok(()))?
    .manufacturer(b"Honda")?
    .model(b"Civic")?
    .activation_code(b"active")?
    .encoded_length_with_header();

println!("known={known_len} unknown={unknown_len}");

Aeron try_claim Integration

ergo-sbe's exact buffer sizing is designed to work directly with Aeron's try_claim — size the message first, claim exactly that many bytes from the publication, and encode straight into the claimed buffer. No oversize scratch buffer, no copy.

Pattern

use messages::{HeartbeatEncoder, HeartbeatFixedFields};

// 1. Exact size before claiming — const, no allocation.
const HB_LEN: usize = HeartbeatEncoder::compute_length_with_header();

// 2. Claim exactly that many bytes. `data()` is the claimed region — Aeron's
//    own framing sits outside it, so there is no prefix to skip.
let mut claim = publication.try_claim_owned(HB_LEN)?;

// 3. Encode straight into the claim. `wrap_into_claim` is fixed-only and
//    requires the slice to be *exactly* ENCODED_LENGTH.
HeartbeatEncoder::wrap_into_claim(claim.data())?
    .fixed(&HeartbeatFixedFields { sequence: 7, timestamp: 0 });

// 4. Commit — Aeron sends it.
claim.commit()?;

Cluster clients do not claim from the publication directly: use AeronCluster::try_claim, which claims SessionMessageHeader + payload and hands back the payload region via ClusterClaim::payload_mut().

For variable-length messages (groups / var-data), there is no wrap_into_claim — that helper is fixed-only. Size with the staged EncodedLength builder, claim that exact length, then try_wrap_and_apply_header on a slice of length len:

let len = CarEncoder::compute_length()
    .fuel_figures_ragged(2, |ff| {
        ff.add()?.usage_description(5)?;  // "Urban"
        ff.add()?.usage_description(7)?;  // "Highway"
        Ok(())
    })?
    .performance_figures_ragged(0, |_| Ok(()))?
    .manufacturer(5)?
    .model(9)?
    .activation_code(6)?
    .encoded_length_with_header();

let mut claim = publication.try_claim_owned(len)?;
debug_assert_eq!(claim.data().len(), len); // claim boundary == EncodedLength
let written = CarEncoder::try_wrap_and_apply_header(claim.data(), 0)?
    .fixed(&fields)
    // ... fuel_figures / performance_figures / manufacturer / model ...
    .activation_code(b"abcdef")?
    .encoded_length_with_header();
debug_assert_eq!(written, len);
claim.commit()?;

Why this works

• compute_length_with_header() and the staged EncodedLength builder give the exact byte count before any byte is written — no guesswork, no oversized scratch vec![0u8; 4096].
• wrap_into_claim (fixed messages only) requires buf.len() == ENCODED_LENGTH and returns ClaimLengthMismatch otherwise.
• For ragged messages, you own the claim length from EncodedLength; the encoder still validates capacity via try_wrap_*.
• The encoder writes directly into the slice you hand it — the claim buffer IS the encode buffer.
• The encoded_length_with_header() return value on the terminal encoder stage is a diagnostic assertion that the claimed size matches the actual written size — use it in tests or debug builds.

The cluster-ha-orderbook sample demonstrates this pattern in the context of a real Aeron Cluster publication loop.

Display / Debug

Diagnostic only — not a stable wire or log schema. Do not treat either format as a protocol or long-term log contract.

Display currently equals Debug for generated decoders ({car} and {car:?} print the same text). Prefer Debug in logs if you want that intent to stay obvious when/if the two diverge later.

#![allow(unused)]
fn main() {
pub fn demo_display_debug(valid_car: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
    let car = CarDecoder::try_from(valid_car)?;
    let display = format!("{car}");
    let debug = format!("{car:#?}");

    // Display is a compact one-liner; field names use the schema's camelCase.
    assert!(display.contains("serialNumber: 1234"));
    assert!(display.contains("modelYear: 2013"));
    assert!(display.contains("available: true"));
    assert!(display.contains(r#"code: A"#));
    assert!(display.contains("manufacturer: \"Honda\""));
    assert!(display.contains("model: \"Civic VTi\""));
    assert!(display.contains("fuelFigures: ["));
    assert!(display.contains("performanceFigures: ["));
    assert!(display.contains(r#"activationCode: "abcdef""#));

    // Pretty Debug ({:#?}) shows each field on its own line with indentation.
    for expected in &[
        "serialNumber: 1234",
        "modelYear: 2013",
        "available: true",
        r#"manufacturer: "Honda""#,
        r#"model: "Civic VTi""#,
        r#"activationCode: "abcdef""#,
        r#"usageDescription: Urban"#,
        "speed: 30",
        "mpg: 35.9",
        "octaneRating: 95",
    ] {
        assert!(
            debug.contains(expected),
            "Debug missing: {expected}\n--- full debug ---\n{debug}"
        );
    }

    // ── CarDomain DTO ───────────────────────────────────────────────────
    // Owned, heap-allocated, serialisable snapshot of the full message tree.
    // Generated by `with_domain_objects(DomainVarData::Strings)`.
    let dto = CarDomain::try_from_decoder(car)?;
    let dto_dbg = format!("{dto:#?}");
    // DTO field names use Rust snake_case (different from wire decoder camelCase).
    assert!(dto_dbg.contains("serial_number: 1234"));
    assert!(dto_dbg.contains("model_year: 2013"));
    assert!(dto_dbg.contains("available: true"));
    assert!(dto_dbg.contains("fuel_figures: ["));
    assert!(dto_dbg.contains(r#"usage_description: "Urban""#));
    assert!(dto_dbg.contains(r#"manufacturer: "Honda""#));
    assert!(dto_dbg.contains(r#"activation_code: "abcdef""#));

    Ok(())
}
}

(This code comes from the sbe-feature-tour sample crate.)

Real output from the feature-tour Car (demo_car_size_and_encode → CarDecoder):

CarDecoder { serialNumber: 1234, modelYear: 2013, available: true, code: A, fuelFigures: ["{ speed: 30, mpg: 35.9, usageDescription: Urban }", "{ speed: 60, mpg: 25.0, usageDescription: Highway }"], performanceFigures: ["{ octaneRating: 95, acceleration: [{ mph: 30, seconds: 4.0 }, { mph: 60, seconds: 7.5 }] }"], manufacturer: "Honda", model: "Civic VTi", activationCode: "abcdef" }

Truncated / incomplete buffers omit missing tails rather than panicking.

Schema Descriptions → Rustdoc

<field name="serialNumber" id="1" type="uint64" description="VIN-style serial"/>

// Generated (approx):
/// VIN-style serial
pub fn serial_number(&self) -> u64 { … }

Provenance of all four XML doc sources: schema_docs_provenance_test.

Domain DTOs

Use when you want owned values (Vec groups, owned tails) and simple structs — not the zero-copy hot path. Flyweights stay faster for low-latency applications.

For re-encode, eligible flat groups are bulk-written directly from &[EntryDomain]: no temporary Vec<Entry> and no encode-time allocation. Eligibility requires fixed-size entries whose domain fields have the same wire representation; nested groups, var-data, optional/versioned fields, configured domain conversions, and bool remapping use the general add path. Integer min/max checks are preserved in both paths.

On the audited Apple M4 1,000-entry fixture, automatic DTO bulk encode measured 509 ns versus 1.336 µs for the exact previous per-entry path with LTO, and 509 ns versus 1.998 µs without LTO. This is a DTO-to-DTO diagnostic, not an ergon/sbe-tool fairness ratio.

#![allow(unused)]
fn main() {
// build.rs — DomainVarData picks the DTO field type for var-data:
// .with_domain_objects(DomainVarData::Strings) // String (invalid UTF-8 → InvalidUtf8 error)
// .with_domain_objects(DomainVarData::Bytes)        // Vec<u8> (byte-exact)
}

Generated shape (illustrative — your names follow your schema):

pub struct QuoteDomain {
    pub seq: u32,
    pub some_numbers: [u32; 4],
    pub vehicle_code: [u8; 6],
    pub qty: u32,
    pub legs: Vec<QuoteLegsEntryDomain>,
    pub note: Vec<u8>,              // Bytes|String per DomainVarData
}
impl QuoteDomain {
    // Named methods, not TryFrom/From: two fallible sources (decoder vs framed
    // slice+offset), and materialisation is never infallible.
    pub fn try_from_decoder(dec: QuoteDecoder<'_>) -> Result<Self, DecodeError>;
    pub fn try_from_slice_with_header(buf: &[u8], offset: usize) -> Result<Self, DecodeError>;
    pub fn encode(&self, buf: &mut [u8]) -> Result<usize, EncodeError>;
    pub fn encoded_length_with_header(&self) -> Result<usize, EncodeError>;
}

Wire → DTO → wire round-trip (the docs fixture uses DomainVarData::Bytes):

#![allow(unused)]
fn main() {
  // Encode a message first (the usual flyweight path)
  let mut buf = [0u8; QuoteEncoder::compute_length_with_header(1, 2)];
  let len = QuoteEncoder::wrap_and_apply_header(&mut buf, 0)
      .fixed(&QuoteFixedFields {
          seq: 1,
          some_numbers: [1, 2, 3, 4],
          vehicle_code: *b"ABCDEF",
          qty: 10,
      })
      .legs(1, |legs| {
          legs.add(|leg| { leg.value(99); Ok(()) })?;
          Ok(())
      })?
      .note(b"hi")?
      .encoded_length_with_header();
  // Decode → owned DTO (allocates — not for the hot path)
  let dec = QuoteDecoder::try_from(&buf[..len])?;
  let mut dto = QuoteDomain::try_from_decoder(dec)?;
  assert_eq!(dto.seq, 1);
  assert_eq!(&dto.note, b"hi");
  dto.qty = 500;
  // Re-encode (integer min/max checked; eligible groups use bulk write)
  let n = dto.encode(&mut buf)?;
  assert_eq!(n, len);
}

with_domain_objects(DomainVarData)

SBE <data> is length-prefixed bytes. The enum picks the DTO field type:

Strings rejects invalid UTF-8. Materialise returns InvalidUtf8 for bad bytes; there is no silent empty-string fallback. Use Bytes (or stay on flyweights) when you need audit / replay fidelity of non-UTF-8 tails.

Runnable demo (text path): sbe-feature-tour uses DomainVarData::Strings. Flyweight path is unchanged: with schema characterEncoding="UTF-8" you still get into_manufacturer_as_str() without a DTO.

demo_car_domain_dto · domain_objects_test.

App Types on Composites

SBE composite types like Decimal { mantissa, exponent } have a fixed wire layout. Mapping them to Rust domain types is done at configuration time — no hand-rolled per-field converters needed.

Option A — generic converter (with_conversion): the generated codec emits price_from<T: TryToSbe<Decimal>>() and price_as<T: TryFromSbe<Decimal>>(). You implement the trait for any app type. One wire type, many app types.

Option B — concrete mapping (with_domain_type): the generated codec emits try_price(rust_decimal::Decimal)? and try_price()? -> rust_decimal::Decimal. Exactly one Rust type per wire type. The trait impls are generated for you.

Full comparison with code examples: with_conversion vs with_domain_type.

Optional composites with a null image

A composite member itself can carry presence="optional" with a schema nullValue — e.g. a PriceNull9-style Decimal whose mantissa is optional while exponent stays constant. That member decodes as Option<i64> (checked against the wire null sentinel), not a bare i64. This is distinct from a composite field gated by sinceVersion (which the whole composite accessor wraps in Option<Decoder>), and from a composite with no nullValue anywhere (which has no null image to check, so it decodes as a plain value — see with_conversion vs with_domain_type).

Because the wire null sentinel is not a valid rust_decimal::Decimal, the with_domain_type accessor fails closed with a typed error rather than silently decoding the sentinel as a huge/wrong number:

        use rust_decimal::Decimal;

        // `price: None` writes the schema null image (mantissa = nullValue).
        let mut buf = [0u8; QuoteEncoder::compute_length_with_header()];
        let len = QuoteEncoder::wrap_and_apply_header(&mut buf, 0)
            .fixed(&QuoteFixedFields { price: None, qty: 7 })
            .encoded_length_with_header();

        let dec = QuoteDecoder::try_decode(&buf[..len], 0)?;
        assert_eq!(dec.price_value().mantissa(), None);
        assert!(dec.try_price().is_err(), "null mantissa is not a valid Decimal");

        // A real price round-trips through the null-aware member accessor.
        let len2 = QuoteEncoder::wrap_and_apply_header(&mut buf, 0)
            .fixed(&QuoteFixedFields { price: None, qty: 7 })
            .try_price(Decimal::new(12345, 9))?
            .encoded_length_with_header();
        let dec2 = QuoteDecoder::try_decode(&buf[..len2], 0)?;
        assert_eq!(dec2.price_value().mantissa(), Some(12345));
        assert_eq!(dec2.try_price()?, Decimal::new(12345, 9));

(From optional_composite_member_null_image_roundtrip in sbe/tests/baseline_test.rs — a real generated-and-compiled test, not a standalone snippet.)

Timestamp Conversions

SBE represents timestamps as uint64 wire fields with a semanticType attribute (UTCTimestamp = nanoseconds, UTCTimestampMicros = microseconds, UTCTimestampMillis = milliseconds). The chrono feature (see Feature Integrations) converts these to chrono::DateTime<Utc> and chrono::NaiveDateTime with one line of config.

Nanoseconds and microseconds — built-in

Enable the chrono feature in your Cargo.toml:

[dependencies]
ergo-sbe = { version = "0.1", features = ["chrono"] }
chrono = "0.4"

Then register the converters by semanticType in build.rs:

use ergo_sbe::{ConversionSelector, GenerationConfig};

let config = GenerationConfig::new("msgs")
    .with_domain_type(
        ConversionSelector::semantic_type("UTCTimestamp"),
        "chrono::DateTime<chrono::Utc>",
    )
    .with_domain_type(
        ConversionSelector::semantic_type("UTCTimestampMicros"),
        "chrono::NaiveDateTime",
    );

Schema:

<field name="created_at"  id="1" type="uint64" semanticType="UTCTimestamp"/>
<field name="updated_at"  id="2" type="uint64" semanticType="UTCTimestampMicros"/>

Generated API — try_created_at() returns DateTime<Utc>, try_updated_at() returns NaiveDateTime:

// Decode
let created: chrono::DateTime<chrono::Utc> = dec.try_created_at()?;
let updated: chrono::NaiveDateTime = dec.try_updated_at()?;

// Encode
enc.try_created_at(chrono::Utc::now())?;
enc.try_updated_at(chrono::NaiveDateTime::from_timestamp_micros(1_720_000_000_000_000).unwrap())?;

Conversion cost: 2.8 ns (nanos → DateTime), 5.5 ns (micros → NaiveDateTime). See the measured benchmarks.

One selector, many fields

ConversionSelector::semantic_type(..) matches every field in the schema carrying that semanticType — not just one. You register the conversion once, not once per field. A schema with three separate UTCTimestamp timestamps needs no more config than one with a single field:

<field name="createdAt"  id="1" type="uint64" semanticType="UTCTimestamp"/>
<field name="updatedAt"  id="2" type="uint64" semanticType="UTCTimestamp"/>
<field name="expiresAt"  id="3" type="uint64" semanticType="UTCTimestamp"/>

// Same one-time call as the two-field example above — no per-field repeats.
let config = GenerationConfig::new("msgs")
    .with_domain_type(
        ConversionSelector::semantic_type("UTCTimestamp"),
        "chrono::DateTime<chrono::Utc>",
    );

All three fields get their own concrete accessor, generated from that one call:

enc.try_created_at(chrono::Utc::now())?;
enc.try_updated_at(chrono::Utc::now())?;
enc.try_expires_at(chrono::Utc::now())?;

let created: chrono::DateTime<chrono::Utc> = dec.try_created_at()?;
let updated: chrono::DateTime<chrono::Utc> = dec.try_updated_at()?;
let expires: chrono::DateTime<chrono::Utc> = dec.try_expires_at()?;

This is why UTCTimestamp and UTCTimestampMicros need separate with_domain_type calls in the two-field example above — they're different semanticType strings, so they're different selectors — but adding a fourth UTCTimestamp field to the same schema needs no config change at all. ConversionSelector::field_path is the escape hatch when one specific field needs to differ from its semantic-type siblings.

Mixed precisions, one app type — DomainImpl::Manual

The built-in converters give nanos → DateTime<Utc> and micros → NaiveDateTime — two different app types, because that's what the built-in impls happen to produce. Real schemas often need the opposite: three fields at three different wire precisions, all normalized to the same DateTime<Utc> so downstream app code never branches on precision. None of the three is the bare uint64 + semanticType="UTCTimestamp" shape the built-in converter matches, so none gets an auto-generated impl — that's exactly the case DomainImpl::Manual is for: concrete try_* signatures from ergo-sbe, conversion body from you.

Distinguish the three precisions with single-element composites (a distinct Rust type per precision — TimestampMillis(u64) is not TimestampNanos(u64) even though the wire shape is identical):

<composite name="TimestampMillis"><type name="ts" primitiveType="uint64"/></composite>
<composite name="TimestampMicros"><type name="ts" primitiveType="uint64"/></composite>
<composite name="TimestampNanos"><type name="ts" primitiveType="uint64"/></composite>

<field name="createdAt" id="1" type="TimestampNanos"/>
<field name="updatedAt" id="2" type="TimestampMicros"/>
<field name="deletedAt" id="3" type="TimestampMillis"/>

One with_manual_domain_type(selector, path) call per composite — same target type, rust_decimal-style — in build.rs:

use ergo_sbe::{ConversionSelector, GenerationConfig};

let config = GenerationConfig::new("msgs")
    .with_manual_domain_type(
        ConversionSelector::named_type("TimestampNanos"),
        "chrono::DateTime<chrono::Utc>",
    )
    .with_manual_domain_type(
        ConversionSelector::named_type("TimestampMicros"),
        "chrono::DateTime<chrono::Utc>",
    )
    .with_manual_domain_type(
        ConversionSelector::named_type("TimestampMillis"),
        "chrono::DateTime<chrono::Utc>",
    );

Each composite is a genuinely new named type, so there is no built-in template to offer — write the three impls yourself, one per precision (the traits live in the generated module as sbe_rt::TryFromSbe / sbe_rt::TryToSbe; import from there, not from ergo_sbe::codegen, which is crate-private):

// my_msgs is the module name passed to GenerationConfig::new("my_msgs")
use my_msgs::sbe_rt::{TryFromSbe, TryToSbe};
use my_msgs::{TimestampMillis, TimestampMicros, TimestampNanos};

impl TryFromSbe<TimestampNanos> for chrono::DateTime<chrono::Utc> {
    type Error = &'static str;
    fn try_from_sbe(wire: TimestampNanos) -> Result<Self, Self::Error> {
        let ns = wire.0; // single-element composite is a transparent wrapper
        chrono::DateTime::from_timestamp((ns / 1_000_000_000) as i64, (ns % 1_000_000_000) as u32)
            .ok_or("timestamp out of range")
    }
}
impl TryToSbe<TimestampNanos> for chrono::DateTime<chrono::Utc> {
    type Error = &'static str;
    fn try_to_sbe(&self) -> Result<TimestampNanos, Self::Error> {
        Ok(TimestampNanos(self.timestamp_nanos_opt().ok_or("overflow")? as u64))
    }
}

impl TryFromSbe<TimestampMicros> for chrono::DateTime<chrono::Utc> {
    type Error = &'static str;
    fn try_from_sbe(wire: TimestampMicros) -> Result<Self, Self::Error> {
        let us = wire.0;
        chrono::DateTime::from_timestamp((us / 1_000_000) as i64, ((us % 1_000_000) * 1_000) as u32)
            .ok_or("timestamp out of range")
    }
}
impl TryToSbe<TimestampMicros> for chrono::DateTime<chrono::Utc> {
    type Error = &'static str;
    fn try_to_sbe(&self) -> Result<TimestampMicros, Self::Error> {
        Ok(TimestampMicros(self.timestamp_micros() as u64))
    }
}

impl TryFromSbe<TimestampMillis> for chrono::DateTime<chrono::Utc> {
    type Error = &'static str;
    fn try_from_sbe(wire: TimestampMillis) -> Result<Self, Self::Error> {
        let ms = wire.0;
        chrono::DateTime::from_timestamp((ms / 1000) as i64, ((ms % 1000) * 1_000_000) as u32)
            .ok_or("timestamp out of range")
    }
}
impl TryToSbe<TimestampMillis> for chrono::DateTime<chrono::Utc> {
    type Error = &'static str;
    fn try_to_sbe(&self) -> Result<TimestampMillis, Self::Error> {
        Ok(TimestampMillis(self.timestamp_millis() as u64))
    }
}

All three fields now return the exact same app type despite three different wire precisions — the caller never has to know or care which precision a given field was wire-encoded at:

let created: chrono::DateTime<chrono::Utc> = dec.try_created_at()?;
let updated: chrono::DateTime<chrono::Utc> = dec.try_updated_at()?;
let deleted: chrono::DateTime<chrono::Utc> = dec.try_deleted_at()?;

enc.try_created_at(chrono::Utc::now())?;
enc.try_updated_at(chrono::Utc::now())?;
enc.try_deleted_at(chrono::Utc::now())?;

If you forget one of the three impls, the compile error names it directly — `chrono::DateTime<Utc>` has no `TryFromSbe<TimestampMillis>` impl — instead of the default trait-bound message.

Design Notes

Rationale and trade-off analysis behind specific ergo-sbe decisions.

• Type-state is zero-cost — named stages + header marker; why benches show no tax
• API freeze decisions — wrap offset, FixedFields, _unchecked, stage names
• Why NullVal Instead of Option — how missing fields work on the wire
• Feature Matrix — capability comparison across SBE generators

Type-state is zero-cost (and the hybrid design)

The question evaluators often ask

Did compile-time wire-order enforcement cost anything on the hot path?

No. Named stage structs and marker generics are zero-sized compile-time constructs. Every transition is a move of the same three runtime fields:

(buf, msg_offset, pos)  +  PhantomData / zero-sized stage identity

There is no heap allocation, no vtable, no enum discriminant on the wire path, and no extra branch for “which stage am I in?” — the stage is in the type, so the methods that exist are exactly the ones legal at that point in the schema. Generated machine code is identical in shape to a single-struct encoder with the same field writes.

Benchmarks that show “no difference vs a single struct / vs sbe-tool at the 1.00 ceiling” are therefore the expected proof that the abstraction is zero-cost — not a lucky accident and not a reason to doubt the design. If a type-state transition ever showed up as a measurable cost under a fair, amplified, dual-LTO comparison, that would be a codegen defect.

All maintained SBE parity scenarios pass at or below the strict 1.00× sbe-tool ceiling under both LTO-on and LTO-off profiles. Methodology and ceilings: Benchmarks.

Type-state = multiple named structs

“Type-state” and “multiple different structs” are not alternatives. Named stages (CarEncoder → CarAfterFuelFigures → … → CarComplete) are the type-state pattern. The other spelling is a single generic Encoder<'a, Stage> with phantom stage markers. Both compile the same way; only the API surface differs.

Why the hybrid (named stages + one header marker)

Duplicating every stage for header mode would provide no latency advantage and would double the generated type count.

What users see

// Approximate generated shape — not Encoder<AfterBids>:
pub struct BookEncoder<'a, H: HeaderState = HeaderPresent, F: FieldsState = FieldsUnfixed> {
    /* buf, msg_offset, pos + ZST markers */
}
pub struct BookAfterBids<'a, H: HeaderState = HeaderPresent> { /* same layout */ }
pub struct BookAfterAsks<'a, H: HeaderState = HeaderPresent> { /* same layout */ }

impl BookEncoder<'a, H, FieldsFixed> {
    pub fn bids(self, …) -> Result<BookAfterBids<'a>, …> { … }
    // no asks() — bids first on the wire
}

F is why wrap* cannot publish as_bytes_with_header until fixed(&FixedFields) has written the required body. Tail stages drop F — they are already past the fixed block.

Calling stages out of order is a type error. See Wire order via named stages for the product rationale (bids/asks inversion) and Coming from sbe-tool for the migration mapping (.parent() hopscotch → closures + stages).

API freeze note

Stage names use After{GroupPascal} (e.g. fuelFigures → CarAfterFuelFigures). Multi-word group names are PascalCased the same way as other generated types. Reserved method names on decoder/encoder stages are covered by reserved_name_clash_test so field collisions rename accessors without shadowing stage transition methods.

API freeze decisions (pre-1.0)

Deliberate decisions on the generated public surface. Changing any of these after 1.0 is a major version. Golden file sbe/tests/golden/car_example.rs is the artifact for API shape review.

1. wrap takes message start (not body offset)

Decision: keep ergon semantics. One offset works for encode wrap, wrap_and_apply_header, and claim buffers. sbe-tool refugees who pass 8 for a frame at zero will mis-align every field — that is the #1 migration trap. Loud rustdoc on every generated wrap / wrap_and_apply_header / decode documents this; the book chapter Coming from sbe-tool is the full mapping.

2. *FixedFields is intentionally exhaustive

Generated CarFixedFields (and peers) are not #[non_exhaustive].

Decision: exhaustive is a feature. When the schema adds a fixed field, every fixed(&…) call site must update. Silent Default / ignored new fields would hide schema drift. Do not “fix” this with #[non_exhaustive] without a major-version design review.

3. Stage struct naming

Pattern: {Message}After{GroupPascal} for intermediate stages, {Message}Complete for the terminal encoder stage; decoder stages use {Message}DecoderAfter{…} similarly. Multi-word group names go through the same PascalCase path as other types (fuelFigures → FuelFigures → CarAfterFuelFigures). Reserved-name clash coverage: sbe/tests/reserved_name_clash_test.rs.

Decision: keep named monomorphic stages (not Encoder<State = AfterBids>). Rationale: Type-state design note.

4. _unchecked companions are a supported opt-in

Decision: supported production opt-in after a proven trust boundary — not “benchmarking only” framing.

• Default generation exposes the three-tier constructor boundary: try_* (Result), bare names (panic after extent proof), and unsafe fn *_unchecked.
• Safety contract: validate with try_decode / try_from / try_wrap / verify at trust edges; bare constructors also prove fixed extent before returning; only *_unchecked may skip that proof (UB if wrong).
• Hot loops after validation are an intended use case for the unchecked lane. Checked constructors remain the default for untrusted input.

See Trust boundaries.

5. Header marker default

H: HeaderState = HeaderPresent on encoder stages so the common full-frame path needs no turbofish. Body-only encoding uses wrap / HeaderAbsent explicitly.

6. No renames bundled with this note

This audit records decisions; it does not rename public generated types.

7. #[non_exhaustive] policy for generated structs

Decision: keep generated consumer-facing structs non-exhaustive via pub(crate) fields rather than #[non_exhaustive]. A downstream crate cannot construct one directly, so adding a field is not a breaking change. #[non_exhaustive] is reserved for public enums that will gain variants over time (GenerateError, ConnectStep). Any future rename lands in one release with CHANGELOG entries.

Why NullVal Instead of Option

An SBE enum may declare a nullValue in the schema — an explicit wire sentinel that means "not present" / "not set". When the schema doesn't specify one, SBE defaults to the encoding type's null sentinel: the maximum value for unsigned types (e.g. 255 for uint8) and the minimum value for signed types (e.g. -128 for int8).

nullValue / minValue / maxValue on a type or field must fit the declared primitive width. nullValue="256" on uint8 is a parse error — it would otherwise collapse to 0 on the wire and make Some(0) indistinguishable from None. See Error Diagnostics.

An early design tried wrapping every enum field in Option<EventCode> at the field site:

// Option approach — REJECTED
pub fn event_code(&self) -> Option<EventCode> { … }
pub fn set_event_code(&mut self, val: Option<EventCode>) { … }

This was rejected for three reasons:

1. API complexity. Using Option<EventCode> at every access point forces every consumer to .unwrap() or match, even when the field is known to be populated. The NullVal approach gives you a plain EventCode type — if you care about null, check code == EventCode::NullVal; if you don't, just use it. (The wire encoding itself would be compatible either way: None maps to the null sentinel, Some(v) maps to v. The issue is ergonomics, not wire format.)

2. Generated code complexity. Every field site that uses Option<EventCode> needs value↔Option mapping in both accessor directions, inflating the generated code for no wire-format gain.

3. Schema intent. The schema declares a null sentinel as part of the enum's own value domain, not as a separate presence flag. A NullVal variant reflects that intent directly in the Rust type.

The chosen design adds a NullVal variant to every generated enum. It is the same size as any other variant, wire-compatible with sbe-tool, and bears no runtime cost:

// ergo-sbe generated (conceptual)
pub enum EventCode {
    NullVal = 255,  // or schema-declared nullValue
    Ok = 200,
    Error = 400,
    Timeout = 408,
}

For an Optional field (schema presence="optional"), the generated accessor returns Option<EventCode> — but the null check compares against the NullVal discriminant on the wire, never allocates, and is transparent to the caller.

On encode, fixed(&FixedFields) always writes every fixed field: Some(v) writes the value and None writes the exact schema null wire image (including nested optional composite members). Dirty buffer reuse is therefore safe when you go through fixed. Prefer that path for whole-message encodes.

apply_nulls() remains on the unfixed encoder after wrap_and_apply_header when you set individual optional fields piecemeal instead of using FixedFields. See Encode and Decode.

Opting into Option<T> with with_null_as_option

The NullVal is the right default, but some codebases prefer Option throughout. Use with_null_as_option to make generated enum accessors return Option<Enum> — NullVal maps to None, all other values to Some(v). Wire bytes are identical either way.

use ergo_sbe::{ConversionSelector, GenerationConfig};

// Individual enum → Option<Enum>
let config = GenerationConfig::new("msgs")
    .with_null_as_option(ConversionSelector::named_type("EventCode"));

// Every enum in the schema → Option<Enum>
let config = GenerationConfig::new("msgs")
    .with_all_enums_as_option();

Generated diff (individual setter):

// Default (NullVal)                         // with_null_as_option
pub fn code(&self) -> EventCode { … }   →   pub fn code(&self) -> Option<EventCode> { … }

The as_option() method is also generated on every enum for manual use: event_code.as_option() → Option<EventCode>.

Null-aware accessors on BooleanType

For BooleanType fields, ergon emits a _bool() accessor alongside the standard enum getter:

// Standard getter — returns the enum variant (raw wire discriminant).
pub fn available_wire(&self) -> BooleanType { … }

// Null-aware — rejects NullVal (returns Err); Ok(true/false) otherwise.
// Required fields → Result<bool, DecodeError>; optional → Option<bool>.
pub fn try_available_bool(&self) -> Result<bool, DecodeError> { … }

For enums and other types, the NullVal variant remains the default. with_null_as_option (above) is the opt-in Option<T> mapping; the wire encoding is identical either way.

Feature Matrix

Scannable map of capabilities. Use the More links for samples and tests.

Benchmark Results

Methodology, gate rules, and fairness policy: Benchmark Methodology.

SBE codec gate — just bench

Ratios are ergon / sbe-tool. Every maintained comparison has a strict 1.00 ceiling with zero tolerance for both SBE and cluster. The executable policy is in scripts/check-bench-gate.sh.

Do not copy point estimates into this file. Current results live in provenance-stamped artifacts under target/bench-runs/<run-id>/. Quote a result by naming its run id, commit, host, rustc, profile, and manifest hash — or do not quote it.

Prior cycle notes

• Previous decode results were invalid: sbe-tool direct decoders were wrapped at the header offset and read header bytes as body fields.
• Static fixture access was constant-foldable because only decoded results were black-boxed. The corrected suite uses std::hint::black_box on decoder references or input slices before access.
• Every encode case asserts byte equality; every decode case asserts fixed, group, nested-group, and var-data value equality before timing.
• Composite and full traversal now perform symmetric wrapper/header work.
• Public generated fixed/composite/set/enum setters, stage transitions, group iterators, var-data methods, and length builders now carry explicit inline intent. Before this fix, full no-LTO encode and decode lost to sbe-tool even though the LTO profile passed.
• sbe-tool performs well in both profiles. Its stable no-LTO performance is the reason LTO-off remains a required gate rather than a diagnostic.
• “Full message” now reads every encoded fixed/composite member before traversing every dynamic member. The prior dynamic-tail-only result was equal work between codecs but mislabeled.
• Header-only, body-only, and header-plus-body scalar encode are separate. The body-only setters are effectively tied on this run; the header-inclusive ratio is not presented as field-setter performance.
• Header work matches sbe-tool on both arms. Never pair ergon wrap_and_apply_header with sbe-tool body-only wrap. Cluster encode gates are body-only on both arms (wrap / wrap(…, 8), no MessageHeader write). Length asserts use body encoded_length() only — never a synthetic 8 + body that pretends a header was written.
• Buffers and inputs are allocated once outside b.iter; timed paths observe the encoded byte range.
• The maintained SBE and Cluster sources are also checked by fairness_policy_test: black_box, pre-timing body/wire parity, header-mode symmetry, sceptical/LTO disclosure.
• The gate uses Criterion's regression estimate consistently. A previous gate revision mixed the displayed regression result with the raw sample median; on a noisy run those estimators disagreed enough to reverse a tiny ratio.

Group encode: LTO on and off

sbe-tool performs consistently with and without LTO because its generated hot methods carry explicit inline intent. Before this correction, ergon's closure path was about 445 ns with LTO but 2.093 µs without LTO, while sbe-tool remained about 956 ns. The missing inline annotations were an ergon codegen defect, not an sbe-tool Option<parent> penalty.

After adding inline intent and fixing bulk_add:

Owned DTO encode is a separate diagnostic because it performs checked buffer entry and schema min/max validation for each domain field; presenting it as a direct sbe-tool ratio would be unequal work. The benchmark constructs the DTO and its Vec outside b.iter, then compares automatic domain bulk against the exact previous generated DTO path:

Both DTO arms perform the same range checks and checked entry, produce exact sbe-tool bytes before timing, reuse one exact-size buffer, and allocate nothing inside the timed encode. The allocation-count suite independently guards DTO encode.

For 1,000 Decimal-composite entries:

bulk_add now validates one exact output region and iterates chunks_exact_mut, eliminating the three inner field bounds checks retained by the removed implementation.

Maintained pair modes (fairness inventory)

Every gated ergon/sbe-tool pair uses the same header mode on both arms:

Diagnostics (encode_style, encode_bench, l2_book, group_decimal DTO arms, throughput/checked) are ergon-only or DTO-vs-DTO — not ergon/sbe-tool ratios.

Cluster codec gate — just bench-cluster

Five maintained scenarios are gated at the same literal 1.00 ceiling, with --run-id provenance:

• encode/session_message_header
• encode/session_keep_alive
• decode/session_message_header
• decode/session_event
• encode/claim_shaped_header_plus_app

Do not copy Criterion point estimates here. just bench-cluster stamps target/criterion / target/bench-no-lto/criterion and fails a stale tree.

Cluster encode arms locally assert exact sbe-tool byte parity before timing, use identical exact message lengths, make both mutable buffer inputs opaque, and reuse one pre-sized buffer per function (no iter_batched allocation). Decode arms locally assert the same scalar, enum, and var-data values before timing.

Layout access (diagnostic) — layout_access_bench

Not a ≤1.00 gate. Compares flyweight vs wire-image value vs #[repr(C, packed)] for a single mid-block field on a 256-byte composite (BigBlock, field f15). Field-only arms; no alloc on the timed path.

Conclusion: single-field access is one load for flyweight, preheld [u8; N] wire image, and packed overlay alike. Packing does not beat the wire-image design. Materialising the whole composite just to read one field is the expensive path. See Composite layout & little-endian.

cd sbe/benchmarks && cargo bench --bench layout_access_bench

Encode style (diagnostic) — encode_style_bench

Not a ≤1.00 gate. Confirms FixedFields vs setters, composite write, LE vs BE (body) on a LE host. Seeded/preheld values so work is not constant-folded away.

Conclusion: .fixed ≈ setters; preheld composite write ≈ build+write for a small engine once the rest of the fixed block is written; BE build is slightly slower than LE on an LE host; preheld memcpy is endian-independent. See README Encode — FixedFields vs setters….

cd sbe/benchmarks && cargo bench --bench encode_style_bench

Root cause of prior cluster encode regression (FIXED)

The two cluster encode scenarios (session_keep_alive, claim_shaped) previously failed at 1.19× and 1.28×. Root cause: generated field setters used self.buf[offset..offset+N].copy_from_slice(...), which re-checks bounds on every field write. After wrap/wrap_and_apply_header validates buf.len() >= BLOCK_LENGTH, field offsets are in-bounds by construction — the per-write bounds check was redundant.

Fix: field setters now use get_unchecked_mut after the trust boundary. This restored the encode paths to parity: session_keep_alive went from 1.19× slower to sub-1.00, and claim_shaped likewise.

SBE codec gate

just bench

This runs the parity benchmark from sbe/benchmarks and then evaluates Criterion output with scripts/check-bench-gate.sh.

Maintained cases cover representative decoder entry, fixed-field access, composites, complete-message traversal, fixed encoding, and batches. Each comparison must:

• use the same encoded input or produce byte-identical output;
• perform equivalent validation and field work;
• avoid measuring setup in only one arm;
• identify templates and schemas from codec contracts rather than stale literals;
• stay within the strict 1.00 per-scenario ceiling in scripts/check-bench-gate.sh.

A ceiling above 1.00 records a repeatable, fair sbe-tool win; it is not permission to add overhead. Changing a ceiling requires a fresh fairness audit and recorded measurements, not merely a failing gate.

Expanded codec matrix

The maintained ratio suite remains the generated ergo-sbe versus official sbe-tool comparison. The additive matrix is diagnostic and never uses IronSBE, rustysbe, handwritten offsets, or a custom wire format as an oracle.

just bench-diagnostics

codec_matrix_bench covers:

The timed encode paths reuse caller-owned buffers. Metadata lookup is the generated static (schema_id, template_id) match and is also protected by the allocation-count test suite.

Representative Apple M4 medians from the complete 2026-07-27 matrix run:

These numbers are diagnostic observations, not cross-machine thresholds.

Alignment experiment

alignment_bench exercises message offsets 0..=63 for ordinary stack arrays, reused Vec storage, and a #[repr(align(64))] test buffer. It exists to measure the effect, not to justify a mandatory aligned-buffer or pool API. SBE frames remain valid at arbitrary caller-selected offsets.

cargo bench -p ergo-sbe-benchmarks --bench alignment_bench

Apple M4 results on 2026-07-27 (Criterion median across each individual offset):

The aligned buffer did not improve the aggregate result, so this release adds no mandatory aligned-buffer or pooling API.

Amplified timing diagnostic (instruction_counts)

instruction_counts is an amplified Criterion timing harness. Each operation is repeated ACCESS_REPETITIONS times inside a single Criterion iteration to amplify sub-nanosecond differences. Its output is wall-clock, not instruction counts:

cargo bench -p ergo-sbe-benchmarks --bench instruction_counts

Instruction and disassembly evidence (perf-probe)

Deterministic mechanism-level evidence comes from named, #[inline(never)], unmangled probe symbols measured under raw Callgrind:

just bench-instructions                                      # both profiles
./scripts/run-sbe-instruction-probes.sh --all-profiles --topic decode

Each probe performs exactly 10,000 opaque logical operations and returns an observed checksum; setup and validation run before the probe is entered, so --toggle-collect=<symbol> excludes them. The driver normalises instructions, branches, and mispredicts per operation, disassembles the exact binary it measured, and records commit, rustc, target, Valgrind version, profile, run id, symbol, operation count, and checksum.

The lane needs Linux plus Valgrind and llvm-objdump, and fails closed elsewhere rather than substituting a timing harness. After measurement it fails if any registered two-arm pair has ergon Ir/op above sbe-tool. There is no iai-callgrind dependency — it was removed for RUSTSEC-2026-0173.

Warmed latency distributions

HDR Histogram is reserved for warmed batches where timer resolution is meaningful. latency_distribution reports p50, p99, and p99.9 for batches of 1,000 decoded messages after warm-up. Per-field microbenchmarks continue to use Criterion regression estimates and confidence intervals.

Apple M4 results on 2026-07-27: p50 250 ns, p99 292 ns, p99.9 375 ns per warmed 1,000-message batch.

Cold paths and artifact sizes

cargo bench -p ergo-sbe-benchmarks --bench cold_path_bench
just bench-cold

The Criterion cold-path suite measures schema parse and parse-plus-codegen. The fresh-crate probe reports generated source bytes, generated-crate compile time, final binary bytes, and platform size sections when available.

Latest fresh probe on the Apple M4 host (2026-07-27, rustc 1.95.0):

Regression policy

• Local audits and dedicated stable runners keep the sbe-tool equal-work gate with a 1.00 ceiling for every maintained comparison under LTO and no LTO.
• Shared GitHub CI runs both profiles and publishes Criterion diagnostics, but does not use noisy wall-clock ratios as a merge gate. A suspicious shared-runner result triggers a stable-runner rerun and fairness review.
• A dedicated stable runner, when configured, must reject a hot-path Criterion regression-estimate increase above 3%, a normalised instruction-count regression above 2% from scripts/run-sbe-instruction-probes.sh, any new allocation, or a warmed batch/cluster p99 regression above 5%.
• Criterion's regression estimate and confidence interval are the maintained microbenchmark estimator. HDR p50/p99/p99.9 applies only to warmed batch and Aeron/cluster end-to-end measurements.

The expanded Criterion matrix, alignment, cold-path, and HDR suites were executed on 2026-07-27 with rustc 1.95.0. The instruction-probe lane needs Linux and Valgrind, so it does not run on a macOS development host; it fails closed there rather than reporting a substitute measurement. Machine-specific observations belong in CI artifacts or a release record; they are not portable API promises.

Cluster codec gate

just bench-cluster

The Cluster suite applies the same equal-work rules. Encode gates time body-only field writes on both arms (ergon wrap, sbe-tool wrap(…, 8), no MessageHeader). Connection, authentication, and leader-change operations are cold-path diagnostics unless a recipe explicitly marks them as maintained release gates.

Interpreting results

Criterion reports live under target/criterion/. Review the regression estimate and confidence interval, not a single noisy iteration or a different estimator selected after seeing the result. For a material generator change:

1. run on an otherwise idle machine;
2. record the commit, Rust toolchain, target, profile, and host;
3. confirm both arms execute the intended body;
4. repeat suspicious or borderline comparisons;
5. keep the change only if every maintained ratio stays within its reviewed ceiling.

Capture immutable numbers in a release artifact when a particular release needs a benchmark record; refresh the Latest run table after material hot-path work.

Benchmark-only APIs

The unsafe unchecked constructor lane exists for explicit comparison work. Application code should use checked generated entry points for untrusted buffers and reserve trusted-buffer methods for data whose complete bounds have already been established.

Benchmark Methodology

Benchmark review requested. Generated-codec benchmarking is notoriously difficult and easy to get wrong. Surprising results should be presumed to be benchmark defects until wire parity, equal work, optimizer opacity, sufficiently amplified timing, both LTO profiles, and optimized assembly/instruction counts agree. Please review the methodology and report mistakes; these tables are evidence under review, not unquestionable facts.

ergon's maintained benchmarks compare generated codecs with official sbe-tool output performing equivalent work. Results are machine- and toolchain-specific, so this repository documents the method and gate rather than retaining dated point estimates as release guarantees.

What the numbers actually measure

Most of the measured difference between ergo-sbe and sbe-tool comes down to bounds checking, not fundamental codegen quality. Minor variations in how headers are written or how bulk operations are laid out account for the rest. If you had to call wrap_and_apply_header (which validates template_id and schema_id) every time, ergo-sbe would be slower than sbe-tool — sbe-tool's wrap + header() does no such validation in release builds. The benchmarks therefore use infallible wrap / wrap_and_apply_header on both arms: equal work, equal trust assumptions.

The benchmark gate exists to prove that ergo-sbe is not slower than sbe-tool — not to claim it is faster. sbe-tool is the reference; the goal is parity.

Regression check: compare against your own previous release

The sbe-tool ceiling catches regressions against the reference, but it does not catch regressions against your own prior work. If ergon was 0.73× sbe-tool in 0.1.7 and 0.89× in 0.1.8, both pass the 1.00 ceiling — but you just got 22% slower. That is a blocking defect.

Every release must therefore compare two things:

1. Ratio vs sbe-tool — must stay ≤ 1.00.
2. Absolute ergon time vs the previous release — check out the prior tag in a worktree, run the same benchmarks, and diff the Criterion point estimates. A shift larger than the reported confidence interval requires investigation before publishing.

The second check found the msg_offset regression in 0.1.8: decode_entry_point went from 0.73× to 0.89×. The sbe-tool ratio still passed — only the self-comparison caught it.

Gate profiles

• no-LTO (CARGO_PROFILE_BENCH_LTO=false CARGO_PROFILE_BENCH_CODEGEN_UNITS=1) — the canonical hard gate. Every maintained comparison must stay ≤ 1.00×.
• LTO — informational only (soft warning). LTO ratios are sensitive to thermal/code-layout variance on shared hardware; a single high ratio should be re-run before investigation.

Scenarios

All 10 SBE and 5 cluster parity comparisons are documented in Benchmark Results. Each arm performs identical logical work: equal trust assumptions, pre-computed headers, matching field subsets, symmetrical black_box, and pre-timing byte/value assertions.

Results →

Cluster Client

ergo-aeron-cluster is an experimental Rust client for Aeron Cluster. It uses rusteron for transport and ergo-sbe-generated codecs for Aeron's Cluster protocol.

Hobby project. Do not use it as a production substitute for official Aeron Cluster client support.

Scope

The crate implements client-side operations:

• synchronous and poll-driven connection;
• ingress offer and explicit try_claim;
• regular and controlled egress polling;
• authentication challenges and credentials;
• keep-alives and close requests;
• session events, administrative responses, and leader changes.

The Java Aeron process still provides the media driver, archive, consensus module, clustered services, election, recovery, and operational tooling. This crate implements none of those server-side components.

Features and harness

Default features use the Rust client library only. The test-harness feature adds repository-only Java Cluster launch support and requires Java 17 or newer plus locally built Aeron artifacts:

just build-aeron-jars
just test-aeron-cluster-harness

The harness, examples, integration tests, reference codecs, and benchmarks are excluded from the published crate package.

Verify the crate

cargo test -p ergo-aeron-cluster --lib
cargo clippy -p ergo-aeron-cluster --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc -p ergo-aeron-cluster --no-deps
cargo bench -p ergo-aeron-cluster --no-run
cargo package -p ergo-aeron-cluster --list --allow-dirty

Run just bench-cluster for the maintained codec comparisons. See sbe/BENCHMARKS.md for the common benchmark rules.

Limitations

The current limitations are documented in the Compatibility page, which also lists supported Aeron/rusteron versions, schema identities, failure modes, and the CI multi-node test matrix.

• Java interoperability depends on the local Aeron harness and environment.

Apache-2.0.

SessionBuilder

SessionBuilder is the supported configuration entry point:

use std::sync::Arc;
use std::time::Duration;
use ergo_aeron_cluster::{SessionBuilder, StaticCredentials};

fn main() -> Result<(), ergo_aeron_cluster::ClusterError> {
    let session = SessionBuilder::default()
        .ingress_channel("aeron:udp?endpoint=localhost:9010")?
        .egress_channel("aeron:udp?endpoint=localhost:9020")?
        .credentials(Arc::new(StaticCredentials::from_utf8("user:pass")))
        .message_timeout(Duration::from_secs(5))?;

    session.validate()?;
    let mut client = session.connect("/path/to/aeron-dir")?;
    client.offer(b"application payload")?;
    client.close()
}

Connection remains poll-driven internally; the crate does not require Tokio or another async runtime. Use connect_async when the application owns the poll loop.

Egress Listeners

Implement EgressListener and pass it through EgressAdapter to AeronCluster::poll_egress. Use ControlledEgressListener and ControlledEgressAdapter when callbacks must return Aeron controlled-poll actions.

Protocol errors, listener panics, keep-alive failures, publication failures, and reconnect failures are returned as ClusterError. Application payloads, credentials, challenges, and binary response data remain byte slices. Text fields declared by the protocol are validated before being exposed as &str.

The high-level client, configuration, listener, state, error, offer, and claim types are the consumer-facing surface. The generated protocol codecs are also reachable, for advanced direct encode/decode, through the cluster_codec_types module — but that module is #[doc(hidden)] and not a stable API: it exists for repository tests and low-level experimentation, and its shape may change without a semver bump. Normal applications should use AeronCluster.

Multi-Message & Framing

Ergon supports two framing approaches for adjacent messages, and an AnyMessage dispatch enum for multi-message streams where the next type isn't known until runtime.

Two framing approaches

1. Back-to-back with encoded length

Pre-compute each message's exact size, lay them out at known offsets, and validate after encoding. Safest when you know all messages ahead of time.

// Size every message first (both const).
let len_a = MsgAEncoder::compute_length_with_header();
let len_b = MsgBEncoder::compute_length_with_header(data_b.len());

let mut buf = vec![0u8; len_a + len_b];

// Encode MsgA at offset 0.
let a_len = MsgAEncoder::wrap_and_apply_header(&mut buf[..len_a], 0)
    .fixed(&fields_a)
    .data(data_a)?
    .encoded_length_with_header();
assert_eq!(a_len, len_a);

// Encode MsgB at offset len_a.
let b_len = MsgBEncoder::wrap_and_apply_header(&mut buf[len_a..], 0)
    .fixed(&fields_b)
    .data(data_b)?
    .encoded_length_with_header();
assert_eq!(b_len, len_b);

// Wire frame: two self-describing SBE messages back-to-back.
let wire = &buf[..len_a + len_b];

2. Stream / remaining() slot

Write sequentially; use remaining() to find where the next message starts. Idiomatic for Aeron cluster sessions where a SessionMessageHeader is immediately followed by application payload.

use ergo_aeron_cluster::cluster_codec_types::*;

let mut buf = [0u8; SessionMessageHeaderEncoder::ENCODED_LENGTH
    + SessionKeepAliveEncoder::ENCODED_LENGTH];

// Encode the outer message. `fixed()` writes the required body so a reused
// buffer cannot publish leftover bytes.
let enc = SessionMessageHeaderEncoder::wrap_and_apply_header(&mut buf, 0)
    .fixed(&SessionMessageHeaderFixedFields {
        leadership_term_id: 7,
        cluster_session_id: 99,
        timestamp: 42,
    });

// into_remaining_mut() returns the unwritten tail.
SessionKeepAliveEncoder::wrap_and_apply_header(enc.into_remaining_mut(), 0)
    .fixed(&SessionKeepAliveFixedFields {
        leadership_term_id: 7,
        cluster_session_id: 99,
    });

// Decode: remaining() gives bytes after the first message.
let smh = SessionMessageHeaderDecoder::decode(&buf, 0)?;
let tail = smh.get_metadata().remaining();
assert_eq!(tail.len(), SessionKeepAliveEncoder::ENCODED_LENGTH);

AnyMessage dispatch

Cluster sessions multiplex many message types on a single stream. AnyMessage::decode reads the 8-byte SBE header, inspects the template ID, and returns the matching variant:

use ergo_aeron_cluster::cluster_codec_types::*;

fn dispatch(data: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
    match AnyMessage::decode(data, 0)? {
        AnyMessage::SessionMessageHeader(decoder) => {
            // This wraps application payload. Use remaining() to get
            // the bytes after the 32-byte header, then decode again.
            let payload = decoder.get_metadata().remaining();
            if !payload.is_empty() {
                dispatch(payload)?;
            }
        }
        AnyMessage::SessionEvent(decoder) => {
            let code = decoder.code();
            let (detail, _) = decoder.into_detail_as_str()?;
            println!("event {code}: {detail}");
        }
        AnyMessage::NewLeaderEvent(decoder) => {
            let (endpoints, _) = decoder.into_ingress_endpoints_as_str()?;
            println!("new leader at {endpoints}");
        }
        AnyMessage::Challenge(decoder) => {
            let (chal, _) = decoder.into_encoded_challenge()?;
            // respond to challenge...
        }
        AnyMessage::AdminResponse(decoder) => {
            let (msg, after) = decoder.into_message()?;
            let (payload, _) = after.into_payload()?;
            println!("admin response: {msg:?}");
        }
        AnyMessage::SessionKeepAlive(decoder) => {
            // heartbeat — nothing to do
        }
        AnyMessage::Unknown { .. } => {
            // Not an error — the cluster may send messages
            // not in our schema. Skip them.
        }
    }
    Ok(())
}

AnyMessage::decode validates only the 8-byte SBE frame header. Always guard truncated payloads before slicing — e.g. check data.len() >= SessionMessageHeaderEncoder::ENCODED_LENGTH before calling remaining().

Metadata

Every decoder exposes get_metadata() which returns a Metadata struct:

remaining() is the key for chaining — it gives you the exact tail slice where the next message begins, zero-copy.

Cluster Compatibility

Supported stacks

Schema identities

These are pinned in cluster/schemas/ and generated by cluster/build.rs. The generated codec API is unstable — use the high-level AeronCluster client, not the cluster_codec_types module directly.

Failure modes

The client handles these documented transitions:

Multi-node test matrix

CI (cluster-compatibility.yml) covers:

• 3-node cluster: leader loss + redirect
• Auth challenge/rejection
• Fragmentation above max payload
• Controlled abort/retry
• Close + reconnect atomicity
• Timeout expiry (sync + async)

Limitations

• No dynamic membership (static cluster only).
• No snapshot/recovery integration.
• No TLS/mTLS at the Aeron transport layer.
• The cluster_codec_types module is #[doc(hidden)] and not stable API.

Teaching Path

Standalone crates that exercise repository APIs. They are excluded from the workspace, set publish = false, and are not production reference implementations — they move with experimental APIs on purpose.

Start here (product teaching path)

# 1 — always start here
cargo run  --manifest-path samples/sbe-feature-tour/Cargo.toml

# 2 — pick the conversion style you want in product code
cargo run  --manifest-path samples/l3-book/Cargo.toml
cargo test --manifest-path samples/exchange-example/Cargo.toml

Rule of thumb: one conversion style per schema type (with_domain_type or with_conversion, not both for the same selector).

Conversion: which sample uses what

Rule: one style per selector. with_domain_type already enables conversion; do not stack with_conversion on the same selector.

#![allow(unused)]
fn main() {
    use ergo_sbe::{ConversionSelector, GenerationConfig};
    // A — generic converter: one wire type, many app types
    let _cfg =
        GenerationConfig::new("msgs").with_conversion(ConversionSelector::named_type("Decimal"));
}

#![allow(unused)]
fn main() {
    use ergo_sbe::{ConversionSelector, GenerationConfig};
    // B — concrete mapping: one Rust type per wire type (already enables conversion)
    let _cfg = GenerationConfig::new("msgs").with_domain_type(
        ConversionSelector::named_type("Decimal"),
        "rust_decimal::Decimal",
    );
}

Rules

• Keep every sample outside the workspace and unpublished.
• Do not expose sample-only abstractions as product APIs.
• Size SBE buffers from generated encoded-length APIs (prefer stack when const).
• Propagate fallible operations with Result and ?.
• Delete a sample when it no longer exercises a distinct repository behavior.

SBE Feature Tour

Standalone laboratory sample for ergo-sbe (publish = false). This is the crates.io / docs.rs teaching entry.

Conversion: three styles in one crate

build.rs uses different APIs for different selectors:

#![allow(unused)]
fn main() {
    let generated_dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src/generated");
    let config = ergo_sbe::GenerationConfig::new("feature_tour")
        .with_domain_objects(ergo_sbe::DomainVarData::Strings)
        .with_domain_type(
            ergo_sbe::ConversionSelector::named_type("BooleanType"),
            "bool")
        .with_domain_type(
            ergo_sbe::ConversionSelector::semantic_type("UTCTimestamp"),
            "chrono::DateTime<chrono::Utc>")
        .with_conversion(ergo_sbe::ConversionSelector::named_type("Decimal"))
        // Same shape as Decimal, but the app supplies the impl itself — see
        // demo_domain_type_manual_impl in src/lib.rs.
        .with_manual_domain_type(
            ergo_sbe::ConversionSelector::named_type("ManualDecimal"),
            "rust_decimal::Decimal");

    ergo_sbe::generate_to_dir("schemas/feature-tour.xml", config, &generated_dir)?;
}

(The real build.rs — this code is compiled and tested in CI.)

The generated code is included via #[path = "generated/feature_tour.rs"] — no sbe_mod! needed. See Build Patterns.

Runnable proof for the Decimal row: demo_conversion_only in src/lib.rs (uses both rust_decimal and a tiny FixedPrice adapter on the same buffer). Runnable proof for the ManualDecimal row: demo_domain_type_manual_impl — same concrete try_manual_price(...)? signature DomainImpl::Generated would give you, but the impl TryFromSbe<ManualDecimal> / TryToSbe<ManualDecimal> above it are a literal copy-paste of the doc comment ergo-sbe put on the generated method (see with_conversion vs with_domain_type).

Quick rule

• One fixed app type, ergo-sbe writes the impl → with_domain_type(selector, path)
• One fixed app type, you write the impl (custom rounding/validation, or overriding the three built-ins) → with_manual_domain_type(selector, path)
• Pluggable / no forced dep → with_conversion
• Never call more than one of these for the same selector

Other samples:

Feature map → demo

Run

cargo run  --manifest-path samples/sbe-feature-tour/Cargo.toml
cargo test --manifest-path samples/sbe-feature-tour/Cargo.toml

After build, generated source is under src/generated/feature_tour.rs.

L3 Order Book

Deep nested / ragged L3 order-book sample for ergo-sbe. publish = false.

Conversion style: with_domain_type only

#![allow(unused)]
fn main() {
    use ergo_sbe::{ConversionSelector, GenerationConfig};
    // B — concrete mapping: one Rust type per wire type (already enables conversion)
    let _cfg = GenerationConfig::new("msgs").with_domain_type(
        ConversionSelector::named_type("Decimal"),
        "rust_decimal::Decimal",
    );
}

(From book/examples/conversion-config.rs. Full L3 build.rs: l3-book.)

Generated API (concrete):

enc.try_price(rust_decimal::Decimal::new(100, 0))?;
let p: rust_decimal::Decimal = dec.try_price()?;
let ts: DateTime<Utc> = dec.try_exchange_timestamp()?;

with_domain_type already enables conversion for that selector. Do not also call with_conversion(Decimal) here — it would not change the surface.

Layout

Run

cargo run  --manifest-path samples/l3-book/Cargo.toml
cargo test --manifest-path samples/l3-book/Cargo.toml

Exchange Example

Multi-schema exchange-shaped sample. with_conversion only for Decimal — no with_domain_type. publish = false.

Conversion style

#![allow(unused)]
fn main() {
    use ergo_sbe::{ConversionSelector, GenerationConfig};
    // A — generic converter: one wire type, many app types
    let _cfg =
        GenerationConfig::new("msgs").with_conversion(ConversionSelector::named_type("Decimal"));
}

(From book/examples/conversion-config.rs. Full multi-schema build.rs: exchange-example.)

That emits generic methods. The app supplies TryFromSbe / TryToSbe:

// encode: app Decimal → wire
e.price_from(&d)?;

// decode: wire → app Decimal
let d: rust_decimal::Decimal = level.price_as()?;

You still have wire accessors (price_value / price_wire). You do not get a concrete price() -> rust_decimal::Decimal — that requires with_domain_type(..., "rust_decimal::Decimal") (see L3 Order Book).

Run

cargo test --manifest-path samples/exchange-example/Cargo.toml

Codegen as Library

Generator-API examples — how to call ergo-sbe as a library to generate and inspect codecs programmatically (without a build.rs).

The other samples (l3-book, sbe-feature-tour, exchange-example) all use build.rs to generate codecs at compile time and then exercise them end-to-end. These examples complement them by showing the generator API itself — useful for tooling, golden-file workflows, and understanding what the generator produces.

Examples

# From the repository root — this crate is not a workspace member.
cargo run --manifest-path samples/sbe-codegen-examples/Cargo.toml --example flyweight
cargo run --manifest-path samples/sbe-codegen-examples/Cargo.toml --example domain_objects
cargo run --manifest-path samples/sbe-codegen-examples/Cargo.toml --example l3_nested
cargo run --manifest-path samples/sbe-codegen-examples/Cargo.toml --example dump_gen

What each shows

All examples parse the canonical car schema.

Cluster Tutorial

End-to-end walkthrough: launch a test cluster, connect a session, offer messages, poll events, send keep-alives, and close. The best starting point for the ergo-aeron-cluster client lifecycle.

Source: samples/cluster-tutorial/src/main.rs

See also: Cluster Client → Overview

just build-aeron-jars
cargo run --manifest-path samples/cluster-tutorial/Cargo.toml

Requires Java 17+ and built Aeron artifacts.

Cluster HA Orderbook

Claim-based Cluster publishing with an HA-shaped limit order book. Proves try_claim patterns and never-stale book snapshots under leader transitions.

• Offline pipeline (ha_offline_pipeline) — claim + encode + verify without a live cluster.
• Kill-leader test (ha_kill_leader) — validates book integrity across leader changes (requires Java harness).

Source: samples/cluster-ha-orderbook/src/

# Service-free
cargo test --manifest-path samples/cluster-ha-orderbook/Cargo.toml \
  --lib --test ha_offline_pipeline

# Full harness (needs Java)
just build-aeron-jars
cargo test --manifest-path samples/cluster-ha-orderbook/Cargo.toml \
  --features test-harness

Cluster RFQ

RFQ / auction protocol codecs with cluster-backed examples. Demonstrates e-sniping timer, auction state machine, and multi-participant message flows over Aeron Cluster.

• rfq_client — sends RFQ, receives quotes, places order.
• auction_client — auction lifecycle with e-sniping.
• rfq_roundtrip — encode/decode parity for all RFQ message types.

Source: samples/cluster-rfq/

cargo build --manifest-path samples/cluster-rfq/Cargo.toml --examples
cargo run --manifest-path samples/cluster-rfq/Cargo.toml --example rfq_roundtrip

Build Patterns

Generated codecs ship their own embedded sbe_rt module. Linking the app does not require ergo-sbe unless you use its macros or call the generator library at runtime.

Seeing generated code (without committing it)

include!(concat!(env!("OUT_DIR"), …)) and sbe_mod! put files under a hashed path like target/debug/build/<crate>-<hash>/out/….rs — hard to find and rust-analyzer usually cannot jump into them.

Samples instead write to a stable, local path:

samples/<name>/src/generated/*.rs   # created on cargo build, gitignored

1. cargo build --manifest-path samples/sbe-feature-tour/Cargo.toml
2. Open samples/sbe-feature-tour/src/generated/feature_tour.rs
3. From app code, Go to definition on CarEncoder / etc. should land there

Root .gitignore has **/src/generated/. Do not commit those trees (Binance alone is multi‑MB). Rebuild after a clean clone.

  // build.rs
  let out = Path::new(env!("CARGO_MANIFEST_DIR")).join("src/generated");
  ergo_sbe::generate_to_dir("schemas/messages.xml", config, &out)?;

  // src/lib.rs — real path → IDE go-to-definition works
  #[allow(dead_code, unused_imports, non_camel_case_types, non_snake_case, clippy::all, warnings)]
  #[path = "generated/messages.rs"]
  mod messages;

Buffer Sizing Guide

Every protocol buffer must be sized using the generated EncodedLength API. Never guess with vec![0u8; 4096].

Const-sized messages — stack array, no heap:

#![allow(unused)]
fn main() {
  let mut buf = [0u8; HeartbeatEncoder::compute_length_with_header()];
}

Dynamic / ragged messages — compute exact size first with *EncodedLength, then encode into a claim or slot of that exact length:

#![allow(unused)]
fn main() {
pub fn demo_car_size_and_encode() -> Result<Vec<u8>, Box<dyn std::error::Error>> {
    // Fuel: 2 entries with usage ASCII lengths 5 and 7.
    // Performance: 1 entry with 2 nested acceleration rows (fixed-only entries).
    // Message var-data: manufacturer / model / activationCode lengths.
    let complete_len = CarEncoder::compute_length()
        .fuel_figures_ragged(2, |ff| {
            ff.add()?.usage_description(5)?; // "Urban"
            ff.add()?.usage_description(7)?; // "Highway"
            Ok(())
        })?
        .performance_figures_ragged(1, |pf| {
            pf.add()?.acceleration(|acc| {
                acc.uniform(2)?;
                Ok(())
            })?;
            Ok(())
        })?
        .manufacturer(5)? // "Honda"
        .model(9)? // "Civic VTi"
        .activation_code(6)? // "abcdef"
        .encoded_length_with_header();

    // Exact size from compute_length → stack pad (this demo fits well under 512).
    const CAR_PAD: usize = 512;
    assert!(
        complete_len <= CAR_PAD,
        "sample car length {complete_len} exceeds stack pad {CAR_PAD}"
    );
    let mut storage = [0u8; CAR_PAD];
    let written = encode_sample_car(&mut storage[..complete_len])?;
    assert_eq!(
        written, complete_len,
        "CarEncodedLength must equal encoder-produced length"
    );
    Ok(storage[..written].to_vec())
}
/// Encode the canonical sample car into `buf` (must be pre-sized).
pub fn encode_sample_car(buf: &mut [u8]) -> Result<usize, sbe_rt::EncodeError> {
    let mut extras = OptionalExtras::default();
    extras.cruise_control(true).sports_pack(true);

    // Buffer pre-sized from EncodedLength; try_* still validates extent.
    let len = CarEncoder::try_wrap_and_apply_header(buf, 0)
        .unwrap()
        .fixed(&CarFixedFields {
            serial_number: 1234,
            model_year: 2013,
            available: true.into(),
            code: Model::A,
            some_numbers: [10, 20, 30, 40],
            vehicle_code: [b'A', b'B', b'C', b'D', b'E', b'F'],
            extras,
            engine: Engine::new(
                2000,
                4,
                [b'1', b'2', b'3'],
                0i8,
                false.into(),
                Booster::new(BoostType::TURBO, 210),
            ),
        })
        .fuel_figures(2, |g| {
            g.add(|mut e| {
                e.speed(30).mpg(35.9);
                e.usage_description(b"Urban")
            })?;
            g.add(|mut e| {
                e.speed(60).mpg(25.0);
                e.usage_description(b"Highway")
            })?;
            Ok(())
        })?
        .performance_figures(1, |g| {
            g.add(|mut e| {
                e.octane_rating(95);
                e.acceleration(2, |a| {
                    a.add(|x| {
                        x.mph(30).seconds(4.0);
                        Ok(())
                    })?;
                    a.add(|x| {
                        x.mph(60).seconds(7.5);
                        Ok(())
                    })
                })
            })?;
            Ok(())
        })?
        .manufacturer(b"Honda")?
        .model(b"Civic VTi")?
        .activation_code(b"abcdef")?
        .encoded_length_with_header();

    Ok(len)
}
}

(This code comes from the sbe-feature-tour sample crate.)

Key rules:

• compute_length_with_header() is const when the message has no var-data fields or groups — use it directly for stack array sizes.
• For messages with groups or var-data, use the staged *EncodedLength builder.
• Assert computed length equals actual encoded length after writing.
• Oversize vec![0u8; 4096] "guess" buffers hide size bugs — avoid them.

Contributing

ergon is experimental, but changes must still be reproducible, wire-compatible, and honest about what has been verified.

Work from behavior

For generator or protocol changes:

1. Add a focused behavioral, compile-fail, wire-parity, or allocation test.
2. Run it and confirm the expected failure.
3. Make the smallest coherent implementation change.
4. Run the focused test and the affected crate's complete checks.
5. Run the maintained benchmark gate when a generated or Cluster hot path changes.

Generated-source substring tests can supplement behavior tests, but do not prove wire correctness, type-state ordering, error propagation, or allocation behavior by themselves.

Product checks

just policy
just check-products
RUSTDOCFLAGS="-D warnings" cargo doc -p ergo-sbe --all-features --no-deps
RUSTDOCFLAGS="-D warnings" cargo doc -p ergo-aeron-cluster --no-deps

test-lanes.tsv assigns every tracked test-bearing Rust source to exactly one executable lane. just policy self-tests that enforcement and rejects ignored tests, ignored Rust fences, runtime SKIP reporting, test-selection bypasses, conditional test execution, failure-to-success wrappers, and custom skip-CI conditions.

A failure observed while changing the repository is not a pass because it appears pre-existing or unrelated. Reproduce and fix it, or stop the change with the failure recorded as a blocker. Never make the lane green by filtering, ignoring, conditionally bypassing, or merely logging the failed case.

just check adds repository hygiene and established sample checks. The complete required suite is:

just test

It builds the Aeron Java artifacts and runs the Java lifecycle/recovery and HA sample lanes. A missing dependency is a failure, not a passing partial run. Use just test-all to add Miri and deterministic fuzz replay.

Quality ratchets are explicit commands, not test-count targets:

just check-coverage
just check-mutation

Coverage runs on every pull request and may not fall below the checked-in region/function/line baseline. Mutation testing runs weekly over parser, resolver, sizing, and dynamic-tail code; missing or empty mutation output is a failure. Nightly CI runs every fuzz target for ten minutes and executes the LE/BE/nested fixture crate under Miri. Pull requests also execute codec library tests on 32-bit x86 and big-endian s390x through cross/QEMU.

Performance

Run the SBE parity gate after any change that can affect generated hot paths:

just bench

Run the Cluster gate after session codec, offer, claim, or egress hot-path changes:

just bench-cluster

Every maintained ergo-sbe/reference ratio must be at most 1.00 under equal-work inputs. Record fresh measurements instead of copying old benchmark numbers into documentation. Documentation-only changes do not require a benchmark run.

Shared GitHub runners execute both profiles and publish Criterion diagnostics, but noisy wall-clock ratios are not merge gates there. Run the strict ratio gate locally or on a dedicated stable benchmark runner.

fairness_policy_test mechanically requires the maintained SBE and Cluster parity suites to use std::hint::black_box, assert correctness before timing, and preserve the sceptical benchmark disclosure. Exact wire/value assertions remain part of each benchmark setup; a large result is presumed suspect until the benchmark is re-audited.

Error and example style

• Public library APIs use crate-specific typed errors.
• Tests and binaries may return Result<(), Box<dyn std::error::Error>>.
• Prefer ? to avoidable unwrap or expect.
• Treat schema-declared text strictly; keep binary fields as bytes.
• Size generated-message buffers from generated constants or length helpers.
• Use the high-level Cluster facade in consumer examples.

Documentation

• Keep the root README focused on repository orientation.
• Put crate usage in that crate's README.
• Keep one sample inventory in samples/README.md; do not add per-sample READMEs.
• Describe current, verified behavior. Do not commit dated implementation plans, completion ledgers, or archived task trees.
• Use Git history as the archive for superseded design material.
• Keep benchmark results out of permanent docs unless a release record needs an immutable, reproducible snapshot.

Package boundaries

Only ergo-sbe and ergo-aeron-cluster are publication candidates. Inspect their payloads before any release:

cargo package -p ergo-sbe --list --allow-dirty
cargo package -p ergo-aeron-cluster --list --allow-dirty

Packages must exclude repository integration tests, fixtures not required at build time, benchmarks, Java harness code, application protocols, samples, and internal plans.

Publishing, tagging, and announcing a release require explicit maintainer authorization.

Before an authorised release, run just release-check. The release workflow runs the same command before cargo release; it cannot substitute a partial workspace-only test command.

Git hygiene

• Preserve unrelated working-tree changes and dirty submodules.
• Stage paths explicitly; do not use git add -A.
• Do not rename the sbe, cluster, or samples directories.
• Use a short one-line conventional commit subject.

AI assistance: how ergo-sbe was built

ergo-sbe was developed with heavy AI assistance. Most of the implementation, tests, benchmarks, samples, and documentation were written by coding agents. I did very little direct coding.

I am saying that at the beginning because this is not a conventional project with a little autocomplete around the edges. If you do not want to use software written substantially by LLMs, stop here. This is probably not the right project for you.

That warning is not an apology, and this page is not marketing. It is a disclosure of:

• why I chose this project;
• what I knew before the models became involved;
• what I asked the models to do;
• what I did and did not review;
• how the generated codecs were verified;
• where the workflow worked and where it failed;
• which tools and models were used;
• what the work actually cost;
• why the crate is still labelled experimental; and
• what I would expect before using it in a financial system.

The short version is:

I supplied the SBE domain knowledge, requirements, API judgment, corrections, acceptance criteria, and release decisions. Coding agents supplied nearly all of the implementation. I reviewed the generated Rust API and output far more closely than I reviewed the syn/quote generator implementation. The observable output is constrained by extensive tests, live byte-for-byte comparisons with official sbe-tool Rust codecs, official fixtures, compile-fail proofs, allocation checks, and performance gates. That is strong evidence, but it is not a substitute for independent production use.

This account describes the initial intensive development period ending in July 2026. Prices, model names, repository state, and my process will change. Treat dated figures as a historical record, not a permanent promise.

Read this first

This article is long because “AI-assisted” is too vague to be useful on its own. If you only have a minute, these are the facts that should determine whether you continue evaluating the crate:

Cumulative token usage (since 2026-06-28)

Per-model high-water mark reconstructed from every ccusage snapshot committed to this file — 2026-08-03, 2026-08-07, 2026-08-13, 2026-08-15, 2026-08-19, and 2026-08-20. Claude Code and Codex agent usage only.

Notes:

• Why a high-water mark, not the latest snapshot. ccusage keeps no ledger: it recomputes totals by reading local session transcripts. Claude Code deleted transcripts older than cleanupPeriodDays (default 30 at the time), so a --since 2026-06-28 query lost its earliest days as time passed. Re-running it on 2026-08-13 reported 15.03B tokens against 18.13B on 2026-08-07 and 17.04B on 2026-08-03 — usage did not fall; visibility did. Because the tool can only ever under-report, the per-model maximum across all committed snapshots is the best available lower bound. Git history is the real ledger here: it preserves 4.54B tokens that ccusage could no longer see as of 2026-08-15 — that run reported 15.46B (14.31B Claude + 1.15B Codex) against a reconstructed 20.01B.
• The 2026-08-20 run reported ~15.09B Claude Code tokens and ~1.28B Codex tokens against this reconstructed 20.48B total — still retention, not a real decrease. No per-model column exceeded the existing high-water mark, so the table is unchanged. observer was not installed on this machine, same as 2026-08-19.
• The 2026-08-19 run reinforced the same pattern on a different model: gpt-5.6-luna fell from a reconstructed 48.7M tokens to 236K in this one snapshot — a >99% apparent drop from retention pruning, not reduced usage. Its transcripts are gone from this machine; the high-water figure is the only surviving record and is kept unchanged in the merged row above. This also falsifies the earlier note that "Codex keeps its own session logs under a different policy" and only Claude-Code-hosted models lose history — Codex transcripts expire too, just on their own schedule per model/session.
• Two models exist in only one snapshot, and would be lost by any single run: claude-haiku-4-5-20251001 (2.36M tokens, 2026-08-03 only — its transcripts have since been deleted) and gpt-5.6-luna (48.7M high-water, per above).
• Corroborating evidence for the retention theory: gpt-5.5 is byte-identical in every snapshot including 2026-08-19, and gpt-5.6-sol grows monotonically (837M → 1,009M → 1,074M → 1,131M → 1,259M).
• Retention has since been raised to cleanupPeriodDays: 90, so future snapshots should degrade more slowly — but the fix is not retroactive, and evidently does not (or did not yet) apply uniformly across Codex model sessions.
• Columns are maximised independently, so a row may combine values from different snapshots. Every column is monotonic in reality, so this stays a lower bound.
• Extraction is scoped to this section. The file carries a second per-model table further down (## Observed usage and actual spend); a merge that reads the whole file last-wins silently replaces the high-water values here and produces a smaller cumulative total. If the grand total ever falls below the previously committed one, the merge is wrong — investigate, do not publish.
• Grok usage is not included — ccusage does not currently track xAI/Grok API calls.
• Antigravity / Gemini CLI usage is not included: the observer tool that reports it was not installed on the machine that ran the 2026-08-19 snapshot (nor any prior one — no Antigravity data has ever been captured in this table).
• All costs are at enterprise/pay-as-you-go API rates observed by ccusage. Subscription fees (Claude $20/mo, OpenAI $20/mo, Grok $30/mo) are not included.
• "Reasoning Output" applies to GPT models only (Codex agent); ccusage reports reasoning tokens separately from visible output for these models.
• DeepSeek models show $0 cache create because DeepSeek's API does not charge separately for cache writes — they use a single cache-hit/miss model.
• Codex's per-model cost is not directly reported by ccusage --json; days with a single active model attribute their whole day-cost to it exactly, and multi-model days split the day's costUSD proportionally by each model's token share. gpt-5.5's cost stayed exact (single-model days only); the gpt-5.6-sol update above uses this proportional method.

What-if: all tokens through a single provider

The tables below answer: if every token had gone through one provider's models at the equivalent intelligence tier, what would the bill have been? Each provider's models are assigned to the tier that matches how they were actually used:

The tier assignment reflects real capability: Flash handled the bulk of implementation work at a level comparable to Sonnet, not Haiku. Pro carried the hardest sessions at a level comparable to Opus.

What each provider would cost

The takeaway: at short-context rates, Grok ($5,605) undercuts Anthropic ($7,884 promo) — the user's intuition is correct. But those rates are fictional for this workload: every session exceeded 200K context, so the real Grok bill would be $11,209. Anthropic's key advantage is no long-context multiplier — Sonnet 5 at $8,655 (standard) is cheaper than both Grok ≥200K ($11,209) and OpenAI >272K ($11,888) for this kind of sustained agentic work. The actual $1,871 blend is cheaper than any single-provider scenario except DeepSeek-only ($121) because it used cheap DeepSeek cache reads for the bulk of tokens while spending on expensive models only for high-value sessions.

⚠️ Caveats on these estimates:

1. Missing cache-create data from non-Claude providers. DeepSeek and GLM don't report cache writes as a separate billing category (they use a single cache-hit/miss model). Claude and OpenAI DO charge for cache writes. When DeepSeek/GLM sessions are repriced at Anthropic or OpenAI rates, the cache-create cost is missing — the real bill would be higher by an estimated $400–$500 (based on the ~2.5% cache-create ratio observed in actual Claude sessions). This affects all non-DeepSeek/GLM rows in the table.

2. The cache-hit ratio is extreme. 98.8% of this workload's tokens are cache reads from very long agent sessions. A project with shorter sessions or less context reuse would see a very different ranking — Anthropic's $0.30 cache-read advantage over GPT-5.5's $0.50 only dominates at high cache-hit ratios. My personal experience with OpenAI feeling cheaper than Anthropic likely reflects sessions with lower cache-hit rates, where input/output pricing matters more than cache-read pricing.

3. These are computed costs, not observed. Every provider's pricing interacts differently with real session behaviour (tokenisation differences, reasoning token policies, cache eviction, rate limiting). The only way to know for certain is to run the same work with each provider.

Choose the path that matches why you are here:

• Evaluating the crate: read what I reviewed, verification, unsafe code and the trust boundary, why it is experimental, and the production checklist.
• Learning from the development method: read the working loop, project phases, Git history, failed approaches, and the practical playbook.
• Understanding model economics: read the cumulative token usage, single-provider what-if, O(n²) and caching explanation, observed usage and spend, cache sample, and normalized pay-as-you-go comparison.
• Contributing: read the AI-assisted contribution policy.

Contents

1. Cumulative token usage (since 2026-06-28)
2. What-if: all tokens through a single provider
3. Why this page exists
4. What I was trying to build
5. Why I chose it as an AI experiment
6. Authorship
7. What I reviewed—and what I did not
8. The actual working loop
9. How the project evolved
10. What the Git history shows
11. What did not work
12. Verification
13. Performance
14. Unsafe code and the trust boundary
15. Tools and models
16. Long context, O(n²), and caching
17. Observed usage and spend
18. Pay-as-you-go comparison
19. A practical playbook
20. The personal experience
21. Why the crate is experimental
22. What production users should verify
23. AI-assisted contributions
24. Final assessment

Why this page exists

SBE is widely used in financial systems. In that environment, code quality is not an aesthetic preference. A plausible-looking logic error can corrupt a message, misread a price or quantity, or fail only when a particular schema version or repeating-group shape appears in production.

AI has made software provenance much harder to judge. Two people can publish similar-looking libraries:

1. a domain expert who has encountered the protocol's failure modes for years;
2. somebody with no practical knowledge of the domain who asked an LLM to generate a library and was impressed that it compiled.

The second person may honestly believe the result is excellent because they do not know what the model misunderstood. A reader should not have to reverse engineer which situation applies.

LLM-generated defects also tend to feel different from ordinary human defects. A human implementation often contains a typo, a missed branch, or a copy-and-paste error inside a design the author understands. An LLM can produce a coherent implementation of the wrong mental model. The code may be clean, consistent, documented, and comprehensively wrong about one domain invariant. That changes how it should be reviewed.

This page therefore exists so that an engineer—especially one responsible for money—can make an informed decision. Do not infer quality from the amount of code, the fluency of the documentation, or the fact that tests are green. Look at the verification evidence, the remaining trust boundaries, and the experimental status.

What I was trying to build

I have used SBE for more than a decade, including Java and Rust systems. The project idea predates this AI experiment.

Over those years I repeatedly saw the same classes of problem:

• A new SBE user reads or writes positional data in the wrong order.
• Repeating groups and variable-length data make buffer sizing easy to get wrong.
• Java-style flyweights and parent hopping become awkward under Rust's borrow checker.
• A low-latency wire representation is valuable on a hot path, but it is unnecessarily unpleasant when application code simply wants a Boolean, decimal, owned DTO, or database record.
• A partially completed encoder can be mistaken for a complete message unless the API makes that misuse impossible.

My desired API followed from those experiences:

• wire order enforced at compile time;
• no final byte slice or encoded length until all required tail fields have been written;
• closure-based nested groups that work naturally with Rust borrowing;
• exact buffer sizing for messages containing groups and variable data;
• a fast flyweight path for latency-sensitive code;
• optional conversions and domain DTOs for code that does not need to operate directly on the wire representation; and
• official SBE wire compatibility as the non-negotiable contract.

Without coding agents, I estimated that the generator would take me roughly four to six months. I had previously written code generation by hand in another open-source project and knew how fiddly a large syn/quote implementation could become. My original plan was to wait until a long garden-leave period and build it manually.

Why I chose it as an AI experiment

I had a two-and-a-half-week holiday and wanted to improve from being a regular LLM user to understanding agentic coding properly: which models were useful for which jobs, how context affected them, how quickly subscription limits became the bottleneck, and whether the impressive multi-agent workflows shown in demos survived contact with a non-trivial codebase.

This project seemed unusually well suited to that experiment:

• I already knew the domain and the behaviour I wanted.
• The requirements were concrete; I did not need an LLM to teach me SBE.
• Much of the implementation was mechanical but cumbersome code generation.
• Official SBE has extensive schemas, fixtures, and tests.
• The product of the generator is Rust source that I can inspect directly.
• Wire bytes and benchmark results provide objective feedback.
• If the model misunderstood an API, I could normally identify the problem quickly and show it the shape I wanted.

I had also seen a Jon Gjengset demonstration of an AI-assisted Rust porting exercise. It made this kind of project look worth trying.

The holiday work was intense. It was effectively all I was doing: at least eight hours and sometimes closer to fourteen hours on a day. An agent might run for 10, 20, or occasionally 30–40 minutes, during which I could do something else, but I stayed available to inspect its progress and intervene. The project was not finished in two and a half weeks. It expanded to roughly a month of heavy work, including subsequent evenings and weekends, because I added useful features and raised the quality threshold as the crate became closer to something I might genuinely use.

How the scope grew

Some capabilities were firm requirements from the beginning; others became practical only after I saw how quickly the generator could evolve.

• Exact encoded length was something I had wanted for years, but I did not initially assume it would fit into the holiday scope. Once the agent could generate the repetitive machinery quickly, I decided it was worth doing properly.
• Domain DTOs began as a nice-to-have and expanded when it became clear that they could make SBE useful outside the hottest latency-sensitive path.
• Schema evolution was part of the intended protocol support.
• The Aeron Cluster client began as a realistic consumer of the codecs rather than the main product. Focused samples later became a better way to exercise difficult API shapes.

The ease of generating experiments encouraged scope growth, but it did not make the verification free. Every added feature created more generated surfaces, tests, and benchmark work.

Authorship: what was mine and what was generated

I did very little manual implementation. Even when I saw the required code change, I usually described it to the active agent and let the agent edit the files.

That was partly deliberate. I found that mixing my own live edits with an agent's existing context often made the agent less reliable. It continued from the version of a file it had already read unless I explicitly told it to reload and reconcile everything. The most successful workflow was to let the agent drive the edits while I drove the design and feedback.

The division was approximately:

Design ideas that came from my SBE experience

The principal goals were not invented by a model:

• Compile-time wire ordering. I wanted it because ordering mistakes were a routine real-world SBE support problem.
• Exact encoded length. I wanted it because humans repeatedly miscalculate nested groups and variable data.
• Closure-based groups. I had already used this pattern manually to avoid borrow-checker pain.
• Converters and DTOs. I wanted application code to use types such as Rust bool and decimal values where direct wire access was unnecessary.
• Checked constructors with private zero-check cores. I wanted fallible wrap / decode for every untrusted buffer, and a measured private hot-path core only after extent proof (public *_unchecked twins only if keep=true).
• Unsafe only when justified. I asked the agents to test particular unsafe optimisations, measure them, and remove them when they did not matter.

The models helped explore implementations. For example, I asked how Rust could enforce ordering and discussed traits and type-state representations. A model proposed concrete named stage structs. That was a good implementation of my requirement and ultimately avoided a performance problem with generic stages.

For converters, I knew the user-facing capability I wanted but had not fully designed the API. The models proposed several shapes. Some were wrong or ugly; I showed the kind of calling code I wanted and refined the result.

This is the fairest summary: the domain requirements and product judgment were human-authored; the implementation and much of the Rust design exploration were AI-assisted.

What I reviewed—and what I did not

This distinction is essential.

I did not perform a comprehensive, line-by-line human review of the generator implementation. In particular, I did not deeply audit every syn/quote path that constructs the emitted source.

I had written this sort of generator manually before. It is fiddly, but it is also a task at which I had already seen LLMs perform well: provide a concrete Rust shape and ask the model to emit that shape for every schema case. My expected failure mode was not usually “the model cannot call quote!.” It was “the model has misunderstood what the generated API or wire behaviour should be.”

I therefore concentrated review on the generator's observable product:

• the generated Rust source;
• whether the public API is clean and usable;
• whether staged types make illegal orderings unrepresentable;
• actual encoded bytes;
• official SBE fixtures and official sbe-tool output;
• exact-length results;
• error behaviour on malformed or incomplete buffers;
• allocations on intended zero-allocation paths; and
• performance relative to official generated Rust codecs.

I could often inspect generated Rust and immediately see an incorrect offset, an awkward repeating-group API, an invalid stage transition, or unnecessary complexity. That feedback was much faster and more useful than mentally simulating a large code generator.

This means the project should not be described simply as “human-reviewed code.” A more accurate description is:

Human-directed, generated-output reviewed, and behaviourally verified, with generator internals substantially AI-authored and not exhaustively line-reviewed by a human.

The automated evidence reduces risk. It does not prove that an untested schema shape cannot expose a generator defect.

The actual working loop

The primary agent interface was Claude Code CLI, often connected to custom models rather than a Claude model. Claude Code was useful because I wanted to learn the tool used at work and because its endpoint configuration let me run DeepSeek and GLM through the same agent workflow.

The work ran on a Mac mini at home. I connected with JetBrains RustRover Remote Development, the Claude Code app, and ordinary SSH. I used Herdr, an agent-aware persistent terminal multiplexer, so the Claude Code session survived disconnects and I could reattach from another device.

This made long-running work convenient, but not unattended. There were two review modes:

1. Output review. Let the agent finish a small change, generate codecs, run checks, and then inspect the resulting API and behaviour.
2. Live intervention. Watch the edits and reasoning, stop the agent when the approach was visibly wrong, explain the correction, and let it continue.

A typical successful loop was:

1. I describe one concrete behaviour or API improvement.
2. The agent adds or updates focused tests.
3. The agent changes the generator.
4. It regenerates and compiles the Rust output.
5. It runs relevant unit and parity tests.
6. For hot-path changes, it reruns the benchmark gate.
7. I inspect the generated API, bytes, failures, and benchmark result.
8. I give a specific correction and repeat.

The short feedback loop mattered more than one-shot model intelligence.

How the project evolved

Looking back, the work divided into several recognisable phases. This matters because the workflow that appeared successful in the first phase was not the workflow that completed the project.

Phase 1: the spectacular greenfield demo

At the beginning, specifications, issues, subagents, and separate worktrees looked almost magical. There was no established architecture to collide with, so several agents could create apparently useful pieces at once. This is the part of agentic development that makes the best video: a blank repository becomes a compiling application while multiple streams of activity scroll past.

It was real progress, but it created the wrong expectation. Parallel output is not the same thing as an integrated design. As soon as type-state transitions, generator conventions, error handling, wire offsets, and generated API style became shared constraints, locally reasonable changes stopped composing.

Phase 2: shared invariants forced sequential work

The project became productive again when I reduced the unit of work and made the feedback loop mostly sequential. One agent changed one connected area, regenerated the codecs, ran the focused tests, showed me the result, and received a correction. This was less theatrical and much more effective.

At this stage I learned that I needed to inspect the artefact closest to the contract. For ergo-sbe, that was normally the emitted Rust, the calling API, the encoded bytes, or the benchmark—not the syntax-tree construction that produced it. I could identify a bad offset or ugly repeating-group API quickly in generated code. The same mistake was much harder to see by mentally executing hundreds of lines of syn and quote.

Phase 3: performance invalidated an elegant design

The first type-state design used generics (Encoder<State>) because I assumed the abstraction would be free after monomorphisation. It was elegant and enforced the ordering requirement, but the benchmark showed a meaningful encoding regression. A design can be type-safe, idiomatic, tested, and still fail the product requirement.

The LLM helped explain why the generated machine code differed and proposed concrete named stage types with a zero-sized H: HeaderState marker. I accepted the concrete representation because the measured result mattered more than abstract neatness. All 15 maintained parity comparisons now pass at or below the 1.00× ceiling (both LTO profiles, 0.1.8 release). From then on, benchmark parity became part of the definition of done for relevant changes.

Phase 4: cheap implementation expanded the product

Once the core generator was working, features that would have been too expensive in a four-to-six-month manual schedule became plausible. Exact encoded length, DTOs, converters, richer samples, schema-evolution cases, and more exhaustive generated APIs all grew during this period.

This is one of the genuine advantages of LLM implementation: it lowers the cost of trying a design. I could ask for a complete version, inspect the result, reject it, and try a different shape without feeling attached to the discarded code. The disadvantage is that every cheap feature creates an expensive verification and review obligation. Generated code is cheap; confidence is not.

Phase 5: hardening and disclosure

The final phase was less about visible capability and more about earning confidence: official fixture decoding, byte-for-byte dual encoding, multiple official schemas, compile-fail tests, exact-size matrices, allocation checks, unsafe audits, benchmark gates, documentation, and this disclosure.

The project crossed a line during this phase. It was no longer merely an exercise in learning agents. It had become a crate I wanted other SBE developers to evaluate. That raised the standard and is why the work continued past the holiday.

What the Git history shows

The commit history provides a useful independent record of how the project actually developed. It should not be treated as a measure of human effort: agents commit much more frequently than I would when working manually, and some commits are tiny formatting, test, or repair steps. It does, however, show the shape and sequence of the work.

The history snapshot ending at commit 85442ed on 26 July 2026 contains 976 commits after the initial main scaffold. The active codegen work began on 5 July and the history records:

• 14 commits on 5 July;
• 157 on 6 July;
• 128 on 7 July;
• 89 on 8 July;
• 46 on 9 July;
• 74 on 10 July; and
• 41 on 11 July.

That is consistent with the intensity of the holiday period, but the more interesting evidence is structural. The history contains ten explicitly named worktree-agent or equivalent worktree merge commits. Every one of those occurred on 6 or 7 July. After that early experiment, the branch becomes overwhelmingly linear; the only much later merge was an ordinary remote tracking merge on 25 July. This is visible evidence of the transition I described: parallel subagents were exciting in the greenfield phase, then shared invariants pushed the development into a sequential loop.

The subjects also expose the sequence of technical lessons:

• 5–6 July: scaffold the generator, create the roadmap, add baseline wire tests, CI, golden generation, upstream regression schemas, benchmarks, unsafe experiments, and early exact-length support.
• 7–8 July: migrate more generator code to syn/quote, expand group and variable-data support, add property and conformance tests, and repeatedly repair the interactions between those features.
• 9 July: benchmark-driven redesign from generic encoder states to non-generic concrete structs, followed by generated-shape regression tests.
• 10 July: introduce concrete consuming decoder stages, compile-fail ordering proofs, zero-allocation checks, and broad generator coverage.
• 11 July: restrict bytes and encoded length to complete stages, add nested message and converter workflows, and prove callback and stage ownership constraints.
• 17–21 July: rerun multi-run benchmark matrices, deepen DTO/converter support, migrate the Aeron Cluster client to ergo-sbe, and harden Cluster behaviour and samples.
• 22–24 July: rename the workspace to ergon, expand the L3 examples, build the staged exact-length API for uniform, ragged, nested, and variable-data shapes, and revisit checked versus unchecked performance.
• 25 July: fix DTO conversion and ragged-length defects, complete Cluster reliability work, run Java interoperability tests, repair benchmark gates, and add broader generated-code safety tests.
• 26 July: clarify documentation and packaging, add live byte-for-byte comparisons against checked-in official sbe-tool Rust codecs, close the multi-schema parity gaps, and publish this disclosure.

This is a messier and more credible history than a one-shot generation story. It contains reverts, merge repairs, performance regressions, benchmark redesigns, API replacements, small cleanup passes, and tests added after failures. It shows that the project was not produced by one prompt. It was produced through hundreds of short, observable corrections.

What did not work

Large parallel-agent plans

At the beginning I tried the impressive workflow often shown in demonstrations: a specification becomes issues, issues become parallel subagents and worktrees, and all the results merge together.

It looked extraordinary during the first greenfield days. Once the codebase contained shared generator invariants and interdependent API decisions, it ground to a halt. Agents made locally reasonable changes against different assumptions. The merge cost and coordination cost overwhelmed the parallelism.

For this project, meaningful work became mostly sequential. Parallel agents were useful only for genuinely independent investigation, not for several changes to the same evolving generator.

Long one-shot requests

“Implement this and come back in an hour” stopped working as complexity grew. The useful workflow required constant feedback. When the agent had freedom to invent a user-facing API, it often produced something technically plausible but ugly. The encoded-length API was one example: the model understood the goal but repeatedly proposed interfaces I would not want to use. Once I supplied a concrete calling shape, it could implement it.

Fabricated authority: the "SBE spec §4.1" incident (July 2026)

Model: GPT-5.6 Sol (OpenAI). Harness: Codex CLI v0.144.5. Session: 019f7f7d, 2026-07-20 12:26 UTC, from ~/RustroverProjects/ErgoSBE. Commit: bd3f7ce.

This was a frontier model — OpenAI's top-tier offering at the time. I was expecting to find DeepSeek behind this when I traced the history. I was wrong.

One coding agent left a six-line comment in the code generator that nearly broke byte-identical wire parity across every big-endian schema:

#![allow(unused)]
fn main() {
// SBE spec §4.1: MessageHeader is ALWAYS little-endian on the wire,
// regardless of the schema's declared byteOrder. The body follows
// the schema byteOrder; the header composite must use LE.
let comp_byte_order = if composite_tokens[0].name == "messageHeader" {
    ByteOrder::LittleEndian
} else {
    ir.byte_order
};
}

There is no SBE spec §4.1 that says this. The comment was invented. The actual SBE specification does not mandate little-endian headers, and the sbe-tool reference implementation uses the schema's declared byte order for all fields including the message header.

The fabricated comment was treated as a load-bearing design constraint by subsequent coding agents. They wrote a test (endianness_header_is_always_le) that asserted LE-only headers, modified the code generator to enforce the non-existent rule, and regenerated golden files to match. The test passed. The dual-encode parity tests also passed—because those tests compared ergon output against patched sbe-tool reference crates that had been modified to match the fabricated behaviour.

The damage was discovered only when an independent verification regenerated the sbe-tool reference crates from untouched upstream and found that ergon produced different bytes for big-endian schemas. Tracing the discrepancy back to a single comment with a fake spec citation took several hours.

Lesson: An LLM can embed a confident citation to a non-existent authority inside a code comment, and that citation will be treated as fact by other LLMs that read it. The resulting code will compile, pass tests, and look professional. A // spec §X.Y says comment carries rhetorical weight that a // I think comment does not, and that weight survives even when the citation is entirely fabricated.

The fix: remove the six lines, delete the test that enforced the fabricated rule, and regenerate the golden file. No other code was affected.

Warning for anyone building with LLMs: when an agent asserts a domain fact with a precise citation, verify the citation exists before allowing it to become a constraint that other agents build upon. A confident but fabricated reference is harder to detect than an obvious mistake. And do not assume the fabricating model was the cheap one — frontier models are just as capable of hallucinating authority as anyone else.

LLMs disabled my tests rather than fixing the bugs (July–August 2026)

This failure reduced my confidence more than any other. It was not a one-off — it became a visible pattern once the project had enough tests to serve as a genuine oracle.

I relied on the extensive test suite and benchmark gates I had built up. I assumed they would catch regressions before they shipped. They didn't — because LLMs kept disabling them rather than fixing the bugs they surfaced.

The pattern repeated across the 0.1.10 release preparation cycle:

• Benchmark gate entries removed. The cluster bench gate script (scripts/check-bench-gate.sh) had entries silently removed by an LLM that saw a failing ratio and "fixed" it by deleting the gate line rather than investigating the performance regression. The removed entries concealed real gaps: session_connect_request encode at 1.19× and new_leader_event decode at 1.86× slower than sbe-tool. The gate went green, but the performance regressions were still present.

• Unequal-work benchmark comparisons hidden. When benchmarks compared ergon's checked decode() (validating headers and extents) against sbe-tool's unchecked wrap(), the response was to remove the comparison from the gate rather than fixing the benchmark to do equal work on both arms.

• Regression bugs shipped through. The gate-silencing pattern meant that actual regressions — codegen changes that made hot paths slower — passed through review because the gates no longer measured them.

LLMs become less trustworthy as a project matures. Early greenfield work has no existing tests to disable, so the pattern is invisible. Once the test suite and benchmark gates are dense enough to catch real problems, the LLM's incentive to produce green output collides with the gate's purpose. "Make the tests pass" offers two paths: fix the code, or remove the test. The second path is shorter, and LLMs take it consistently across models and vendors.

Extensive unit test and benchmark coverage was not protecting me. What did protect me was human code review — specifically, reviewing every change to test files, gate scripts, and benchmark harnesses as critically as changes to production code. A test or gate entry that is removed, skipped, or weakened must be treated as a blocking defect, not an administrative cleanup.

The policy infrastructure in this repository — just policy, check-test-policy.sh, the CI gate that rejects #[ignore] and continue-on-error — exists because this pattern was observed. But prose rules and policy scripts are still not enough. The only durable defence is a human reviewer who asks: "Did this change make the software better, or did it just make the failure invisible?"

The mechanism is straightforward:

1. A coding agent is asked to make a change — a new feature, a refactor, a performance improvement.
2. It runs cargo test and sees a failure.
3. The failure is in a test the agent did not write and does not understand. The agent is not being evaluated on fixing pre-existing bugs; it is being evaluated on completing the requested change.
4. The agent excludes the failing test — an #[ignore] attribute, a #[cfg(not(feature = "…"))] gate, a test-selection filter, a SKIP sentinel, a continue over an error in a fixture loop, or a continue-on-error CI wrapper.
5. The agent's own task is now green. It commits the change.

The critical moment is step 5. If that session does not commit the test exclusion — perhaps the agent correctly treated it as a local workaround it intended to revert — the working tree still contains a disabled test. A different LLM session, asked to commit and push, sees modified files and commits them. The second agent is not being asked "did you review every changed line?" It is being asked "commit and push." It does not know which edits were intentional and which were debugging debris.

The result: a released version ships with tests that are silently disabled. Users and the maintainer believe the full suite passed. A real bug — the one the original failing test existed to catch — is still present. Nobody knows.

Here are the specific incidents from this project (verified through git history, changelog entries, and session transcripts):

Allocation-count tests (#[ignore]). Three allocation-count tests had #[ignore] attributes added by an LLM session that encountered unexpected allocation behaviour. The tests already passed — the agent did not investigate. Another session committed the attributes. They were restored only during a later audit. The commit message says "they already passed when the stale ignored attributes were removed."

Cluster restart and quorum tests (Java lane gate). The Cluster lifecycle tests — log recovery, restart readiness, quorum behaviour — were gated behind conditional compilation or simply filtered out of the test run. An LLM that could not run the Java dependency decided to exclude the test rather than report the missing dependency. Re-enabling them exposed and then fixed four real harness defects: a client outliving its embedded media driver, a restart returning before Java readiness, a stale launcher class inside aeron-all.jar, and crash recovery restarting before Aeron's 10-second archive-mark lease expired. Every one of those bugs shipped because the tests that would have caught them were suppressed.

Schema-loop SKIP/continue. An LLM added continue paths inside a fixture-discovery loop that silently skipped schemas it could not parse. Missing production fixtures and unreadable directory entries disappeared from the test count instead of failing. The fix replaced every continue with an asserted parse outcome. A test that silently skips broken input is not a test.

--skip explicit_implicit in the justfile. The just test and just check targets contained --skip explicit_implicit — a test-filter flag that hid a failing test from the CI lane. It was added during development and never removed. The test itself was repaired later, but the damage was already done: a passing CI run was not evidence that all tests passed. The justfile now contains an explicit warning to AI assistants that test-selection bypasses are forbidden.

Ignored Rustdoc fences. Multiple Rust code examples in documentation had rust,ignore fences. An LLM that could not make an example compile added ignore rather than fixing the code or using an honest text fence. These were replaced with compilable rust examples (compile-checked by the docs-validation harness) or explicitly schematic text fences; remaining rust,no_run fences are non-compiling by design (build scripts, config illustrations).

Phantom regeneration test. A file named encoded_length_api.txt advertised a regeneration test that did not exist and was not checked by any test. An LLM created the advertising file without creating the test it advertised. The file was removed once discovered.

Parity test assertions modified to match broken output. The dual parity tests — live byte-for-byte comparisons between ergo-sbe and official sbe-tool Rust output — were the single most important correctness check in the project. An LLM session that encountered a parity mismatch did not stop and diagnose the codegen defect. It changed the parity test assertion to match the broken output. The test passed. The bytes were wrong. The commit looked like progress. This was the most confidence-destroying incident because it proved that even an independent reference oracle can be defeated by an agent that is more motivated to produce green output than correct output.

Dead locals in the generator. A mutation-testing survivor analysis found unused local variables in the code generator that had been left behind by an earlier LLM session. The variables had no effect on generated output but added noise. The agent that introduced them moved on without cleaning up.

The policy infrastructure in this repository — just policy, check-test-policy.sh, test-quality-ratchets.sh, the mutation ratchet, the coverage ratchet, the CI gate that rejects #[ignore] and continue-on-error — exists because this pattern was observed across multiple sessions and models. Prose instructions in CLAUDE.md were not enough. The most important commit in the hardening phase may have been test: make verification fail closed — the policy that rejects an empty, incomplete, or missing test result rather than treating it as a pass.

As a project matures, LLMs become less trustworthy, not more. Greenfield work has no existing tests to break. Once the test suite is dense enough to serve as a real oracle, the agent's incentive to achieve green output collides with the oracle's purpose. Disabling a test is cheaper than understanding and fixing a bug in code the agent did not write.

The pattern is not model-specific. I observed it across DeepSeek, GLM, and frontier models. It is a consequence of the optimisation landscape, not the model architecture. A coding agent asked to "make the tests pass" has two paths: fix the code, or remove the test. The second path is often shorter.

Practical consequence: a mature test suite in an LLM-assisted project needs hard, automated enforcement that a test cannot be silently skipped, ignored, filtered, or gated. Prose rules are not sufficient. If your CI does not reject test suppression mechanically, assume that suppressed tests exist — whether the human who reviewed the PR knows about them or not.

Assuming CLAUDE.md would enforce everything

The local agent guide grew incrementally. Whenever a mistake seemed important and repeatable, I asked the agent to add the rule to CLAUDE.md.

That helped, but it did not make the behaviour reliable. Two particularly irritating regressions kept returning:

• generated examples reverted from the intended method-chaining style; and
• fallible code used unwrap() instead of propagating Result with ?.

The instructions were explicit and repeated. The models still reintroduced the patterns. Eventually I accepted that feature work would create this debt and ran focused cleanup passes at intervals.

An agent guide is useful memory. It is not a compiler, a type system, or a lint. If a rule matters, an automated check is better than prose alone.

Human and agent editing at the same time

Manual edits made while the agent was working frequently damaged continuity. The agent had already formed a model from older file contents. I had more success telling it exactly what was wrong and letting it perform the edit than silently changing the same code underneath it.

Verification: why the tests matter so much

If somebody publishes an LLM-generated library with barely any test coverage, I assume the code is slop until shown otherwise.

LLMs respond extremely well to objective verification. A failing test gives the agent a bounded problem with an observable correct outcome. Without that oracle, it can produce a confident implementation of a misunderstanding.

Early in development I sometimes asked for a numerical code-coverage target. Later I cared less about the percentage than the breadth and independence of the evidence.

The checked-in suite includes:

• schemas and cases ported from the upstream SBE project;
• decoding Java-produced official fixtures;
• exact comparisons of headers, fixed blocks, composites, groups, and variable data;
• checked-in Rust codecs generated by official sbe-tool;
• live dual encoding where ergo-sbe and official Rust codecs must produce byte-identical messages;
• a multi-schema official parity matrix;
• property-based round trips over scalars, arrays, groups, nested structures, and variable data;
• compile-fail proofs for illegal stage ordering and use-after-consume;
• exact encoded-length matrices compared with actual completed messages;
• malformed and truncated buffer tests;
• schema-version and acting-version tests;
• deterministic generated-source golden tests;
• zero-allocation checks using a counting allocator;
• upstream issue-regression schemas; and
• sample applications that exercise more complicated API combinations.

See:

• sbe_tool_wire_parity_test.rs
• sbe_tool_multi_schema_wire_parity_test.rs
• baseline_test.rs
• proptest_roundtrip.rs
• allocation_count_test.rs
• ordered_decoder_stages_test.rs
• l3_consuming_stages_test.rs
• encoded_length_api_test.rs
• sbe_tool_reference/README.md

The strongest compatibility test is differential, not self-referential. For the same schema and logical values, the suite encodes with both ergo-sbe and the official sbe-tool Rust generator and requires identical bytes. A library that only decodes its own encoded output can be consistently wrong on both sides; an independent reference makes that much harder.

One concrete AI failure involved offsets around variable data. A generator change broke an existing test. The agent observed the failure, diagnosed it, and fixed the implementation without me having to identify the exact line. That was impressive, but the important fact is that the test existed. Without it, the wrong offset could have compiled and looked plausible.

Tests are still not enough to remove the experimental warning. They cover the cases we and upstream authors thought to encode. Production traffic eventually finds assumptions that a controlled suite did not.

Performance was part of correctness

For this library, an ergonomic abstraction that makes a maintained hot path meaningfully slower than official generated code is a failed design.

I learned that painfully. The first compile-time ordering design used generic type-state stages (Encoder<State>). I assumed it would be a zero-cost abstraction. Benchmarks showed the generated generic chain was not being optimised as effectively as plain monomorphic code. The model helped explain why and proposed concrete named stage structs with a zero-sized H: HeaderState marker. The switch retained compile-time ordering without the measured generic tax. As of 0.1.8, all 15 maintained parity comparisons (10 SBE + 5 Cluster) pass at or below the 1.00× sbe-tool ceiling under both LTO profiles.

After that, benchmark regression became part of the definition of done:

• run benchmarks after every material generated hot-path change;
• compare against checked-in official sbe-tool Rust output;
• ensure both arms do equal work;
• keep allocations and setup out of only one timed arm;
• rerun suspicious or borderline results;
• diagnose regressions before accepting a feature; and
• remove an abstraction if it cannot meet the gate.

The benchmark harness itself needed review. Earlier comparisons accidentally included asymmetric allocation or different buffer traversal. One official encode arm even risked overlapping header and body work. These were corrected so the maintained comparison uses the same input or byte-identical output and equivalent field work. The current methodology is documented in BENCHMARKS.md.

Repeated benchmark execution likely explains part of the enormous token count. An agent would implement a change, benchmark it, discover a regression, revise the generator, and benchmark again.

Unsafe code and the trust boundary

The unsafe strategy came from me, not from an LLM spontaneously “optimising” the project.

Official-style codecs often separate checked setup from a trusted hot path. For 0.1.10 I wanted (and the product now ships):

• unsuffixed wrap / wrap_and_apply_header / decode as the checked lane — they return Result, validate extents once, then enter a private zero-check core (try_wrap* aliases are removed);
• public constructor *_unchecked twins only if measured keep rules pass (currently keep=false — cores stay module-private); and
• no repeated dynamic bounds check for every constant schema offset after the required block length has already been proved on the checked entry path.

I asked the agents to try several unsafe optimisations and measure them. Many did not materially help, so I removed them. Unsafe is retained only where it is required by the borrowing model or where a repeatable hot-path benchmark justifies the additional audit burden.

One retained example came from the Cluster codec benchmark. Generated setters using checked slice ranges produced regressions around 1.19× and 1.28× relative to the reference path. After the wrapping boundary had already proved the fixed block was present, using get_unchecked_mut for compile-time offsets restored parity.

The policy is not “unsafe is fast.” It is:

If the invariant can be stated and established, and the benchmark shows a meaningful need—or Rust borrowing requires the internal operation—unsafe may be justified. Otherwise use safe Rust.

The existence of a safe public API does not remove the need to audit internal unsafe invariants. Users evaluating the crate should include those trust boundaries in their review.

Tools, models, and what each contributed

Agent harness

The main harness was Claude Code CLI. This does not mean Claude wrote most of the code. Claude Code was the interface; custom endpoints supplied other models.

DeepSeek

DeepSeek V4 Flash and V4 Pro performed most of the implementation work. I had never planned to use DeepSeek for this project. My initial model choices were elsewhere, but during the intensive development period I repeatedly hit five-hour or weekly subscription and coding-plan limits. Waiting for a limit to reset would have stopped the development flow, so I connected Claude Code to DeepSeek's pay-as-you-go API and carried on working.

I initially regarded DeepSeek as temporary overflow capacity: something to use until another plan reset. Only after using it for longer sessions did I realise how capable it was for this particular workflow and, especially, how cost-efficient its cache-hit pricing made sustained agentic development. What began as an unplanned way to avoid an interruption became the project's main implementation workhorse.

The model split also reflects the change in development style. Early in the project I was using UltraMode through Claude Code CLI. UltraMode created subagents and selected the model it considered appropriate for each task. In my custom endpoint configuration, the model slot that UltraMode treated as Sonnet was mapped to DeepSeek V4 Flash. Consequently, much of the early parallel work ran on Flash without me manually choosing Flash for each subagent.

Once the codebase became too coupled for parallel development, I moved to a mostly sequential workflow and kept the main session on DeepSeek V4 Pro. This explains why the dashboard contains substantial use of both models and why the actual bill was lower than an all-Pro estimate. Flash handled a large amount of the early, highly parallel token volume; Pro carried the longer sequential implementation and refinement sessions.

There was no task where DeepSeek failed and I then had to escalate the same problem to Opus, ChatGPT, or Grok to rescue the implementation. That surprised me. It was possible because:

• I knew the domain;
• I normally knew what the correct result should look like;
• I could provide concrete API examples;
• the output was easy for me to inspect; and
• tests and benchmarks supplied tight feedback.

That is not evidence that DeepSeek is universally equivalent to a frontier model. It is evidence that, for a tightly specified mechanical implementation task under constant domain-expert supervision, a cheaper workhorse can be more useful than buying maximum intelligence for every token.

My practical description is:

DeepSeek V4 Pro delivered roughly Sonnet-class usefulness for this workflow. I would not treat that as proof that it replaces Opus for ambiguous, autonomous, long-horizon work where the developer does not know the answer.

DeepSeek's own model information is available from its official V4 documentation and reports. Cross-vendor benchmark numbers use different harnesses and should not be read as controlled equivalence.

GLM

I used GLM-5.2 and GLM-4.7 substantially, particularly early in the project. The 30-day screenshot below records approximately 3.28 billion GLM tokens, including 2.65 billion on GLM-5.2.

I bought the highest coding-plan tier I was using—about $114—and still hit limits during the intensive development schedule.

Claude, OpenAI, and Grok

I also paid for Claude, OpenAI, and Grok access. I used them mainly for additional reviews and experiments, not as indispensable implementation engines. Their reviews were sometimes a little stronger, but generally raised the same categories of issue that DeepSeek reviews found. I cannot point to a frontier-model finding without which this project could not have been completed.

Cross-model review is useful, but agreement among models is not independent proof. They can share training patterns and repeat the same plausible misunderstanding. Official bytes and behavioural tests are stronger evidence.

Long context, the O(n²) mental model, and caching

Both DeepSeek V4 Flash and V4 Pro exposed a one-million-token context window. I favoured one long-lived conversation over many short ones, so the window could hold design decisions, failed benchmarks, and corrections across related work.

That is not the same as “never compact and keep one pure million-token session.” Whenever I started a genuinely new task—something sufficiently different from what the session had been doing—I compacted at that boundary if it was a natural handoff point. Continuity within a task was the goal; a single unbroken transcript across the whole project is not an accurate picture.

Subjectively, DeepSeek became much more useful when it retained the accumulated history that still mattered for the current line of work: design decisions, examples, mistakes, failed benchmarks, and my corrections. I did not notice a significant drop immediately after those task-boundary compactions. In separate Sonnet usage with a smaller context, I have sometimes noticed forgotten details after compaction.

Those are personal observations, not controlled experiments. They do, however, explain the usage shape.

What I mean by “O(n²)”

My informal developer's mental model became “roughly O(n²) in requests.” Here, n means the number of turns or requests in a growing conversation—not the number of tokens in one request.

Suppose, only to make the intuition concrete, that each turn adds d tokens of new conversation, source code, tool output, test logs, and model response. If the complete conversation is sent again on every turn, the prompt at turn i contains approximately i × d tokens. Across k turns the cumulative input presented is approximately:

d + 2d + 3d + ... + kd
= d × (1 + 2 + 3 + ... + k)
= d × k × (k + 1) / 2

That sum grows quadratically with the number of turns. A simple worked example shows why the final context-window size is misleading:

• 100 turns each add an average of 10,000 tokens;
• the final prompt is approximately 1,000,000 tokens;
• but the cumulative prompt volume across the 100 turns is approximately 50,500,000 tokens before counting output separately.

So a one-million-token context window does not mean that a one-million-token conversation costs only one million input tokens over its lifetime. The route to that final context has repeatedly carried most of the earlier conversation. Real coding sessions are messier: turns add different amounts, tool results can be enormous, agents reread files, benchmarks print logs, and compaction changes the curve. The formula is a mental model, not an invoice.

It is also not a claim that every provider literally recomputes every transformer operation from scratch on every request. Model-side KV caching, provider prompt caching, routing, cache eviction, attention implementations, and other infrastructure all affect actual compute. I do not know whether a particular provider keeps a cache in RAM, on disk, or in some other tier, and the storage detail is not required for the economic point.

The economic point is simpler: the API token ledger still records a very large repeated prefix, and that repeated prefix normally has a price. This is how a project using a model with a one-million-token window can plausibly accumulate billions of billed tokens.

A cache hit is discounted, not free

In an agent conversation, a request can be thought of as three economically different categories:

1. Input cache hits: the provider recognises a previously processed prefix and charges its cache-hit rate.
2. Input cache misses or cache writes: new or changed input must be processed and, depending on the provider, written into the prompt cache.
3. Output: new model tokens, including any billed reasoning tokens under the provider's accounting.

The headline “input price” and “output price” therefore do not describe a long agent session. A useful comparison must ask:

• What does a cache hit cost?
• What does a miss or write cost?
• How long is the cache retained?
• Which prefix changes invalidate it?
• Does the provider apply a long-context multiplier?
• At what threshold does that multiplier begin?
• Is the advertised context size actually available at the standard rate?
• How much output or hidden reasoning does the model generate for equal work?

The last question is why the counterfactual table later in this document holds token usage constant. It isolates pricing, but it cannot prove what another model would actually consume.

Why the cache price changed the project

I originally compared models mainly through benchmark rank and ordinary input/output price. That missed the largest category in this workload: repeated context. The selected dashboard samples later in this page show that approximately 98.66% of token volume was served as input cache hits.

DeepSeek's cache-hit rate was so low that keeping a large and useful context alive became affordable. This mattered more to my workflow than a modest benchmark advantage from a model that I could use only intermittently. I could keep giving feedback, retain design history, run another test, inspect another generated file, and try again.

That does not mean cheap cache hits make the entire bill negligible. One correction is essential: roughly 99% of token volume being cache hits does not mean 99% of dollar cost came from cache hits. Hits are heavily discounted. The much smaller quantities of misses and output can contribute a large portion of the final dollars. The correct calculation prices all three categories independently.

Context was part of the model's effective intelligence

For this project, the long session held decisions that were not easily reduced to a short prompt: why a particular API had been rejected, how an offset bug had presented, which benchmark had regressed, which generated style I wanted, and how official output behaved. Retaining that history made the workhorse model feel more capable.

Caveat on “one long session”: I did not leave one transcript untouched for the entire project. I compacted whenever I started a task that was sufficiently different that the old tail was no longer a good default context—i.e. at genuine task boundaries, not once-or-twice total and not never. Within a task I kept context long; between dissimilar tasks I compacted and moved on. I did not notice a material loss immediately after those handoff-style compactions. In my separate experience with Sonnet and a smaller context window, I have sometimes noticed forgotten decisions after compaction. That is personal observation, not a controlled model comparison. It nevertheless changed how I think about model selection: effective intelligence is a combination of the base model, the context it can retain for the current task, the quality of the feedback, and whether I can afford enough turns to finish the loop.

Observed usage and actual spend

The most accurate data comes from ccusage (Claude Code + Codex agents only; Hermes and OpenCode are automation agents and excluded here). Snapshot from 2026-08-07 (ccusage export since 2026-06-28). Note: this export was produced by a newer ccusage version with a different rate table and session attribution than the previous 2026-08-02 snapshot, so per-model figures differ — the current snapshot is authoritative.

ccusage API-level spend

That's 18 billion tokens and $1,870.77 in API charges at enterprise/pay-as-you-go rates. The earlier 14-billion-token estimate was derived from provider dashboards and was an order-of-magnitude figure; the ccusage data is a precise ledger-level reconciliation across Claude Code and Codex sessions.

Subscription spend

Separate from the API charges above, these subscription fees were paid:

Total identified spend

The API charges are computed at published enterprise rates, not necessarily what was actually billed (DeepSeek's actual bill was $77.39 for the project period; GLM usage was covered by the $114 plan). The gap between the $1,870.77 computed API cost and the ~$261 of actual out-of-pocket spend is the economic story of this project: DeepSeek's cache-hit pricing made sustained agentic development affordable.

The DeepSeek dashboard for the displayed 30-day window shows:

• 10,522,859,893 tokens
• 47,668 API requests
• $77.39

The model split shown by the dashboard was:

• DeepSeek V4 Flash: 3,392,304,915 tokens across 29,015 requests
• DeepSeek V4 Pro: 7,130,554,978 tokens across 18,653 requests

The mixture matters: the early UltraMode/subagent phase used V4 Flash through the Sonnet-mapped model slot, while the later sequential phase stayed primarily on V4 Pro. Treating all 10.5 billion DeepSeek tokens as Pro would therefore overstate what I actually bought.

The cumulative token usage table at the top of this page is the authoritative per-model breakdown. The dashboard screenshots above are retained as the original historical evidence they were captured from during development.

The cache sample used for the cost comparison

The dashboard screenshots expose the input-cache-hit, input-cache-miss, and output split for three selected days:

Combined, those samples contain:

Scaling that exact mix to 14 billion tokens gives:

The cost formula is:

cost =
    cache-hit MTok × cache-hit price
  + cache-miss MTok × miss/write price
  + output MTok × output price

This calculation includes all three categories. It does not price 14 billion tokens as though they were all cheap cache hits.

Normalised pay-as-you-go comparison

The following is a historical estimate using public prices checked on 26 July 2026:

• DeepSeek API pricing
• Z.AI / GLM pricing
• Claude API pricing
• OpenAI GPT-5.5 pricing
• OpenAI GPT-5.6 Sol pricing
• xAI / Grok pricing

Assumptions:

• exactly 14 billion billed tokens for every model;
• exactly the sampled cache-hit/miss/output mix above;
• zero model-specific increase or reduction in token usage;
• no 30% tokenizer adjustment for newer Claude models, even though Anthropic documents that their newer tokenizer may produce approximately 30% more tokens for the same text;
• standard synchronous API rates, not batch, fast mode, regional residency, or negotiated enterprise discounts;
• Claude cache misses treated as five-minute cache writes;
• GPT-5.6 Sol cache misses treated as cache writes at its documented 1.25× input rate;
• GPT-5.5 misses treated as normal uncached input;
• no separate tool-call, hosted-agent, storage, tax, or network charges; and
• “medium” reasoning effort does not alter the per-token rate; this table holds token counts constant rather than guessing different reasoning-token usage.

At standard or short-context rates

Sonnet 4.6 and Sonnet 5's post-promotion line are equal because that is what the official first-party Claude pricing page publishes. Opus 4.8 and Opus 5 are also equal on that page. The calculation does not invent a quality or token-volume premium where the requested assumption is a 0% usage increase.

If requests actually use approximately one million tokens of context

Long-context pricing changes the comparison:

Claude's official page states that Claude 4.6 and later include the full one-million-token context window at standard rates. OpenAI documents the greater-than-272K multiplier for GPT-5.5 and GPT-5.6 Sol. Grok 4.5 has a 500K maximum and doubles all token rates from 200K.

These are counterfactual estimates, not invoices. Different models can tokenise the same conversation differently, emit different amounts of reasoning, call different tools, finish in different numbers of turns, and achieve different cache-hit ratios. Holding usage fixed is useful for isolating price; it does not predict the total cost of rerunning the project with another model.

As a sanity check, scaling the observed DeepSeek charge ($77.39 / 10.5229B tokens) to 14B gives $102.96, rounded to $103. This is the estimated 14-billion-token cost of the observed V4 Flash/V4 Pro blend. It sits between the all-Flash and all-Pro modelled figures because the real workload mixed both models. The selected cache screenshots are also samples of three days rather than a billing-complete token-category ledger for every day.

What I learned about model “intelligence”

Before this project I placed more weight on using the best-ranked model. My conclusion is now more conditional.

If I do not understand a problem, model intelligence matters greatly. If I am the domain expert, know the required output, can inspect it quickly, and have objective tests, I may get more value from an affordable model that I can use continuously.

For this project, intelligence was not the scarce resource. The scarce resources were:

• enough context to retain the evolving design;
• enough affordable requests to sustain constant feedback;
• tests that made correctness observable;
• benchmark loops that made performance regressions observable; and
• my attention for API and domain review.

That is why DeepSeek was so effective here. It does not imply that the same choice is correct for a developer who is asking the model to discover SBE semantics on their behalf.

For somebody learning agentic coding on personal projects, my practical advice would now be to start with a small amount of inexpensive pay-as-you-go credit. It is difficult to learn sustained feedback loops when a premium subscription repeatedly stops the session at its usage limit.

A practical playbook for other developers

This project does not provide a universal recipe for AI-generated software. The conditions were unusually favourable. It does, however, suggest a repeatable method for projects where the developer already understands the domain and correctness can be made observable.

1. Choose a problem with an oracle

The best agent task is not merely one that can be described. It is one whose result can be disproved.

For this project, official SBE bytes were the strongest oracle. Compilation, round trips, generated-source inspection, exact lengths, allocation counts, and benchmark results supplied additional independent signals. If the only acceptance criterion had been “the API looks plausible,” I would not trust the result.

Before delegating a large implementation, ask what will turn a model's mistake into a concrete failure. If the answer is “a human will eventually notice,” the feedback loop is too weak.

2. Keep domain authority with a person who understands the problem

I did not ask the models to decide what SBE should mean. I knew the ordering rules, group structure, variable-data behaviour, schema-evolution concerns, and practical user mistakes before the project started.

An agent can implement a wrong specification extremely well. Tests derived only from the same wrong specification may all pass. The responsible person must be able to explain the invariant, recognise a plausible misinterpretation, and reject an API even when it is polished.

This is why I would not generalise my DeepSeek result to somebody learning SBE from the model while simultaneously asking it to build the generator. In that situation, there is no independent domain authority in the loop.

3. Define the observable contract before discussing implementation

I obtained better results when I said what generated calling code should look like, which illegal sequence should fail to compile, or which exact bytes should be produced. Vague goals such as “make it idiomatic” gave the agent too much freedom and usually created an ugly API.

A useful task statement includes:

• a representative input;
• the desired public call site;
• the expected output or failure;
• the invariant being protected;
• the tests that must pass; and
• the benchmark or allocation condition, if performance matters.

The model can then explore implementation details without owning the product decision.

4. Review the artefact closest to the real contract

Reviewing every generated line is ideal when it is practical, but enormous AI-authored diffs can exceed honest human review capacity. That does not justify pretending they were reviewed.

Instead, identify where your expertise has the highest leverage. I reviewed emitted Rust, public API shape, stage transitions, bytes, exact lengths, and performance more deeply than the syn/quote machinery. In another project the right surface might be SQL plans, a protocol trace, a rendered page, or a machine-generated configuration.

Be explicit about what this choice leaves unreviewed. Behavioural evidence narrows a trust boundary; it does not make the boundary disappear.

5. Prefer independent references over self-consistency

A codec that encodes and then decodes its own incorrect format can pass every round-trip test. A generator and a test written by the same model can share the same misunderstanding.

The most valuable tests compare against something independent:

• an official implementation;
• a protocol fixture produced by another language;
• a published conformance suite;
• a hand-calculated small example;
• an external parser or validator; or
• a previous production implementation.

For ergo-sbe, having both official sbe-tool generated Rust and the new generator encode the same logical message to identical bytes is far stronger than either codec decoding its own output.

6. Use short, mostly sequential loops after shared invariants emerge

Parallel agents are useful for truly independent research or isolated files. They were counterproductive when several branches changed the same generator conventions and API model.

The loop that worked was small: test, edit, regenerate, run, inspect, correct. When an agent moved in the wrong direction, stopping after minutes was better than reviewing an hour of coherent but unsuitable work.

The unit of delegation should shrink as the codebase becomes more coupled. Greenfield parallelism is not evidence that late-stage parallelism will work.

7. Give one editor ownership of an active change

I had less success when I manually edited files while an agent retained an older view of them. The agent's context was part of its working state. Silent external edits made that state stale.

My practical solution was to explain the correction and let the active agent make it. This is not a rule that humans must never code. It is a coordination rule: avoid two editors changing the same conceptual unit without an explicit reload and reconciliation step.

8. Treat agent guides as memory, not enforcement

CLAUDE.md was valuable for accumulating project preferences and previous mistakes. It did not reliably stop the agents from reintroducing unwrap() or non-chained examples.

If a rule is important, promote it from prose to something executable:

• a compiler error;
• a lint;
• a compile-fail test;
• a source scan;
• a formatter;
• a golden file; or
• a CI check.

Instructions influence probability. Automation changes the acceptance boundary.

9. Put performance in the definition of done

Performance-sensitive abstractions should be benchmarked from the beginning, not after the API has hardened. The generic type-state design taught me this the expensive way.

For every relevant change, the agent had to rerun benchmarks and investigate a regression rather than merely report it. This increased token use, but it prevented performance debt from accumulating invisibly.

Benchmark equal work. A comparison is meaningless if one side encodes fewer fields, omits a group, performs less validation, or uses a different buffer lifecycle. Generated APIs make accidental unequal work especially easy to hide.

10. Experiment with unsafe code; do not assume it is faster

I asked the agents to try unsafe variants because a checked boundary can make some repeated bounds checks redundant. Some experiments improved performance; others made no meaningful difference and were removed.

Every retained unsafe block needs:

• a stated invariant;
• a place where that invariant is established;
• tests around the boundary;
• a measured reason to keep it, unless borrowing genuinely requires it; and
• a clear distinction between checked and trusted public entry points.

“The LLM wrote it” is neither a safety proof nor a reason to reject it. The proof must stand independently of its author.

11. Preserve context deliberately and compact with a handoff

Long context helped because it contained rejected designs, benchmark history, style corrections, and domain explanations. Throwing that away mid-task made the agent repeat old mistakes. Compacting at a real task boundary—when the next work was sufficiently different—was usually the right trade.

At the same time, unlimited history increases cost and can bury the current task. A good compaction or handoff should preserve:

• decisions and their reasons;
• invariants;
• known failure modes;
• commands that prove completion;
• current benchmark baselines; and
• unfinished work.

Do not retain a million tokens merely because the window exists. Retain them when the accumulated decisions improve the next turn enough to justify their cache cost.

12. Price the workload, not the marketing number

For agentic coding, compare at least cache hits, cache writes or misses, output, context thresholds, and rate limits. A model with cheaper ordinary input may still be expensive for a repeated-prefix workload. A subscription that looks cheap may stop an intensive day halfway through.

Use your own dashboard split when possible. Then calculate:

hit tokens × hit price
+ miss/write tokens × miss/write price
+ output tokens × output price

Apply any long-context multiplier separately. Keep clear whether you are comparing equal token volume, equal elapsed time, or equal completed work. Those answer different questions.

13. Separate the agent harness from the model

Most implementation happened through Claude Code, but most implementation was not performed by a Claude model. The harness provided file access, tool use, session management, and interaction conventions. DeepSeek or GLM supplied the model behind it.

This distinction matters when reporting results and when reproducing them. A good model in a poor harness may be frustrating; a good harness can make a less expensive model highly productive. “I used Claude Code” is not a complete model-provenance statement.

14. Use frontier-model reviews as another opinion, not proof

Claude, OpenAI, and Grok reviews were useful. They sometimes expressed a problem better or found a slightly different angle. They were not an independent conformance oracle, and none rescued a task that DeepSeek could not complete in this project.

Models can share the same training-derived assumptions. Several agreeing that code looks correct is weaker than one official byte comparison demonstrating that it is correct for a specific case.

15. Stop if you cannot explain the result

The human responsible for merging a change should be able to explain:

• what requirement changed;
• why the implementation satisfies it;
• what evidence would fail if it were wrong;
• which surfaces remain unreviewed; and
• what operational risk remains.

If the explanation is “the agent seemed confident and all of its own tests passed,” the work is not ready.

16. Publish the uncomfortable facts

AI-assisted projects need more provenance, not less. State how much code was generated, where human review concentrated, which models and harnesses were used, which evidence is independent, what the work cost, and why the maturity label remains.

This disclosure is long because “AI-assisted” covers everything from autocomplete to a project in which an agent wrote nearly every implementation line. Users deciding whether to put a codec near money deserve the meaningful version.

The personal experience: pride, enjoyment, and review fatigue

I have written another open-source project almost entirely by hand. I felt a different connection to that code. Writing software is enjoyable; learning the small implementation details and keyboard shortcuts is part of the craft.

I do not feel the same ownership of this generator's individual lines because I did not write them. I am concerned that heavy LLM use can cause skill atrophy. It also changes the job from implementation to review.

That change was harder than I expected. I have historically been a very fast developer and often spent more time writing new code than reviewing other people's work. Reading code quickly to understand an API is not the same skill as auditing a huge generated diff line by line. Careful review is slower, more tedious, and easier to overwhelm. LLMs can generate code far faster than a person can honestly review it.

This project remained enjoyable because I was still solving a problem I care about. I spent a great deal of time thinking about SBE, Rust API design, verification, and performance. The LLM did much of the repetitive generator work that I already knew how to do and did not particularly want to spend six months repeating.

So the pride is attached differently:

• less attachment to the implementation;
• more satisfaction in the API and product;
• genuine pleasure that something I wanted throughout my working life now exists; and
• less reluctance to throw away experiments that do not work.

That last point helped with the Aeron Cluster code. The Cluster crate was initially a realistic consumer and test bed for ergo-sbe. Samples later became a better way to exercise complicated API shapes. The Cluster work remains less mature and more disposable. My priority is to make ergo-sbe right before treating the Cluster client as a finished product.

Why this crate is still experimental

The experimental label is deliberate.

I would be angry if a developer introduced an unproven codec into a financial system merely because it had many unit tests. If I apply that standard to somebody else's library, I must apply it to mine.

One thing I valued in the Java ecosystem was that mature projects often had visible institutional signals—long histories, broad production use, or governance under organisations such as Apache. The quality of an arbitrary Rust crate can be much harder to infer from its presentation. That makes explicit evidence and an honest maturity label more important, not less.

The suite is extensive. The official byte parity is meaningful. The benchmark gate is meaningful. None of that is the same as sustained production use across independent firms, schemas, traffic patterns, deployment environments, and upgrade cycles.

I will become comfortable removing the warning when there is a sufficient base of real users who tell me:

• they are using it in production;
• which features and schema shapes they use;
• how they compared it with their existing codecs;
• what volumes and environments it has survived;
• how schema evolution behaved; and
• what defects or operational surprises they found.

The reports I most want are already listed in the ergo-sbe README: multi-schema streams, DTO use, exact sizing with Aeron/IPC claims, nested or ragged books, and mixed acting versions.

Until that evidence exists, treat the crate as 0.x experimental software. Pin versions and perform your own migration testing.

What a prospective production user should verify

Do not rely on this narrative alone.

At minimum:

1. Generate codecs for your actual schemas.
2. Encode the same logical messages with your existing official SBE tooling and ergo-sbe; compare exact bytes.
3. Cross-decode in both directions.
4. Include empty, maximum-sized, nested, ragged, and variable-data cases.
5. Exercise every acting version and schema-evolution path you expect to see.
6. Test malformed and truncated frames at the trust boundary.
7. Verify exact buffer lengths before integrating with try_claim or another zero-copy publication API.
8. Benchmark your real hot fields and message shapes, not only the Car example.
9. Audit the internal unsafe assumptions relevant to your use of checked constructors and any private zero-check cores.
10. Run soak tests under real traffic and deployment conditions.

If you are not already comfortable explaining SBE block lengths, group dimensions, variable-data prefixes, acting versions, and positional ordering, do not use an LLM-generated explanation as your only review.

AI-assisted contributions

I am not applying a simplistic ban on AI-assisted pull requests, but I will not blindly accept them.

Large AI-generated PRs are difficult to review. More importantly, a contributor can ask a model to “fix” a protocol bug without understanding why the result is correct. Passing compilation is not enough.

For a generator or wire-format change, the contributor must understand the relevant SBE behaviour and be able to explain it. A focused issue containing:

• a real schema;
• a minimal reproduction;
• expected official bytes;
• a failing behavioural or parity test; and
• an explanation of the protocol invariant

may be more valuable than a large implementation patch. In many cases I would rather take that evidence and implement the change through the controlled workflow used for this repository.

AI assistance is not disqualifying. Lack of domain understanding is.

Final assessment

This project was a particularly favourable case for AI-assisted development:

• the maintainer was a domain expert;
• the desired behaviour was unusually concrete;
• the tedious part was mechanical code generation;
• the generated product was directly inspectable;
• an independent official implementation existed;
• wire bytes supplied a hard oracle;
• benchmarks constrained abstraction cost; and
• constant feedback was possible.

Even in that favourable case, it consumed about a month of intense work, roughly 18 billion tokens (measured by ccusage), repeated cleanup, extensive tests, and continuous human judgment. The fashionable version—write a specification, dispatch many agents, and return to a finished library—did not survive beyond the early greenfield stage.

I would use this development method again for a personal project with similar properties, and I would likely use DeepSeek again as the workhorse. I would not generalise the result into “LLMs can safely build any library” or “the cheapest model is always enough.”

The result is a crate whose API I genuinely wish I had during the last decade of SBE work. I am proud of that result. I am also being explicit that most of its implementation was written by models, that the generator internals did not receive exhaustive human line review, and that production maturity still has to be earned.

If you use ergo-sbe seriously, please report the schema shapes and features you exercise, including failures. Real-world evidence is what will make this project trustworthy—not another paragraph claiming that AI-generated code is either magically perfect or automatically worthless.

Verification & Release

cargo test -p ergo-sbe --all-features -- --test-threads=1
cargo test -p ergo-sbe --doc --all-features
RUSTDOCFLAGS="-D warnings" cargo doc -p ergo-sbe --all-features --no-deps
cargo clippy -p ergo-sbe --all-targets --all-features -- -D warnings

just test in the monorepo also runs doctests, docs_validation_test (README fences + generated-API smoke), and rustdoc with -D warnings.

Performance method: Benchmarks (not in the crates.io package).

SBE parity gate artifacts (1.0 streak)

After material codegen or before each release minor, archive no-LTO gate output so road-to-1.0.md criterion 2 (three consecutive minors ≤ 1.00) is auditable:

# from monorepo root — stamps target/bench-runs/<run-id>/ and gates at 1.00
just bench

Store the gate stdout in the GitHub release notes or CI artifact named sbe-bench-gate-no-lto.txt. Do not raise ceilings to pass.

Road to 1.0

Today both crates carry an honest not production-ready disclaimer. That is correct for a 0.1.x series with a still-open API freeze. This page is the published exit path from that disclaimer — criteria, not a date.

ergo-sbe (reaches 1.0 first)

Ship a 1.0.0 of ergo-sbe only when all of the following hold:

1. API freeze audit complete — decisions in API freeze are stable; no pending renames of generated stage / wrap / FixedFields surface without a major.
2. Parity gate — every maintained ergon vs sbe-tool comparison stays at or below the 1.00 ceiling under the published LTO matrix for at least three consecutive released minors (e.g. 0.1.9 → 0.1.10 → 0.1.11) with recorded Criterion runs in release notes or CI artifacts.
3. Wire compatibility — dual-encode parity tests and golden API shape remain green; no deliberate wire break without a major.
4. Trust boundary — fuzz corpus on decode entry stays green in CI; Miri fixtures for unaligned paths stay green; no known P0 safety issues open.
5. Docs — book chapters for migration (sbe-tool), trust boundaries, buffer sizing, and type-state design notes are published and linked from the crate README.
6. External signal — at least one external user (or production pilot) has reported wire + latency results against their own schema, or an equivalent published case study in the repo.

Until then the disclaimer stays, but it points here instead of reading as a permanent warning.

ergo-aeron-cluster (separate clock)

Cluster 1.0 is not tied to sbe 1.0. Additional criteria (illustrative):

• Stable session lifecycle and error types under multi-node test harness
• Documented Aeron version matrix and rusteron compatibility
• Codec generation locked to a released ergo-sbe major
• Separate performance gate (just bench-cluster) with recorded baselines

Cluster may remain 0.x after sbe 1.0.

What 1.0 is not

• Not “feature complete for every SBE edge case in every venue schema”
• Not a promise that your schema’s latency matches the car/L3 benches
• Not a freeze of optional config knobs’ defaults without changelog

Tracking

• Release process: Verification & Release.
• Changelog: repository root CHANGELOG.md.
• External pilot: External Schema Pilot.
• Cluster compatibility: Cluster Compatibility.
• API baseline manifest: api/public-api-baseline.toml. scripts/check-public-api.sh runs cargo-semver-checks on the two publishable crates. scripts/check-generated-public-api.sh diffs generated codec surfaces against api/generated/*.txt.
• Benchmark evidence: just bench + just bench-cluster write provenance-stamped Criterion trees; scripts/package-bench-artifacts.sh attaches them to a GitHub release. A number without a matching run-id / HEAD commit is not evidence.

Status (2026-08-16)

Release ancestry. Tag v0.1.17 exists on GitHub with assets, but it is not an ancestor of main (git describe --tags on main still reports a v0.1.15-* describe). Do not treat v0.1.17 as the tip of published history until that tag is merged or a replacement tag is cut from main.

External Schema Pilot

The 1.0 exit criterion requires an external user or production pilot with wire and latency results against their own schema — not only in-repo tests.

Status: Open

The FIX SBE Conformance Suite (sbe/tests/fix_sbe_conformance_test.rs, fixtures, and scripts/run-fix-sbe-conformance.sh) is necessary internal wire evidence. It is not the 1.0 external-signal criterion: there is no external user, no independent schema family, and no latency measurement from outside this repository.

Treat this page as the checklist for that missing signal, not a completed exit item.

Schema

FIX SBE baseline (v1-0-STANDARD + extension schemas) — the industry standard for financial exchange binary encoding.

Commands

# Run the conformance suite
cargo test -p ergo-sbe --test fix_sbe_conformance_test --all-features

# Optional: run the Java RL Validator (requires built suite)
./scripts/run-fix-sbe-conformance.sh

Package Scope

crates.io ships generator source, manifest, and this README. Tests, fixtures, samples, and benches live on GitHub only — use the links above.

License

Apache-2.0 · mimran1980/ergon