L3 Order Book

Deep nested / ragged L3 order-book sample for ergo-sbe. publish = false.

Conversion style: with_domain_type only

#![allow(unused)]
fn main() {
    use ergo_sbe::{ConversionSelector, GenerationConfig};
    // B — concrete mapping: one Rust type per wire type (already enables conversion)
    let _cfg = GenerationConfig::new("msgs").with_domain_type(
        ConversionSelector::named_type("Decimal"),
        "rust_decimal::Decimal",
    );
}

(From book/examples/conversion-config.rs. Full L3 build.rs: l3-book.)

Generated API (concrete):

enc.price(rust_decimal::Decimal::new(100, 0))?;
let p: rust_decimal::Decimal = dec.price()?;
let ts: DateTime<Utc> = dec.exchange_timestamp()?;

with_domain_type already enables conversion for that selector. Do not also call with_conversion(Decimal) here — it would not change the surface.

Encoding: size exactly, then write

Nothing here guesses a buffer size. The staged L3BookEncodedLength builder walks the same shape the encoder will write — bids_ragged with a nested orders count per level, then asks_ragged, then the var-data symbol — and returns the exact header-inclusive length:

pub fn book_encoded_length(
    bids: &Side<'_>,
    asks: &Side<'_>,
    symbol: &str,
) -> Result<usize, sbe_rt::EncodeError> {
    let after_bids = L3BookEncoder::compute_length().bids_ragged(bids.len() as u16, |g| {
        for (_, _, orders) in bids {
            g.add()?.orders(|og| {
                og.uniform(orders.len())?;
                Ok(())
            })?;
        }
        Ok(())
    })?;
    let after_asks = after_bids.asks_ragged(asks.len() as u16, |g| {
        for (_, _, orders) in asks {
            g.add()?.orders(|og| {
                og.uniform(orders.len())?;
                Ok(())
            })?;
        }
        Ok(())
    })?;
    let complete = after_asks.symbol(symbol.len())?;
    Ok(complete.encoded_length_with_header())
}

_ragged is the point for an order book: every level has a different number of orders, so the count cannot be multiplied out. g.add()?.orders(|og| …) declares one level and then that level’s own order count.

The encoder then writes into a buffer of precisely that length, in wire order, one chained expression:

pub fn encode_book(
    buf: &mut [u8],
    bids: &Side<'_>,
    asks: &Side<'_>,
    symbol: &str,
) -> Result<usize, sbe_rt::EncodeError> {
    let complete = L3BookEncoder::try_wrap_and_apply_header(buf, 0)?
        .fixed(&L3BookFixedFields {
            exchange_timestamp: 1_720_000_000_000_000_000u64,
            sequence: 42,
            is_active: true.into(),
        })
        .bids(bids.len() as u16, |g| {
            for (price, size, orders) in bids {
                g.add(|mut e| {
                    e.price(*price)
                        .map_err(|_| sbe_rt::EncodeError::DomainConversionFailed {
                            field: "price",
                            reason: "conversion",
                        })?;
                    e.size(*size)
                        .map_err(|_| sbe_rt::EncodeError::DomainConversionFailed {
                            field: "size",
                            reason: "conversion",
                        })?;
                    e.orders(orders.len() as u16, |og| {
                        for (oid, qty) in *orders {
                            let raw_qty =
                                SbeDecimal::new(qty.mantissa() as i64, -(qty.scale() as i8));
                            og.add_struct(&L3BookBidsOrdersEntry {
                                order_id: *oid,
                                quantity: raw_qty,
                            })?;
                        }
                        Ok(())
                    })
                })?;
            }
            Ok(())
        })?
        .asks(asks.len() as u16, |g| {
            for (price, size, orders) in asks {
                g.add(|mut e| {
                    e.price(*price)
                        .map_err(|_| sbe_rt::EncodeError::DomainConversionFailed {
                            field: "price",
                            reason: "conversion",
                        })?;
                    e.size(*size)
                        .map_err(|_| sbe_rt::EncodeError::DomainConversionFailed {
                            field: "size",
                            reason: "conversion",
                        })?;
                    e.orders(orders.len() as u16, |og| {
                        for (oid, qty) in *orders {
                            let raw_qty =
                                SbeDecimal::new(qty.mantissa() as i64, -(qty.scale() as i8));
                            og.add_struct(&L3BookAsksOrdersEntry {
                                order_id: *oid,
                                quantity: raw_qty,
                            })?;
                        }
                        Ok(())
                    })
                })?;
            }
            Ok(())
        })?
        .symbol_as_str(symbol)?;
    Ok(complete.encoded_length_with_header())
}

Both functions end with encoded_length_with_header(), and the tests assert the two agree — a length that does not match the bytes written is a bug in the sizing API, not something to paper over with a larger buffer.

symbol declares characterEncoding="ASCII" in the schema, so the generated encoder carries a symbol_as_str(&str) setter alongside the raw symbol(&[u8]) one. Every caller here always holds text, so encode_book takes symbol: &str and writes through symbol_as_str — the ASCII check runs before any byte is written rather than leaving it to the caller. Both setters write identical bytes for the same text; reach for the raw symbol(&[u8]) setter only when the caller genuinely starts from bytes.

let len = book_encoded_length(bids, asks, symbol)?;   // exact, header-inclusive
let mut storage = vec![0u8; len];                     // no oversize, no truncate
let actual = encode_book(&mut storage, bids, asks, symbol)?;
assert_eq!(len, actual);

See Exact sizing and Buffer sizing.

Decoding: random access or one staged chain

L3Book is the shape that makes the decoder lanes worth distinguishing: two sibling groups, each entry carrying a nested orders group, then a trailing var-data symbol. Reaching symbol means walking past every order of every level.

tests/l3_tests.rs encodes one ragged fixture and decodes it through the generated lanes, asserting they produce an identical snapshot. Note what the snapshot holds: the schema’s domain types (rust_decimal::Decimal, DateTime<Utc>, bool) because that is what with_domain_type generates, and a borrowed &'a str for symbol — copying it would defeat the flyweight.

/// Everything the fixture contains, in wire order — in the schema's *domain*
/// types, not its wire types.
///
/// This sample generates with `with_domain_type`, so `price` is
/// `rust_decimal::Decimal`, `exchangeTimestamp` is `DateTime<Utc>` and
/// `isActive` is `bool`. Comparing at that level is what proves the
/// conversions round-trip; comparing raw mantissas would pass even if the
/// domain layer were broken.
///
/// `symbol` is borrowed, not owned: every lane hands back `&'a str` pointing
/// into the wire buffer. Copying it would defeat the flyweight. The `Vec`s
/// are the *test's* comparison scaffolding — a decode has to materialise
/// something to compare — not part of the decode path, which allocates
/// nothing.
#[derive(Debug, PartialEq, Eq)]
struct Snapshot<'a> {
    timestamp: chrono::DateTime<chrono::Utc>,
    sequence: u64,
    is_active: bool,
    /// `(price, size, [(order_id, quantity)])` per level.
    bids: Vec<OwnedLevel>,
    asks: Vec<OwnedLevel>,
    symbol: &'a str,
}

/// A decoded level, owned so lanes can be compared. The decode itself borrows
/// — see `best_bid_and_depth` for the shape a hot path actually uses.
type OwnedLevel = (Rd, Rd, Vec<(u64, Rd)>);

/// Ragged on purpose: 3 bid levels with 2/1/3 orders and 2 ask levels with
/// 1/2. A uniform fixture would let a lane compute entry positions by
/// multiplication and still look correct.
fn fixture() -> (Vec<u8>, Vec<OwnedLevel>, Vec<OwnedLevel>) {
    let o_b1 = [(101u64, d(5)), (102, d(7))];
    let o_b2 = [(103u64, d(25))];
    let o_b3 = [(104u64, d(1)), (105, d(2)), (106, d(3))];
    let bids: &[l3_book::Level<'_>] = &[
        (d(50800), d(15), &o_b1),
        (d(50750), d(40), &o_b2),
        (d(50700), d(60), &o_b3),
    ];
    let o_a1 = [(201u64, d(10))];
    let o_a2 = [(202u64, d(20)), (203, d(30))];
    let asks: &[l3_book::Level<'_>] = &[(d(50850), d(20), &o_a1), (d(50900), d(35), &o_a2)];
    let symbol = "BTCUSDT";

    // Exact sizing through the staged builder, then encode into a buffer of
    // precisely that length: no oversize, no truncate.
    let len = l3_book::book_encoded_length(bids, asks, symbol).expect("length");
    let mut storage = vec![0u8; len];
    let actual = l3_book::encode_book(&mut storage, bids, asks, symbol).expect("encode");
    assert_eq!(len, actual, "computed length must match encoded length");

    let own = |lv: &(Rd, Rd, &[(u64, Rd)])| (lv.0, lv.1, lv.2.to_vec());
    (
        storage,
        bids.iter().map(own).collect(),
        asks.iter().map(own).collect(),
    )
}

Random access — any order, Sync, re-walks

/// Lane 1 — random access (the base decoder from `try_decode`).
///
/// Getters may be called in any order and the decoder is `Sync`. Each
/// dynamic-tail getter re-walks from the fixed block, so reading `bids`,
/// `asks` and `symbol` walks the bid orders three times over. Correct
/// everywhere, cheapest when you want one or two fields.
fn decode_random_access(wire: &[u8]) -> Result<Snapshot<'_>, Box<dyn std::error::Error>> {
    let dec = L3BookDecoder::try_decode(wire, 0)?;

    // Deliberately out of wire order: symbol (the last tail) first, then back
    // to the groups. Only a lane that resolves tails independently can do
    // this. The `&str` borrows the wire buffer — nothing is copied.
    let symbol = dec.symbol_as_str()?;

    let mut bids = Vec::new();
    for level in dec.bids()? {
        let level = level?;
        let mut orders = Vec::new();
        // `orders` is fixed-stride, so its whole region was proven in bounds
        // when the group was entered and the iterator yields entries directly.
        // The outer `bids`/`asks` iterators yield `Result` because a dynamic
        // entry's extent can only be known once it is reached.
        for order in level.orders()? {
            orders.push((order.order_id(), order.quantity()?));
        }
        bids.push((level.price()?, level.size()?, orders));
    }

    let mut asks = Vec::new();
    for level in dec.asks()? {
        let level = level?;
        let mut orders = Vec::new();
        for order in level.orders()? {
            orders.push((order.order_id(), order.quantity()?));
        }
        asks.push((level.price()?, level.size()?, orders));
    }

    Ok(Snapshot {
        timestamp: dec.exchange_timestamp()?,
        sequence: dec.sequence(),
        is_active: dec.is_active()?,
        bids,
        asks,
        symbol,
    })
}

Note it reads symbol — the last tail — first, then goes back to the groups. Only a lane that resolves each tail independently can do that. The cost is that every dynamic-tail getter re-walks from the fixed block.

Staged — wrong order is a compile error

/// Sequential decode — `into_*` / `skip_*`, compile-time order. The group
/// iterator is the stage: `into_asks` skips unread bids.
fn decode_staged(wire: &[u8]) -> Result<Snapshot<'_>, Box<dyn std::error::Error>> {
    let dec = L3BookDecoder::try_decode(wire, 0)?;
    let timestamp = dec.exchange_timestamp()?;
    let sequence = dec.sequence();
    let is_active = dec.is_active()?;

    // Levels carry a nested `orders` group, so they have no stride and take a
    // visit closure — the completion it returns is where the next level starts.
    // `orders` entries are fixed-stride, so they are a plain iterator.
    let mut bids = Vec::new();
    let mut asks = Vec::new();
    let (symbol, _complete) = dec
        .into_bids(|level| -> Result<_, Box<dyn std::error::Error>> {
            let price = level.price()?;
            let size = level.size()?;
            let mut orders = Vec::new();
            let mut order_iter = level.into_orders()?;
            for order in &mut order_iter {
                orders.push((order.order_id(), order.quantity()?));
            }
            bids.push((price, size, orders));
            Ok(order_iter.finish()?)
        })?
        .into_asks(|level| -> Result<_, Box<dyn std::error::Error>> {
            let price = level.price()?;
            let size = level.size()?;
            let mut orders = Vec::new();
            let mut order_iter = level.into_orders()?;
            for order in &mut order_iter {
                orders.push((order.order_id(), order.quantity()?));
            }
            asks.push((price, size, orders));
            Ok(order_iter.finish()?)
        })?
        .into_symbol_as_str()?;

    Ok(Snapshot {
        timestamp,
        sequence,
        is_active,
        bids,
        asks,
        symbol,
    })
}

into_symbol exists only on the stage reached after asks completes, so the out-of-order read above is not expressible here. One wire-order pass.

Memoized — any order, each boundary walked once

/// Lane 3 — memoized (`decoder.memoized()`).
///
/// Same getter names as the base lane, plus a progressive cache of discovered
/// tail boundaries. Read here in the worst order for the base lane — symbol
/// first, then back to the groups, then symbol again — which is exactly the
/// shape the cache exists for.
fn decode_memoized(wire: &[u8]) -> Result<Snapshot<'_>, Box<dyn std::error::Error>> {
    let dec = L3BookDecoder::try_decode(wire, 0)?.memoized();

    let symbol = dec.symbol_as_str()?;

    let mut bids = Vec::new();
    for level in dec.bids()? {
        let level = level?;
        let mut orders = Vec::new();
        for order in level.orders()? {
            orders.push((order.order_id(), order.quantity()?));
        }
        bids.push((level.price()?, level.size()?, orders));
    }

    let mut asks = Vec::new();
    for level in dec.asks()? {
        let level = level?;
        let mut orders = Vec::new();
        for order in level.orders()? {
            orders.push((order.order_id(), order.quantity()?));
        }
        asks.push((level.price()?, level.size()?, orders));
    }

    // Re-reading a tail is free once its boundary is known: the base lane
    // would re-walk every bid and ask order to get back here.
    assert_eq!(dec.symbol_as_str()?, symbol, "re-read must be identical");

    Ok(Snapshot {
        timestamp: dec.exchange_timestamp()?,
        sequence: dec.sequence(),
        is_active: dec.is_active()?,
        bids,
        asks,
        symbol,
    })
}

Same getter names as random access. Read in the order shown — symbol, then back to the groups, then symbol again — the base lane re-walks every bid and ask order to return to symbol; this lane reuses the boundary it already found. Build it once and pass &L3BookMemoizedDecoder around: calling .memoized() in each function creates a separate empty cache.

Ordered — one spelling, all the way down

/// Lane 4 — ordered (`decoder.ordered()`). One callback per tail, in wire
/// order, with the *same spelling* at every tail, plus an `EntryInfo` giving
/// each entry's position.
///
/// Compare with `decode_staged`: there, `bids` takes a visit closure and
/// `orders` is an iterator, because their shapes genuinely differ. Here every
/// message-level tail — fixed block, both groups, and the var-data — reads the
/// same way. That uniformity is why this lane exists: code that walks whole
/// messages writes one form instead of branching on each tail's shape.
///
/// It goes all the way down. A `bids` level carries its own tail (the nested
/// `orders` group), so it has an `ordered()` of its own, and the spelling does
/// not change at the entry boundary. Returning that nested walk is the
/// completion the parent owes — no `.done()` on the way out. The message
/// chain ends on `encoded_length_with_header()`, like encode.
///
/// The staged `level.into_orders()` iterator is still generated beside it and
/// is still the better tool when you want to `break` early — `orders` is
/// fixed-stride, so its iterator is an `ExactSizeIterator` and stopping short
/// still reaches the next tail by arithmetic. The lane adds a spelling; it
/// removes nothing.
///
/// `EntryInfo::count` is the wire-declared `numInGroup`, read from the group's
/// dimension header before the walk starts, so the `Vec` is sized once rather
/// than grown. It costs no scan. `bids` and `asks` still need separate bodies
/// because their entry decoders are distinct types — the lane unifies how a
/// tail is *reached*, not what its entries are.
fn decode_ordered(wire: &[u8]) -> Result<Snapshot<'_>, Box<dyn std::error::Error>> {
    // Ordered callbacks return `DecodeError`; a domain getter fails with its
    // conversion's own error, mapped the way the generated DTO maps it.
    fn domain<T, E>(field: &'static str, r: Result<T, E>) -> Result<T, sbe_rt::DecodeError> {
        r.map_err(|_| sbe_rt::DecodeError::DomainConversionFailed {
            field,
            reason: "domain conversion rejected wire value",
        })
    }

    let mut timestamp = None;
    let mut sequence = 0u64;
    let mut is_active = false;
    let mut bids: Vec<OwnedLevel> = Vec::new();
    let mut asks: Vec<OwnedLevel> = Vec::new();
    let mut symbol: Option<&str> = None;

    let len = L3BookDecoder::try_decode(wire, 0)?
        .ordered()
        .fixed(|d| {
            timestamp = Some(domain("exchange_timestamp", d.exchange_timestamp())?);
            sequence = d.sequence();
            is_active = domain("is_active", d.is_active())?;
            Ok(())
        })?
        .bids(|level, info| {
            if info.is_first() {
                bids.reserve_exact(info.count);
            }
            let price = domain("price", level.price())?;
            let size = domain("size", level.size())?;
            // Nested `ordered()` returns the completion the parent owes —
            // no `.done()` on the way out.
            let mut orders = Vec::new();
            let complete = level.ordered().orders(|order, _oinfo| {
                orders.push((order.order_id(), domain("quantity", order.quantity())?));
                Ok(())
            })?;
            bids.push((price, size, orders));
            Ok(complete)
        })?
        .asks(|level, info| {
            if info.is_first() {
                asks.reserve_exact(info.count);
            }
            let price = domain("price", level.price())?;
            let size = domain("size", level.size())?;
            let mut orders = Vec::new();
            let complete = level.ordered().orders(|order, _oinfo| {
                orders.push((order.order_id(), domain("quantity", order.quantity())?));
                Ok(())
            })?;
            asks.push((price, size, orders));
            Ok(complete)
        })?
        .symbol_as_str(|s| {
            symbol = Some(s);
            Ok(())
        })?
        .encoded_length_with_header();
    debug_assert_eq!(len, wire.len());

    Ok(Snapshot {
        timestamp: timestamp.ok_or("fixed callback must run")?,
        sequence,
        is_active,
        bids,
        asks,
        symbol: symbol.ok_or("symbol callback must run")?,
    })
}

Compare it with the staged lane above: there bids takes a visit closure and orders is an iterator, because those two shapes genuinely differ. Here every message-level tail — fixed block, both groups, then the var-data — reads the same way, and each entry callback also receives an EntryInfo carrying index, the wire-declared count, and the acting block_length. Count and block length come from the group decoder the staged walk already opened; the index advances with the walk. Using info.count to reserve space needs no entry scan. Empty groups never deliver EntryInfo; <group>_count() on the ordered stage still reports the declared count.

It recurses. A bids level carries its own tail — the nested orders group — so the level decoder has an ordered() too, and the callback above uses it. The spelling does not change at the entry boundary, and returning that nested walk is the completion the parent closure owes — no .done() on the way out. An entry with no tails of its own (a fixed-stride entry) has nothing to order and gets no lane. An entry whose existing getter is named ordered also keeps its getter and uses the staged spelling for its tails.

The staged spelling is still generated beside it: level.into_orders() hands back a real ExactSizeIterator you can break out of and still reach the next tail by arithmetic. Pick the iterator when you want to stop early and continue at the next tail, or the ordered lane when you want callbacks throughout.

It is a façade over the staged stages, so it keeps their single traversal and compile-time tail order. The message chain ends on encoded_length_with_header(), like encode. Dynamic groups fill EntryInfo from that walk; fixed-stride groups reuse the iterator’s count and stride. Neither shape pre-scans entries or re-opens the dimension header. The optimizer and measured workload determine the callback wrapper’s final cost. Message-level fixed receives a fixed-fields-only view. Callbacks return DecodeError; convert to an application error after the walk.

On a hot path, collect nothing

The four functions above build owned Vecs because a test has to materialise something to compare. Real consumption does not: the decoders are flyweights over the wire buffer, &str and &[u8] borrow from it, and a full nested walk needs no allocation at all.

/// The shape you actually want on a hot path: no `Vec`, no `String`, no copy.
///
/// The lane functions above build owned collections because a test has to
/// materialise something to compare. Real consumption does not. Here every
/// level and every nested order is visited, `symbol` is used as a borrowed
/// `&str`, and the only state is a handful of scalars in registers — the
/// generated decoders are flyweights over the wire buffer and allocate
/// nothing, so this whole walk is allocation-free.
///
/// A price is compared as a price, not as a raw mantissa: the wire `Decimal`
/// carries its own exponent, and `1.00` (mantissa 100, exponent −2) is less
/// than `99` (mantissa 99, exponent 0). The fallible domain conversion is the
/// one place that knows how to align the two.
fn best_bid_and_depth(wire: &[u8]) -> Result<(Rd, u64, usize, &str), sbe_rt::DecodeError> {
    let mut best_bid: Option<Rd> = None;
    let mut total_orders = 0u64;
    let mut levels = 0usize;

    let (symbol, _done) = L3BookDecoder::try_decode(wire, 0)?
        .into_bids(|level| -> Result<_, sbe_rt::DecodeError> {
            levels += 1;
            let px = level
                .price()
                .map_err(|reason| sbe_rt::DecodeError::DomainConversionFailed {
                    field: "price",
                    reason,
                })?;
            if best_bid.is_none_or(|b| px > b) {
                best_bid = Some(px);
            }
            let mut orders = level.into_orders()?;
            // Fixed stride, so the count is exact without walking.
            let entries = &mut orders;
            total_orders += entries.len() as u64;
            orders.finish()
        })?
        .skip_asks()?
        .into_symbol_as_str()?;

    Ok((
        best_bid.ok_or(sbe_rt::DecodeError::DomainConversionFailed {
            field: "bids",
            reason: "no bid levels",
        })?,
        total_orders,
        levels,
        symbol,
    ))
}

hot_path_walk_borrows_everything asserts the returned &str points inside the wire buffer rather than at a copy. Repository-wide, allocation_count_test pins the same property for generated decode under a counting allocator.

Which one

Full comparison, including how each differs from sbe-tool’s single limit cursor: Decoder lanes.

Layout

Run

cargo run  --manifest-path samples/l3-book/Cargo.toml
cargo test --manifest-path samples/l3-book/Cargo.toml