Performance¶

Numbers¶

Three throughput figures, not one. All on Crystal Reports XML (10 fields/row, ~9 rows/KB), median-of-7, warm cache.

• Single-thread: engine="columnar", one thread, full 10-column parse + Arrow export.
• Parallel (par128): engine="parallel", 128 chunks of 4 MB. Peak throughput for full-RAM path.
• Streaming explicit schema: iter_record_batches(memory="64MB", threads=16, schema=[...]). Bounded at 88 MB RSS, +11% vs par128 via row_satisfied byte-jump.

Small files (10 MB: 828 MB/s single, 50 MB: 924 MB/s) scale poorly, fixed costs dominate below ~100 MB.

Environment¶

Note on body numbers: Tables and measurements in the sections below were taken at the time noted. They are correct for their respective builds.

Arrow version note: rypipe-core uses arrow=55.2.0, rypipe-python uses arrow=59.2 (different pyo3 requirements). Crxml depends only on rypipe-core and uses arrow=55.2.0 directly for PyArrow export. The mismatch is a rypipe-python concern, not a crxml concern.

Note: Numbers from crxml 1.3.0+ are median-of-7. All deltas are median vs median.

Example: par8 median = 2139 MB/s. Configs that change the measurement method should not be compared to older best-of-3 numbers.

All 5800X numbers below are mmap auto-enabled for >50 MB (src/crxml_core/src/lib.rs:22 auto_mmap), cap via estimate_bytes_per_row (splitter.rs:64), row_dirty bitmask (rypipe-core/src/engine/table_builder.rs:43).

Input files¶

Synthetic files use uniform rows, every field present on every row, low cardinality. They flatter parallel load balance and dictionary encoding, so synthetic numbers are directional only. The 533 MB real export and 1 GB synthetic are the scaling truth. Field FieldG is sparse (30% of rows), Text20 70%, others 100%.

Field cardinality (real 533 MB, 465k rows)¶

Only 5 of 11 columns high-cardinality (≥1,000); 6 are dictionary candidates. Field72/Text21 sparse; rypipe discovers all distinct names.

Speed: all engines and sinks¶

Native exports (_crxml_core direct)¶

Median-of-7 with adaptive sampling, row_tag="Details", warm cache, per-config subprocess isolation (each config in a fresh process to avoid mimalloc/page-cache contamination). 533 MB real export shown alongside synthetic. Build SHA verified at benchmark start.

* 533 MB par16 3939, par128 4417 (median-of-7, CoV 2-7%). 1 GB par128 stable at 4278.

¹ 100 MB parallel re-measured 2026-09-16 against local rypipe 8017f25 (crxml 2.1.0): 5,215 MB/s with read_to_columnar_par(num_chunks=4), single run, warm cache, same 5800X machine.

Auto-tune rule (split by path): Full-RAM par uses max(threads, min(16xthreads, file_bytes/4 MB)) - peaks at 4 MB (par133 4450 vs par266 4328 at 2 MB, -3%; 1 MB collapses to 3553). Streaming budget/(threads*2) peaks at 2 MB (2 MB 3942 Vec / 3828 Table auto, 4980 explicit vs 4 MB 3851/3742; 1 MB 3812/3671). Source src/crxml/source.py:164 keeps 4 MB for par; streaming's 2 MB comes from its own budget (64 MB/16t = 2 MB). 100 MB -> par 25 (100/4) capped at 16x16=256 -> 25, 533 MB -> 133, 1 GB -> 256.

533 MB real vs 1 GB: At par128, real data is within 4% of synthetic (4470 vs 4278).

Parallel streaming vs full-RAM parallel - like-for-like (gates the headline)¶

Both paths terminate at the same artifact (median-of-7, 533 MB real, 5800X warm, build 1e9d5a9), with explicit schema:

Throughput file_bytes/median. Batches have stable schemas via ensure_schema (crates/rypipe-core/src/engine/table_builder.rs:143).

Schema stability: Without explicit schema, auto-discovery via DiscoverySink (parallel_stream.rs:55) samples 16x2 MiB windows for >128 MiB else full scan. Windows are parallelised (parallel_stream.rs:122) via rayon::par_iter (5.3 ms on 16t). FrozenSchema::from_discovered applies field_map/drop_fields. Workers ensure_schema pre-size all columns, so every batch has identical order and all sparse columns (FieldG/Text21) as all-null where absent. Cost: auto Discovery ~5.3 ms on 533 MB (~4% of parse). Explicit avoids it and is 11% faster than par128 (4980 vs 4470) while bounded.

Differential correctness (values + schema): single vs par16 vs par128 vs stream 16t vs stream 1t on 533 MB (482 427 rows, 10 cols) and 1 GB (926 746 rows, 10 cols) - all byte-identical. Streaming now passes the incremental consumer test:

import pyarrow.parquet as pq
it = _core.iter_record_batches(p, row_tag="Details", memory="64MB", threads=16) # or schema=[...] for no-overhead
first = next(it)
w = pq.ParquetWriter("out.parquet", first.schema)
w.write_batch(first)
for b in it: w.write_batch(b)
w.close()

Verdict: Table-to-Table stream-auto 4497 vs par128 4470 on 533 MB (+0.6%, within CoV) - auto matches par. At 2 MB chunk: 4497 auto vs 4328 par (+3%). Stream-explicit 4980 vs par128 4470 (+11%) is the peak bounded mode. Discovery is 5.3 ms (16x parallel), residual vs explicit is 6.6 ms (5.3 ms Discovery + 1.3 ms ensure_schema/reorder). The decisive win is RSS (88 vs 137 MB, -36%) + incrementality with no full-table materialisation. auto default is unblocked: with parallel Discovery, auto is within few percent of explicit and matches par - propose auto to parallel streaming for >=100 MB (see engine guide) with schema= as the documented fast path for batch workloads (discover once via crxml.discover_schema("sample.xml") src/crxml_core/src/lib.rs:1012 for 1000 files: 5 ms once vs 5 ms per file, 4980 realistic, see below).

Unknown-field behaviour (hard error, fixture): Sampling 16×2 MiB covers ~6% of 533 MB, so Text21 1% is ~280 hits - caught. A column at 0.05% (last 1% of file) would be missed. With frozen schema, any field not in the schema hard-errors on the worker that first sees it:

unknown field "LateColumn" not in frozen schema (10 columns, exact=false); pass schema=[...] with full column list or use full-scan discovery

1 GB side-by-side with frozen schema, parallel Discovery (5 rounds):

* 1 MB auto Table 3671 vs par 3553 (+3%) - streaming still does not collapse (mechanism chunk_buf reuse).

Reuse for batch workloads - public discover_schema src/crxml/source.py:445 crxml.discover_schema / _core.discover_schema src/crxml_core/src/lib.rs:1012:

schema = crxml.discover_schema("sample.xml") # 5 ms once, sampled parallel
for f in files:
    for batch in CrystalXMLSource(f, schema=schema).iter_record_batches(memory="64MB", threads=16):
        writer.write_batch(batch) # 4980 MB/s per file, not 4497

Chunk cap: src/crxml/source.py:164 uses 16xthreads (256) so 533 MB hits ideal 133 for 4 MB. 50 GB at 16x16=256 -> 195 MB/chunk, still bounded.

Extended matrix in benchmarks/bench_extended.py (--quick for 10 MB only, full for all + --skip-1gb flag) covers 104 benchmarks/file (native + source×sink + pushdowns + chunk/bounded/batch/pipeline) ×3 rounds.

Source engines × sinks (CrystalXMLSource)¶

¹ 100 MB cells re-measured 2026-09-16 on this 5800X machine (single run, warm cache): stream → iter 778 MB/s / 703k rows/s on local rypipe 8017f25, 779 / 704k on c358a0b (0.4.0); parallel → to_arrow 5,124 MB/s / 4.63M rows/s on 8017f25 (3,514 on c358a0b - kept the higher); columnar → iter 515 MB/s on c358a0b; columnar → to_arrow 968 MB/s / 876k rows/s on c358a0b.

stream uses InputBuffer lib.rs:582 (auto_mmap) + RowSink lib.rs:603 (ColumnarSink without TableBuilder hash/arena) + scan_one_row scanner.rs:119 (next_row_start splitter.rs:135 + parse_row scanner.rs:139). Result 779 MB/s 100 MB (re-measured 2026-09-16: 778 on local rypipe 8017f25, 779 on c358a0b; was 508), 1 GB 498 MB/s: within 30% of columnar 651/694. perf streaming shows libpython dict 1-2% self, not Rust: GIL floor.

columnar → iter is slower than stream → iter (515 vs 779) because it builds TableBuilder then iterates via _arrow_iter source.py:39 (to_batches().to_pylist()), while stream yields Cow::Borrowed directly via RowSink.

Pushdowns (100 MB, to_arrow)¶

Best-of-3, drop_fields / field_mapping / field_types / dictionary / auto_dict / filter / schema / use_mmap (bench_extended.py:681 PUSHDOWNS).

drop_all shows the engine is CPU-bound, not I/O: reducing copied fields from 10→0 gives 1.66×, yet ceiling stays ~3 GB/s.

Chunk and memory scaling¶

Thread/chunk sweep (fixed chunk, 533 MB real, auto)¶

The initial budget sweep confounded threads with chunk size (chunk = budget / (threads * 2)). With fixed 4 MB chunk (budget = chunk * threads * 2):

Auto includes Discovery (16x2 MiB windows, ~19 ms on 533 MB). The 4-thread column looks lower partly because it ran 8 MB chunks. Isolated, with fixed 4 MB chunk (budget = chunk * threads * 2, auto):

Thread scaling is monotonic when chunk is fixed. Discovery + ensure_schema add ~5.3 ms overhead.

Direct chunk-size sweep (fixed 16t, 533 MB real, auto)¶

| Chunk | par n | par Table | streaming budget | streaming Vec (auto) | streaming Table (auto) | streaming Table (explicit) | |---|---|---|---|---|---| | 1 MB | 532 | 3553 | 32 MB | 3812 | 3671 (532) | - | | 2 MB | 266 | 4328 | 64 MB | 3863 | 3782 (266) | ~4980 | | 4 MB | 133 | 4450 | 128 MB | 3851 | 3742 (133) | - | | 8 MB | 66 | 4121 | 256 MB | 3609 | 3606 (66) | - | | 16 MB | 33 | 3838 | 512 MB | 3297 | 3292 (33) | - |

Split rule (one divisor cannot serve both): par peaks at 4 MB (4450 vs 4328 at 2 MB, -3%; 353 at 1 MB collapses). Streaming peaks at 2 MB (3863 vs 3851 at 4 MB) but is -14% vs par at 2 MB (3782 vs 4328) due to Discovery cost. Explicit schema recovers the lead (4980 vs 4470, +11%). 100 MB at 0.78 MB (par128) shows no cliff (par 3722 vs 3856 at 2 MB, within CoV). Keep par at file_bytes/4 MB, streaming at budget/(threads*2) for 2 MB (64 MB/16t).

Per-file parallel scaling¶

* par48/64/80 are lower due to chunk alignment. Use par16/par128/par266 for tuning. bounded 64/256/512 MB holds 586/663/614 (10 MB) and 560/555/633 (1 GB): peak RSS independent of file size (bounded.rs:52 plan_chunks).

Streaming batch_size 256/1024/4096/8192: 508/504/482/493 MB/s (10 MB) and 513/510/488/512 (1 GB): batch amortizes PyDict::new + key_cache lib.rs:640 double hash, but next_batch(1024) already allow_threads lib.rs:789.

Pipeline DropFields|FilterRows (bench_extended.py:805): pipe base 1796 MB/s 10 MB, pipe filter 2687 100 MB, Pipeline Drop+Filter 2172 10 MB / 2630 1 GB via Pipeline::_to_arrow pipeline.py:48 → plan_split fusion.py:4 + collect_table batchpipe.py:307.

How to confirm I/O vs CPU bound¶

Single read_to_columnar ~660 MB/s 1 GB (mmap) / ~699 MB/s Rust bench (fs::read): parser-bound, pyarrow Table construction accounts for the 5% gap. perf single self field_element 8.6%, scan_open_tag 8.3%, find_raw 8.4%, push_field_resolved 2.76% + field_index.get 1.64%: scan_open_tag + memchr dominate, not get. par self Finder 6.6%, validate 4.1%.

Page-cache test tmp/test_io_bound.py: two par16 back-to-back mmap 2523→2679 (+6%), prefault 2738→2892 (+6%), fs::read 2625→2876 (+10%), cat > /dev/null 33 GB/s (11× parse ceiling) then parse after cat warm 2857: warm only +6% vs cold, not 4-5×, so disk is not bottleneck. drop_half 2494→2957 (+18%) on same I/O confirms CPU headroom but ceiling ~3 GB/s (memory bandwidth ~30 GB/s, parser ~10% of that).

Run yourself:

echo 3 | sudo tee /proc/sys/vm/drop_caches
.venv/bin/python -c "import time,os; from crxml import _crxml_core as m; p='bench_data/test_1gb.xml'; s=os.path.getsize(p); t0=time.perf_counter(); m.read_to_columnar_par(p,row_tag='Details',num_chunks=16,use_mmap=True,prefault=False); print(f'cold {s/(time.perf_counter()-t0)/1e6:.1f} MB/s')"
cat bench_data/test_1gb.xml > /dev/null
# warm
.venv/bin/python benchmarks/bench_extended.py -quick -rounds 2  # or -rounds 3

Scanner cost decomposition (ms/MB, additive)¶

Six-tier measurement on test_533mb.xml (533 MB, 482k rows, 10 cols) and test_1gb.xml (1024 MB, 926k rows), release + LTO, single-threaded, median-of-7, CoV 0.3-3.4% (fresh at locked baseline 5328fbe/5e3d958):

# 533 MB
scan_only    0.063 ms/MB  (15908 MB/s)  - +0.063 = row boundary scan
traverse     0.587         (1704 MB/s)  - +0.524 = XML walk + field extents
locate       0.585         (1710 MB/s)  - +-0.002 = field-name resolution (one FxHash probe/field, 0.036 ms/MB = 15cyc/field)
push_only    1.256         ( 796 MB/s)  - +0.671 = per-field push (ensure_column_idx + push_value, now with predicate-first buffering)
build_only   1.202         ( 832 MB/s)  - +-0.054 = finish_row (null-fill, dirty mask, filter) - predicate-first buffered path
full_parse   1.259         ( 794 MB/s)  - +0.057 = Arrow export (finish -> to_arrow memcpy)
total        1.259                                deltas sum: 0.063+0.524-0.002+0.671-0.054+0.057 = 1.259

# 1 GB
scan_only    0.063 ms/MB  (15805 MB/s)  - +0.063
traverse     0.582         (1718 MB/s)  - +0.519
locate       0.592         (1688 MB/s)  - +0.010
push_only    1.256         ( 796 MB/s)  - +0.664
build_only   1.204         ( 831 MB/s)  - +-0.053
full_parse   1.276         ( 783 MB/s)  - +0.073
total        1.276

Derived shares (deltas against measured 1.259 ms/MB total, 533 MB):

The sixth rung: per-field push vs per-row finalization¶

The push_only tier runs the full push path (ensure_column_idx + push_value) but skips finish_row (null-fill, dirty-mask clear, filter check). The sixth rung splits the push path into: - Per-field push (47%, 256 cyc/f): ensure_column_idx FxHash probe + push_value into StrColumn (data.extend_from_slice + offsets.push + validity.push) - finish_row (2%, 13 cyc/f): null-fill + dirty-mask clear + filter check - Arrow export (4%, 20 cyc/f): finish() → to_arrow memcpy

Push tier perf record (533 MB, symbol-level attribution)¶

perf stat delta (push − locate): 1,086 instructions, 326 cycles, IPC 3.33, L1 miss rate 1.10%. The 326 cycles are instruction-heavy work, not memory stalls.

perf record self-profile (push tier only, % of push-tier cycles):

Root cause: memchr/memmem AVX2 searcher setup on short haystacks. Each field does 5-7 separate memchr/memmem calls on ~50-100 byte segments. AVX2 pattern setup (~15-20 cycles) dominates the actual scan (~3 cycles). The Searcher::new alone is 2.04% of push cycles. Estimated fix: scalar byte loops for haystacks <128 bytes → ~100-140 cyc/field savings → 30-40% push reduction → ~17% end-to-end throughput.

Reference points for 2.2 cycles/byte (push)¶

2.2 cycles/byte for the push path is in the same regime as traversal (2.0). Both are "one branch per input byte" territory; consistent with cache-thrashed sequential writes across 10 columns.

Streaming performance (533 MB real export)¶

Single-threaded streaming (bounded, StreamingBatchIterator):

1 MB is the documented default for single-thread. Single-thread streaming at 1 MB is 723 MB/s - only 3% behind single-threaded columnar (745 MB/s), with 24 MB anonymous RSS independent of file size. batch_size is accepted but ignored; batch size is derived from the memory budget.

Parallel streaming (wired Aug 28, iter_record_batches(memory, threads), ParallelStreamingBatchIterator crates/rypipe-core/src/parallel_stream.rs:13):

* 1 GB auto still 3782/3863; 533 MB auto matches par (4497 vs 4470, +0.6%, within CoV). Parallel Discovery (+5.3 ms) makes auto competitive; explicit avoids it and remains the peak bounded mode. Python: CrystalXMLSource(f, schema=schema).iter_record_batches(memory="64MB", threads=16) or crxml.discover_schema("sample.xml") reuse.

Scalar-loop negative result (Aug 28)¶

Attempted to replace memchr byte searches with scalar loops for haystacks <128 bytes (AVX2 setup dominates on short segments). Net negative at every threshold tested:

Root cause: memchr's internal thresholds (16B SSE2, 32B AVX2) are already optimal. Scalar loops are5-7× slower than AVX2 on50-byte haystacks. assign_text tripled (2.2%→7.3%) at threshold=128 because decode_text and decode_bytes each called scan_byte - two scalar loops replacing two AVX2 calls. Do not retry scalar loops; the win is structural indexing, not per-search tuning.

Chunk-count sweep (post split-scan fix, 533 MB, median-of-7)¶

Peak at par128 (4470 MB/s, 4.16 MB chunks, CoV 3.0%). par192 marginally 4214 but 2.6× noisier; par266 at 2 MB is 4328 (−3% vs 4 MB); 1 MB collapses to 3553. Auto-tune rule: max(threads, min(8×threads, file_bytes/4 MB)) - 533 MB → 128 (capped vs ideal 133), 1 GB → 128. Streaming's 2 MB optimum is separate (64 MB/16t) - one divisor cannot serve both, keep split.

Traverse tier memchr profile (structural indexing justification)¶

Traverse is 59% memchr (vs push's 26%). Combined: 36.5% of total parse is memchr on short haystacks. Halving memchr → ~18% end-to-end throughput. Structural indexing (one-pass SIMD mask computation replacing 5-7 sequential searches) is the only remaining lever and targets both tiers.

Engine selection guide (per goal)¶

Schema stability & reuse: Every iter_record_batches batch has identical schema (parallel_stream.rs:59 opts.schema -> engine.rs:79 ensure_schema). Without explicit schema=[...], auto-discovery schema.rs:90 via DiscoverySink parallel_stream.rs:55 (<=128 MiB full scan else 16x2 MiB sampled, parallel, needs_value=false) captures all columns (FieldG 30%, Text21 1%). Hard-error on unknown field (engine.rs:510 MergeError naming column, unknown field "LateColumn" not in frozen schema... pass schema=[...]) - fixture with LateColumn only in last 1% verifies loud failure vs silent drop. Column order is schema order if explicit, else discovery file order. Provide schema= to avoid 5.3 ms; reuse via crxml.discover_schema("sample.xml") src/crxml/source.py:445 / _core.discover_schema src/crxml_core/src/lib.rs:1012 (see below). 1 MB chunks par collapses (3553) while streaming does not (3812, +7%) due to chunk_buf reuse.

The ceiling¶

At 0.7 GB/s single / 4.2 GB/s parallel on a ~30 GB/s memory bus, this parser is CPU-bound (tokenizing + FxHash + memchr on short haystacks + StrColumn::push arena). The memchr scanner handles the heavy lifting, row_dirty:Vec<bool> keeps finish_row minimal (<1%), field_index:Vec+HashMap provides single-probe field lookup.

Measured ceilings (5800X, warm, median-of-7, CoV 2-7%):

* 100 MB/1 GB par16 variance thermal; use par128 for ceilings. Explicit is +11% vs par128 and defines the ceiling.

¹ 100 MB par16 cell re-measured 2026-09-16 against local rypipe 8017f25 (crxml 2.1.0): 5,215 MB/s with read_to_columnar_par(num_chunks=4) (par4 rather than par16), single run, warm cache, same 5800X machine.

Median of 7 runs (adaptive: keep sampling until 1.31×CoV ≤5% capped at 31, halving floor costs 4× rounds). Observed CoV across configurations: median 5%, max 26% (10 MB par8)†. Per-cell floor = 1.31×CoV (95% for two medians, n=7): 2.5% CoV →3.3% floor, 5%→6.6%, 26%→34%†. Cells with CoV>8% marked † (untrustworthy for tuning). Deltas below the cell's own floor are reported as no measurable difference.

† 10 MB parallel is too small (2.8k rows/chunk, 20 ms); rayon work-stealing variance + frequency/thermal drift + CCX scheduling dominate. Fix: 20 repeats inside one timed region per median_of call, taskset -c 0-15 (pin all 16 logical CPUs without restricting) + thermal settle, or drop 10 MB from parallel tables (a number you cannot act on should not be in a tuning guide).

Correctness & Coverage¶

Every number is backed by tests/test_differential.py vs xml.etree.ElementTree (ragged, empty, entities, unicode, comments with fake row tags) and splitter 18 tests. benchmarks/bench_extended.py (all or --quick) covers 104 benchmarks/file (native + source×sink + pushdowns + chunk/bounded/batch/pipeline) ×3 rounds, best-of-N after warmup.