Changelog¶
1.0.0 (2026-07-06)¶
Bug Fixes¶
-
auto_dict plan lost in parallel merge:
ColumnarEngine::new()defaulted toauto_dict: false, makingauto_dict_upgrade()a no-op. Fixed by usingColumnarEngine::with_plan(est, plan)to carry the build plan forward. -
Text field parsing in bounded path: The parser was capturing whitespace-only text nodes as field values instead of looking for
<TextValue>children. Fixed to correctly consumeTextValueinner text. -
Stream engine column discovery: Engine used first-row columns as schema; sparse columns appearing only in later rows caused crashes. Schema is now discovered across all rows.
-
Publishing workflow missing
columnarfeature:maturin build --features mmaprisked overriding pyproject.toml's feature list and silently droppingcolumnarfrom the published wheel. CI now builds from pyproject.toml defaults.
Features¶
-
prefaultparameter: All engines acceptprefault: bool.True=MADV_WILLNEED(speed),False=MADV_SEQUENTIAL(lower RSS). Defaults: True for columnar/parallel, False for bounded. -
Parallel engine profiling:
get_par_profile()returns nanosecond timing for split-scan, off-GIL parse, and on-GIL assembly phases (gated behindprofileCargo feature). -
Bounded mode RSS rewrite: Mmap used only for initial split scan, then dropped. Chunks read via
File::seek/read_exact. Peak RSS tracks thememory=budget, not the file size. -
sort_columns()on engine: Ensures all batch engines produce identical column order for schema-match fast path inconcat_tables().
Performance¶
-
Splitter SIMD optimization:
next_row_start()searches for<tagin onememmem::findpass instead ofmemchr(b'<')(24M iterations to 465k matches). Split phase 40% faster. Total parallel throughput improved 22% (327 to 472 MB/s on 533 MB real file). -
find_special_regions()single-pass: Scans once forb"<!"prefix instead of two separate scans for<!--and<![CDATA[. -
concat_tables()schema fast path: Skipspromote_options='default'when schemas already match.
Chores¶
- Build features (
columnar,mmap) now enabled by default in pyproject.toml.profileremains opt-in. - Removed
--features mmapfrom CI publishing workflow. - Added
docs/performance.mdwith environment block, per-engine speed tables, memory decomposition, and throughput ceiling. - Correctness harness validates all engines against stream oracle across 29 test cases plus 465k-row real-file cross-check.
- README restyled to match seoslug format: quick start first, narrative "why" section, comparison table, features table, framework support,
---section rulers.