Skip to content

Changelog

0.4.0 (2026-09-16)

Features

  • Strict memory budget for streaming. CrystalXMLSource.iter_record_batches(..., strict=True) makes the memory budget a hard limit via MemoryBudget::with_strict: oversized batches raise MemoryError instead of passing through. Default False keeps the soft-budget behavior.
  • Split-point contract restored for degenerate inputs. CrystalXmlSplitter overrides find_split_points to guarantee the documented contract (sorted, unique, first = 0, last = file length) for inputs shorter than the planned chunk count, where the rypipe-core 0.4.0 default can emit a duplicate leading 0.

Changed

  • rypipe-core updated to 0.4.0 (registry). Adapts to the 0.4.0 API: CompareOp/FieldType parse via str::parse, FilterPredicate::Strip carries the new TrimMode field, Error::Memory maps to Python MemoryError, and the bindings adopt the pyo3 0.29 renames (PyObject → Py<PyAny>, with_gil → attach, allow_threads → detach) alongside the arrow 59 bump.
  • Small-file chunk planning. plan_chunk_count floors at the thread count in 0.4.0, so tiny inputs no longer collapse to a single chunk; the splitter tests now assert the contract rather than the old single-chunk fallback.
  • Regenerated bench_data/test_10mb.xml with the current generator and aligned the 10 MB row-count expectations (21,047 → 9,010) in test_integration.py and test_source.py; the 50 MB / 100 MB fixtures already matched.

0.3.1 (2026-09-11)

Bug Fixes

  • FromStr import. Added missing use std::str::FromStr to lib.rs and plan_kwargs.rs for FieldType and CompareOp parsing against rypipe-core 0.3.
  • from_str return type. Adapted all from_str(...).ok_or_else(...) calls to chain .ok() first, since FromStr::from_str returns Result<T, ()>, not Option<T>.
  • Null filter semantics. _ConstantPredicate now correctly returns True for != when the field value is None (Python semantics: None != "NYC" is True).
  • Invalid operator test. Updated test_invalid_op_constant_raises to use "xor" instead of ">", which is now a supported operator.

Changed

  • Publishing CI: test blocker. The PyPI publishing workflow now runs tests as a required gate before building.

0.3.0 (2026-09-11)

Features

  • CrystalXMLSource inherits from rypipe.Adapter. Framework-native integration: adapters get caching and pipelines for free by subclassing rypipe.Adapter.
  • Filter pushdown: is_null, is_type, Any/All/Not combinators. Supports null-type checks and logical combinators for complex filter expressions.
  • Recursive plan_kwargs for nested combinators. plan_kwargs now handles nested FilterRowsAny/All/Not correctly.
  • Public API: to_arrow, to_pandas, to_polars, to_parquet. Direct access to all output formats from the engine.
  • Binary memory units. Accepts TB in addition to KB, MB, GB for large dataset streaming.
  • chunksize param on to_pandas(). Controls chunk size for bounded-memory pandas conversion.

Breaking Changes

  • to_dataframe alias removed. Use to_pandas() instead.
  • read() convenience function removed. Use CrystalXMLSource or the rypipe.Adapter pattern instead.
  • rypipe as a hard dependency. crxml now requires rypipe to be installed.

Bug Fixes

  • rypipe-core dependency corrected. Now points to crates.io version 0.3 instead of local path.

Dependencies

  • rypipe-core updated to 0.3.0 (expression API, streaming sinks, observer hooks).

2.0.0 (2026-09-02)

Framework rebrand

  • rypipe reframed as a "data ingestion framework" (was "columnar engine"). Code and traits unchanged; reframing reflects that adapters extend rypipe rather than merely using it.

Performance

  • Single-thread: ~953 MB/s on 533 MB Crystal Reports XML (up from ~800 MB/s).
  • Parallel projected: ~6,879-7,630 MB/s drop_half/rename/schema on 533 MB.
  • Incremental dict unification: auto_dict parallel gap closed from 45% to 16%. Per-chunk upgrade in parallel, then O(dict_size) unification.
  • row_satisfied projection short-circuit: Scanner byte-jumps to row close when all wanted columns arrive. +64% on drop_half parallel.
  • expect_slot layout prediction: memcmp raw bytes instead of full attribute scan + hash lookup. ~25 ns to ~8 ns per field.
  • F1 precomputed close_finder: Eliminated per-row Vec+Finder allocation. +10% single-thread.
  • F2 fast-path find_attr_value: Single-attribute fast path. +3% single-thread.
  • Engine-provided Splitter default: Eliminates bug class that caused TSV and crxml regressions.

Bug Fixes

  • Sparse-column KeyError (from 1.2.1): to_arrow() crashed when later rows had fields the first row lacked. Fixed to union all row keys.
  • Compare filter docs corrected: Compare filters do not force the merge path; they are applied per-row during parse AND re-applied post-export.
  • auto_dict docs corrected: Incremental dict path preserves fast export when schemas consistent.

Dependencies

  • rypipe-core upgraded to 2.0.0 (was 0.1.1).

Documentation

  • 134 audit findings fixed across 5 rounds (rypipe + crxml).
  • Architecture docs expanded from 1,058 to 2,000 lines.
  • Writing-adapters reworked into 7 dedicated pages.
  • All em dashes replaced with ;:,()-.

1.2.0 (2026-08-23)

Refactor

  • Extracted the columnar engine into the sibling rypipe workspace (rypipe-core, rypipe-xml, rypipe-python).
  • rypipe-core is now consumed from git as a versioned dependency (version = "0.1", mmap feature) instead of a path dependency: building crxml no longer requires a sibling rypipe checkout.
  • Embedded the Crystal Reports XML adapter (previously the separate rypipe-xml crate) directly in crxml_core.
  • Renamed the internal plan type from BuildPlan to rypipe_core::ExecutionPlan.
  • Compare filters now use arrow::compute kernels instead of pyarrow.compute.

Removed

  • Deleted src/crxml_core/src/columnar.rs and src/crxml_core/src/splitter.rs; their logic lives in rypipe now.

Kept

  • The streaming CrxmlReader remains in crxml_core.

Packaging

  • sdist now ships LICENSE explicitly (PEP 639 license expression) so PyPI accepts the upload.
  • CI installs rypipe from PyPI for integration tests instead of cloning a sibling checkout.

Testing

  • All existing tests pass.

1.0.0 (2026-07-06)

Bug Fixes

  • auto_dict plan lost in parallel merge: ColumnarEngine::new() defaulted to auto_dict: false, making auto_dict_upgrade() a no-op. Fixed by using ColumnarEngine::with_plan(est, plan) to carry the build plan forward.

  • Text field parsing in bounded path: The parser was capturing whitespace-only text nodes as field values instead of looking for <TextValue> children. Fixed to correctly consume TextValue inner text.

  • Stream engine column discovery: Engine used first-row columns as schema; sparse columns appearing only in later rows caused crashes. Schema is now discovered across all rows.

  • Publishing workflow missing columnar feature: maturin build --features mmap risked overriding pyproject.toml's feature list and silently dropping columnar from the published wheel. CI now builds from pyproject.toml defaults.

Features

  • prefault parameter: All engines accept prefault: bool. True = MADV_WILLNEED (speed), False = MADV_SEQUENTIAL (lower RSS). Defaults: True for columnar/parallel, False for bounded.

  • Parallel engine profiling: get_par_profile() returns nanosecond timing for split-scan, off-GIL parse, and on-GIL assembly phases (gated behind profile Cargo feature).

  • Bounded mode RSS rewrite: Mmap used only for initial split scan, then dropped. Chunks read via File::seek/read_exact. Peak RSS tracks the memory= budget, not the file size.

  • sort_columns() on engine: Ensures all batch engines produce identical column order for schema-match fast path in concat_tables().

Performance

  • Splitter SIMD optimization: next_row_start() searches for <tag in one memmem::find pass instead of memchr(b'<') (24M iterations to 465k matches). Split phase 40% faster. Total parallel throughput improved 22% (327 to 472 MB/s on 533 MB real file).

  • find_special_regions() single-pass: Scans once for b"<!" prefix instead of two separate scans for <!-- and <![CDATA[.

  • concat_tables() schema fast path: Skips promote_options='default' when schemas already match.

Chores

  • Build features (columnar, mmap) now enabled by default in pyproject.toml. profile remains opt-in.
  • Removed --features mmap from CI publishing workflow.
  • Added docs/performance.md with environment block, per-engine speed tables, memory decomposition, and throughput ceiling.
  • Correctness harness validates all engines against stream oracle across 29 test cases plus 465k-row real-file cross-check.
  • README restyled to match seoslug format: quick start first, narrative "why" section, comparison table, features table, framework support, --- section rulers.

0.1.0 (2026-06-01)