r/elixir • u/Shoddy_One4465 • 11d ago

Just released ExArrow v0.8.0

Just released ExArrow v0.8.0 — it now supports larger-than-memory Parquet on the BEAM, plus column/predicate/row-group pushdown, Zstd (and other codec) writes, footer-only metadata, and lazy multi-file / directory streams. Peak memory scales with the row groups you select, not the whole file. Also public IPC.File.write/3 and RecordBatch.concat/1. Hex ~> 0.8.0.

https://hex.pm/packages/ex_arrow | https://github.com/thanos/ex_arrow | Full notes: https://github.com/thanos/ex_arrow/releases/tag/v0.8.0

18 Upvotes

96% Upvoted

1 comment sorted by

1

u/eileenmnoonan 10d ago

Oooh this looks really good. A couple years ago I was tinkering around on https://github.com/enoonan/elixir_lancedb and was wishing something like this existed.