spark-extension

[2.15.0] - 2025-12-13

Added

• Support encrypted parquet files (#324)

Changed

• Remove support for Spark 3.0 and Spark 3.1 (#332)
• Make all undocumented unintended public API parts private (#331)
• Reading Parquet metadata can use Parquet Hadoop version different to version coming with Spark (#330)

[2.14.2] - 2025-07-21

Changed

• Fixed release process (#320)

[2.14.1] - 2025-07-17

Changed

• Fixed release process (#319)

[2.14.0] - 2025-07-17

Added

• Support for Spark 4.0 (#269, #272, #293)

Changed

• Improve backticks (#265)

  New: This escapes backticks that already exist in column names.

  Change: This does not quote columns that only contain letters, numbers and underscores, which were quoted before.

• Move Python dependencies into setup.py, build jar from setup.py (#301)

Fixes

• Support diff for Spark Connect implemened via PySpark Dataset API (#251)

Added

• Add ignore columns to diff in Python API (#252)
• Check that the Java / Scala package is installed when needed by Python (#250)