DataFusion Comet 0.10.0 Changelog

This release consists of 183 commits from 26 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

• fix: [Iceberg] Fix decimal corruption #1985 (andygrove)
• fix: broken link in development.md #2024 (petern48)
• fix: [iceberg] Add LogicalTypeAnnotation in ParquetColumnSpec #2000 (huaxingao)
• fix: hdfs read into buffer fully #2031 (parthchandra)
• fix: Refactor arithmetic serde and fix correctness issues with EvalMode::TRY #2018 (andygrove)
• fix: clean up [iceberg] integration APIs #2032 (huaxingao)
• fix: zero Arrow Array offset before sending across FFI #2052 (mbutrovich)
• fix: [iceberg] more fixes for Iceberg integration APIs. #2078 (parthchandra)
• fix: Add support for StringDecode in Spark 4.0.0 #2075 (peter-toth)
• fix: Avoid double free in CometUnifiedShuffleMemoryAllocator #2122 (andygrove)
• fix: Remove duplicate serde code #2098 (andygrove)
• fix: Improve logic for determining when an UnpackOrDeepCopy is needed #2142 (andygrove)
• fix: Add CopyExec to inputs to SortMergeJoinExec #2155 (andygrove)
• fix: Fix repeatedly url-decode path when reading parquet from s3 using native parquet reader #2138 (Kontinuation)
• fix: [iceberg] Switch to OSS Spark and run Iceberg Spark tests in parallel #1987 (hsiang-c)
• fix: [iceberg] Fall back to spark for schemas with empty structs #2204 (andygrove)
• fix: Fix failing TPC-DS workflow in PR CI runs #2207 (andygrove)
• fix: [iceberg] order query result deterministically #2208 (hsiang-c)
• fix: use spark.comet.batchSize instead of conf.arrowMaxRecordsPerBatch for data that is coming from Java #2196 (rluvaton)
• fix: if expr nullable #2217 (Asura7969)
• fix: Support auto scan mode with Spark 4.0.0 #1975 (andygrove)
• fix: Make Sha2 fallback message more user-friendly #2213 (rishvin)
• fix: separate type checking for CometExchange and CometColumnarExchange #2241 (mbutrovich)
• fix: Fix potential resource leak in native shuffle block reader #2247 (andygrove)
• fix: Remove unreachable code in CometScanRule #2252 (andygrove)
• fix: Fall back to native_comet for encrypted Parquet scans #2250 (andygrove)
• fix: Fall back to native_comet when object store not supported by native_iceberg_compat #2251 (andygrove)
• fix: split expr.proto file (new) #2267 (kination)
• fix: handle cast to dictionary vector introduced by case when #2044 (parthchandra)
• fix: Remove check for custom S3 endpoints #2288 (andygrove)
• fix: implement lazy evaluation in Coalesce function #2270 (coderfender)
• fix: Update benchmarking scripts #2293 (andygrove)
• fix: Fix regression in NativeConfigSuite #2299 (andygrove)
• fix: Validating object store configs should not throw exception #2308 (andygrove)
• fix: TakeOrderedAndProjectExec is not reporting all fallback reasons #2323 (kazuyukitanimura)
• fix: Fallback length function with binary input #2349 (wForget)

Performance related:

• perf: Optimize AvgDecimalGroupsAccumulator #1893 (leung-ming)
• perf: Optimize SumDecimalGroupsAccumulator::update_single #2069 (leung-ming)
• perf: Avoid FFI copy in ScanExec when reading data from exchanges #2268 (andygrove)

Implemented enhancements:

• feat: Add from_unixtime support #1943 (kazuyukitanimura)
• feat: randn expression support #2010 (akupchinskiy)
• feat: monotonically_increasing_id and spark_partition_id implementation #2037 (akupchinskiy)
• feat: support map_entries #2059 (comphead)
• feat: Support Array Literal #2057 (comphead)
• feat: Add new trait for operator serde #2115 (andygrove)
• feat: limit with offset support #2070 (akupchinskiy)
• feat: Include scan implementation name in CometScan nodeName #2141 (andygrove)

Documentation updates:

• docs: Update benchmark results for 0.9.0 #1959 (andygrove)
• doc: Add comment about local clippy run before submitting a pull request #1961 (akupchinskiy)
• docs: Minor improvements to Spark SQL test docs #1980 (andygrove)
• docs: Update Maven links for 0.9.0 release #1988 (andygrove)
• docs: Documentation updates for 0.9.0 release #1981 (andygrove)
• docs: Add guide showing comparison between Comet and Gluten #2012 (andygrove)
• docs: Remove legacy comment in docs #2022 (andygrove)
• docs: Update Gluten comparision to clarify that Velox is open-source #2043 (andygrove)

Other:

• chore: Start 0.10.0 development #1958 (andygrove)
• build: Fix release dockerfile #1960 (andygrove)
• test: Run Iceberg Spark tests only when PR title contains [iceberg] #1976 (hsiang-c)
• chore: Reuse comet allocator #1973 (EmilyMatt)
• chore: update CopyExec with maintains_input_order, supports_limit_pushdown and cardinality_effect #1979 (rluvaton)
• chore: extract CreateArray from QueryPlanSerde #1991 (tglanz)
• chore: use DF scalar functions for StartsWith, EndsWith, Contains, DF LikeExpr #1887 (mbutrovich)
• refactor: standardize div_ceil #1999 (tglanz)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

    75	Andy Grove
    27	dependabot[bot]
    11	Oleks V
     9	Zhen Wang
     7	hsiang-c
     5	Artem Kupchinskiy
     5	B Vadlamani
     5	Kazantsev Maksim
     5	Matt Butrovich
     5	Parth Chandra
     4	Rishab Joshi
     3	Peter Toth
     3	Tal Glanzman
     2	Dharan Aditya
     2	Huaxin Gao
     2	KAZUYUKI TANIMURA
     2	Leung Ming
     2	Raz Luvaton
     2	Yu-Chuan Hung
     1	Asura7969
     1	Emily Matheys
     1	K.I. (Dennis) Jung
     1	Kristin Cowalcijk
     1	Peter Nguyen
     1	codetyri0n
     1	Šimon Lukašík

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.