Spark 3.5.2 is the second maintenance release containing security and correctness fixes. This release is based on the branch-3.5 maintenance branch of Spark. We strongly recommend all 3.5 users to upgrade to this stable release.

Notable changes

[SPARK-45988]: Fix pyspark.pandas.tests.computation.test_apply_func in Python 3.11
[SPARK-45989]: Fix pyspark.pandas.tests.connect.computation.test_parity_apply_func in Python 3.11
[SPARK-46411]: Change to use bcprov/bcpkix-jdk18on for test
[SPARK-47368]: Remove inferTimestampNTZ config check in ParquetRowConverter
[SPARK-47370]: Add migration doc: TimestampNTZ type inference on Parquet files
[SPARK-47435]: SPARK-45561 causes mysql unsigned tinyint overflow
[SPARK-47440]: SQLServer does not support LIKE operator in binary comparison
[SPARK-47473]: Correctness issue of converting postgres INFINITIES timestamps
[SPARK-47494]: Add migration doc for the behavior change of Parquet timestamp inference since Spark 3.3
[SPARK-47537]: Use MySQL Connector/J for MySQL DB instead of MariaDB Connector/J
[SPARK-47666]: Fix NPE when reading mysql bit array as LongType
[SPARK-47770]: Fix GenerateMIMAIgnore.isPackagePrivateModule to return false instead of failing
[SPARK-47774]: Remove redundant rules from MimaExcludes
[SPARK-47847]: Deprecate spark.network.remoteReadNioBufferConversion
[SPARK-48016]: Fix a bug in try_divide function when with decimals
[SPARK-48068]: mypy should have --python-executable parameter
[SPARK-48083]: session.copyFromLocalToFs failure with 3.5 client <> 4.0 server
[SPARK-48084]: pyspark.ml.connect.evaluation not working in 3.5 client <> 4.0 server
[SPARK-48086]: Different Arrow versions in client and server
[SPARK-48087]: Python UDTF incompatibility in 3.5 client <> 4.0 server
[SPARK-48088]: Skip tests being failed in client 3.5 <> server 4.0
[SPARK-48089]: Streaming query listener not working in 3.5 client <> 4.0 server
[SPARK-48090]: Streaming exception catch failure in 3.5 client <> 4.0 server
[SPARK-48109]: Enable k8s-integration-tests only for kubernetes module change
[SPARK-48116]: Run pyspark-pandas* only in PR builder and Daily Python CIs
[SPARK-48132]: Run k8s-integration-tests only in PR builder and Daily CIs
[SPARK-48133]: Run sparkr only in PR builders and Daily CIs
[SPARK-48138]: Disable a flaky SparkSessionE2ESuite.interrupt tag test
[SPARK-48167]: Skip known behaviour change by SPARK-46122
[SPARK-48178]: Run build/scala-213/java-11-17 jobs of branch-3.5 only if needed
[SPARK-48192]: Enable TPC-DS and docker tests in forked repository
[SPARK-48930]: Redact awsAccessKeyId by including accesskey pattern
[SPARK-49054]: Column default value should support current_* functions
[SPARK-47305]: PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation
[SPARK-47307]: Spark 3.3 produces invalid base64
[SPARK-47318]: AuthEngine key exchange needs additional KDF round
[SPARK-47385]: Tuple encoder produces wrong results with Option inputs
[SPARK-47398]: AQE doesn’t allow for extension of InMemoryTableScanExec
[SPARK-47434]: Streaming Statistics link redirect causing 302 error
[SPARK-47455]: Fix Resource Handling of scalaStyleOnCompileConfig in SparkBuild.scala
[SPARK-47463]: An error occurred while pushing down the filter of if expression for iceberg datasource.
[SPARK-47503]: Spark history sever fails to display query for cached JDBC relation named in quotes
[SPARK-47507]: Upgrade ORC to 1.9.3
[SPARK-47521]: Use Utils.tryWithResource during reading shuffle data from external storage
[SPARK-47561]: Fix analyzer rule order issues about Alias
[SPARK-47633]: Cache miss for queries using JOIN LATERAL with join condition
[SPARK-47636]: Use Java 17 instead of 17-jre image in K8s Dockerfile
[SPARK-47646]: try_to_number fails with NPE for malformed input
[SPARK-47676]: Clean up the removed VersionsSuite references
[SPARK-47762]: Add pyspark.sql.connect.protobuf into setup.py
[SPARK-47824]: Nondeterminism in pyspark.pandas.series.asof
[SPARK-47828]: DataFrameWriterV2.overwrite fails with invalid plan
[SPARK-47840]: Remove foldable propagation across Streaming Aggregate/Join nodes
[SPARK-47895]: group by all should be idempotent
[SPARK-47904]: Preserve case in Avro schema when using enableStableIdentifiersForUnionType
[SPARK-47910]: Memory leak when interrupting shuffle write using zstd compression
[SPARK-47921]: Fix ExecuteJobTag creation in ExecuteHolder
[SPARK-47927]: Nullability after join not respected in UDF
[SPARK-48019]: ColumnVectors with dictionaries and nulls are not read/copied correctly
[SPARK-48037]: SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[SPARK-48105]: Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store
[SPARK-48128]: BitwiseCount / bit_count generated code for boolean inputs fails to compile
[SPARK-48172]: Fix escaping issues in JDBCDialects
[SPARK-48173]: CheckAnalsis should see the entire query plan
[SPARK-48179]: Pin nbsphinx to 0.9.3
[SPARK-48184]: Always set the seed of dataframe.sample in Client side
[SPARK-48197]: avoid assert error for invalid lambda function
[SPARK-48237]: After executing test-dependencies.sh, the dir dev/pr-deps should be deleted
[SPARK-48241]: CSV parsing failure with char/varchar type columns
[SPARK-48248]: Fix nested array to respect legacy conf of inferArrayTypeFromFirstElement
[SPARK-48265]: Infer window group limit batch should do constant folding
[SPARK-48273]: Late rewrite of PlanWithUnresolvedIdentifier
[SPARK-48286]: Analyze ‘exists’ default expression instead of ‘current’ default expression in structField to v2 column conversion
[SPARK-48294]: Make nestedTypeMissingElementTypeError case insensitive
[SPARK-48297]: Char/Varchar breaks in TRANSFORM clause
[SPARK-48237]: After executing test-dependencies.sh, the dir dev/pr-deps should be deleted
[SPARK-48241]: CSV parsing failure with char/varchar type columns
[SPARK-48248]: Fix nested array to respect legacy conf of inferArrayTypeFromFirstElement
[SPARK-48265]: Infer window group limit batch should do constant folding
[SPARK-48273]: Late rewrite of PlanWithUnresolvedIdentifier
[SPARK-48286]: Analyze ‘exists’ default expression instead of ‘current’ default expression in structField to v2 column conversion
[SPARK-48294]: Make nestedTypeMissingElementTypeError case insensitive
[SPARK-48297]: Char/Varchar breaks in TRANSFORM clause
[SPARK-48308]: Unify getting data schema without partition columns in FileSourceStrategy
[SPARK-48428]: IllegalStateException due to nested column aliasing
[SPARK-48481]: OptimizeOneRowPlan should not be effective for streaming DataFrame
[SPARK-48484]: V2Write use the same TaskAttemptId for different task attempts
[SPARK-48498]: Always do char padding in predicates
[SPARK-48608]: Spark 3.5: fails to build with value defaultValueNotConstantError is not a member of object org.apache.spark.sql.errors.QueryCompilationErrors
[SPARK-48642]: False reported SparkOutOfMemoryError caused by killing task on spilling
[SPARK-48666]: A filter should not be pushed down if it contains Unevaluable expression
[SPARK-48709]: Varchar resolution mismatch for DataSourceV2 CTAS
[SPARK-48719]: Wrong result in regr_slope & regr_intercept aggregate with tuples has NULL
[SPARK-48791]: Perf regression due to accumulator registration overhead using CopyOnWriteArrayList
[SPARK-48843]: Infinite loop with GlobalLimit/BindParameters
[SPARK-48845]: GenericUDF Can not CatchException From Child UDFs
[SPARK-48863]: ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class org.apache.spark.sql.catalyst.util.ArrayData when parsing JSON with “spark.sql.json.enablePartialResults” enabled
[SPARK-48871]: Fix INVALID_NON_DETERMINISTIC_EXPRESSIONS validation in CheckAnalysis
[SPARK-48921]: ScalaUDF in subquery should run through analyzer
[SPARK-48991]: FileStreamSink.hasMetadata handles invalid path
[SPARK-49000]: Aggregation with DISTINCT gives wrong results when dealing with literals
[SPARK-49005]: Use 17-jammy instead of 17 to prevent Python 3.12
[SPARK-49065]: Rebasing in legacy formatters/parsers must support non JVM default time zones
[SPARK-49094]: ignoreCorruptFiles file source option is partially supported for orc format
[SPARK-49099]: Refactor CatalogManager.setCurrentNamespace
[SPARK-45587]: Skip UNIDOC and MIMA in build GitHub Action job
[SPARK-45593]: Building a runnable distribution from master code running spark-sql raise error “java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.InternalFutureFailureAccess”
[SPARK-47172]: Upgrade Transport block cipher mode to GCM
[SPARK-47299]: Use the same versions.json in the dropdown of different versions of PySpark documents
[SPARK-47734]: Fix flaky pyspark.sql.dataframe.DataFrame.writeStream doctest by stopping streaming query
[SPARK-47825]: Make KinesisTestUtils & WriteInputFormatTestDataGenerator deprecated
[SPARK-47897]: ExpressionSet performance regression in scala 2.12
[SPARK-48081]: Fix ClassCastException in NTile.checkInputDataTypes() when argument is non-foldable or of wrong type
[SPARK-48292]: Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
[SPARK-48391]: Use addAll instead of add function in TaskMetrics to accelerate
[SPARK-48424]: Make dev/is-changed.py to return true if it fails
[SPARK-48586]: Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()
[SPARK-48610]: Remove ExplainUtils.processPlan synchronize
[SPARK-48806]: Pass actual exception when url_decode fails
[SPARK-47481]: Fix Python linter
[SPARK-48535]: Update doc to log warning for join null related config usage
[SPARK-48934]: Python datetime types converted incorrectly for setting timeout in applyInPandasWithState

Dependency changes

[SPARK-45445]: Upgrade snappy to 1.1.10.5
[SPARK-46335]: Upgrade Maven to 3.9.6 for MNG-7913
[SPARK-47083]: Upgrade commons-codec to 1.16.1
[SPARK-47111]: Upgrade PostgreSQL JDBC driver to 42.7.2 and docker image to 16.2
[SPARK-47182]: Exclude commons-(io|lang3) transitive dependencies from commons-compress and avro-*
[SPARK-47428]: Upgrade Jetty to 9.4.54.v20240208
[SPARK-47432]: Add pyarrow upper bound requirement, <13.0.0
[SPARK-47790]: Upgrade commons-io to 2.16.1
[SPARK-48494]: Update airlift:aircompressor to 0.27
[SPARK-48920]: Upgrade ORC to 1.9.4

You can consult JIRA for the detailed changes.

We would like to acknowledge all community members for contributing patches to this release.