build: Add spark-4.1 profile and shims

[manuzhang]

Which issue does this PR close?

First step of #2792.

Rationale for this change

What changes are included in this PR?

1. Add spark-4.1 profile with minor shim version spark-4.1
2. Move src/main/spark-4.0 to src/main/spark-4.x for common shim classes in spark-4.0 and spark-4.1.
3. Add CometSumShim and ShimSQLConf for spark-4.0 and spark-4.1 specific shims respectively
4. Add MapStatusBuilder.scala to access org.apache.spark.scheduler.MapStatus in java. MapStatus has added a constructor argument in Spark 4.1, and only kept compatibility for scala codes.

---

Summary of Changes in 4.1.1.diff (generated by Copilot)

This patch updates the Comet project to support Spark 4.1. The changes largely involve build configuration updates, integration into Spark's session initialization, and extensive test exclusions for features not yet supported or behaving differently in Comet.

Build Configuration

• pom.xml:
  • Updated spark.version.short to 4.1.
  • Added dependency for comet-spark-spark4.1.
• sql/core/pom.xml:
  • Added comet-spark-spark4.1 dependency to the SQL core module.

Core Spark Integration

• SparkSession.scala:
  • Added isCometEnabled check (defaults to true if ENABLE_COMET env var is not set or true).
  • Added loadCometExtension to automatically inject org.apache.comet.CometSparkSessionExtensions when enabling Comet.
• SparkPlanInfo.scala:
  • Added support for CometScanExec to properly extract metadata for Spark UI/history.

Test Infrastructure

• IgnoreComet.scala:
  • Introduced new Scalatest tags: IgnoreComet, IgnoreCometNativeIcebergCompat, IgnoreCometNativeDataFusion, IgnoreCometNativeScan.
  • Added IgnoreCometSuite trait to disable entire test suites when Comet is enabled.

Test Exclusions

A large number of existing Spark SQL tests have been modified to skip when Comet is enabled. Common reasons cited in the diffs include:

• Unsupported Features:
  • SubqueryBroadcastExec ([INTERNAL_ERROR] Custom columnar rules cannot transform shuffle node to something else #1737, Support SubqueryBroadcastExec in Comet #242)
  • Spill support in SortMergeJoin
  • Native CometSort peak execution memory updates
  • Datetime rebase mode
  • Parquet column index
  • RLE encoding, DELTA encoding
  • Variant types (issues/2209)
• Behavioral Differences:
  • Explain output differences
  • Shuffle partition size and metrics changes
  • Exception handling (e.g. Cast context)
• Pending Fixes:
  • References to various GitHub issues (e.g., Investigate failing tests with Spark 4.0.0 #1948, Fix listagg-collation.sql test in Spark 4.0.0 #1947, [Spark 4.0.0] Cast in ANSI mode uses wrong exception class #2218, Fix Spark 4.0 sql tests #551).

Modified Test Files

The following logical areas of tests had significant exclusions:

• Adaptive Query Execution (AQE)
• Join Suites (DataFrameJoin, JoinHint, JoinSuite)
• Parquet Data Source (Filter, Encoding, Query, Schema)
• SQL Query execution and metrics
• Collation support

How are these changes tested?

Added UTs.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.02%. Comparing base (f09f8af) to head (26a609e).
⚠️ Report is 918 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2829      +/-   ##
============================================
+ Coverage     56.12%   60.02%   +3.90%     
- Complexity      976     1469     +493     
============================================
  Files           119      176      +57     
  Lines         11743    16174    +4431     
  Branches       2251     2682     +431     
============================================
+ Hits           6591     9709    +3118     
- Misses         4012     5113    +1101     
- Partials       1140     1352     +212     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[manuzhang]

@andygrove @coderfender Please help review. Do we need to pass all tests now?

[andygrove]

Test failure:

CometExec3_4PlusSuite:
- subquery limit: limit with offset should return correct results (565 milliseconds)
- offset (602 milliseconds)
- test BloomFilterMightContain can take a constant value input (231 milliseconds)
- test NULL inputs for BloomFilterMightContain (251 milliseconds)
- test BloomFilterMightContain from random input *** FAILED *** (511 milliseconds)
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 41.0 failed 1 times, most recent failure: Lost task 1.0 in stage 41.0 (TID 102) (localhost executor driver): org.apache.comet.CometNativeException: assertion `left == right` failed: Unsupported BloomFilter version: 2, expecting version: 1
  left: 2
 right: 1
        at comet::errors::init::{{closure}}(__internal__:0)
        at std::panicking::panic_with_hook(__internal__:0)
        at std::panicking::panic_handler::{{closure}}(__internal__:0)
        at std::sys::backtrace::__rust_end_short_backtrace(__internal__:0)
        at __rustc::rust_begin_unwind(__internal__:0)
        at core::panicking::panic_fmt(__internal__:0)
        at core::panicking::assert_failed_inner(__internal__:0)
        at core::panicking::assert_failed(__internal__:0)
        at <datafusion_comet_spark_expr::bloom_filter::spark_bloom_filter::SparkBloomFilter as core::convert::From<&[u8]>>::from(__internal__:0)

[andygrove]

@andygrove @coderfender Please help review. Do we need to pass all tests now?

Yes, we either need tests to pass, or we can potentially disable some specific tests by updating the diff file, and file issues to resolve those failures.

[manuzhang]

@andygrove Sure, since Spark 4.1.1 is to be released soon, I will check again after upgrading to 4.1.1

[guixiaowen]

@manuzhang Spark 4.1.1 has already been released. Do you want to continue working on this integration?

[manuzhang]

@guixiaowen I've already upgraded to Spark 4.1.1, but I'm blocked by OOM failure in some tests. Anyway, I will rebase and try again.

[andygrove]

Thanks for working on this @manuzhang. I have a new PR up to enable Spark 4.1 that can replace this one. Perhaps you could help review?

#4093