JDK21 branch merge#2268
Open
vishramachandran wants to merge 8 commits into
Open
Conversation
feat(build): JDK21 support with Spark 4.0 upgrade
…to wifi (#2272) Pin spark driver/executor in unit tests to loopback so tests pass regardless of WiFi/VPN state. Without this, InetAddress.getLocalHost resolves to the WiFi IP on macOS, causing Spark internal RPC to bind on a non-loopback address that may not be reachable.
There was a problem hiding this comment.
Pull request overview
This PR migrates the FiloDB build/runtime from JDK 11 to JDK 21, including upgrades of associated build tooling and libraries (sbt 1.9.8, sbt-assembly 2.1.5, Spark 4.0.0, Arrow 15.0.2, ANTLR 4.13.1, etc.), regenerates ANTLR parser/lexer artifacts, refactors HLL aggregators to use byte arrays as Spark Aggregator buffers, and adds JDK 21 module --add-opens flags. CI is updated to JDK 21 and several test setups were tweaked to bind Spark to loopback for stability.
Changes:
- Build/tooling upgrades for JDK 21 compatibility (sbt, sbt-assembly, Spark 4, Arrow 15, ANTLR 4.13.1) and required
--add-opensJVM flags. - Refactored
HllSketchAgg/HllSketchMergeAggto useArray[Byte]Aggregator buffers (encoder change toEncoders.BINARY) and switched query Flight IO scheduler to virtual threads. - Test cleanups for Spark-based specs (loopback driver host, helper
localSparkConf) and minor cleanups (StringBuilder predicates, unused imports).
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/scala.yml | Switch CI to JDK 21 (temurin). |
| .jvmopts | Add JVM heap/GC defaults. |
| project/build.properties | Bump sbt to 1.9.8. |
| project/build.sbt | Add scala-xml Always version scheme. |
| project/plugins.sbt | Upgrade sbt-assembly, kanela, gatling, sbt-protobuf. |
| project/Dependencies.scala | Bump Spark to 4.0.0, Arrow to 15.0.2, ANTLR runtime to 4.13.1. |
| project/FiloSettings.scala | Drop -release:11, target --release 21, add JDK 21 module-opens, update assembly API usage. |
| project/FiloBuild.scala | Filter -Xfatal-warnings for core, fork spark-jobs tests, add publishM2 stub. |
| core/.../UnsafeUtils.scala | Wrap deprecated objectFieldOffset(Field) usage. |
| coordinator/.../QueryScheduler.scala | Use virtual-thread-per-task executor for Flight IO. |
| coordinator/.../FiloDBSinglePartitionFlightProducer.scala | Use ServerAuthHandler.NO_OP and prune unused imports. |
| prometheus/.../AntlrParser.scala | Use nonEmpty/isEmpty on StringBuilder. |
| prometheus/antlr/PromQL{Lexer,Parser,Visitor,BaseVisitor}.java, PromQL.interp | Regenerated via ANTLR 4.13.1 (with absolute path in headers). |
| spark-jobs/.../HllSketchAgg.scala, HllSketchMergeAgg.scala | Use Array[Byte] buffer with Encoders.BINARY. |
| spark-jobs/.../LabelChurnFinderSpec.scala | Add shared sparkSession, localSparkConf() helper. |
| spark-jobs/.../LabelStatsKafkaProducerSpec.scala | Add localSparkConf() helper pinning to loopback. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
29
to
34
| val sparkConf = new SparkConf(loadDefaults = false) | ||
| sparkConf.setMaster("local[2]") | ||
| .setMaster("local[2]") | ||
| .setAppName("LabelStatsKafkaProducerSpec") | ||
| .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") | ||
| .set("spark.ui.enabled", "false") |
| @@ -1,4 +1,4 @@ | |||
| // Generated from PromQL.g4 by ANTLR 4.9.1 | |||
| // Generated from /Users/vishr/projects/apple/FiloDB/prometheus/src/main/java/filodb/prometheus/antlr/PromQL.g4 by ANTLR 4.13.1 | |||
Comment on lines
124
to
212
| @@ -126,12 +140,10 @@ class LabelChurnFinderSpec extends AnyFunSpec with Matchers with BeforeAndAfterA | |||
| ("bulk_ws", "_metric_", 1, 1), | |||
| ("bulk_ws", "pod", numPods/2, numPods/2), | |||
| ) | |||
| spark.stop() | |||
| } | |||
|
|
|||
| it ("should run LCF job for multiple namespaces and workspaces") { | |||
| val sparkConf = new SparkConf(loadDefaults = true) | |||
| sparkConf.setMaster("local[2]") | |||
| val sparkConf = localSparkConf() | |||
| val filterConfig = ConfigFactory.parseString( | |||
| s""" | |||
| |filodb.labelchurnfinder.pk-filters.0._ns_ = "bulk_ns.*" | |||
| @@ -140,11 +152,11 @@ class LabelChurnFinderSpec extends AnyFunSpec with Matchers with BeforeAndAfterA | |||
| val settings2 = new DownsamplerSettings(filterConfig.withFallback(jobConfig.withFallback(baseConf))) | |||
| val lcf = new LabelChurnFinder(settings2) | |||
| sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") | |||
| val spark = SparkSession.builder() | |||
| sparkSession = SparkSession.builder() | |||
| .appName("LabelChurnFinder") | |||
| .config(sparkConf) | |||
| .getOrCreate() | |||
| val result = lcf.countsFromSketches(lcf.computeLabelStats(spark)).collect() | |||
| val result = lcf.countsFromSketches(lcf.computeLabelStats(sparkSession)).collect() | |||
| result.length shouldEqual 6 | |||
| val cards = result.map { row => (row.getAs[String](WsCol), | |||
| row.getAs[String](LabelCol), | |||
| @@ -158,12 +170,10 @@ class LabelChurnFinderSpec extends AnyFunSpec with Matchers with BeforeAndAfterA | |||
| ("bulk_ws", "_metric_", 1, 1), | |||
| ("bulk_ws", "pod", numPods/2, numPods/2), | |||
| ) | |||
| spark.stop() | |||
| } | |||
|
|
|||
| it ("should run LCF job for different time range") { | |||
| val sparkConf = new SparkConf(loadDefaults = true) | |||
| sparkConf.setMaster("local[2]") | |||
| val sparkConf = localSparkConf() | |||
| val filterConfig = ConfigFactory.parseString( | |||
| s""" | |||
| |filodb.labelchurnfinder.pk-filters.0._ns_ = "bulk_ns.*" | |||
| @@ -173,11 +183,11 @@ class LabelChurnFinderSpec extends AnyFunSpec with Matchers with BeforeAndAfterA | |||
| val settings2 = new DownsamplerSettings(filterConfig.withFallback(jobConfig.withFallback(baseConf))) | |||
| val lcf = new LabelChurnFinder(settings2) | |||
| sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") | |||
| val spark = SparkSession.builder() | |||
| sparkSession = SparkSession.builder() | |||
| .appName("LabelChurnFinder") | |||
| .config(sparkConf) | |||
| .getOrCreate() | |||
| val result = lcf.countsFromSketches(lcf.computeLabelStats(spark)).collect() | |||
| val result = lcf.countsFromSketches(lcf.computeLabelStats(sparkSession)).collect() | |||
| result.length shouldEqual 6 | |||
| val cards = result.map { row => (row.getAs[String](WsCol), | |||
| row.getAs[String](LabelCol), | |||
| @@ -191,14 +201,12 @@ class LabelChurnFinderSpec extends AnyFunSpec with Matchers with BeforeAndAfterA | |||
| ("bulk_ws", "_metric_", 1, 1), | |||
| ("bulk_ws", "pod", numPods/2, numPods/2) | |||
| ) | |||
| spark.stop() | |||
| } | |||
|
|
|||
| it ("should call publishLabelStats when actionOnLabelStats is invoked") { | |||
| val sparkConf = new SparkConf(loadDefaults = true) | |||
| sparkConf.setMaster("local[2]") | |||
| val sparkConf = localSparkConf() | |||
| sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") | |||
| val spark = SparkSession.builder() | |||
| sparkSession = SparkSession.builder() | |||
| .appName("LabelChurnFinder") | |||
| .config(sparkConf) | |||
| .getOrCreate() | |||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.