when upgrading to scala 3 (3.7+, 3.8+) my project stops compiling #25386
|
I have a really really big project (sadly can't share) that will start hanging once I upgrade from scala 2 to scala 3 (3.7,3.8) it just stops doing anything at all and hangs in I already tried to raise the log level, bug it does not print out anything at all. |
Replies: 1 comment 9 replies
|
You could use |
|
attaching the debugger should be possible. by default the compiler runs in the same JVM as sbt, so start sbt with but yeah, it takes a bit of setup. maybe you can manually find which match expression is causing the issue. look for large matches with many cases (or with many nested patterns). maybe try a bisect-like approach, comment out half of the large suspicious match expressions to narrow it down. it would be valuable to have a reproducer. without, there's probably not much we can do, unfortunately. something like copilot or claude might be able to do the work if you explain it how to reproduce the issue and the approach to narrow it down. perhaps also gemini-cli which has a generous free tier IIUC, i never tried that. |
|
well basically I tried to comment out EVERY match call and this did not work aswell, so I need to debug anyway, but that will probably take me a while. |
|
it looks like it is stuck in playframeworks generated code of the router I see a lot of these sird class. so I have some clues at least. because we written some kind of code for that. som I guess that at least looks like some useful information for me. Maybe I can add a reproducer. something like that looks fishy: package envisia.workflow.analysis.base.models
// YearRange.scala
import java.time.Year
import envisia.erp.base.routing.{ EnvisiaQueryString, ExtractorImplicit }
import play.api.routing.sird.QueryStringParameterExtractor
case class YearRange(start: Year, end: Year)
object YearRange extends ExtractorImplicit {
val yearRange: QueryStringParameterExtractor[YearRange] = EnvisiaQueryString { implicit qs =>
val now = Year.now()
YearRange(
getYear("start").getOrElse(now),
getYear("end").getOrElse(now)
)
}
}
// MonthRange.scala
import java.time.{ LocalDate, Month }
import envisia.erp.base.routing.{ EnvisiaQueryString, ExtractorImplicit }
import play.api.routing.sird.QueryStringParameterExtractor
case class MonthRange(start: Month, end: Month)
object MonthRange extends ExtractorImplicit {
def now(): MonthRange = {
val month = Month.from(LocalDate.now())
MonthRange(month, month)
}
val monthRange: QueryStringParameterExtractor[MonthRange] = EnvisiaQueryString { implicit qs =>
val now = Month.from(LocalDate.now())
MonthRange(
getMonth("start").getOrElse(now),
getMonth("end").getOrElse(now)
)
}
}
package envisia.erp.base.routing
import java.time._
import java.time.format.DateTimeParseException
import envisia.utils.date.DateHelper
import play.api.routing.sird.{ QueryString, QueryStringParameterExtractor }
import scala.util.Try
import scala.util.control.Exception.allCatch
trait ExtractorImplicit {
def getString(name: String)(implicit qs: Map[String, Seq[String]]): Option[String] = {
qs.get(name).map(_.mkString).filter(_ != "")
}
def getStringList(name: String)(implicit qs: Map[String, Seq[String]]): Option[List[String]] = {
getString(name).flatMap(v => Try(v.split(",").toList).toOption)
}
def getLocalDate(name: String)(implicit qs: Map[String, Seq[String]]): Option[LocalDate] = {
getString(name).flatMap(DateHelper.safeParse)
}
def getInt(name: String)(implicit qs: Map[String, Seq[String]]): Option[Int] = {
qs.get(name).map(_.mkString).flatMap(v => allCatch.opt(v.toInt))
}
def getLong(name: String)(implicit qs: Map[String, Seq[String]]): Option[Long] = {
qs.get(name).map(_.mkString).flatMap(v => allCatch.opt(v.toLong))
}
def getBoolean(name: String)(implicit qs: Map[String, Seq[String]]): Option[Boolean] = {
qs.get(name).map(_.mkString).flatMap {
case "true" => Some(true)
case "false" => Some(false)
case _ => None
}
}
def getEnum(name: String, `enum`: Enumeration)(implicit qs: Map[String, Seq[String]]): Option[`enum`.Value] = {
getString(name).flatMap { str =>
`enum`.values.find(_.toString.toLowerCase == str.toLowerCase)
}
}
def getYear(name: String)(implicit qs: Map[String, Seq[String]]): Option[Year] = {
def parseYear(s: String): Option[Year] = {
try {
Some(Year.parse(s))
} catch {
case _: DateTimeParseException => None
}
}
getString(name).flatMap(parseYear)
}
def getYearMonth(name: String)(implicit qs: Map[String, Seq[String]]): Option[YearMonth] = {
def parseYearMonth(s: String): Option[YearMonth] = {
try {
Some(YearMonth.parse(s))
} catch {
case _: DateTimeParseException => None
}
}
getString(name).flatMap(parseYearMonth)
}
def getMonth(name: String)(implicit qs: Map[String, Seq[String]]): Option[Month] = {
def parseMonth(s: Int): Option[Month] = {
try {
Some(Month.of(s))
} catch {
case _: DateTimeException => None
}
}
getInt(name).flatMap(parseMonth)
}
}
object ExtractorImplicit extends ExtractorImplicit
object EnvisiaQueryString {
def apply[T](block: QueryString => T): QueryStringParameterExtractor[T] = { (qs: QueryString) =>
Some(block(qs) :: Nil)
}
}
trait PageRouterExtension {
object page extends QueryStringParameterExtractor[Long]() {
override def unapplySeq(qs: QueryString): Option[Seq[Long]] = {
EnvisiaQueryString { implicit qs =>
ExtractorImplicit.getLong("page").getOrElse(1L)
}.unapplySeq(qs)
}
}
}
package envisia.erp.base.routing
import java.time.Year
import java.util.UUID
import play.api.mvc.PathBindable.Parsing
import play.api.routing.SimpleRouter
import play.api.routing.sird.PathBindableExtractor
import play.utils.UriEncoding
trait EnvisiaRouter extends SimpleRouter with PageRouterExtension {
val uuid: PathBindableExtractor[UUID] = new PathBindableExtractor[UUID]
private object escapeBindable
extends Parsing[String](
UriEncoding.decodePath(_, "UTF-8"),
UriEncoding.encodePathSegment(_, "UTF-8"),
(key: String, e: Exception) => "Cannot parse parameter %s as String: %s".format(key, e.getMessage)
)
val escape: PathBindableExtractor[String] = new PathBindableExtractor[String]()(escapeBindable)
private object yearBindable
extends Parsing[Year](
(value) => {
if (value forall Character.isDigit) Year.of(value.toInt)
else throw new Exception("not a valid integer")
},
_.getValue.toString,
(key: String, e: Exception) => "Cannot parse parameter %s as Year: %s".format(key, e.getMessage)
)
val year: PathBindableExtractor[Year] = new PathBindableExtractor[Year]()(yearBindable)
}
at least that looks like methods that will always be called. I need to narrow that down now. but it looks like the year thingy is the first error that is looping I guess. |
|
Here's what claude sonnet came up with after 30 minutes. Take it with a pile of salt :-) no effort done from my side here (sorry). Theory The hang is caused by the SpaceEngine's reachabilityCheckable analysis performing non-memoized decompose(List[T]) calls in O(N²) complexity for large Play SIRD routers. What the stack trace reveals Thread pool-46-thread-9 consumed 135s of CPU in 140s (96% utilization). The hot path is: The core problem SpaceEngine.decompose(tp: Type) at line 632 is not memoized, unlike Typ.decompose (which memoizes per-object). But Typ objects are value types created fresh by project(pat) for each case pattern. So for N routes in the router, N different Typ(List[T], false) objects List[T] is a sealed type → isDecomposableToChildren = true → decompose checks inhabited(::[T]) via containsUninhabitedField(::[T]) → calls tp.fields → triggers computeAsSeenFrom (expensive membersNeedAsSeenFrom computation). Why SIRD triggers this Play's QueryStringParameterExtractor.unapplySeq returns Option[Seq[T]]. The internal unapplySeq signature becomes List[T] (line 597: defn.ListType.appliedTo(elemTp) :: Nil). The binding variable yr: YearRange appears in the Prod's sub-spaces as Typ(List[YearRange], Similarly, PathExtractor.unapplySeq returns Option[List[String]], contributing Typ(List[String], false) spaces. Why it's quadratic For a router with N routes, checkReachability processes each route against all O(N) previous routes. For each comparison, simplify(Prod(..., [Typ(List[T])])) is called, triggering decompose(List[T]) (non-memoized). Total: O(N²) calls to the expensive decompose(List[T]) For N=200 routes, that's ~40,000 calls. If each decompose(List[T]) call takes ~3ms (due to computeAsSeenFrom), that's ~120s. This matches the observed 135s. Why "loops back to MonthRange" The Or(prevs_LazyList).foldLeft(covered)(minus) iterates through all previous routes. Routes using monthRange and yearRange appear many times in prevs. Each step forces more LazyList elements that contain Typ(List[MonthRange]) and Typ(List[YearRange]) spaces needing Why it's Scala 3 specific Scala 2's pattern match analysis didn't use the SpaceEngine reachability algorithm. Scala 3's SpaceEngine was a complete rewrite with thorough reachability checking but without adequate memoization for the non-memoized decompose(tp: Type) path, especially when combined Reproducer hypothesis A minimal reproducer would be something like: The more routes with unapplySeq patterns, the worse the hang — quadratic in the number of routes. |
|
ah so it's basically not a "hang" more like a due to the way how sird works it need a lot of time in scala3 bascially. I never successfully compiled the project btw. I always stopped it after ~40 minutes or so, so that looks like really bad. and I thin the stack trace did hang mostly in the analysisRouter: I hope at least that this is the problem and I think I will replace all the "Extractor" functions to use them directly inside the controller functions, which might resolve the problem somewhat. |
|
btw. after repalcing every queryString parsing (thanks claude) with something like this: it did compile in scala3 the first time. I would say a clean compile is as fast as scala2 now, which probably means that this thing what we did was problematic in scala2 as well, just way way way way worse in scala3. but first I need to test these things and probably start backporting things in this branch... tons of changes already and the querystring parsing should probably go into the controller. I just wanted to have a quick demo to test if it will compile. |
Here's what claude sonnet came up with after 30 minutes. Take it with a pile of salt :-) no effort done from my side here (sorry).
Theory
The hang is caused by the SpaceEngine's reachabilityCheckable analysis performing non-memoized decompose(List[T]) calls in O(N²) complexity for large Play SIRD routers.
What the stack trace reveals
Thread pool-46-thread-9 consumed 135s of CPU in 140s (96% utilization). The hot path is: