[Signal] Apache Spark — AQE skew detection never fires on large tables — the 256MB threshold is too high

[vara-bonthu]

Framework

Apache Spark

Tag

Performance

Headline

AQE skew detection never fires on large tables — the 256MB threshold is too high

Detail

spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes defaults to 256MB. On multi-TB fact tables, partition skew typically starts at 1–5GB — well above the threshold, so AQE never triggers skew join optimization. Lower to 64MB and set spark.sql.adaptive.skewJoin.skewedPartitionFactor=3 to catch skewed joins early. One config change on a 2TB join can cut runtime by half.

Metric

2× faster

Metric Label

skewed join with threshold tuned

Your name

Vara

Contribution agreement

• I agree that this signal may be edited for clarity and published on datasignal.dev with my name and GitHub handle as credit.