Spark shuffle is one of the most common sources of performance problems in distributed data processing. In this guide, we'll walk through what shuffle actually is, how to detect when it's causing problems, and practical optimisation strategies that work in production.
What Is Shuffle?
Shuffle is the process of redistributing data across partitions — it happens whenever an operation requires data from one partition to be moved to another. Common operations that trigger shuffle include groupBy, join, repartition, distinct, and orderBy.
During a shuffle, each executor writes its output to local disk, and then the downstream stage fetches the relevant partitions. This write-read cycle is what makes shuffle expensive — it involves disk I/O, network transfer, and serialisation/deserialisation overhead.
Detecting Shuffle Problems
The Spark UI is your first line of defence. In the Stages tab, look for:
- High Shuffle Read/Write — If shuffle read or write is in the hundreds of GB for a small dataset, something is wrong
- Skew — Some tasks processing vastly more data than others in the same stage
- Spill — Data spilling to disk during shuffle indicates memory pressure
Practical Optimisations
1. Reduce Shuffle Data
The best optimisation is to avoid shuffling data you don't need. Push filters and aggregations upstream — filter early, aggregate before join, and use broadcast joins for small tables.
2. Optimise Partitioning
Right-size your partitions. The sweet spot is typically 128–256 MB per partition. Use spark.sql.shuffle.partitions to control the number of output partitions after shuffle. For Databricks, auto-optimisation features like spark.databricks.adaptive.autoOptimizeShuffle can help.
3. Tune Shuffle Configuration
Key parameters to adjust:
spark.shuffle.file.buffer— Default 32k, increase to 64k or 128k for large shufflesspark.reducer.maxSizeInFlight— Default 48m, controls how much data each reducer fetches at oncespark.shuffle.spill.compress— Always enabled, uselz4orsnappycompression
4. Leverage Adaptive Query Execution (AQE)
AQE in Spark 3.x dynamically optimises query plans at runtime. Enable it with spark.sql.adaptive.enabled=true. It automatically coalesces partitions, handles skew joins, and switches join strategies based on runtime statistics.
Need help optimising your Spark workloads? Let's talk →