Apache Spark WTF??? — ⭕ Skip 🔺 Game 🟥

“Round and round (clap)
둥글게 둥글게 (짝)
dunggeulge dunggeulge (jjak)
Round and round (clap)
둥글게 둥글게 (짝)
dunggeulge dunggeulge (jjak)
Let’s dance as we spin around
빙글빙글 돌아가며 춤을 춥시다
binggeulbinggeul doragamyeo chumeul chupsida”
(Mingle Game Song “Round and Round”)

---

When Spark Gets Smart About Doing Less

Apache Spark is built for speed. At its core, it’s a high-performance, distributed data processing engine that transforms your code into directed acyclic graphs (DAGs) — a fancy way of organizing computation into a clear, one-way flow. These DAGs are then sliced into jobs, stages, and tasks, which Spark executes across your cluster.

Under normal circumstances, Spark dutifully follows this pipeline from start to finish. But Spark isn’t just fast — it’s clever. In certain situations, it realizes, “Wait a minute… I don’t need to do this again.” And that’s when execution skipping kicks in.

Execution skipping helps Spark avoid:

• Recomputing results that are already available
• Reading the same data again and again when it hasn’t changed
• Reprocessing intermediate steps that are cached or checkpointed
• Launching duplicate or speculative tasks that aren’t needed

Skipping in Spark doesn’t mean “ignoring work.” It means cleverly avoiding unnecessary computation based on what’s already known or stored.

It can happen at multiple levels:

• Job Skipping: Spark may skip running an entire job if a result is already cached or can be short-circuited due to smart optimizations.
• Stage Skipping: If outputs of a stage are already materialized (say, via a checkpoint or persisted RDD), Spark skips re-executing that stage.
• Task Skipping: Spark might skip individual tasks when it detects that certain partitions don’t need to be processed — for example, if they’re filtered out or already computed.

These skipping behaviors are powered by:

• Spark’s understanding of DAG lineage
• The state of the cluster and cache
• Data source characteristics
• And internal optimizations by Catalyst and AQE (Adaptive Query Execution)

Here are five common situations where Spark confidently says, “No thanks, I’ve already handled that.”

• Cached Query Results: When you re-run a query and the results are already sitting comfortably in memory, Spark skips the entire computation and serves the answer instantly.
• Persisted Iterative Workloads: Algorithms like PageRank often perform the same transformations over and over. By using .persist() or .checkpoint(), Spark holds onto intermediate results, avoiding expensive recomputation across iterations.
• Common Subqueries: If multiple queries or stages in your pipeline share the same subquery or transformation, Spark can recognize and reuse the result instead of recalculating it each time. It’s like memoization for your data pipeline.
• Partition Pruning: When your filters exclude entire partitions (like WHERE date = '2025-01-01'), Spark intelligently skips reading any files or partitions outside that range. No I/O, no shuffle, no problem.
• Smart File Skipping: Data formats like Parquet include rich metadata (like min/max values, row counts, column stats). Spark uses this info to avoid scanning files — or even individual blocks — that it knows won’t match your query.

Now, let’s take a closer more hands-on look at how each of these skipping techniques actually works behind the scenes…

(Ready to geek out? Let’s go.)

🏃‍♂️ Cache Me If You Can

aaaa

💽 Persist, Don’t Insist

aaa

👀 Seen It, Skipped It

aaa

✂️ No Barking at the Wrong (Pruned) Tree

aaa

🙈 Peek and Not Seek

aaaa

🤨 Stage Complete … Somehow