Apache Spark WTF??? · 4 May 2026

Apache Spark WTF??? 🪦 Back From The Dead 💀

The screen is glowing. The Spark UI has not moved in ten minutes. You check the driver logs, and there it is — the dreaded obituary:

15 min read

Apache SparkDatabricksSpark 3.5Performance

0% read·15 min in this view

“Straight out the casket, risin’ up
Open up your eyes, can’t you see me?
What the fuck is this? Madness, pick up my bones
Erase my name from off the tombstones”
(from “Back from the Dead” by House of Pain)

🧟 You Can’t Disrespect it, I’m the Resurrected

To understand a Spark resurrection, you first need to understand how Spark handles a burial. There are three main characters in this tragedy, and they rarely agree on the time of death.

The Executor Heartbeater: The Intern with the Candle

Deep inside every executor lives the Heartbeater. When it starts, it uses a randomized initial delay so thousands of executors don’t all scream “I AM ALIVE” at the exact same millisecond — like zombies rushing the only open bar at a data conference.

This heartbeat isn’t a simple ping. It carries the executor ID, accumulator updates, the BlockManagerId, and metric peaks. It arrives like a frantic worker carrying a clipboard and a bag of suspicious metrics. As long as the Driver sees that little candle bobbing in the dark, it assumes the house is fine.

The Driver HeartbeatReceiver: The Clerk with the Ledger

The Driver’s RPC endpoint, the HeartbeatReceiver, decides who is still part of polite society. It maintains a map — executorLastSeen—that looks like this:

Executor 1: Seen 8s ago. Fine.
Executor 2: Seen 11s ago. Fine-ish.
Executor 7: Seen 128s ago. Suspiciously corpse-like.

If a heartbeat arrives from an executor already struck from the ledger, things get spicy. You get the distributed-systems equivalent of a text from someone whose funeral you attended yesterday:

Executor 7: “Hey, still alive.”
Driver: “Absolutely not.”

The Scheduler Undertaker: The Bureaucrat

Once the Driver decides an executor has been silent too long, the Scheduler starts the paperwork. It marks tasks as failed, cleans up BlockManager states, and erases cached data. This is the bureaucratic death certificate.

The horror is that two parts of the system disagree on the funeral. The executor JVM may still be warm and processing data while the Driver has already reassigned its work to someone else.

The Two Clocks of Doom

Spark heartbeat behavior is governed by innocent-looking configs. They are tiny clocks strapped to a distributed grenade.

spark.executor.heartbeatInterval**: The Pulse [**default: 10s]This is how often the executor raises its hand to say: “Still here, still processing, still emotionally unavailable.” But heartbeating is a privilege. If the JVM is frozen in a Stop-the-World GC pause or turned into a “potato” by CPU throttling, that ten-second rhythm becomes fiction. Reality says: “I’ll heartbeat whenever the CPU gives me permission to exist.”

spark.network.timeout: The Official Time of Death [default: 120s]This is the Driver saying: “If I don’t hear from you for two minutes, I’m calling the priest.” This serves as the fallback for shuffle connections, RPC asks, and block manager lookups. If the liveness paperwork doesn’t arrive, Spark isn’t sentimental. It sees silence; it concludes Corpse.

spark.network.timeoutInterval: The Coroner’s Rounds [default: 60s]The Driver isn’t watching the monitor continuously; it walks through the cemetery periodically. This is why logs look weird: Removing executor 7: 157912 ms exceeds timeout 120000 ms The timeout was 120s, but the coroner only does rounds every 60s. Spark is not a high-precision hospital monitor; it is a sleepy night guard with a flashlight.

ﮩﮩ٨ـ♡ﮩ٨ Skip the Autopsy ’Cause I Never O.D.’d

The Zombie Paradox

The Zombie Executor exists because “alive” is not a single truth in a distributed system. There is “alive” to the OS, to the JVM, to the Spark Driver, and finally, to the Resource Layer — the landlord (Kubernetes, YARN, or Databricks) that owns your cluster. These layers do not share one clock, one brain, or one definition of existence.

Different Observers, Different Truths

Imagine Executor 7 hits a massive Stop-the-World GC pause, brutal CPU pressure, or I/O wait so heavy it cannot even blink. From the outside, the vital signs look fine:

PID exists? Yes.
JVM still owns memory? Yes.
Resource Layer says container is Running? Yes.
Spark Driver says executor is alive? Absolutely not.

The Awkward Reunion

From the executor’s perspective, it was just stuck. When it finally wakes up from its GC coma, CPU starvation, or network fog, it resumes its duties:

Executor: “I brought metrics!”
Driver: “You are dead.”
Executor: “I have a heartbeat!”
Driver: “Since when do ghosts have heartbeats?”

The Scheduler has already moved on, tasks have been retried elsewhere, and the funeral is over. Now both are standing in the same room making eye contact. This is why blindly increasing timeouts isn’t a silver bullet — larger timeouts reduce false deaths but force Spark to wait forever to clean up genuine corpses. You don’t want false funerals, but you also don’t want immortal corpses spookily consuming resources.

The SPARK-24387 Ghost Story

For a real campfire story, look at SPARK-24387. This isn’t just a bug report; it’s a crime scene transcript.

The report describes a job where an executor’s heartbeat is lost. The Driver removes it, but because removal, cleanup, and scheduling happen at different speeds, a nightmare window opens:

HeartbeatReceiver: “This executor is dead.”
TaskScheduler: “Noted. Removing.”
Cluster Backend: “Killing it… eventually.”
Scheduler State: “Wait, I see a free executor!”
Retry Attempt: “Hold my beer, I’m going in.”
Executor: “Back from the dead, baby.”

The Driver thinks it’s a corpse. The Scheduler sees available real estate. And the Executor? It just clocked back in for the morning shift. This is the zombie paradox in one tiny nightmare: when reality fails to converge, the “dead” are the first ones back to work.

👻 Ghost-Busting in Spark 3.5 & 4.x

If you’re reading this thinking, “Wait, is my cluster just a haunted house of immortal corpses?”— do not panic yet. The Zombie Paradox remains a law of distributed physics, but modern Spark (3.5+) has become a professional exorcist. The resurrected executor is no longer welcome at the party.

The “Access Denied” Treatment

In older Spark horror stories, a timed-out executor could look usable again if the Driver’s internal state hadn’t converged. Modern Spark is less forgiving. Once the scheduler backend removes an executor, any subsequent task updates from that ID are ignored.

In human terms:

Executor: “I finished the task!”
Driver: “I have no record of you. Your work is invalid.”

The executor may still be alive at the JVM level, but Spark treats its messages as noise from an “Unknown Executor” and drops them immediately.

The Ghost-Touch BlockManager

The real danger of a resurrected executor isn’t a heartbeat; it’s the attempt to reattach cluster state — cached blocks or shuffle data — after the Driver has already erased it.

Modern Spark hits this with a “double-tap.” If the scheduler is asked to remove an executor, it explicitly purges that ID from the BlockManagerMaster simultaneously. A worker without a valid BlockManager state is like a ghost trying to pick up a coffee cup: its hand passes straight through the handle. It can’t serve data, and it can’t store results.

Decommissioning Over Surprise Funerals

Modern Spark prioritizes Decommissioning over sudden death. When supported by the cluster manager (Kubernetes, YARN, or Databricks), Spark tries to retire executors gracefully. Instead of a sudden flatline, Spark attempts to migrate shuffle and cached blocks before the worker leaves.

This doesn’t fix every GC coma or “Kubernetes potato” transformation, but it changes the logic from “Executor vanished. Good luck, everyone.” to “Executor is retiring. Move the useful stuff before the coffin closes.”

Closing the “Hold My Beer” Window

The SPARK-24387 nightmare happened because removal, scheduler state, and process death weren’t synchronized. Modern Spark narrows that window through aggressive state cleanup and stricter tracking of executors “pending removal.”

So yes, a JVM may still wake up after a timeout and try to rejoin the living. But Spark is now much more likely to answer: “Cute heartbeat. Still dead.” The zombie can still twitch; it just can’t get back on the schedule.

💀 Why Executors Rise from the Grave

Distributed systems are essentially a collection of computers lying to each other about the time. When an executor “returns from the dead,” it is not a glitch; it is a failure of consensus. The Driver, JVM, OS, and Cluster Manager are all looking at the same process through different lenses and rules for “survival.”

To fix a ghost, you must first identify which layer is lying.

1. The Internal Coma: Stop-the-World GC

The classic cause of resurrection is not a crash, but a pause. The JVM isn’t dead; it’s stuck in a Full GC existential crisis. During a Stop-the-World pause, the Heartbeater thread freezes. The Driver sees silence exceeding spark.network.timeout and reaches for the shovel.

Eventually, the JVM wakes up — but Spark has already signed the death certificate. Broadcast joins often trigger this: send an 8 GB “love letter” to every executor and suddenly every JVM is “chewing glass”— allocating, spilling, and collecting just to stay afloat.

2. The Overwhelmed Manager: Driver Overload

Sometimes the executor is innocent. It is sending telegrams, but the Driver is buried under unopened mail. A massive fleet of tiny executors can effectively DDoS the Driver with heartbeats and task metrics.

The LiveListenerBus can also become a crime scene. If driver-side event processing, UI updates, and metrics pile up, the Driver hits backpressure. It isn’t dead; it’s just drowning in paperwork and ignoring the phone when the executors call.

3. The Fog of War: Network & Timeout Traps

If the network between the Driver and Executor is auditioning for a horror movie, spark.network.timeout becomes the master clock for Spark’s anxiety. If a shuffle storm or network congestion delays the heartbeat packet beyond this window, the executor is marked as lost.

Beware: Zero is not Zen. Never assume 0 means the same thing everywhere. Depending on the config, it can mean disabled, infinite, or “use another default.” Blindly using zeros can create immortal zombies that hang on a socket forever, quietly consuming cluster slots while doing zero work.

4. The Physical Body: Shuffle & Disk I/O

If the JVM is the soul, disk I/O is the body. High-volume shuffle acts like a localized DoS attack. Imagine a moving company forcing a grand piano through a narrow hallway; the heartbeat is the guy on the guest list trying to squeeze past with a tiny envelope.

When memory fills up and Spark spills to disk, the resulting high I/O wait creates “micro-stutters.” These aren’t GC pauses, but they have the same effect: they delay the Heartbeater just enough for the Driver to start measuring the coffin.

5. The Aggressive Landlord: Kubernetes & YARN

Sometimes Spark didn’t kill the executor; the platform did. On Kubernetes, tight CPU limits lead to throttling. The threads are “runnable,” but the OS refuses to give them the CPU cycles needed to breathe.

Memory is even nastier. Spark may see a healthy heap, but the Kernel OOM Killer watches the total container: JVM, off-heap, direct buffers, and Python workers. If you cross the line, the landlord issues a SIGKILL with no dramatic farewell.

The Driver says: “Heartbeat timeout.”
The Landlord says: “Eviction.”

📋 The Forensic Checklist

When an executor vanishes and then reappears like a digital phantom, follow the data, not your gut. Your goal is to identify the killer and understand how the “body” managed to crawl back into the cluster.

1. Driver Log Clues: The Bureaucrat’s Record

The Driver is the first to notice the silence. Its logs capture the exact moment Spark initiated the funeral paperwork.

HeartbeatReceiver: Removing executor [ID] with no recent heartbeatsThe smoking gun. Note the timestamp and the ms since last heartbeat—it tells you exactly how long the silence lasted.
TaskSchedulerImpl: Lost executor [ID] The moment the Scheduler is officially informed that the worker is gone.
ExecutorLostFailure Tasks on that executor are now orphans. This is the point where a heartbeat problem officially becomes a Stage performance problem.

2. Executor Log Clues: The Victim’s Last Words

If the executor survived long enough to write logs, this is where the resurrection drama lives.

Issue communicating with driver in heartbeater The executor tried to speak, but the Driver didn’t answer. Suspect Driver overload, network fog, or a timeout mismatch.
Exit as unable to send heartbeats... more than [X] times The executor hit spark.executor.heartbeat.maxFailures (default 60) and shut itself down. It realized it was cut off from the Driver and decided to exit to stop wasting cluster resources.
OutOfMemoryError / GC overhead limit exceeded The coma went too deep; the JVM wasn’t just slow—it lost the memory war.
Container killed by YARN / OOMKilled The assassination markers. The platform ended the process before Spark could say goodbye.

3. Spark UI Clues: The Crime Scene Map

The Spark UI provides the visual history of the disaster via the Stages and Executors tabs.

GC Time (The Orange Bar): If GC time is a large fraction of task duration, you are staring at a GC Zombie.
Task Skew & Input Size: One executor getting 10x the data? That is your prime suspect for memory pressure and “Back from the Dead” timeouts.
Spill (Memory/Disk): High spill means Spark is leaning on the disk. If node metrics show high I/O wait, the heartbeat thread likely got trapped behind disk misery.
Executor Timeline: Look for gaps. If all tasks on an executor stop simultaneously while the process persists, suspect a Stop-the-World pause or CPU starvation.

4. Platform Clues: The Landlord’s Ledger

The truth is often outside Spark’s own logs.

YARN Container Diagnostics: Check the ResourceManager UI for “Physical memory limits exceeded.”
Kubernetes Pod Status: Run kubectl describe pod to find OOMKilled, Evicted, or NodePressure.
CPU Throttling: Check your monitoring dashboards. If CPU usage is pinned at the container limit and nr_throttled is high, the heartbeater was effectively suffocated.
Disk I/O Wait: High iowait points to disk bottlenecks: heavy spills, slow local storage, or “noisy neighbors” on shared nodes.
Network Retransmits: High retransmits mean heartbeat packets are getting lost in the fog. The executor is shouting; the Driver just can’t hear it.

⏱️ Don’t Just Crank the Timeout and Call It Architecture

If your first instinct is to set the timeout to 30 minutes to “stop the errors,” you aren’t an architect; you’re a necromancer. You are keeping corpses on life support. To fix the Zombie Paradox, you must treat the trauma, not just give the ghost more time to wander the cluster.

1. Timeout Tuning That Does Not Summon Demons

Before touching configs, remember this golden rule:

spark.executor.heartbeatInterval << spark.network.timeout

The defaults (10s vs 120s) give the executor many chances to miss a beat before the Driver calls the Undertaker. Do not go absurdly high; if your timeout is huge, Spark will take forever to notice genuinely dead executors, wasting hours on a doomed job. Only increase timeouts when you have proof the executor is healthy but delayed (e.g., GC logs or CPU throttling). If the executor is OOMing, a longer timeout just gives it more time to suffer.

2. Memory Fixes: Feed the Beast Properly

If zombies are born in GC comas, stop the comas.

Fix Skew First: Skewed data creates skewed memory pressure. Use salting, repartitioning, or AQE to balance the load.
Reduce Broadcast Size: If you are broadcasting an 8 GB “love letter,” you are essentially throwing a brick through every executor’s window. Use a shuffle join instead.
Tune Memory Overhead: On K8s/YARN, spark.executor.memoryOverhead covers off-heap allocations, Netty buffers, and Python workers. If this is too small, the landlord (the OS) will murder the container while the JVM heap still looks innocent.

3. Driver Fixes: Respect the Manager

If the Driver is overwhelmed, the whole cemetery goes to hell.

Consolidate Executors: Avoid thousands of tiny executors. Fewer, larger executors reduce the Driver’s administrative burden: fewer heartbeats, fewer records, and less metadata.
Beef Up the Hardware: Give the Driver enough CPU and memory. It is the brain of the operation; do not run it on a potato.
Reduce Event Pressure: Avoid tiny-task storms. If you have a million tasks lasting 100ms each, the Driver spends its life drowning in paperwork instead of managing the job.

4. Network and Shuffle Fixes: Clear the Hallway

If the “moving company” (Shuffle) is blocking the heartbeat, clear the hallway.

Fix Skew Before Tuning: If one executor is drowning while others are idle, a timeout increase is just giving the drowning executor a nicer watch. Balance the shuffle first.
Optimize Shuffles: Filter early, project fewer columns, and avoid accidental wide dependencies.
Avoid Immortal Zeros: Do not set timeouts to 0 or 24h unless you want executors that hang forever on a socket. If an executor genuinely hangs, you want Spark to kill it and retry.

5. Kubernetes / YARN Fixes: Landlord Relations

CPU Headroom: On Kubernetes, tight CPU limits cause throttling. The heartbeat thread becomes “runnable” but is practically ignored by the OS. Give executors enough CPU headroom to breathe.
Identify the Killer: Do not let platform kills masquerade as Spark timeouts. Check for OOMKilled or YARN memory violations. If the landlord is murdering the process, no Spark timeout will save it.
Use Decommissioning: When supported, enable Spark’s decommissioning features. This migrates shuffle and cached blocks before an executor exits, turning sudden funerals into planned retirements.

👓 The Executor Wasn’t Dead — Your Observability Was

We’ve spent this deep dive hunting ghosts, but the truth is simpler: the executor was never actually dead — it was just trapped in a different reality.

In a distributed system, “alive” is a shifting consensus, not a binary fact. The Driver thinks you’re dead if the paperwork is ten seconds late, while the JVM and OS still see a warm body. It’s not a crash; it’s a glitch in the bureaucratic matrix where the death certificate was issued, but the worker refused to stop breathing.

Spark is essentially a stressed-out priest trying to run a funeral while the corpse keeps sitting up in the casket to ask for a coffee refill. The “Back from the Dead” phenomenon is just that awkward moment when the funeral is over, but the executor is still standing in the lobby holding a tray of metrics. Don’t trust the tombstone — blindly cranking up spark.network.timeout just builds a longer, more expensive hallway for the ghosts to walk in. If you give a zombie a bigger timeout window, you don’t get a healthy worker; you just get a zombie who is late for its own burial.

In the world of Big Data, we obsess over data accuracy but ignore infrastructure accuracy. A heartbeat is just a message, and a timeout is just an assumption. The next time your Spark UI shows a resurrection, remember it’s not magic — it’s just physics with a bad attitude.

Your executors are hard to kill; make sure your architecture is, too.

“Y no estaba muerto no, no
Y no estaba muerto no, no
Y no estaba muerto no, no
Estaba tomando cañas, lerelele
…
No estaba muerto, estaba de parranda”
(from “El Muerto Vivo” by Peret)