Apache Spark WTF??? 🤪 Insane In The DataFrame 📜

“Rip that dataframe, execute the explain
More gigas, spike heap, it’s going insane
Insane in the dataframe
Insane in the Spark brain”
(from “Insane in the brain” by Cypress Hill)

---

🕵️‍♂ ️The Usual Suspects: Heap, Off-Heap, Driver, Executor, and Other Liars

Before we start blaming Spark 3.5+ for crimes against humanity, we need to do the boring detective work.

I know. Nobody came here for responsible debugging. You came for distributed systems violence. But an OOM in Spark is not one error. It’s a whole family of tiny disasters wearing the same trench coat.

Sometimes the JVM heap explodes.Sometimes the executor gets murdered by Kubernetes. Sometimes Python eats the memory and Spark pretends it has never met Python in its life. Sometimes the driver dies because it tried to remember too many things.Sometimes your heap looks healthy, your Spark UI smiles politely, and then Kubernetes appears from the ceiling like Batman and deletes the pod.

1. The first question: who died?

When someone says: Spark failed with OOM. That is not a diagnosis. That is like saying: The patient is dead. Useful? Yes. Enough? Absolutely not, doctor.

You need to know who actuallydied:

• A driver OOM smells like collect(), large broadcasts, Spark UI metadata, event logs, query plans, accumulators, or too many retained executions.
• An executor OOM smells like skew, joins, aggregations, sorts, bad partition sizing, writer fanout, table-format internals, or one partition carrying the emotional weight of the whole cluster.
• A Kubernetes OOMKilled smells like container memory, not necessarily JVM heap. The JVM may be innocent. The pod may still be dead.
• And a Python worker OOM smells like Arrow, Pandas UDFs, toPandas(), native memory, and that beautiful moment where the JVM says: I have no idea what happened. I was in the other room.

Spark’s own documentation is very explicit that executor and driver container memory is not just heap. Memory overhead includes non-heap memory, off-heap memory, PySpark/Python processes, and other processes running inside the same container. The container limit is effectively heap plus memory overhead, not heap alone.

That sentence should be printed on a mug. Preferably one you could throw at the wall after your third OOMKilled.

2. The second question: what kind of memory are we talking about?

Spark memory started as a small club with two rooms:

• execution memory, where joins, shuffles, sorts, and aggregations were sweating on the dance floor,
• and storage memory, where cached RDDs, cached DataFrames, and broadcast blocks were sitting on couches refusing to leave.

In early Spark, those rooms had stricter walls. If execution needed space but storage was chilling with unused room, Spark was not always great at moving people around.

Around Spark 1.6, the Unified Memory Manager knocked down part of the wall. Execution and storage began sharing a common pool. Execution could evict cached blocks when things got intense, while storage could use free space when the dance floor was quiet.

In Spark 3.x and+, the club got bigger and weirder. Now it is not just execution and storage. You also have driver metadata, query plans, listeners, accumulators, off-heap memory, Python workers, Arrow, native libraries, and container overhead.

So old Spark memory was a two-room venue. Modern Spark memory is a suspicious nightclub with a VIP room, guests overstaying, Python workers smoking outside, native memory hiding in the basement, and Kubernetes at the door counting everyone.

3. The third question: did Spark OOM, or did the container OOM?

This distinction is extremely important and extremely annoying.

A JVM OOM usually leaves a Java-shaped corpse: OutOfMemoryError: Java heap space or GC overhead limit exceeded or Spark’s own SparkOutOfMemoryError. That is painful, but at least the JVM had the decency to write a note.

A Kubernetes OOM kill is different. The container exceeds its memory limit, and Kubernetes kills it from the outside. No dramatic JVM stack trace. No polite Spark exception. Just:

Reason: OOMKilled  
Exit Code: 137

This is the distributed systems equivalent of: He was alive when I left the room.

That is why Spark jobs using PySpark, Arrow, native libraries, or heavy off-heap memory can look absolutely fine in the Spark UI while the container memory graph is climbing the walls.

Spark’s configuration docs say that spark.executor.pyspark.memory, when set, limits Python memory per executor; if it is not set, Spark does not limit Python memory and the application must avoid exceeding the overhead memory shared with other non-JVM processes.

This is especially relevant in Spark 3.5+ because modern Spark workloads are rarely “just JVM Spark.” They are Spark plus Python, Arrow, vectorized UDFs, table formats, native state stores, Kubernetes limits, and one dashboard that insists everything is fine.

4. A practical smell test for Spark OOMs

Before touching any config, ask these questions.

If the driver died

Suspect: collect() / toPandas(), large broadcast creation, huge query, plansAQE plan updates, Spark UI metadata, SQL listeners, accumulators, event logs, too many retained jobs/executions.

Driver OOMs are especially evil because increasing executor memory does absolutely nothing. That is the classic Spark ritual:

1. Driver dies.
2. Engineer increases executor memory.
3. Driver dies again, but now more expensively.

Beautiful. Tragic. Cloud-billed.

If one executor died repeatedly

Suspect: data skew, large partitions, wide aggregations, sorts, joins, writer fanout, one huge input file, bad partitioning, table-format writer internals.

If the same task keeps failing, do not start by increasing the whole cluster. Start by asking: Why is this one task carrying a piano up the stairs?

If Kubernetes killed the pod

Suspect: off-heap memory, Python worker memory, Arrow batches, direct buffers, memory overhead too low, container limit too tight.

The Spark UI may not show the whole murder.

Check: container RSS, pod events.Check executor logs, JVM heap and non-heap, whether Python is living rent-free in your executor.

If the History Server or UI exploded

Suspect: giant event logs, giant plan strings, too many SQL executions, too many accumulators, listener retention, large UI metadata.

This is the modern Spark 3.5+ horror genre: the job may not be the problem. The description of the job may be the problem.

5. The tiny checklist before you panic

When Spark screams OOM, do not immediately do this:

--executor-memory 500g

That is not debugging. That is negotiating with a raccoon. 🦝

Start here:

1. Who died? Driver, executor, Python worker, pod, history server?
2. How did it die? Java heap space, SparkOutOfMemoryError, GC overhead, OOMKilled, exit 137?
3. Where did memory grow? JVM heap, off-heap, Python, container RSS, event log, UI metadata?
4. Is the data actually big? Or is the plan, partition, metadata, delete file set, bucket fanout, or single record big?
5. Did this start after moving to Spark 3.5+? If yes, look suspiciously at AQE, UI metrics, V2 readers, Arrow, table formats, and memory overhead.
6. Can I prove it? Heap dump, Spark event log, executor logs, pod metrics, query plan, table metadata.

Because in Spark 3.5+, OOM is not a root cause. It is a doorbell. And behind the door might be a skewed join, a Python worker, a Kubernetes limit, a 3.5 GB JSON whale, or a tiny piece of metadata wearing a fake mustache.

Open carefully. 🙈

---

🍽️ Case File #1: When the Spark UI Eats the Driver

In SPARK-55353 case, a complex SQL query ran on Spark 3.2.2 with 2 GB of driver memory, but failed on Spark 3.5.7 even with 8 GB. The query had around 200 stages, several UDFs, 12 left joins with subqueries, and more than 100 output columns.

According to the heap dump analysis, the driver contained 15.6 million AccumulatorMetadata objects, consuming around 1.5 GB, retained by SQLAppStatusListener. The report links this growth to extra Spark SQL metrics added across Spark 3.3, 3.4, and 3.5. The JIRA was still open when checked and affects Spark 3.5.7.

The data did not kill the driver.
The dashboard’s memory of the data killed the driver. 📊🔪

The crime scene

The obvious suspect in a driver OOM is usually something humiliating:

df.collect()

Or its fancy cousin:

df.toPandas()

Or the classic:

bigThing.broadcast()

You know, the usual:

“I accidentally invited the entire data lake into the driver heap.”

But this case is nastier. The heap dump did not primarily point to rows collected to the driver. It pointed to Spark’s own SQL UI/status machinery: SQLAppStatusListener, holding millions of accumulator metadata objects.

The Spark Web UI is supposed to help you monitor jobs, stages, storage, executors, SQL, streaming, and runtime details.

That is the wholesome version.

The horror version is:

Every useful metric has a cost.
Every task whispers statistics into the driver’s ear.
Sometimes the driver remembers too much.

Why accumulators are in the murder room

Accumulators aggregate information from tasks back to the driver. In user code, they are counters. In Spark SQL internals, they also power many UI metrics:

output rows  
spill size  
shuffle read/write  
scan time  
operator metrics  
driver-side metrics

Now multiply:

big SQL query  
  × many stages  
  × many tasks  
  × many operators  
  × many metrics  
  × retained UI metadata  
  = driver heap confetti cannon 🎉💥

Metrics are good. Metrics are how we debug the monster.

But occasionally, the monster eats the notebook.

The tragedy of “just add driver memory”

The upgrade story is brutal:

Before:  
Spark 3.2.2  
driver memory = 2 GB  
query runs

After:  
Spark 3.5.7  
driver memory = 8 GB  
query dies

That is how platform teams develop a thousand-yard stare.

Because if metadata scales with tasks, stages, operators, and metrics, increasing memory may only postpone the funeral. You are not fixing the shape of the problem. You are buying a larger coffin.

Spark exposes spark.sql.ui.retainedExecutions, defaulting to 1000, to control how many SQL executions are retained in the UI. That can matter in long-running apps or shared sessions. The JIRA also proposes a more direct config, spark.sql.ui.appStatusListener.enabled, to control whether SQLAppStatusListener is loaded at all.

Important: that config is a proposal from an open JIRA, not a guaranteed knob in every Spark build.

In other words:

Do not copy-paste random configs from a bug report into production and call it architecture.
That is how haunted clusters are born. 👻

---

Why this is a Spark 3.5+ horror 🧟‍♂️

Old-school Spark OOMs were often satisfyingly physical: a huge shuffle, a terrible join, a skewed key the size of Jupiter, or a groupByKey() written by someone who enjoys chaos.

You could point at the data and say:

“There. That thing is too large.”

This case is different. The dangerous object is not necessarily the data.

It is the metadata about the data.

The observability layer is no longer just a window.

Sometimes it is a hungry window.

---

How to recognize the beast 👃🔥

You may be in this story if the driver OOMs while executors look suspiciously innocent.

Other smells:

Complex SQL:  
  many stages  
  many tasks  
  many joins  
  many SQL operators  
  many generated columns

Spark UI or SQL tab is huge.

Event log is suspiciously large.

Heap dump shows:  
  SQLAppStatusListener  
  SQLExecutionUIData  
  AccumulatorMetadata  
  SQL metrics  
  listener/status/store objects

Increasing executor memory does nothing.

Increasing driver memory helps briefly,  
then sadness returns.

The key phrase is:

java.lang.OutOfMemoryError: Java heap space

on the driver.

Not executor.
Not container.
Driver.

That changes the investigation.

---

What to inspect before sacrificing more RAM 🧪

Do not start with:

--driver-memory 200g

Start with evidence.

Look at the heap dump. Yes, heap dumps feel like reading a phone book written by demons. Do it anyway.

Look for dominators like:

SQLAppStatusListener  
AppStatusStore  
ElementTrackingStore  
SQLExecutionUIData  
AccumulatorMetadata  
SQLMetric

Then check UI retention settings, especially in notebooks, shared clusters, and long-running Spark applications. If the query was generated by a BI tool, orchestration framework, dbt macro, feature platform, LLM, or one very caffeinated engineer, inspect the generated SQL.

Also compare versions carefully. If your path looks like:

Spark 3.2 / 3.3 ➜ Spark 3.5+

and suddenly the driver needs much more heap for similar workloads, do not immediately blame the data. Blame the timeline. Then investigate the internals.

What can you do? There is no universal button labeled:

spark.sql.please.stop.eating.driver.memory=true

Although honestly, there should be.

But you can respond sensibly.

Reduce retained UI history where appropriate:

spark.sql.ui.retainedExecutions=100

or lower in very long-lived interactive environments. This may reduce UI memory pressure, but it will not magically solve every listener-retention problem.

Shorten application lifecycles when possible. Long-lived Spark sessions are convenient, but they are also terrariums for metadata. If a driver runs for days while thousands of queries pass through it, the UI/status store can become a museum of every bad decision your workload ever made.

Watch generated SQL. If one query has hundreds of stages and a suspicious number of joins, ask why. Break it up if that improves stability. Materialize intermediate results when justified. Cache or checkpoint intentionally. Do not let one SQL statement become the final boss of the JVM.

And use heap dumps as truth serum. If AccumulatorMetadata and SQLAppStatusListener dominate, you are in this story. If UnsafeRow, byte[], hash maps, broadcast relations, or user objects dominate, you are in a different one.

Do not debug all OOMs with the same hammer.

The tiny moral

The Spark UI is not free. Metrics are not free. Listeners are not free. And in a large enough query, with enough stages, tasks, SQL operators, and retained metadata, the thing designed to help you understand the job can become the thing that kills the job.

That is not a reason to hate observability. It’s a reason to respect its appetite. Because sometimes Spark does not run out of memory processing your data. Sometimes Spark runs out of memory remembering how heroic it was while processing your data.

---

---

“Last five months of my life was the bad part.
Next ten years of my life, I’m making mad art.
My priority is high priority.
I’m hi-qualified — night prizefight, voluntarily.”
(from “Crazy” by Madchild)