Apache Spark WTF??? — 𓁈 Looks Like An Exception 𓅓

“Okay, we’re gonna do one (article) more, yeah!”
(adapted from “Walk Like an Egyptian” by The Bangles)

---

𓂀 Unearthing the Levels of the Great Pyramid

To truly understand the construction of the Great Pyramid of Spark CSV Ingestion, one must look at the ancient hierarchy of labor that transforms raw desert silt into gilded knowledge. Each level represents a specialized guild of the empire, working in harmony to build a monument that can withstand the test of any query.

L0: The Decree of the Pharaoh (The User API)

At the sun-drenched summit stands the Pharaoh (the Developer). Work at this level is born from divine intent — purely declarative and lazy. When we invoke spark.read.csv(), we are not moving stone; we are issuing a Decree to the kingdom’s resources.

• The Actor: DataFrameReader.
• The Duty: To capture the Pharaoh’s intent — paths, delimiters, and configurations — and manifest them as a Logical Plan. This stage is an abstract architectural blueprint where the pyramid exists only in thought, yet to be physically realized.
• The Curse: The Phantom Job. If the Pharaoh demands inferSchema= true, Spark is forced to launch a “Job 0.” This scout must scan the entire quarry just to guess the data’s nature, effectively increasing the I/O cost before construction even begins. To lift this curse, define a manual StructType schema to bypass the scan entirely.

L1: The Architect’s Blueprint (Options & Schema)

Before a single laborer is summoned, the High Architect must define the geometry. This level translates the Pharaoh’s decree into the rigid laws of the physical world.

• The Actor: CSVOptions.
• The Duty: To validate sacred parameters like quote, escape, and header. The Architect locks in the Schema (StructType), ensuring every layer below knows the exact weight and dimensions of the data they must carry.
• The Curse: The Hidden Tongue. If the Architect assumes the stone is etched in UTF-8 but the quarry uses UTF-16, the blueprints become an illegible mess. The resulting “Garbled Text” corrupts the knowledge at the source, turning familiar names into a chaotic jumble of symbols that no laborer can recognize.

L2: The Surveying of the Land (Files & Splits)

The Royal Surveyors step onto the vast, arid plains of the Data Lake. Their task is to divide the monumental burden into manageable, parallel tasks for the thousands of workers across the empire.

• The Actor: InMemoryFileIndex & BatchScanExec.
• The Duty: They perform Partition Pruning, ignoring entire valleys (directories) that do not match the Pharaoh’s filters. Most importantly, they mark the Sacred Splits, assigning byte-offsets to each laborer to ensure no two executors carve the same stone.
• The Curse: The Fragmented Earth. If the quarry consists of millions of tiny pebbles (Small File Syndrome) rather than great blocks, metadata overhead consumes the Scribes. The workers spend more time reading maps than moving stone, and the construction grinds to a halt.

L3: The Quarry of Raw Matter (Hadoop & Line Readers)

In the deep pits of the cluster, the Laborers (Executors) begin the physical toil. This is the level where raw, unrefined matter is pulled from the earth and moved into the stream of the Nile.

• The Actor: Hadoop LineReader.
• The Duty: They stream raw bytes from storage, scanning the silt for record boundaries ($\n$ or $\r\n$). This is the first point of contact between Spark’s logic and the physical weight of the data.
• The Curse: The Monolithic Block. If the data is compressed with GZIP, it is as if the quarry has produced one single, seamless slab of stone. Because GZIP is “non-splittable,” the law of parallelism is broken. A single laborer must drag the entire massive block alone while the rest of the empire’s workers stand idle, unable to help. If parallelism is your goal, choose splittable compression rituals like BZIP2 or LZ4 instead. This curse also extends to the ZIP format.

L4: The Scribes of the Decryption (Univocity Wrapper)

The raw stones are brought to the Scribes’ Workshop. The material is messy and full of traps — fissures hidden inside quotes and escaped characters that would confuse a common worker.

• The Actor: UnivocityParser (The Spark Wrapper).
• The Duty: This layer bridges Spark’s SQL logic with the Oracle’s raw power. The Scribes maintain the “State” of the parse; if they see an open quote, they protect the text within, ensuring the block’s structure remains true. They also enforce the mode (Permissive vs. FailFast).
• The Curse: The Bottomless Well. In PERMISSIVE mode, a stone carved with too many columns causes the Scribes to hoard the “dust” and debris in a “rescue column” (_corrupt_record). This bloats the worker’s memory until the entire construction collapses under the weight of its own waste. Trigger this ritual by setting columnNameOfCorruptRecord in your options to capture these shards without crashing the worker.

L5: The Oracle of the Commas (Univocity Core)

At the heart of the structure sits the Oracle, a foreign mercenary hired for its specialized speed. Here, the continuous lines of text are finally shattered into individual fields.

• The Actor: com.univocity.parsers.csv.CsvParser (The Third-Party Engine).
• The Duty: Using ancient, highly optimized wisdom, the Oracle identifies delimiters and strips quotes with “Zero-Copy” efficiency. It produces an Array[String]—the data is now separated, but it remains “Cold Stone,” lacking the life of a true Data Type.
• The Curse: The Quoted Labyrinth. If a quote is never closed, the Oracle swallows miles of stone as a single field. This leads to a Memory Overflow as the JVM tries to build a massive String object. Crucially, while Univocity is “Zero-Copy” and points to existing buffers to save memory, an unclosed quote forces it to eventually allocate a giant heap object that can choke the executor.

L6: The Alchemical Forge (Bytecode & Type Conversion)

In the deepest foundation lies the Great Forge, where the final transformation occurs. This is the ultimate alchemy: turning “Base Metal” (Strings) into “Gilded Types.”

• The Actor: RowConverter & Whole-Stage CodeGen.
• The Duty: Spark generates a custom furnace — specialized Java Bytecode — on the fly for our specific schema. It casts the string "1" into a binary Integer. The result is the InternalRow, the final, gilded block of the pyramid.
• The Curse: The Heavy Casting. Casting into complex objects or massive decimals instead of simple integers creates “Garbage Collection Slag.” The forge overheats as workers spend more time cleaning waste from memory than pouring gold, slowing the final assembly.

---

𓆑 An Ancient Curse

Why Univocity

In the construction of the Great Pyramid, Spark is a master of many things — planning, scheduling, and massive-scale logistics — but it is not a specialized stone-cutter. For the delicate and treacherous task of splitting lines into fields, it recruits a foreign mercenary: the Univocity library.

To a common laborer, finding a comma seems easy. But the CSV is a labyrinth of edge cases. Consider a single line: 1, "Cairo, Egypt", "He said, ""Build it higher"""

A basic parser would see five commas and shatter the stone into five broken pieces. Univocity is used because it possesses the ancient wisdom to handle:

• Nested Quotes: Knowing that a comma inside "" is data, not a delimiter.
• Escaped Characters: Handling double-quotes ("") as a single character.
• Multiline Records: Remembering that a newline inside a quote is a “Sacred Newline” that belongs to the same record.

Writing a parser that handles these edge cases is easy; writing one that does it at the speed of the Nile’s current is nearly impossible.

• Zero-Copy Efficiency: Univocity is built for extreme performance. It minimizes the creation of temporary objects, ensuring the “Forge” at L6 isn’t choked by memory slag (Garbage Collection).
• Battle-Tested Robustness: Rather than Spark’s priests writing their own parser from scratch and risking bugs, they use the world’s fastest and most reliable CSV engine.

The Curse of the EOF

Here’s the trap: Univocity is a streaming parser. It thrives on continuous text ending with a proper EOF. Feed it tiny, isolated fragments, and instead of shrugging, it summons an internal EOFException for each piece—like a curse echoing through the tombs. As noted in Nuthin’ But A ‘GC’ Thang, Spark may swallow these exceptions, but the JVM quietly remembers them, absorbing each as a silent malediction.

We’ve seen this overhead firsthand. In Stranded On Dates Rows, parsing millions of invalid dates forced the JVM into a relentless ritual of creating and unwinding ParseException objects. Millions of exceptions—even if never surfaced—can throttle execution, because exception creation itself is costly.

Now multiply that by millions of CSV lines:

• One Parser Call Per Line: Each fragment triggers a full parsing cycle.
• Massive Memory Churn: Temporary exceptions and short-lived strings flood the heap, driving the GC into constant motion.
• CPU Exhaustion: Stack trace generation and exception handling steal cycles from real work.
• Silent Chaos: Spark hides the exceptions, but the cluster quietly grinds to a crawl.

A Plague Upon the Whole Kingdom

This silent epidemic doesn’t just afflict our CSV read stage — it can ripple through the entire Spark application. Millions of short‑lived exception objects fuel garbage collection storms, triggering pause times that freeze executors like sphinxes in the heat. The curse was recorded in SPARK‑51579, though, like many of my offerings to the community, it vanished into the void. The parser’s mismanagement of newlines and the legion of silent exceptions continue to weigh heavily on the JVM, unnoticed by all but the brave explorers of performance.

𓆣 Seeing is Believing

The Script

As always, I’ve prepared a small repo for this journey, containing a simple main.py script that guides us through the labyrinth of Spark CSV parsing, calculation, and execution. Here’s what it does, step by step:

We begin by summoning Spark itself:

spark = (  
    SparkSession.builder  
        .config(  
            "spark.driver.extraJavaOptions",  
            "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005"  
        )  
        .getOrCreate()  
)  
spark.sparkContext.setLogLevel("WARN")

This starts a Spark session with debugging enabled, allowing us to attach a debugger to the underlying JVM. The JDWP agent listens on port 5005, pauses the JVM until we attach a debugger, and lets us step through every line of code, inspect variables, and track down elusive exceptions — even those that occur deep in the JVM process.

Next, we construct a “massive” DataFrame, the digital equivalent of building the pyramid’s foundation:

ROW_COUNT = spark.sparkContext.defaultParallelism * 10_000_000  
df = spark.range(ROW_COUNT).selectExpr(  
    "1 as id",  
    "'Tutankhamun' as name",  
    "'18th Dynasty' as dynasty",  
    "'1332 BC' as reign_start",  
    "'1323 BC' as reign_end",  
    "'1341 BC' as birth_year",  
    "'1323 BC' as death_year",  
    "'Valley of the Kings' as burial_site"  
)

Every row is a perfect copy of Tutankhamun’s story, replicated millions of times — our synthetic quarry stone for the CSV pyramid.

We then write the DataFrame to disk in compressed CSV format:

df.write.mode("overwrite").option("header", True).option("compression", "gzip").csv(OUTPUT_PATH)

Afterwards, the CSV is read back into Spark:

df3 = spark.read.csv(OUTPUT_PATH, multiLine=False, header=True, inferSchema=True)

Schema inference turns the raw strings into typed columns, preparing the data for the next stage: alchemy and calculation.

We convert the BC strings into integers and calculate durations:

def bc_to_int(col):  
    return -F.regexp_replace(col, " BC", "").cast("int")  
  
df4 = (  
    df3  
    .withColumn("reign_start_int", bc_to_int(F.col("reign_start")))  
    .withColumn("reign_end_int", bc_to_int(F.col("reign_end")))  
    .withColumn("birth_year_int", bc_to_int(F.col("birth_year")))  
    .withColumn("death_year_int", bc_to_int(F.col("death_year")))  
    .withColumn("reign_duration", F.col("reign_end_int") - F.col("reign_start_int") + F.lit(1))  
    .withColumn("age_at_death_calc", F.col("death_year_int") - F.col("birth_year_int") + F.lit(1))  
)  
df4.cache().show(1, truncate=False)

The durations are inclusive, reflecting historical reality: every year counts.

Finally, we perform a “noop write”, a trick to force Spark to execute all transformations without creating extra files:

df4.write.format("noop").mode("overwrite").option("header", True).save()

The script measures the total runtime, giving us a sense of how long it takes to quarry, transport, parse, and transform this massive dataset.

Debugging the Hidden Depths

Because this is a PySpark application, exceptions inside the JVM — like EOFException from Univocity parsers—are often invisible in Python. That’s where the debug agent comes in handy:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

• transport=dt_socket: communicates over a network socket.
• server=y: JVM waits for a debugger to connect.
• suspend=y: JVM pauses immediately until debugger attaches.
• address=5005: TCP port for the debugger connection.

With this setup, we can attach our favorite Java/Scala-compatible IDE (IntelliJ, VS Code, Eclipse) to the Spark driver JVM. Set breakpoints in the constructors of com.univocity.parsers.common.input.EOFException or other classes to catch runtime exceptions that would otherwise be lost in the haze of the JVM.

Remote Debugging

If the Spark driver runs on a remote node, fear not! We can redirect the debug port via SSH tunneling:

ssh -L 5005:localhost:5005 user@remote-node

This way, our local IDE sees the remote JVM as if it were running on our machine, allowing us to step through the parsing of CSVs and witness the secrets of the data pyramid unfold.

---

𓏛 The Scroll of Endless Lines

Even the most careful scribe can stumble when a scroll is torn across pages. In Spark, this is the source of the dreaded EOFException when parsing CSVs. Our ancient parchment — the CSV file — doesn’t always respect the tidy row boundaries we expect.

But by enabling multiLine=True in the parser, we cast a magical spell that allows the scribe to read the entire record across lines, ensuring that even torn or multiline fields are fully captured before committing. The curse of unexpected EOFs is thus lifted, and the scroll flows uninterrupted into Spark’s temple of knowledge.

The Curse of the Broken Record

When a CSV contains fields spanning multiple lines (for example, a quoted description with line breaks), the parser faces a labyrinth. Each line read from the scroll (via Hadoop’s LineReader) is treated as a separate entity:

• The parser expects one record per line.
• A multiline field breaks this assumption.
• When the parser reaches the end of the split or file without closing a quote, it cries EOFException—the scroll is incomplete, the scribe confused.

The Magic of the Multiline Spell

Enter the multiline spell: multiLine=True. This setting transforms the parsing ritual:

1. From line-by-line to record-by-record

• Instead of feeding one line at a time to the Univocity parser, Spark now treats all lines in a split as part of a possible record.
• The parser keeps reading until the closing quote is found. The record is only complete when the field boundaries are satisfied.

2. EOFException banished

• Because the parser now patiently follows the full record, the end-of-file is no longer a surprise.
• Even if the record stretches across multiple lines or split boundaries, the parser assembles it before committing.
• The curse is lifted: no more sudden, mysterious errors halting our procession of data.

3. Handling split boundaries

• Multiline mode requires careful lookahead and buffer management.
• Internally, Spark may extend the read past the split boundary to capture the remainder of an incomplete record.
• The logical plan still slices the kingdom into partitions, but the parser ensures that no stone is left unaccounted.

4. The Scalability Trade-off

• Setting multiLine=True eliminates the EOFException, but it comes at a cost: Spark can no longer split files at arbitrary line breaks.
• This can create larger partitions and reduce parallelism, especially when dealing with a few massive files.
• Additionally, each split must fit entirely in memory, placing extra pressure on the cluster.

Wisdom from the Ancients

• Without multiLine: Each line of the CSV is read and parsed one by one, like rigid stone blocks stacked without flexibility. If a single field spans multiple lines, the parser sees the next line as a new record, leaving the previous record incomplete. The result? A collapsed wall, and the dreaded EOFException rises like a curse from the depths of the tomb.
• With multiLine=True: The parser treats the record as a whole, reading line after line until the entire field is complete. Blocks bend and connect like magical stones, flowing across line breaks. Even if a field stretches across multiple scrolls (lines), the parser patiently assembles it into a single record. The temple—our DataFrame—is constructed flawlessly, each record perfectly enshrined, and the ancient curse of unexpected EOFs is lifted.

---

𓉐 Leaving the Tomb of the Dormant Exceptions

Now that I have uncovered the secret of the Multiline Spell, I lay to rest the tomb of the EOFExceptions… for now. The ancient curse that haunted Spark parsing has been tamed, yet the sands of the data valley still hold whispers of hidden mysteries.

The scrolls now flow seamlessly, but some chambers remain uncharted, some secrets buried beneath layers of partitions, waiting for intrepid explorers to uncover them. Each line of CSV, each transformation, is a stone to be aligned, a glyph to be read.

To those daring enough to venture deeper into the Spark valley: may your torches shine bright and your curiosity never wane. Though EOFExceptions sleep, the valley’s mysteries are endless — and some shadows may never fully vanish.

---

“Breakpoints in the right places, see?
(oh-EOF, ooh-EOF-oh-EOF)
Looks like an Exception”
(adapted from “Walk Like an Egyptian” by The Bangles)

---