Apache Spark WTF??? ❤️ The Love Boat ⛵

“Sail, exciting and new
Come aboard, we’re expecting you
Sail, like the sweetest reward
Let it flow, it flows spectacularly”
(adapted from “Love Boat Theam” by Jack Jones)

---

⛵ Before Leaving the Dock

What Sail Actually Is

At its core, Sail is an open-source, Rust-native computation framework designed to be a drop-in replacement for traditional Apache Spark SQL and DataFrame workloads. It speaks the same language as Spark because it implements the Spark Connect protocol — meaning our existing PySpark clients, DataFrame APIs, and SQL scripts generally should keep working without rewriting our business logic.

Under the hood, Sail takes everything you thought you knew about distributed compute frameworks and flips the script. Instead of relying on a heavyweight Java Virtual Machine (JVM) with all its attendant garbage collection pauses, heap tuning, and process boundaries, Sail is crafted entirely in Rust — a systems programming language with compile-time memory safety and zero-cost abstractions.

What does that mean for us passengers? Less overhead, fewer surprises, and performance that hits the hardware like a speedboat hitting open water.

Spark Connect & Compatibility

Spark Connect is the bridge that lets Sail pretend to be Spark’s backend. When our PySpark client emits a query or DataFrame transformation, it doesn’t talk to a JVM executor anymore — it talks over gRPC to Sail’s server component. That server then takes our logical plan and compiles it into optimized execution using Rust’s ecosystem (like Arrow and DataFusion) rather than Spark’s JVM engine.

This is the big architectural win: zero-copy data movement, which means almost no serialization and none of the usual performance penalties that come with it. Because Sail uses the columnar Arrow format natively, Python UDFs can operate on the same memory buffers as the core engine — no need for Py4J bridges, serialization round-trips, or intermediate process boundaries.

Why Rust Outweighs the JVM

As before mentioned, Sail is built in Rust, which makes it faster, lighter, and more nimble than the old, bulky JVM ships. Rust brings several key benefits that directly tackle the old headaches Spark sailors know all too well:

• Memory safety without garbage collection: Rust’s ownership model keeps the ship tight and seaworthy, eliminating whole classes of bugs and avoiding those dreaded GC storms that can grind a job to a halt at the worst possible moment.
• Predictable performance & tiny footprint: Sail workers boot in seconds and sip only a few megabytes at idle — perfect for cloud fleets where elasticity and fast scaling matter.
• High-throughput execution: Thanks to Apache Arrow’s columnar format and SIMD-friendly processing, Sail slices through data like a sleek hydrofoil, far more efficiently than JVM row-based engines.
• In-process Python execution: No more JVM ↔ Python dinghy rides. With Python embedded viaPyO3, LakeSail keeps the whole crew on deck, skipping the classic PySpark bottleneck. Built and packaged with maturin, the Rust–Python bridge runs like a well-crewed love boat — smooth sailing, no pickle overboard.

---

🛳️ ️ The DataFusion Hull: Compact by Design

If the JVM is a steel-plated freighter, DataFusion is the carbon-fiber hull that lets Sail cut through the water.

In Spark, the planner builds a sophisticated JVM execution tree that still resides inside a heavyweight, memory-hungry runtime. Sail, by contrast, uses DataFusion — a Rust-native query engine — to turn SQL and DataFrames directly into lean, native execution.

But here’s the underrated part: Sail isn’t just using a faster executor. It’s building on an entire Arrow-native ecosystem. That means fewer internal format shifts, fewer hidden conversions, and a pipeline that stays in one representation end-to-end. Performance gains compound instead of getting eaten by glue code.

The Vectorized Engine Room: Moving Containers, Not Buckets

Traditional Spark often operates row-by-row. At scale, that’s like passing buckets of water down a long line — functional, but painfully inefficient.

Sail runs on vectorized execution. It processes data in record batches — large, columnar chunks already stored in Apache Arrow format. By processing entire vectors at once, Sail achieves:

• SIMD-Friendly Processing: The CPU performs the same operation on multiple data points simultaneously.
• Minimal Branching: The engine determines data types once per batch, not once per row.
• Better Cache Locality: Contiguous columnar memory keeps the CPU’s high-speed cache full.

Instead of moving one row at a time, Sail moves entire shipping containers of data through the engine in a single pass. More cargo, fewer trips.

🤔 But Doesn’t Spark Use Arrow Too?

Yes — but not at its core.

Spark’s execution engine (Catalyst + Tungsten) runs inside the JVM using its own internal binary row format, like UnsafeRow. Arrow mainly appears at the boundaries — especially to accelerate PySpark features like Pandas UDFs and toPandas() — where it acts as a high-speed serialization bridge between JVM and Python.

Sail, by contrast, builds directly on Apache Arrow via DataFusion. Arrow isn’t a bridge layer — it is the in-memory execution format.

Spark uses Arrow to cross boundaries.
Sail builds the engine directly on top of it.

🤔 What About Spark’s Off-Heap Memory — Isn’t That Columnar?

Not really.

Spark’s Tungsten engine uses off-heap memory to reduce JVM object overhead and GC pressure, but it still stores data primarily in a compact row-oriented binary format like UnsafeRow. While Spark supports columnar execution in certain paths (like vectorized Parquet readers), Arrow is not its universal in-memory representation.

Sail is columnar and Arrow-native end-to-end via DataFusion — there are no row/columnar impedance mismatches inside the engine.

That architectural simplicity matters. When you don’t constantly convert between formats, execution stays cleaner and more predictable.

Pruning the Dead Weight: The Disciplined Planner

DataFusion gives Sail a highly optimized physical planner that is aggressively selective. You never carry more weight than necessary.

• Predicate Pushdown: Filters are applied at the earliest possible moment — often directly at S3 or Parquet read time. We throw the bilge water overboard before it hits the deck.
• Projection Pushdown: If your query needs two columns out of a hundred, Sail reads only those two.
• Type-Specific Kernels: Integer math uses integer kernels. Float math uses float kernels. No generic “handle-everything” overhead.

Less data touched. Fewer instructions executed. Better throughput per core.

Adaptive Scaling: No Infrastructure Wake

Because DataFusion is written in Rust, it doesn’t drag along a massive runtime environment.

In Spark, even small workloads require spinning up a full JVM ecosystem — like launching a cruise liner to cross a quiet harbor.

Sail starts in milliseconds and consumes a fraction of the memory footprint. That makes it ideal for cloud-native elasticity:

• Ephemeral Batch Jobs: Start, process, vanish.
• AI Preprocessing: Feed models with a lean, fast sidecar.
• Containerized Harmony: Fits into small K8s pods where a JVM might OOM immediately.

Native Execution: No Middleman, No Warm-up

Spark translates logic into JVM bytecode, which the JIT then optimizes while the program is already running. That’s why some jobs “warm up” before hitting stride.

With DataFusion inside Sail, you execute native Rust code from the first millisecond. There’s no interpreter, no bytecode translation layer, and no waiting for the JIT to get clever.

From the moment you weigh anchor, the engine is already running at peak efficiency.

---

🧠 Memory Safety: No Oil Changes at Sea

Spark runs on the JVM. That means no matter how much wind you’ve caught, eventually, the engine room yells: “All stop! We’re cleaning the filters!”

That’s Garbage Collection. While the crew is busy scrubbing the deck, your data processing sits dead in the water. It’s the “Stop-the-World” tax, and on a long voyage, it’s a momentum killer.

The Sail Difference: No Overboard Discarding

In Sail, there is no cleanup crew because nothing gets thrown overboard in the first place. By inheriting Rust’s ownership-based memory model, Sail provides memory safety at compile time. We aren’t just managing memory better; we’re eliminating the concept of “garbage” entirely.

Ownership vs. The Cleanup Crew

Think of classic Spark’s JVM galleon: objects fly around freely, references point everywhere, and the garbage collector occasionally halts the entire ship to tidy up. Add ~12 bytes of header per object — a quiet “luggage tax” that turns into gigabytes of heap weight at scale.

Now step onto the Sail racing yacht:

• One Hand for the Ship: Every value has exactly one owner. When ownership moves, it transfers completely. When the owner goes out of scope, the memory is freed immediately. No shared ambiguity, no background cleanup crew — just deterministic, compile-time–guaranteed safety.
• Strict Logbooks: Rust’s compiler checks lifetimes before the program ever runs, ensuring references can’t outlive the data they point to. If it might be unsafe, it simply won’t compile. Memory safety is proven before launch.
• Deterministic Release: Memory is reclaimed the instant it’s no longer needed — no heap sweeps, no stop-the-world pauses, no surprise latency spikes, and no object-header bloat. What you allocate is what you use — and when you’re done, it’s gone.

Deterministic Memory = Smooth Sailing

In the Spark engine room, memory management is a game of probability. You tune your heap fractions, cross your fingers, and hope the Garbage Collector doesn’t decide to take a “union break” right as a massive shuffle hits. When the JVM hits a memory ceiling, it often panics. Spark starts “spilling” data to disk — the digital equivalent of frantically throwing cargo into the hold just to keep the deck clear. It keeps you afloat, but it’s slow, loud, and kills your momentum.

Sail doesn’t panic because Sail knows its limits.

Because Rust handles memory deterministically, Sail doesn’t have to guess how much room is left in the hull — it calculates it to the byte. There is no “Stop-the-World” tax where the entire ship sits dead in the water while the crew scrubs the deck. Instead, memory is reclaimed the exact microsecond it’s no longer needed.

This isn’t just about being “leaner.” It’s about predictability. By eliminating the overhead of object headers and the chaos of unpredictable background sweeps, Sail removes the “latency cliffs” that haunt production pipelines. You don’t have to babysit your executors or spend your weekend tuning heap ratios to avoid a crash.

When your memory management is deterministic, your performance becomes a constant, not a variable. That is the “Calm Seas Effect”: you stop fighting the ship and start enjoying the speed.

Packing the Hull for Speed

To understand why Sail feels faster, we have to look at how a computer actually “thinks.” Your CPU is a world-class engine — but it can only move as fast as it’s fed data. Reaching into main memory (RAM) is slow. It’s like the whole love boat pausing mid-party so housekeeping can tidy up.

To avoid that, the processor keeps a tiny, ultra-fast cache — think of it as a high-speed locker right inside the engine room. The closer the data is to that locker, the faster the engine runs.

On a traditional JVM-based ship, the cargo is scattered. Data lives as row-based objects spread across the heap. So the crew has to chase pointers — running all over the vessel to grab the next piece of a single column. When the processor checks its locker and finds it empty, the engine stalls. It waits.

Sail, powered by Apache Arrow, packs the hull differently.

Instead of heavy row-objects with metadata baggage, Sail stores data in tight, columnar arrays — clean, contiguous blocks of pure values. That changes everything:

• Shoulder-to-Shoulder Data: Values from the same column sit right next to each other. Grab one, and you’ve practically grabbed the next dozen. The locker fills efficiently.
• Predictable Flow: Modern CPUs guess what you’ll need next. When memory is contiguous, the processor can prefetch data before it’s even requested.
• No Metadata Luggage: JVM objects carry headers — tiny per-object taxes that waste precious cache space. Arrow arrays are mostly raw signal, no noise.

The result isn’t just lower memory usage — it’s fewer engine stalls. By packing the hull intelligently, Sail keeps the processor at full throttle, crunching through data in a smooth, uninterrupted flow instead of waiting around for the next elevator from RAM.

Concurrency Without Mutiny

In the old-school JVM world, shared state is the primary cause of “mutiny.” When you have multiple threads trying to grab the same piece of data at once, you get data races— the digital equivalent of two sailors pulling the same rope in opposite directions until it snaps.

To prevent this, Spark relies on complex locks and “synchronized” blocks. It works, but it creates bottlenecks.

Sail uses Rust’s “Fearless Concurrency” to keep the deck clear.

Rust’s compiler acts as the ultimate Captain. It doesn’t just hope the threads behave; it mathematically proves they won’t clash before the ship even leaves the port.

• No Data Races: If two parts of the engine try to claim the same resource unsafely, the code simply won’t compile. You find the “mutiny” at the dock, not in the middle of a storm.
• Actor-Style Event Loops: Sail’s control plane uses an “actor” model. Instead of threads fighting over a single steering wheel, tasks are passed cleanly between components in an organized queue.
• No Null Pointer Exceptions: We’ve all seen the “Man Overboard” of the Java world — the NullPointerException. In Sail’s memory model, those simply don’t exist. If a value might be missing, the code is forced to handle that possibility upfront.

The Upshot: Spark trusts the runtime to handle the chaos and hopes for the best. Sail proves correctness before the first byte of data is even processed. You get a ship that doesn’t just run fast — it runs with total structural integrity.

Shuffle & Spill: The Calm Seas Effect

Now let’s talk about the shuffle — that dramatic moment in a distributed query when data has to move between nodes. In Spark, this is often where the waters get choppy. If memory pressure rises, buffers fill, and the engine starts spilling to disk. Suddenly your blazing-fast computation is waiting on storage I/O. That’s when the sea turns rough.

Sail takes a different route: a pipelined, in-memory shuffle.

Because Sail’s memory footprint is tighter and more predictable (thank you, Arrow and deterministic allocation), more shuffle data stays in RAM — the fast lane. Tasks stream data directly between operators, and when a task finishes, its memory is released immediately. The pipes stay clear. No backup at the docks.

The payoff is practical:

• Lower I/O amplification: Less disk thrashing, less waiting.
• Lower cloud bills: No massive EBS throughput just to juggle temporary spill files.
• Fewer mystery slowdowns: Your 10-minute query doesn’t randomly turn into a two-hour voyage.

Instead of fighting the storm at shuffle time, Sail keeps the waters calm — moving data efficiently without tossing half of it overboard to disk.

---

😌 Smooth Sailing: High Throughput via Tokio

If you want a ship that can host thousands of guests without mutiny, you need a world-class coordinator.

In traditional JVM systems, concurrency often means spawning more threads — like hiring a new, heavyweight sailor for every task. It works… until the deck gets crowded. Too many threads lead to context-switching overhead, memory pressure, and unpredictable latency.

Sail stays light by running on Tokio.

Tokio is Rust’s high-performance asynchronous runtime. It lets Sail handle massive concurrency — network calls, shuffle streams, task scheduling — without spawning a thread per operation.

Work-Stealing: No Core Left Idle

Tokio uses a multi-threaded, work-stealing scheduler. Instead of creating thousands of OS threads, Sail typically runs one worker thread per CPU core. Each thread maintains its own queue of async tasks. If one thread finishes early, it doesn’t nap — it “steals” tasks from a busier neighbor.

The result:

• Better CPU utilization
• Fewer idle cores
• Balanced workloads under pressure

Every core pulls its weight. No sailor stands around watching the waves.

Non-Blocking I/O: The Engine Never Waits

The biggest hidden bottleneck in distributed systems isn’t compute — it’s waiting on I/O.

• The blocking model: A thread asks S3 or disk for data… and then sits there doing nothing until it arrives.
• The Tokio model: A task sends the request and immediately yields control. The CPU moves on to useful work — joins, filters, aggregations — while the data is in flight. When the data arrives, the runtime wakes the task back up.

The engine never idles just because the shore is slow. This is especially powerful in distributed shuffle and remote storage reads, where network latency would otherwise waste CPU cycles.

Cooperative Multitasking: Choreography Over Chaos

In JVM systems, the operating system forcibly preempts threads (preemptive scheduling). That can introduce jitter and unpredictable pauses.

Tokio uses cooperative multitasking.

Tasks yield at explicit .await points — natural suspension boundaries in async Rust. Instead of the OS yanking threads mid-execution, tasks step aside voluntarily.

The result:

• Lower scheduling overhead
• More predictable tail latency
• Fewer “mystery slowdowns”

It’s choreography, not chaos.

Backpressure: Throughput Without Capsizing

High throughput is useless if memory explodes. Sail uses Tokio’s asynchronous channels to move data between components — driver, workers, shuffle streams, task runners. These channels enforce backpressure.

If a downstream stage (like shuffle or aggregation) slows down, upstream producers naturally throttle. Data flow adapts automatically.

That means:

• No unbounded memory growth
• No runaway buffers
• Stable performance under load

The system regulates itself instead of flooding the hull.

Spark Threads vs. Tokio Async — What Actually Changes?

In Spark’s traditional JVM model, concurrency is largely thread-driven. More parallelism often means more threads. Each thread carries its own stack memory, scheduling overhead, and context-switching cost. When a thread blocks on I/O, it simply waits — consuming memory while doing no useful work. As concurrency scales up, thread coordination and OS-level context switching start to eat into throughput and inflate tail latency.

Sail, powered by Tokio, takes a different path. Concurrency is task-driven, not thread-driven. Instead of spawning thousands of heavyweight threads, Sail runs a small, fixed pool of worker threads — typically one per CPU core. On top of those, Tokio schedules thousands of lightweight async tasks. When a task waits on I/O, it immediately yields control. Another task is scheduled onto the same core — no OS intervention, no extra stack allocation, no idle CPU slice.

So What Changes in Practice?

In Spark:

• Blocking ties up a thread.
• Scaling concurrency usually means increasing thread count.
• The OS frequently preempts and reschedules threads.
• Context switching becomes a hidden performance tax.

In Sail (via Tokio):

• Blocking yields control and keeps the core busy.
• Scaling concurrency means scheduling more lightweight async tasks — not more threads.
• Tasks cooperatively yield at explicit .await points.
• Task switching is cheap, predictable, and handled mostly in user space.

This isn’t just an implementation detail — it directly impacts throughput in real workloads involving network I/O, shuffle streams, and object storage.

Spark scales by multiplying threads. Sail scales by multiplying tasks — without multiplying overhead.

Spark adds more sailors to the deck. Sail teaches the same crew to move in perfect sync.

---

🐍 In-Process Python: The VIP Deck

In Spark, Python rides in a separate dinghy. When you run a Python UDF, Spark serializes the data (“pickles” it), sends it over a socket to a separate Python process, executes the logic, and then ships the results back. Even with Arrow optimizations, you’re still crossing process boundaries. That round trip — serialization, socket transfer, deserialization — is where the infamous PySpark bottleneck lives.

Using PyO3 and Maturin, Sail embeds the CPython interpreter directly inside — you read right, inside — the same process as the Rust engine. There’s no sidecar, no separate worker, no socket hop. Python isn’t being towed behind the yacht anymore; it’s on deck, working shoulder to shoulder with the engine crew.

The Zero-Copy Bridge: PyO3 + Arrow

PyO3 allows Rust to host a real CPython interpreter in-process. Maturin compiles the Rust core into a native Python extension that installs cleanly via pip. But the real magic happens in memory.

Because Sail is Arrow-native, and because Python runs in the same process, both Rust and Python can operate on the same underlying memory buffers. There’s no socket. No cross-process marshaling. No duplicated datasets.

In PySpark:

• JVM memory
• Arrow buffers
• Python worker memory

In Sail:

• One process
• One memory space
• Shared Arrow buffers

We’re not “moving” data. We’re handing Python a pointer to bytes Rust already owns.

No serialization tax. No memory doubling. No socket jitter.

Subinterpreters: Breaking the Old GIL Ceiling

Historically, Python’s Global Interpreter Lock (GIL) is a mechanism in CPython that allows only one thread to execute Python bytecode at a time within a single process. Even on a multi-core machine, multiple Python threads can’t run CPU-bound code in parallel because the GIL serializes execution to protect shared memory inside the interpreter.

Sail leans into the emerging subinterpreter model (PEP 684), where multiple isolated Python interpreters can exist inside one process.

What that means:

• Parallel Python: Multiple subinterpreters can run on separate cores.
• Isolation: A failure in one interpreter doesn’t poison the others.
• Tokio-Orchestrated Scheduling: Async tasks and Python execution coexist cleanly inside the same runtime.

Instead of spawning multiple OS-level Python worker processes (like PySpark), Sail keeps everything inside one coordinated engine room.

Sail vs. PySpark: Two Architectures

In PySpark, you’re juggling:

• JVM executors
• Py4J bridge
• Separate Python worker processes
• Socket communication
• Serialization overhead

Every boundary crossing adds latency and memory pressure. And honestly — this is exactly why PySpark never fully convinced me. It always felt like a necessary workaround to make Python coexist with a JVM-first engine — a kind of architectural compromise you accept because you want the Python ecosystem. It works, yes. But it always felt bolted on rather than built in.

In Sail, the stack is unified:

• Rust engine
• Embedded CPython
• Arrow-native memory
• Tokio-driven scheduling

Python isn’t a sidecar. It’s a first-class citizen in the engine. And I’ll be honest — I’m still slightly in awe of how cleanly both the main Python interpreter and multiple subinterpreters coexist with Rust inside the same single process. No duct tape. No awkward bridge. Just runtimes sharing memory and coordinating natively, like they were meant to sail together from day one.

Why This Actually Matters

For AI-heavy or Python-centric pipelines, the difference is dramatic.

You can:

• Run heavy preprocessing in Rust
• Hand off the same memory to Python for model inference
• Avoid data copies between stages

The result isn’t just faster UDFs — it’s architectural simplicity. Fewer moving parts. Fewer failure points. More predictable performance.

Sail doesn’t “optimize” the Python boundary. It removes the boundary.

---

✋ The Honest Tradeoffs: No Boat Is All Sunsets

I won’t pretend it’s all tropical drinks and glassy water. Moving to a high-performance vessel like Sail means adjusting how you hold the helm.

Upfront Rigor (The Captain’s Exam)

Rust’s ownership model is not the “anything goes” world of the JVM. You don’t just write code and hope the garbage collector sorts it out later. You must prove to the compiler that your memory and concurrency are safe before you ever set sail.

It’s more work at the dock — but it’s exactly why the hull doesn’t leak in production.

Retiring the Old Knobs

If you’ve grown fond of tuning spark.executor.memoryOverhead, fiddling with heap fractions, or debating GC strategies, Sail may feel… strangely quiet.

Many of those knobs simply don’t exist. There’s no heap to tune. No GC to appease. No executor memory fractions to micromanage.

For some, losing that control feels unsettling. For most, it feels like a vacation from configuration anxiety.

Specialized Cargo (The UDF Reality)

Spark will happily run almost any Java or Python UDF you throw at it — even if performance suffers quietly.

Sail supports Python beautifully, but if you want absolute peak performance for custom logic, writing it in Rust unlocks the full power of the engine. That’s not a limitation — it’s leverage — but it does mean the fastest path sometimes requires a specialist, not a general deckhand.

The Optimizer Maturity Gap

Here’s a serious one. Spark’s Catalyst optimizer has over a decade of battle testing. It combines rule-based and cost-based optimizations, sophisticated join reordering, and Adaptive Query Execution (AQE) that dynamically reshapes plans at runtime based on real statistics.

That ecosystem maturity matters — especially for extremely complex analytical workloads.

DataFusion’s optimizer is fast and improving rapidly, but it does not (yet) match the full breadth and depth of Catalyst’s cost model, adaptive behaviors, and edge-case heuristics.

If your workload leans heavily on:

• Deeply nested queries
• Complex multi-way joins
• Skew-heavy datasets
• AQE-driven reshuffling

You should benchmark carefully. This is less about raw speed — and more about planner sophistication at extreme scale.

A Smaller Fleet (Ecosystem Maturity)

Spark has been sailing for over a decade. It has connectors for nearly every obscure port and legacy system imaginable.

Sail is newer — sleek, modern, and focused. The major routes are covered (S3, Iceberg, Delta, Parquet), but some niche plugins and long-tail integrations may still be under construction.

If your stack depends on deeply entrenched legacy connectors, you’ll want to evaluate carefully.

The Evolving Manifest

Sail is built for the modern SQL + DataFrame era. If your pipeline leans heavily on old-school RDD transformations and low-level Spark internals, you may find those wooden lifeboats weren’t loaded onto this carbon-fiber yacht.

That’s a deliberate design choice — but it’s still a tradeoff.

---

⚓ Anchors Up

That’s the tour of the vessel — hull, engine room, sails, and VIP deck.

Now the real journey begins. In the next episode of this series, we’ll leave the brochure behind and head into open water — a proper head-to-head between Java + Spark and Rust + Sail. No marketing gloss. Just real workloads, real measurements, and real tradeoffs.

Two ships. Two philosophies. One ocean.

And before we set sail, a big thank you to my colleague David López González, whose contagious excitement about LakeSail helped spark this entire journey. Some voyages start with code — this one started with enthusiasm.

Anchors aweigh. The race begins now. 🚢

---

“LakeSail soon will be making another run.
LakeSail promises something for everyone.
Set a course for adventure,
Your mind on a new partner dance.
Your jobs won’t hurt anymore.
It’s an open source engine and it’s friendly sure.
It’s SAIL! Welcome aboard! It’s SAIL!”
(adapted from “Love Boat Theam” by Jack Jones)

---