Apache Spark WTF??? 🚀 Battle Java-Rust Exclusive 💥

“Armed with honesty, Spark in its majesty,
Sail moves elegantly, challenging its enemy”
(adapted from “Battle Axe Exclusive” by Swollen Members)

---

⛵ The Rust + Sail Arsenal

As we saw in the previous article, the LakeSail + Rust combination isn’t just a new engine — it’s a completely different class of warship. While Apache Spark grew around the JVM ecosystem, Sail was designed from the keel up for modern data workloads. The result is a vessel optimized for raw speed, efficient memory handling, and extremely high throughput. Before we send it into battle, it’s worth reviewing the ammunition that makes this ship so dangerous.

Native Rust Execution

At the heart of Sail is Rust — a systems programming language built for performance and safety. Unlike JVM-based systems, Sail executes fully compiled native code from the first millisecond. There is no bytecode interpretation and no warm-up period waiting for a JIT compiler to optimize hotspots. What you get is predictable performance and extremely low overhead, which is exactly what you want when launching large distributed workloads.

Deterministic Memory (No Garbage Collection)

Rust eliminates one of the JVM’s most infamous sources of turbulence: garbage collection. Through Rust’s ownership and borrowing model, memory is managed deterministically at compile time. Data is freed immediately when it goes out of scope, meaning Sail avoids the stop-the-world pauses and unpredictable latency spikes that can appear during heavy Spark workloads.

Arrow-Native Columnar Memory

Sail is built directly on Apache Arrow, which acts as its universal in-memory format. Instead of juggling multiple internal representations, Sail keeps data in contiguous columnar arrays from ingestion to execution. This layout dramatically improves CPU cache locality and allows vectorized operations to process large batches of values at once.

DataFusion Query Engine

Under Sail’s deck sits Apache DataFusion, a high-performance Rust query engine. DataFusion compiles logical plans into efficient physical execution pipelines that aggressively apply projection pushdown, predicate pushdown, and vectorized kernels. The result is a lean execution plan that touches less data and executes fewer instructions per operation.

Massive Concurrency with Tokio

For coordination and task scheduling, Sail relies on Tokio, Rust’s high-performance asynchronous runtime. Instead of spawning thousands of heavyweight threads, Tokio schedules lightweight async tasks across a small set of worker threads. This allows Sail to handle massive concurrency — network reads, shuffle streams, task orchestration — while keeping CPU cores fully utilized and overhead minimal.

In-Process Python

One of Sail’s most intriguing weapons is its Python integration. Using PyO3, Sail embeds a full CPython interpreter directly inside the engine. Combined with Arrow memory buffers, Python code can operate on the same data in the same process as the Rust engine. No sockets. No serialization round-trips. No separate worker processes.

Zero-Copy Data Flow

Because Sail’s entire pipeline is Arrow-native and runs within a unified runtime, data can move through the system without copying or converting formats. Operators consume and produce Arrow record batches directly, meaning the engine spends far less time shuffling bytes and far more time executing actual computation.

Lean Distributed Execution

Finally, Sail’s lightweight architecture means workers start quickly and consume far less memory than traditional JVM executors. This makes the engine particularly well suited for cloud-native environments where rapid scaling and efficient resource usage matter.

Taken together, these capabilities form the core arsenal of the Rust + Sail fleet: native execution, deterministic memory, columnar vectorization, async concurrency, and zero-copy data movement. On paper, it’s an extremely formidable ship. But naval battles aren’t won on paper. Now it’s time to see how this arsenal performs in open water …

---

☕ Refitting the Battleship: The Modern Spark Arsenal

Before we assume the Rust cruiser sails uncontested, we need to acknowledge something important: the Apache Spark flagship hasn’t been sitting idle in port. The JVM fleet has been quietly refitted with modern engines, lighter armor, and much smarter fire-control systems. With the latest generation of Java, GraalVM, and the low-latency Z Garbage Collector, the old carrier is sailing with a much more advanced propulsion system than the one many of us remember.

JVM Performance Has Come a Long Way

Modern Java is a very different beast from the JVM many people benchmarked against years ago. The runtime has seen major improvements in JIT compilation, vectorization, and memory management. Hot code paths are aggressively optimized at runtime, allowing Spark workloads to reach extremely high throughput once execution stabilizes.

In long-running analytical jobs — exactly the kind of workloads Spark was designed for — the JVM’s adaptive optimization can produce surprisingly efficient machine code.

ZGC: The Quiet Engine Room

One of the biggest historical criticisms of JVM systems has always been garbage collection pauses. The Z Garbage Collector changes that story significantly.

ZGC is designed for extremely low latency, performing most of its work concurrently while the application continues running. Pause times are typically measured in microseconds rather than seconds, even on very large heaps. For large distributed data workloads, this means far fewer of the dreaded “stop-the-world” moments that previously caused unpredictable performance cliffs.

GraalVM: A Smarter Compiler

The introduction of GraalVM adds another powerful weapon to the JVM stack. Graal provides an advanced optimizing compiler capable of generating highly efficient machine code and performing deeper optimizations than traditional HotSpot compilers.

In compute-heavy workloads, Graal’s improvements in inlining, escape analysis, and vectorization can significantly increase throughput — particularly in tight execution loops like those found in query engines.

Spark’s Mature Execution Engine

Beyond the JVM itself, Spark brings something Sail is still developing: a decade of optimization work in its execution engine. The Catalyst optimizer, vectorized readers, whole-stage code generation, and adaptive query execution all contribute to efficient physical plans.

These optimizations allow Spark to dynamically reshape execution strategies during runtime — rebalancing joins, mitigating skew, and adjusting shuffle strategies based on real statistics.

A Battle-Tested Ecosystem

Finally, there’s the ecosystem advantage. Spark has spent more than a decade integrating with nearly every storage system, data format, and distributed environment imaginable. That maturity often translates into stability and predictable performance under real-world workloads.

All of this means the Java battleship is far from obsolete. With modern JVM technology under the hood, Spark is sailing with a much more advanced engine room than many people assume. Which makes the upcoming benchmark far more interesting.

Because this isn’t just Rust vs the JVM anymore. It’s a duel between a purpose-built modern cruiser and a heavily upgraded aircraft carrier that’s been ruling these waters for years.

---

🤔 The Benchmark Battlefield?

Before firing the first salvo between Spark and LakeSail, we need to talk about the battlefield itself: benchmarks. Because in distributed systems, the benchmark you choose often determines the story you can tell.

The Sail Benchmark: Calm Waters or Cherry-Picked Seas?

The official Sail benchmark results rely on a derived TPC-H benchmark comparing Sail against Spark on a 100 GB dataset (scale factor 100), where Sail reportedly completes the workload roughly 4× faster while using less memory and avoiding disk spill.

Sounds impressive — but there are a few reasons to look at these results with a slightly raised eyebrow.

Why TPC-H and Not TPC-DS?

TPC-H is a classic analytical benchmark consisting of 22 SQL queries over 8 tables, designed to simulate decision-support workloads. But modern distributed engines are far more often evaluated with TPC-DS, which contains:

• 99 queries
• More complex joins and nested subqueries
• Skewed data patterns
• More realistic data warehouse workloads

Choosing TPC-H isn’t wrong — but it’s the simpler test. Engines with strong vectorized execution often shine there. So if you want to demonstrate raw execution speed, TPC-H is the easier ocean to dominate.

The Curious Word: “Derived”

Another interesting detail in Sail’s benchmark description is the wording: “derived TPC-H benchmark.”

I didn’t realize at the beginning, but … that phrase matters.

TPC benchmarks are extremely strict about reproducibility. When a benchmark is labeled “derived”, it usually means:

• the queries were modified,
• the execution setup diverges from the official spec,
• or the methodology is not compliant with the official TPC rules.

None of that automatically invalidates the result — but it does mean the benchmark cannot be considered an official TPC result.

In benchmark land, that distinction is important.

Missing Details: The Fog of War

Another issue is the lack of environment transparency. The published benchmark description includes:

• hardware specs
• dataset size
• storage configuration

But several crucial pieces are not mentioned:

• Spark version
• Java version
• JVM configuration
• GC settings
• whether AQE is enabled
• whether vectorized readers are enabled
• whether off-heap memory is enabled

Meanwhile Sail is presented with its modern stack (Rust, Arrow, DataFusion, etc.).

Without those details, it’s hard to know if the comparison is:

• Rust+Sail vs modern Java+Spark
• or Rust+Sail vs an older default Java+Spark configuration

In performance benchmarking, those differences can easily swing results by 2–5×.

What Feels … a Bit Fishy?

None of this means the results are wrong. Sail’s architecture really can be extremely fast. But several things make the benchmark more of a marketing signal than a final verdict:

• Simpler benchmark (TPC-H) rather than TPC-DS
• Derived workload rather than official spec
• Incomplete environment disclosure
• Single-node test scenario

The Sail benchmark is a great demo. But it’s not the kind of evidence that settles a performance war. Which is exactly why we need a more neutral battlefield.

---

🫡 The Benchmark We’ll Use: LakeBench

To test both aircraft carriers under the same conditions, we’ll use LakeBench, an open-source benchmarking framework developed by Microsoft.

LakeBench is interesting because it’s designed to benchmark modern data-lake workloads, not just isolated SQL queries.

It includes several workload families:

• TPC-H analytical queries
• TPC-DS decision-support workloads
• ELTBench pipelines simulating real data engineering tasks
• incremental merges and maintenance operations

That means it can test more than just query execution speed. It can test the entire lifecycle of data engineering workloads.

What Makes LakeBench Good

Several aspects make LakeBench attractive for this experiment:

1. End-to-End Workloads. Instead of just SELECT queries, LakeBench can simulate data ingestion, transformations, merges, maintenance operations, analytical queries. That’s much closer to what real pipelines look like.
2. Consistent Benchmark Harness. LakeBench runs multiple engines through the same benchmark framework, which reduces the risk of inconsistent dataset generation, different query definitions, different measurement logic. That consistency is extremely valuable when comparing engines.
3. Built-In Telemetry. LakeBench captures useful metadata like compute size, execution duration, estimated cost, cluster information. That allows us to compare not just speed but efficiency and resource usage.

But It’s Not Perfect Either

No benchmark is.

LakeBench still has a few limitations worth acknowledging:

• Synthetic workloads. TPC benchmarks are still synthetic models of workloads, not real production traces. Research has repeatedly shown that standard benchmarks like TPC-H and TPC-DS don’t fully reflect real query patterns or data distributions. (arXiv)
• Configuration sensitivity. Distributed engines are extremely sensitive to cluster configuration, partitioning, caching, storage layout. Small changes can dramatically affect results.
• Benchmark ≠ Reality. Even the best benchmark cannot fully capture messy real data, skewed datasets, broken schemas, unpredictable workloads. Real pipelines are often far more chaotic than any benchmark.

---

---

“We know the future without the use of the tarot card”
(adapted from “Battle Axe Exclusive” by Swollen Members)