Apache Spark WTF???🔥Heiss Wie Feuer (Hot As Fire)🔥

“Flame ist der Name, weil mein Spark so heiß ist.”
“Flame is the name because my Spark is so hot.”
(adapted from “Heiss wie Feuer” by Dj Stylewarz)

---

🍡Appetizers: Spark’s Review

Before diving into the mechanics of Apache Spark scheduling, it helps to set the table with a few key concepts. At its core, Spark is a distributed data processing engine, designed to handle massive datasets by splitting work across many nodes in a cluster. But — and this is important — Spark doesn’t automatically parallelize everything you do. Actions in Spark, such as collect(), count(), or save(), trigger computation, but transformations like map or filter are lazy. They build up a logical plan but don’t run until an action forces execution.

This approach has several advantages:

1. Optimized execution plans: By waiting until an action is called, Spark can analyze the entire logical plan and optimize it. This includes reordering operations, combining them where possible, and minimizing data shuffles. Essentially, Spark tries to plan the “best route” through your data before actually moving a single byte.
2. Resource efficiency: Lazy evaluation avoids unnecessary computation. If you define multiple transformations but only perform one action, Spark only computes what’s needed for that action. This can save significant CPU and memory, especially on large clusters.

However, there are trade-offs:

1. Non-intuitive performance behavior: Because Spark delays computation until an action, it’s possible to define a huge chain of transformations and not notice inefficiencies until the action runs. Long chains of transformations can also lead to complex execution plans that are harder to debug.
2. Job scheduling overhead: Spark’s scheduler must decide how to split jobs into stages and tasks, and how to allocate them across available executors. While the scheduler is powerful, it can introduce latency, particularly for small jobs or when resources are constrained.

Spark divides work into stages and tasks. Each stage represents a set of transformations that can be executed without shuffling data, while tasks are the smallest unit of work executed on partitions. Spark schedules jobs stage by stage, respecting dependencies between stages. This design ensures correctness and allows Spark to recover from failures efficiently, but it also means that some parts of a job cannot run in parallel until upstream stages complete.

Once Spark has broken jobs into stages and tasks, it needs to decide which job runs first when multiple jobs are submitted concurrently. This is where the Job Pool Scheduler comes into play. Its role is to manage multiple active jobs by assigning them to job pools, controlling their order of execution, and ensuring fair or orderly sharing of cluster resources.

---

🍽️Main Course: The Choice is Yours

⏱️The Role of the Scheduler

At the heart of Spark’s execution model lies the scheduler. Its mission is simple to state but tricky to pull off: decide what runs where and when across the cluster.

When you submit a Spark job, it doesn’t run as one giant block of code. Spark breaks it down into stages, and each stage into many smaller tasks. These tasks can run in parallel, but only if the cluster’s resources are managed wisely. That’s where the scheduler steps in. In short, the scheduler doesn’t crunch the data — it decides who gets to crunch it, where, and in what order.

Here’s what the scheduler does:

• Task allocation: It maps tasks to available executors, aiming to keep all resources busy.
• Data locality: It tries to run tasks as close as possible to the data they need, reducing network overhead.
• Fairness vs. priority: Depending on configuration, it either processes jobs in strict order (FIFO) or spreads resources more evenly (Fair).
• Cluster utilization: It ensures no CPU core is left twiddling its thumbs while tasks are waiting.

Spark supports two main types of job scheduling: FIFO (First In, First Out) and FAIR scheduling. The choice between these two type affects more than just which job finishes first — it also determines how long jobs sit in the queue and how responsive the cluster feels when multiple users submit workloads or when a single application triggers several jobs in parallel.

FIFO vs FAIR

🥩FIFO Scheduling

FIFO (First-In-First-Out) is Spark’s default scheduling mode. Jobs are queued in the order they’re submitted, and the scheduler gives priority to the one at the front of the line. That job consumes available resources — executors and cores — until it either:

1. finishes execution, or
2. hits a stage boundary where tasks can’t proceed yet (for example, waiting on a shuffle).

Only then does the scheduler allocate resources to the next job in the queue.

In practice, this means if Job A comes first with 1,000 tasks, and Job B arrives just after with 10 tasks, Job B waits. Even though Job B could finish in seconds, it’s stuck until Job A has progressed enough to release resources.

👍 Pros

• Simple & predictable: You always know jobs will run in the order they arrive. No surprises.
• Minimal scheduler overhead: Since Spark doesn’t have to juggle priorities or rebalance resources, there’s less bookkeeping.

👎 Cons

• Starvation risk for small jobs: Short, interactive jobs may wait behind a long-running job, making the cluster feel unresponsive.
• Unfair resource allocation: One heavy job can hog the cluster, even if lightweight jobs could slip through quickly.
• Not multi-user friendly: In a shared cluster, users submitting small ad-hoc queries may get frustrated if a massive batch job is blocking them.

When to use it
FIFO works well in single-user pipelines where jobs are batched intentionally and order matters, or in lightly loaded clusters where contention is rare. But in multi-tenant or interactive environments, FIFO quickly shows its limits — it’s like standing in line behind someone with a cart overflowing with groceries when all you want to do is pay for a single bottle of water.

🐟FAIR Scheduling

Fair scheduling is designed for multi-user or multi-job environments where responsiveness matters. Instead of letting the first job monopolize resources, Spark tries to share the cluster capacity among all active jobs.

Here’s the flow:

1. Each job is placed into a pool (by default, all jobs go into a single pool unless you configure multiple).
2. The scheduler continuously divides available executors and cores among jobs in proportion to their pool weights.
3. When a new job arrives, Spark rebalances resources dynamically so that the newcomer gets a fair slice, rather than waiting until others complete.
4. If one job has no runnable tasks (e.g., waiting on a shuffle), its share is temporarily given to other jobs, improving utilization.

Example:
 Imagine 10 cores and two jobs:

• Job A with 100 tasks.
• Job B with 20 tasks.

With fair scheduling:

• Initially, each job gets ~5 cores. Both run simultaneously.
• If Job B finishes early, its freed cores are reassigned to Job A automatically.
• If a third job (Job C) appears, Spark re-slices the pie so each job gets ~3–4 cores.

This constant reallocation makes the cluster feel much more responsive than FIFO.

👍 Pros

• Improved cluster utilization: Idle resources are quickly reassigned to jobs that can use them.
• Responsive for small jobs: Short queries aren’t stuck waiting behind big batch jobs.
• Multi-tenant friendly: Fairness across users and jobs makes shared clusters practical.
• Dynamic balancing: Resources are continuously redistributed as jobs progress.

👎 Cons

• Configuration overhead: To get the most out of it, you often define pools with priorities and weights. This adds operational complexity.
• Job runtime variability: Because resources are reallocated on the fly, job completion time can fluctuate depending on cluster load.
• Not always optimal for single-job pipelines: If you only ever run one job at a time, fair scheduling doesn’t add value.

When to use it
 Fair scheduling is especially effective in shared Spark clusters, where multiple users compete for resources and workloads range from large batch ETL to small, interactive queries or streaming jobs. By balancing allocation, it prevents starvation and ensures quick turnaround for lightweight tasks, even when heavy jobs are in progress. The trade-off is that while no job is left waiting indefinitely, overall progress may slow slightly — exchanging simplicity for flexibility and responsiveness.

---

🍰Desserts

Alright, everything’s cooking nicely so far, but… could a different scheduling policy in Spark make our jobs run faster? Let’s find out!

To test this, we’ll whip up a simple test: we’ll launch three jobs of different durations. First, we’ll run them one by one (no threading, just like waiting for each dish to be served). Then, we’ll add some multithreading spice and test both FIFO and FAIR scheduling to see who wins the race to the dessert table.

⚠️Words of Caution

• We can tell Spark what jobs to run, but we don’t control when they start or finish — so timing each one isn’t straightforward. In Scala/Java, we can attach listeners to track job events, but in PySpark that’s not possible because our code runs in a separate Python process, while Spark’s scheduler and event bus live inside the JVM. Since Python objects can’t subscribe to those JVM events, PySpark doesn’t support listeners — we’ll need to rely on the status tracker, REST API, or event logs to get job info after execution.
• sc.setJobDescription() is used to label jobs so we can easily identify them in the Spark UI (super handy when debugging or profiling performance). However, it’s not thread-safe — if multiple threads call it at the same time, their descriptions can clash or overwrite each other. So when running parallel jobs, we can’t rely on it to tag each one correctly. The safe approach is to keep our own job descriptions and timings in Python instead.
• Once a SparkSession is created, the value of spark.scheduler.modecan’t be changed on the fly. This is because the scheduler backend — which manages how jobs are queued and assigned to executors — is initialized during the session startup. By the time our code is running, Spark has already built the internal scheduling structures (like pools and listeners) based on that initial configuration. Changing the mode afterward would require rebuilding those components, which Spark doesn’t support dynamically. So, if we want to test different scheduling policies, we’ll need to create separate Spark sessions (or contexts), each with its own spark.scheduler.mode setting.

🍳 Spark Scheduling Test Recipe

Ready to whip up some Spark magic? Follow this tasty recipe to test how different scheduling modes (FIFO, FAIR, and DEFAULT) handle multiple jobs running in parallel. Let’s get cooking! 👨‍🍳✨

1. Gather your ingredients. Import all the essentials: pyspark.sql, concurrent.futures, datetime, and pytest.
2. Mix your base. Create a SparkSession with the desired scheduling mode — FIFO, FAIR, or the DEFAULT single-threaded baseline.
3. Preheat the kitchen. Spark needs a warm-up! Run a tiny job to initialize the environment before the real cooking begins.
4. Prepare your recipe function. Define a helper function that sets the scheduler pool, cooks up an RDD, counts it, and notes the start and finish times.
5. Set up your kitchen team. Create a ThreadPoolExecutor — your crew of cooks — ready to handle multiple dishes (jobs) in parallel.
6. Start cooking the main course. Launch three jobs — high, medium, and low workload — and let them simmer in the thread pool.
7. Check the timers. Collect each job’s start, end, and duration to see who finished their dish first.
8. Clean up the kitchen. Once the feast is done, stop the Spark session and admire your scheduling results.

Spark Scheduling Test Recipe

The repository heiss-wie-feuer contains the test setup, which includes the test_scheduler_modes.py pytest file and the fairscheduler.xml configuration file used to define the fair scheduler behavior.

The configuration used in the tests run on mylaptop with 16 cores defines three FAIR scheduling pools: low, medium, and high.

• Weights (1, 2, 3): When all pools are active, Spark allocates slots roughly in proportion to these weights — about 3 for low, 5 for medium, and 8 for high (total ≈16). The high pool gets more resources but doesn’t monopolize them.
• minShare (1, 2, 4): Ensures each pool always has a minimum number of slots available when it has pending tasks, preventing starvation.

Because the machine has enough cores, all tasks could often run simultaneously. This means FAIR and FIFO showed similar total runtimes, and the weighted allocation only becomes visible when demand exceeds available cores.

🔍 Analysis

DEFAULT (num_workers=1). With only one worker, jobs run sequentially — each starts after the previous one finishes.

• There’s no concurrency and no resource contention.
• Total runtime is simply the sum of all job durations (~95.6s).
• Useful as a baseline for comparison.

FIFO (num_workers=3). With three worker threads, all jobs are submitted concurrently.

• The scheduler processes them in submission order: the largest job gets assigned tasks first.
• However, in this test scenario, there’s no resource contention — all tasks can fit into available slots.
• Because of that, FIFO and FAIR end up finishing at almost the same total time (~98s).
• Smaller jobs complete earlier while the largest job continues, but the “queue prioritization” of FIFO doesn’t significantly impact performance since there’s enough capacity for all.

FAIR (num_workers=3). FAIR scheduling aims to distribute resources evenly among active jobs.

• In this setup, its advantages are not fully visible because resources are not limited — all jobs can run simultaneously without competition.
• The slight improvement in total time (~88s) is likely due to execution timing and internal scheduling overheads rather than true fairness effects.
• The difference between FAIR and FIFO only becomes clear when:
• Active jobs demand more slots than are available, or
• Weights and minShare settings restrict how many slots each pool can use.

Spark Job Timelines

⚙️ On Multithreading Variability

Using Python’s ThreadPoolExecutor to submit jobs in parallel introduces variability because:

• Thread scheduling by the OS is non-deterministic — submission timing may vary by milliseconds.
• Spark picks up jobs asynchronously, so slight differences in submission time can change task allocation order.
• Executor availability, JVM warm-up, and GC cycles can also affect when tasks start.
• As a result, measured durations can fluctuate slightly across runs, even with identical configurations.

🧠 Key Takeaways

• When all tasks fit in the cluster’s available slots, FAIR and FIFO behave almost identically.
• The benefit of FAIR appears only under resource contention or custom pool configurations.
• Multithreading adds minor timing variability, so results should be interpreted as approximate trends, not absolute values.

---

🥂 Cheers!!! (Alles klar?)

I don’t want you to leave with an empty stomach (or a mixed feeling) about Spark’s scheduling strategies. Maybe you were expecting jaw-dropping performance gains from FAIR scheduling — but here’s the truth: FAIR is called FAIR for a reason. Its purpose is to distribute resources evenly among competing jobs, ensuring no single one hogs all the slots.

When multiple users or applications are submitting jobs simultaneously, FAIR truly shines — it keeps things balanced and responsive. But when it’s just you running your own jobs, you’re essentially competing against yourself, so fairness doesn’t bring much extra benefit.

The same logic applies to using multithreading: it can help organize and overlap work, but won’t magically make everything faster. Still, it can improve performance in specific cases — like writing to different sinks, running I/O-bound tasks, or coordinating lightweight operations in parallel.

So, while FAIR scheduling won’t transform your single-user workloads, it’s a solid ingredient for multi-tenant clusters. Think of it as good kitchen etiquette — everyone gets a fair share of the stove 🔥, and no dish gets burnt waiting in the corner.

👋Greetings

I’d like to thank Pere Martí Agut, Ramón del Rey García, and all my fellow diners at the Spark (Fabric-flavored) table for the project that inspired this article.

---

“Hier kommt die Fackel und die Flame”
“Here comes the torch and the flame”
(from “Heiss wie Feuer” by Dj Stylewarz)