Apache Spark WTF??? — Who Moved My Class?

Welcome back to uncovering some of Spark’s darkest and lesser-known secrets. In this episode, we’ll delve deep into a concerning issue regarding a particularly mischievous broadcast join. We aim to understand how broadcast joins trully function and, more precisely, how Spark’s autobroadcast performs its (black) magic. We’ll explore potential pitfalls and challenges that may arise, and most importantly, why they occur.

So, lower the blinds, dim the lights, and grip your wand tightly, as we depart from the Leaky Driver tavern and venture into the uncharted, forbidden forest of Broadcast Joins. Here, before you can (broad)cast any spell, you might face totally unexpected attacks and be torn to pieces by despicable and remorseless creatures known as… OutOfMemory demons!”

(Soundtrack for this article: I Put a Spell On You)

Photo by Maksym Kaharlytskyi on Unsplash