Apache Spark WTF??? — It Works On My Machine
Local environments are crucial, not only on a personal level to learn, practice and quickly develop PoCs, but also in order to debug, fix…
The Case
Setting up a Spark cluster using Docker is quite straighforward. There isn’t also much fuss either in downloading a Spark distribution to our local Linux laptop and starting it executing the start-master.sh and start-worker.sh scripts included.
Whether you spin up a local cluster using Docker or downloading the Spark distribution, the problem remains, how can easily debug Spark internal code or be able to debug our own code when it’s executed on an executor node? Thus, we want to set up a local Spark cluster from our IDE (Intellij), so that we could later analyze, debug, set breakpoints, etc. throughout both, our own code and the Spark code.
DIY (= Do It Yourself)
Let’s create a simple application to run in Spark local mode on our Intellij environment following the next steps:
1] Open the “New Project” dialog panel (File > New > Project). The introduce a name for the project and select
Conclusion
Sick and tired of debugging some error or performance issue? I might help you! Send me your case! I’m genuinely passionate about intriguing errors and performance challenges — those are the things that keep me up at night and also thinking while walking my dog. I’ve been solving interesting cases for the last 15 years. so … don’ hesitate to reach out to me at https://www.linkedin.com/in/angel-alvarez-pascua.