Apache Spark WTF??? 🪳Bug Fixion🐛

🧮 Just Applying a Formula?

The brief sounded harmless: build a PySpark module that loads millions of rows from an Oracle table, applies a fixed (but, naturally, parameterizable) formula per row, and writes the result to another Oracle table.

A nap-worthy task, right? Yeah… no. A few commits later, the sweet little ETL job was gone — replaced by something darker, heavier, and just a bit mean.

• Stage 1 — Triage the universe: Pull 12M and 26M-row CSVs from an S3-ish bucket to decide what to process and which formula each row deserves.
• Stage 2 — IDs from the abyss: Mint a monotonically increasing UUID for each new row, seeded from the previous run’s max.
• Stage 3 — Surprise, Surprise … more math: Read a tiny Oracle table of formulas.
• Stage 4 — Key the chaos: Join those three sources to map keys ➜ formulas.
• Stage 5 — Welcome to heavy metal: Scan 1.7B rows from one Oracle table and 65M from another.
• Stage 6 — Not in the spec: Perform several joins and aggregations to compute sums.
• Stage 7 — The Aftermath: Write six output tables — the new max UUID, the two input CSVs, processed results plus their shiny aggregations.

Every “just one more thing” came with its own boss fight. We paused, we plotted, we whispered, “why me?” — as our tiny three-executor Kubernetes cluster begged for mercy.

Roll up your sleeves. Time to bleed a little.

🎒 Preparing for the Long Journey

To experience the suffering safely, we’ll recreate the setup — test environment, fake data, and all. Not for realism, but for scale: millions of CSV rows and something similar to 1.7 billion in Oracle.

🧪 Prepare the Test Dataset

Before we could stress-test our tiny cluster, we needed data — lots of it. Handcrafting millions of rows sounded noble, but also like a one-way ticket to madness. So instead, we reached for TPC-H dbgen, the battle-tested tool for generating scalable synthetic datasets.

We’ll use its modern Rust-based incarnation, tpchgen-cli — fast, simple, and conveniently painful to install on the first try.

🐧 On Linux

# Update and install dependencies  
sudo apt update  
sudo apt install -y build-essential pkg-config libssl-dev cargo  
  
# Install Rust if you don't have it yet  
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh  
source $HOME/.cargo/env  
  
# Verify versions (optional, but reassuring)  
rustc --version  
cargo --version  
  
# (Optional) Set a custom build directory to avoid path issues  
export CARGO_TARGET_DIR=$HOME/.cargo/tmp_build  
  
# Install tpchgen-cli  
cargo install tpchgen-cli  
  
# Generate our synthetic dataset  
tpchgen-cli -s 100 --format parquet --output-dir ./tpch_data

🪟 On Windows (via WSL). If you’re living dangerously and using Ubuntu on WSL, the process is nearly identical — just remember, you’re technically in Linux land.

# Update your package lists  
sudo apt update  
  
# Install the essentials: compiler tools, pkg-config, SSL dev libraries, and Cargo (Rust’s package manager)  
sudo apt install -y build-essential pkg-config libssl-dev cargo  
  
# Install Rust (this script handles the full setup)  
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh  
  
# Reload your environment so Cargo is available immediately  
source $HOME/.cargo/env  
  
# Verify Rust and Cargo were installed correctly  
rustc --version  
cargo --version  
  
# (Optional) Use a temp build directory to avoid permission issues inside WSL  
export CARGO_TARGET_DIR=$HOME/.cargo/tmp_build  
  
# Install tpchgen-cli from crates.io  
cargo install tpchgen-cli  
  
# Generate our synthetic dataset  
tpchgen-cli -s 100 --format parquet --output-dir ./tpch_data

And that’s it — our shiny new +38.6 GB folder of Parquet files is ready for battle.

⚙️ Prepare the Environment

With the data ready, it was time to summon the beast — a local environment that looked just enough like production to hurt. What followed was less of a setup and more of a ritual: Docker, Java, Hadoop, Oracle, S3 emulators, PyCharm tweaks, and enough environment variables to summon Cthulhu.

Below is the condensed chaos — for Windows (via WSL) or Linux users who want to recreate the crime scene.

🐳 1. Spin up Oracle in Docker. Because Spark loves JDBC — and JDBC loves pain — we start with an Oracle container:

docker run --name oracle-test -d -p 1521:1521 -e ORACLE_PASSWORD=test gvenzl/oracle-free

This gives you an Oracle 23c instance running locally, ready for all your testing (and emotional damage).

🩸 When the “E” in ETL Stands for Excruciating Exports

Reading from S3

oye, estaba contestando a uno que ha escrito un artículo en Linkedin diciendo algo que no es cierto ..

Y he caído … oye, ¿después de cargar los csvs, hacíamos repartition, no?

igual tuneando estos parámetros, nos lo podemos ahorrar

Reading from Oracle

yyyy

extraer datos particionados via jdbc no implica que Spark sea consciente de que los datos están particionados por ese campo

https://chatgpt.com/c/69fd9561-3cec-83eb-8442-2529766c2692

😖 When the ‘T’ in ETL stands for Torturous Transformations

uuid

filter

join: broadcast a 3 bandas?

applying the formula

😩 When the “L” in ETL Stands for Lamentable Loads

truncate

🔨 Nailing It In Kubernetes

---

“La-da-dee-da-da-dee-da
I’m dead serious
Let those peaks blow your mind
La-da-dee-da-da-dee-da
I’m dead serious
The Spark session starts and ends so fast”
(adapted from “Pump it” by Black Eyed Peas)