Apache Spark WTF??? · 5 June 2026

Apache Spark WTF??? 🙏 Losing My Precision 🧮

Oh life, it’s bigger. It’s bigger than DoubleType.

20 min read

Apache SparkSpark 4PySpark 3.5Performance

0% read·20 min in this view

“That’s me in the corner
That’s me in the spotlight, losing my precision”
(adapted from “Losing my religion” by R.E.M.)

😇 The Holy Trinity of Spark Numbers: Float, Double, and Decimal

Before we accuse Apache Spark of skimming from the collection plate, we need to understand its two numeric faiths: Approximate binary floating point and fixed-precision decimal arithmetic.

FloatType— The Candle. A 4-byte, single-precision floating-point number. Fast, compact, and good enough for ML features, sensor readings, or scientific-ish data where tiny errors are acceptable.

DoubleType— The Flashlight. An 8-byte, double-precision floating-point number. The usual analytics workhorse: faster than decimals, more precise than floats, but still approximate.

DecimalType— The Contract. A fixed-precision decimal number backed by java.math.BigDecimal. This is the one you want for invoices, taxes, balances, and anything that might make Finance send you a calendar invite titled “quick question.”

The Base-2 Confession: Why Floats and Doubles Stray

Computers think in binary. Humans bill in decimal.

That mismatch matters because numbers like 0.1 and 0.2 are simple in base 10, but become repeating fractions in base 2. So floating-point types store approximations.

FloatType tries its best and may give you: 0.300000011920929.

DoubleType brings more precision — 53 bits of it — but can still betray you with: 0.30000000000000004.

⛪ Welcome to the 38-Digit Cathedral

DecimalType protects you from binary floating-point sins, but it is not an infinite void. It has walls, rules, and a maximum occupancy sign at the door: 38 digits.

In Spark, decimals are defined with a strict two-part contract:

DECIMAL(precision, scale)

Precision is the total number of digits allowed, counting both sides of the decimal point.
Scale is the number of digits reserved to the right of the decimal point.

That leaves the whole-number side with:

integer digits = precision - scale

So allocation matters:

DECIMAL(10, 2) gives you 8 digits before the dot and 2 after it.
12345678.90 fits.
123456789.00 commits structural heresy: it needs 9 integer digits.
DECIMAL(10, 2) gives you 8 digits before the dot and 2 after it.
12345678.90 fits.
123456789.00 commits structural heresy: it needs 9 integer digits.
DECIMAL(38, 18) gives you 20 digits before the dot and 18 after it. That is the cathedral’s largest legal blueprint.

🙏 Spiritual Warning: if a database column is an ID, do not store it as a decimal just because it contains digits. Use a string when the value is an identifier, or a long/integer type when it is truly numeric and fits.

The Default Parameter Trap

If you create a decimal type in PySpark without explicit parameters, you may stumble into a silent structural trap:

from pyspark.sql.types import DecimalType  
default_decimal = DecimalType()

Your soul expects a flexible, money-friendly decimal.

Spark gives you: DECIMAL(10, 0). That means 10 total digits and zero digits after the decimal point. No cents. No fractions. No mercy.

So if a value like 12.34 is forced into that schema, Spark has to fit it into scale 0, and your cents are rounded away.

Always declare financial schemas explicitly:

DecimalType(18, 2)

or whatever precision and scale your domain actually requires.

Auditing the Lineage

When auditing decimal pipelines, do not trust only the displayed value. A console output like 246.900000 may look clean, but the type underneath may have quietly changed.

Audit the type lineage with typeof() or .printSchema():

SELECT  
  typeof(CAST('123.45' AS DECIMAL(10,2))) AS raw_type,  
  typeof(  
    CAST('123.45' AS DECIMAL(10,2))  
    * CAST('2.0000' AS DECIMAL(10,4))  
  ) AS multiplied_type;

The value may look innocent: 246.900000.
But the result type has expanded to: DECIMAL(21,6).

Trust the schema. Audit the lineage. Fear the implicit cast.

🤓 When Spark Does Math: The Secret Decimal Liturgy

When decimals enter arithmetic, Spark doesn’t just compute the numbers — it dynamically negotiates a completely new data type for the output.

In Spark SQL, typeof() is your holy water. If you aren’t checking the mutating types of your intermediate steps, you’re flying blind.

Here is how Spark handles operations across the decimal registry:

1. Addition & Subtraction: Carrying the Weight

For addition and subtraction, Spark needs enough room to accommodate the largest integer part, the largest fractional part, and one extra digit for a potential carry.

🪤 The Trap: Adding 999.99 (DECIMAL(5,2)) and 0.01 (DECIMAL(3,2)) yields 1000.00. That tiny decimal opens a portal, forcing Spark to automatically scale the result type up to a DECIMAL(6,2) to avoid overflow. Even simple math demands a bigger boat.

2. Multiplication: The Scale Explosion

Multiplication is where things get volatile. When you multiply two decimals, both precision and scale stack up rapidly:

Scale_result = Scale_1 + Scale_2

Precision_result = Precision_1 + Precision_2 + 1

If you multiply DECIMAL(10,2) by DECIMAL(10,4), the result naturally asks for a scale of 6. But if you multiply two high-precision monsters like DECIMAL(38,18), the theoretical math demands 77 digits. Spark politely points to the max occupancy sign (38 Digits) and forces that monster into a tight 38-digit sleeping bag.

3. Division: The Infinite Sermon

Division is mathematically messy because many simple fractions don’t end (1/3=0.3333…). Spark cuts off the infinite scroll by enforcing a predictable, finite precision and scale formula so your cluster doesn’t run a loop into eternity.

4. Remainder & Modulus: The Micro-Negotiations

Even remainder (%) and positive modulus (pmod) require rigorous type evaluations. Once decimals are involved, every single operator triggers an under-the-hood schema negotiation.

🔪 The Scale Sacrifice: Why Spark Truncates Your Fractions

You chose DECIMAL(38,18) and felt spiritually safe. Then you multiplied two values, and Spark returned fewer decimal places. Before accusing the engine of stealing pennies, remember: Spark sacrificed scale to protect the integer soul of your data.

The 38-Digit Studio Apartment

The fundamental engineering conflict stems from a hard physical constraint:

Maximum Precision ≤ 38

When you execute mathematical operations — especially multiplication or division — the required exact precision expands dramatically. Consider multiplying two maximum-width columns:

DECIMAL(38, 18) x DECIMAL(38, 18)

Each number allocates 20 digits before the decimal point and 18 digits after it. Mathematically, the exact product can require up to 77 total digits of precision. Spark does not have an unbounded database layer to store a DECIMAL(77, 36). It must cram that calculation into a 38-digit studio flat.

The Fractional Sacrifice: `allowPrecisionLoss`

When an intermediate operation breaches the 38-digit maximum, Spark runs a protective triage algorithm governed by a crucial configuration:

spark.sql.decimalOperations.allowPrecisionLoss  # Defaults to true

true (The Default): Spark prioritizes the whole integer body over the fraction. It systematically truncates and throws away trailing decimal candles (scale) to ensure the whole number portion doesn’t overflow or drop dead into a NULL. This invisible truncation is where most “Spark stole my pennies!” tickets come from.
false: Spark refuses to compromise. If an exact mathematical representation is structurally impossible within 38 digits, Spark rejects the calculation and returns NULL (or fails outright under ANSI mode).

Integer Digits Get the VIP Seats

When a calculation forces Spark to dump digits to fit inside the 38-character ceiling, it runs a strict hierarchy of significance. It will always protect the integer part over the fractional part. Losing fractional precision changes the accuracy of a number; losing integer digits changes its entire magnitude:

Truncating Scale: 1234567890.123456➔1234567890.1234 (An annoying accounting variance).
Truncating Integers: 1234567890.123456➔4567890.123456 (A catastrophic financial systems failure).

In Spark SQL, integer digits get the front pews. If an operation asks for DECIMAL(48,20), Spark preserves the 28 integer digits first, then squeezes the scale into the 10 remaining slots, producing DECIMAL(38,10).

The Dangerous Anti-Pattern: `DECIMAL(38,18)` Everywhere

A common data-team reflex is to define every decimal as DECIMAL(38,18) “just to be safe.”

Ironically, this can make scale loss more likely.

Why? Because every value enters arithmetic already occupying the whole 38-digit cathedral. The moment you multiply, divide, or aggregate, Spark has almost no structural headroom left, so it may reduce scale to keep the result legal.

Use schemas that match the data instead:

from pyspark.sql.types import StructType, StructField, DecimalType, StringType  
  
schema = StructType([  
    StructField("invoice_id", StringType(), True),  
    StructField("invoice_amount", DecimalType(18, 2), True),  
    StructField("tax_rate", DecimalType(5, 4), True),  
    StructField("exchange_rate", DecimalType(18, 8), True),  
    StructField("calculated_yield", DecimalType(28, 8), True),  
])

Tighter raw schemas leave Spark room to breathe during intermediate math.

Do not maximize every decimal. Engineer the headroom.

📏 ANSI Mode: The Strict Nun with a Ruler

There are two kinds of pipelines: one quietly turns bad casts, overflows, and corrupted decimals into NULL; the other slams a ruler on the desk and says, “Absolutely not.”

That second pipeline is ANSI mode: silent data corruption, upgraded to a loud stack trace.

The Main Switch: `spark.sql.ansi.enabled`

This configuration changes Spark’s entire moral philosophy:

false means legacy permissive mode. Spark forgives bad casts, numeric overflows, and corrupted decimals by quietly returning NULL.
true means ANSI mode. Spark sees the same nonsense and says: “I saw what you did. We need to talk.” Then it throws a runtime exception.

⚖️ The Spark 4.0 Reckoning: in Spark 3.x, ANSI mode was off by default. In Spark 4.0, it is on by default. Legacy pipelines that once passed silently may now fail loudly during migration.

This is not Spark being dramatic. This is Spark finally developing boundaries.

Ingestion Triage: `try_cast`

You want ANSI strictness, but you do not want one malformed row to crash the whole sermon. Enter try_cast.

It applies the cast, but when a value is malformed or overflows, it returns NULL for that cell instead of throwing.

parsed_df = raw_df.selectExpr(  
    "raw_amount",  
    "try_cast(raw_amount AS DECIMAL(5,2)) AS clean_amount"  
)  
quarantine_df = parsed_df.where(  
    "raw_amount IS NOT NULL AND clean_amount IS NULL"  
)

That gives you the best of both worlds: strict parsing, surviving executors, and a quarantine table for the sinners.

Table Writes vs. Queries: `storeAssignmentPolicy`

CAST controls conversions inside your query. Table writes have their own bouncer: spark.sql.storeAssignmentPolicy.

It supports three modes: LEGACY, ANSI, and STRICT.

Since Spark 3.0, the default is ANSI. That means Spark validates implicit type conversions when data crosses the table boundary, rejecting unreasonable assignments before dirty values land in storage.

So if your upstream metric overflows the target column, this setting decides whether Spark lets it slide, rejects it, or becomes extremely strict about possible precision loss.

Queries are one courtroom. Table writes are another.

The Configuration Matrix: ANSI vs. `allowPrecisionLoss`

Do not confuse these two knobs. They guard different gates.

spark.sql.ansi.enabled controls runtime validation. Bad casts, numeric overflows, and invalid operations either fail loudly or sneak through as permissive NULLs, depending on the mode.
spark.sql.decimalOperations.allowPrecisionLoss controls decimal arithmetic negotiation. When an intermediate decimal result needs more than 38 digits, this setting decides whether Spark may reduce fractional scale to make the result fit.

ANSI is the courtroom. allowPrecisionLoss is the tailor squeezing your decimal into a 38-digit robe.

⚔️ The Battle of the Fives: `round` vs. `bround`

After Spark has done the math, negotiated the decimal types, and survived the 38-digit cathedral, you still face one final question: How should we round the result?

This matters most when a number lands exactly on a five.

Spark gives you two different philosophies:

round uses HALF_UP rounding. This is the familiar schoolbook rule: when the discarded part is exactly halfway, push the number up.
SELECT round(2.5, 0); -- 3
Simple. Intuitive. Slightly biased. If you round millions or billions of halfway values this way, the errors tend to drift upward.
bround uses HALF_EVEN rounding. This is banker’s rounding: halfway values go to the nearest even neighbor.e
SELECT bround(2.5, 0); -- 2
SELECT bround(3.5, 0); -- 4
The goal is not vibes. It is bias reduction. By sending some .5 cases down and others up, bround avoids systematically inflating large aggregations over time.

So the rule is simple:

Use round when your business rule explicitly says “half up.”
Use bround when you want banker’s rounding and less aggregate drift.

The five is not innocent. Choose your rounding theology before Finance chooses it for you.

🧎‍♂️ The BigDecimal Confession Booth: Cross-Language Boundaries

We must separate three distinct entities that share a room but live under completely different laws:

Java/Scala java.math.BigDecimal: An infinite digital scroll.
Python decimal.Decimal: A localized execution context.
Spark SQL DecimalType: A rigid form with exactly 38 boxes.

Confusing them is how you accidentally turn an enterprise financial pipeline into a Gregorian scream.

Two Worlds, One Bridge: Application Objects vs. Catalyst Engine

A JVM BigDecimal can grow to hundreds of digits. But once it enters Spark SQL, it crosses into Catalyst territory: fixed schemas, optimized memory, distributed execution, and a hard decimal ceiling.

No matter whether you write Java, Scala, or Python, Spark DecimalType stops at 38 digits.

Your application object may dream bigger. Catalyst does not.

The Native Language Inference Traps

Do not assume your host language protects you from Spark’s decimal rules.

The Scala Dataset Trap: a Scala BigDecimal may feel unbounded in your case class, but Spark still has to encode it as a SQL decimal. Once it enters Catalyst, the 38-digit ceiling applies.

The Python Inference Trap: when PySpark infers schema from native decimal.Decimal objects, it maps them to DECIMAL(38,18) by default.

That sounds safe, but it can be dangerous. Wide inferred decimals leave less room for downstream arithmetic, making scale reduction more likely.

Never let schema inference define your financial boundaries. Declare them.

The Infiltration of Primitive Heretics

The most common precision leak happens when a host-language primitive sneaks into a decimal expression.

In PySpark or Scala, multiplying a DecimalType column by a raw literal like 0.9 may introduce a floating-point value — the exact binary approximation machine we were trying to avoid.

Once decimal math mixes with float or double math, Spark’s type coercion can pull the expression into approximate floating-point territory.

🛠️ The Fix: keep constants explicit.

from decimal import Decimal  
from pyspark.sql.functions import lit  
  
df = df.withColumn(  
    "discounted_amount",  
    col("amount") * lit(Decimal("0.90")).cast("decimal(3,2)")  
)

Or in SQL:

amount * CAST('0.90' AS DECIMAL(3,2))

Do not let 0.9 wander into your financial calculation wearing a fake mustache. Make the multiplier a decimal contract.

Gateways: Pandas and Arrow

Every boundary crossing is a chance for decimal transmutation.

When Spark data moves through .toPandas(), Arrow, Pandas, or local Python loops, do not assume your DecimalType semantics survived unchanged.

Depending on the conversion path, decimals may arrive as Python Decimal objects, Arrow decimal values, Pandas object columns, or — if someone “helpfully” casts them — plain float64. That last one is the heresy.

Once your money becomes a binary float, you are back in approximation land. So after every gateway, inspect the types again.

Spark schema before the bridge. Pandas dtypes after the bridge.
Trust no boundary.

🛢 The JDBC Trap: When Databases Enter the Chapel

So far, we have sinned mostly inside Spark’s own monastery.

Real pipelines are messier. They talk to Oracle tables, MySQL schemas, PostgreSQL columns, SQL Server ledgers, and JDBC drivers with strong opinions.

The moment Spark reads from an external database, it must translate foreign column metadata into Spark SQL types. That translation depends on the database dialect, the driver, and the declared precision and scale.

Most days, this works. Some days, a harmless-looking NUMBER, NUMERIC, or DECIMAL walks in wearing too many digits for Spark’s 38-digit cathedral.

That is when the chapel gets interesting.

The Clash of Decimal Religions

External databases may allow decimal shapes that Spark simply cannot host. MySQL DECIMAL can reach 65 digits. PostgreSQL NUMERIC can be even more flexible. Oracle NUMBER is its own shapeshifting beast, with metadata that can confuse ordinary mapping logic.

Spark, meanwhile, has one sacred limit: 38 digits. So when JDBC hands Spark a giant NUMBER, NUMERIC, or DECIMAL, compatibility is not guaranteed.

Your database brought 65 relatives to the wedding. Spark only has 38 chairs.

Three Symptoms of a JDBC Breakdown

When database numbers clash with Spark’s decimal rules, the failure usually shows up in one of three ways.

Symptom 1: Immediate Cluster Explosion. Spark reads the database metadata, sees a NUMBER, NUMERIC, or DECIMAL shape that cannot fit inside its 38-digit limit, and fails before processing a single row. Painful, but mercifully loud.

Symptom 2: Silent Scale Squeezing. Spark manages to ingest the value, but later decimal math hits the 38-digit ceiling. To preserve the integer part, Spark may reduce fractional scale. Your pipeline keeps running. Your sub-cents quietly leave the chapel.

Symptom 3: The “Too Late for Double” Panic Move. A common rescue attempt is to cast the monster column to Double. But if Spark has already rejected the source decimal during JDBC schema mapping, your cast never gets a chance to run. The rescue boat cannot help if the passenger is rejected at the port.

When the source schema is too wide, fix the mapping at the JDBC boundary: narrow it in the source query, use customSchema, or intentionally read the column as a string and parse it yourself.

Strategies for Defensive Ingestion

Do not let JDBC improvise your financial schema. Control the boundary.

1. Cast at the source. Instead of reading a raw table, use a source-side SQL query that casts oversized numbers into sane business types before Spark sees them.

2. Override the JDBC mapping. Use Spark’s customSchema option when the database metadata is technically valid but too dramatic for your pipeline.

3. Read giant decimals as strings. If a value may exceed Spark’s 38-digit ceiling, ingest it as text first. Then validate, parse what fits, and quarantine the numeric monsters.

4. Split the number’s anatomy. For extreme precision cases — crypto quantities, scientific values, custom ledgers — split the integer body and fractional tail into separate string columns. You lose native decimal math, but you preserve every digit.

5. Unmask the fake mustaches. Some numbers are not numbers. Account numbers, card references, transaction IDs, invoice codes, barcodes, and phone numbers should usually be strings. You will never calculate the average invoice ID.

The Return Journey: Writing Back to the Database

The JDBC trap is a two-way street. Reading from a database is only half the story. Writing back can hurt too.

If Spark calculates a highly precise decimal and writes it into a narrower database column, the sink will not magically understand your intentions. It may reject the row, round the value, or behave according to database-specific rules you definitely do not want to discover in production.

Before calling .save(), make the contract explicit: round, validate, and cast your Spark columns to match the target schema exactly.

Do not let the database sink improvise your final numbers. Bless them before departure.

🧮 Aggregations: When Tiny Decimal Sins Become Cathedral-Sized Problems

So far, we have sinned one row at a time: a tax rate here, a discount there, a suspicious sub-cent in the corner. But Spark was built for scale.

When billions of rows hit SUM, AVG, or grouped financial totals, tiny decimal decisions stop being tiny. A rounded cent, clipped scale, or silent NULL can become a compliance discrepancy with its own meeting series.

The Rounding Altar: Before vs. After Aggregation

The order of operations changes the money.

Imagine three transactions: 0.005, 0.005, 0.005.

If you round each row first: 0.01 + 0.01 + 0.01 = 0.03.

If you sum first: 0.005 + 0.005 + 0.005 = 0.015. Then round once: 0.02.

Both answers can be valid. They are just different business rules wearing the same spreadsheet costume.

Auto-Widening: Room to Grow

Spark knows aggregates can outgrow their row-level containers.

For SUM, Spark adds 10 digits of precision to decimal inputs.

For AVG, Spark expands both precision and scale, because averages often create deeper fractions.

Helpful? Yes. Infinite? No.

Everything still lives inside the 38-digit cathedral. If your input decimals are already oversized, Spark may hit the ceiling during aggregation.

The Cursed Duet: Multiplying Before Summing

The classic failure pattern is: SUM(quantity * price).

Spark evaluates from the inside out. First, quantity * price creates a widened decimal product. Then SUM tries to add even more precision.

If your base columns are over-allocated, this combo can slam into the 38-digit wall fast.

Constrain the intermediate product before grouping:

SUM(CAST(quantity * price AS DECIMAL(18, 2)))

But choose that cast deliberately. It is a business rule, not a bandage.

`SUM` Over `DoubleType`: The Distributed Chaos Trap

Some teams escape decimal errors by casting money to DoubleType. This avoids the 38-digit cathedral. It also reopens the floating-point haunted house.

Floating-point addition is order-sensitive. In distributed systems, aggregation order can change with partitioning, shuffles, and execution plans. The tiny trailing bits can move.

For dashboards, maybe fine. For financial reconciliation, absolutely cursed.

Audit the input scale.
Audit the intermediate casts.
Audit the aggregate schema.

At small scale, precision bugs are anecdotes.
At Spark scale, they become accounting events.

🙃 The Literal Problem: `0.1` Is Not Always the Same Creature

In Spark, 0.1 may look harmless, but the way it enters the system matters.

In pure Spark SQL, a fractional literal like 0.1 is treated as an exact decimal literal.

But in PySpark or Scala APIs, a host-language literal like F.lit(0.9) starts life as a Python or JVM floating-point value. That means it may enter your expression as an approximate binary number — the exact gremlin we were trying to avoid.

So this is risky:

df.withColumn("discounted", F.col("amount") * F.lit(0.9))

Make the contract explicit instead:

from decimal import Decimal  
from pyspark.sql import functions as F  
  
df.withColumn(  
    "discounted",  
    F.col("amount") * F.lit(Decimal("0.90")).cast("decimal(3,2)")  
)

Or in SQL:

amount * CAST('0.90' AS DECIMAL(3,2))

Mixed Types: The Law of Precedence

When an exact decimal meets an approximate float or double, Spark must choose one shared type. And the approximate type can win.

That means one casual floating-point literal can pull the entire expression back into binary approximation land.

The same trap appears in UNION ALL. If one branch returns DECIMAL and another returns DOUBLE, Spark must reconcile the columns. Your pristine decimal branch may be coerced into approximate floating-point territory so both sides can coexist.

One unchecked branch can secularize the whole data lake.

Ingestion: Avoid the Sacrament of Convenience

The literal problem has a cousin at the landing gate: inferSchema=True.

Schema inference is useful for exploration. It is not a production contract.

If a CSV or JSON field looks like a currency fraction, Spark may infer a floating-point type, and every downstream calculation inherits that original sin.

Declare schemas explicitly at ingestion.

The Rule of Literal Discipline

Never trust a number by how it looks on screen.

An exact decimal and an approximate double can walk into a query together. Only the double may walk out.

Keep financial constants in explicit strings, Decimal objects, or declared DecimalType columns from ingestion to aggregation.

🧪 The Precision Exorcism Lab

Enough theology. Time to summon the bugs. These labs are designed to expose the places where Spark decimals mutate, round, overflow, or quietly lose their tiny fractional souls.

Phase 1: Environment & Foundational Truths

Lab 0: The Session Audit.Print your active Spark configs: ANSI mode, allowPrecisionLoss, store assignment policy, Spark version, and runtime. Platform upgrades can silently flip defaults.

Lab 1: The 0.1 + 0.2 Confession.Compare floating-point math against decimal math. Binary floats approximate human decimals; DecimalType avoids that specific sin.

Lab 2: Precision and Scale Are Not Vibes.Try fitting 1000.00 into DECIMAL(5,2). The container only has room for 3 integer digits, so Spark must reject it or return NULL, depending on settings.

Lab 3: The DecimalType() Default Trap.Create an unparameterized DecimalType(). Spark gives you DECIMAL(10,0): ten digits, zero fractional places. No cents. No mercy.

Phase 2: Math, Triage, and Mutations

Lab 4: Multiplication — The Scale Explosion.Multiply decimals and inspect the output type. Products widen fast, and wide inputs can slam into Spark’s 38-digit ceiling.

Lab 5: allowPrecisionLoss— Triage Controls.Run the same overflowing decimal math with allowPrecisionLoss=true and false. With the default, Spark may reduce scale to fit. Without it, Spark refuses the compromise.

Lab 6: Division — The Infinite Sermon.Divide values like 1 / 3. Spark cannot store infinite digits, so it assigns a finite decimal result type.

Phase 3: Rounding & Aggregation Layouts

Lab 7: round vs. bround. Compare commercial rounding with banker’s rounding.round uses HALF_UP. bround uses HALF_EVEN. The five is not innocent.

Lab 8: Round Before vs. After Aggregation.Compare: SUM(round(x, 2)) against: round(SUM(x), 2). Both may be valid. They are different business rules.

Lab 9: Aggregation Type Mutation.Inspect how SUM and AVG reshape decimal types. Aggregations widen schemas because row-level containers are too small for distributed totals.

Phase 4: Border Security & Ingestion Boundaries

Lab 10: The Python Float Infiltration.Multiply a decimal column by F.lit(0.9), then inspect the type. A native Python float can drag exact decimal math back into approximate floating-point territory.

Lab 11: The Union Contamination.Union a decimal branch with a double branch. Spark must reconcile the types, and one approximate branch can poison the whole stream.

Lab 12: Dirty Ingestion with try_cast. Parse corrupt strings and oversized values with try_cast. Bad cells become NULL, letting you quarantine failures without crashing the job.

Lab 13: The JDBC Monster Simulation.Simulate database decimals wider than Spark’s 38-digit limit. Test source-side casts, string ingestion, and explicit schema overrides.

Phase 5: Integration Smoke Tests

Lab 14: The One-Cell Smoke Test.Run a tiny startup check that prints configs, tests float vs. decimal behavior, checks arithmetic type mutation, and validates aggregate output types. One cell. Many demons.

Lab 15: The Ugly-Number Stress Test.Feed your business logic hostile values: half-cents, repeating divisions, malformed strings, max-width decimals, oversized integers, and near-overflow balances.

📜 The 10 Amendments of Working with Decimals

1. Thou Shalt Use Decimal, Not Double

Floating points are approximations. Use DecimalType for money, ledgers, taxes, and exact business values.

The Blueprint: no more 0.1 + 0.2 surprises.

2. Thou Shalt Know Thy Precision and Scale

DECIMAL(p, s) is a contract.

p = total digits.
s = digits after the decimal point.
p - s = digits before the decimal point.

The Blueprint: DECIMAL(18,2) is not 18 digits before the dot.

3. Thou Shalt Respect the 38-Digit Ceiling

Spark decimals stop at 38 digits.

The Blueprint: 38 is the sacred maximum. Plan headroom before multiplication and aggregation.

4. Thou Shalt Round with Intention

Rounding is a business rule, not a formatting choice.

The Blueprint: round() is HALF_UP; bround() is HALF_EVEN. Banker’s rounding counts too, my child.

5. Thou Shalt Round at the Right Altar

Rounding each row and then summing is not the same as summing first and rounding once.

The Blueprint: pick the sequence your business actually means.

6. Thou Shalt Use Decimal Literals

Raw Python or Scala floats are tiny gremlins in binary costumes.

The Blueprint: use CAST('0.90' AS DECIMAL(3,2)), Python Decimal("0.90"), or explicit DecimalType. Cast it. Do not trust it.

7. Thou Shalt Know Thy Types

Never trust only the value printed on screen.

The Blueprint: typeof() and .printSchema() are your flashlights in the data dungeon.

8. Thou Shalt Tame Thy Sources

JDBC databases may bring decimals too large or too weird for Spark’s 38 chairs.

The Blueprint: cast at the source, use customSchema, read monsters as strings, or redesign the ingestion boundary.

9. Thou Shalt Quarantine Bad Inputs

Malformed values and overflows should not dissolve quietly into mystery NULLs.

The Blueprint: use try_cast to separate broken data from genuinely missing data. Quarantine the sinners immediately.

10. Thou Shalt Test the Ugly Numbers

The happy path lies.

The Blueprint: test half-cents, repeating divisions, bad strings, overflows, max-width decimals, and huge multiplications. Ugly tests save beautiful reports.

⛪ Final Thoughts: Leaving the Cathedral

Precision is not a cosmetic fix you apply at the end because the numbers look messy. It’s architecture.

Spark has hard boundaries. If you let it infer financial schemas, mix decimals with floating-point literals, or ignore the 38-digit ceiling, it will enforce its rules without apology.

Sometimes that means vanished cents.
Sometimes it means a silent NULL fog.
Sometimes it means a midnight cluster exception wearing ceremonial robes.

So lock down your ingestion schemas. Quarantine malformed inputs with try_cast. Audit intermediate expressions with typeof() and .printSchema().

Protect the boundaries. Respect the engine. Keep the ledger flawless.

Amen. 🙏

🕊️ A Spiritual Commendation

Before we close this ledger, a reverent greeting to my fellow distributed-systems practitioners, Mariano and Sumit.

And above all, our deepest commendations to Guillem, who spent consecutive days and sleepless nights praying at the altar of the capricious Decimal God, suffering the unspeakable agony of silent trailing precision loss.

May your schemas always be explicit, your partitions stable, and your pews forever full of integer digits.

Your sacrifice has been noted in the master catalog.

“And so we pray, Spark won’t take my decimals away.
And so we pray, for holy casts and schemas that obey.
And so we pray, while my decimals say “bye, bye,” (Wait!)
Pray that my cents make it to the end of the day.”
(adapted from “We pray” by Coldpay ft. Tini)

The 10 Amendments of Working with Decimals