Apache Spark WTF??? — 🕵️♂️ Murder By Proxy? 🕵
“Murder!”, she wrote.
We were assured that nothing had changed in the network or I/O layer. Could it be a sneaky proxy misconfiguration, something quietly slowing down the transfers, or a subtle API version change? Our investigation had only just begun …
⏱️ The Time Theory
When debugging intermittent timeouts, increasing timeouts and retries helps reveal whether failures are due to transient latency rather than real outages. If errors drop, the system may simply be operating too close to its timing limits. Could it be that some calls just need a bit more time to be handled?
When Apache Spark writes data to AWS S3, several software layers are involved, each with its own notions of timeouts, retries, and failure handling. When something goes wrong, the resulting behavior is rarely the fault of a single component — it’s the emergent outcome of all of them interacting.
Three layers immediately came under scrutiny.
The AWS SDK Layer
AWS provides a Java SDK for accessing services such as S3. The original SDK v1 was synchronous and monolithic, whereas SDK v2 introduced a modular design, native asynchronous support, and far more granular control over retries and timeouts. Modern Hadoop distributions rely on AWS SDK v2, which brings its own sophisticated retry mechanisms — largely operating independently of Hadoop itself.
In SDK v2, timeouts are layered:
- API Call Timeout: the maximum total time for a request, including retries.
- API Call Attempt Timeout: the maximum time for a single request attempt.
- HTTP Client Timeouts: low-level connect, read, and write timeouts.
By default:
- No API call timeout is set.
- No API call attempt timeout is set.
- HTTP client timeouts should be configured explicitly to avoid long-hanging requests.
✅ Without explicit configuration, a stalled request may linger far longer than expected.
Retry behavior in SDK v2 depends on the retry mode:
- legacy: the older exponential backoff strategy.
- standard (default): fixed maximum attempts with throttling-aware exponential backoff.
- adaptive: dynamically adjusts retry timing based on observed network and throttling conditions.
In standard mode, the default is 3 attempts (1 initial request + 2 retries). Retries are triggered for specific HTTP errors and network exceptions.
✅ When API timeouts are configured, the total time spent retrying is bounded by those limits.
The Hadoop-AWS / S3A Connector Layer
The hadoop-aws module enables Hadoop to interact with S3 via the S3A filesystem. It wraps the AWS SDK but also adds its own retry and timeout logic on top.
S3A defines several important timeouts:
fs.s3a.connection.establish.timeout(~30s): TCP connection setup.fs.s3a.connection.request.timeout(~60s): waiting for an HTTP response.fs.s3a.connection.timeout(~200s): socket read/write timeout.fs.s3a.connection.acquisition.timeout(~60s): waiting for a pooled connection.
If any of these timeouts are hit, the operation fails and — depending on the error — may be retried.
S3A retry behavior works as follows:
- Retryable failures (network timeouts, EOFs, transient HTTP errors) are retried up to
fs.s3a.retry.limit. - Retries use exponential backoff (
fs.s3a.retry.interval). - Throttling errors have their own specialized retry handling.
- Only idempotent operations are retried to avoid partial or inconsistent state.
⚠️ By the time S3A observes a failure, the AWS SDK may already have retried the request multiple times behind the scenes.
The Apache Spark Layer
Spark itself retries failed tasks, not individual S3 operations. Network and API failures are expected to be handled by Hadoop and the AWS SDK.
Key points:
- Spark retries failed tasks up to
spark.task.maxFailures(default: 4). - Retries occur at the task level (map, reduce, shuffle, output write).
- Spark does not implement its own deep retry logic for S3 API calls.
This leads to an important distinction:
- If an S3 operation throws an exception, Spark may retry the entire task.
- If an S3 operation hangs without throwing, Spark has no way to detect it and may appear stuck indefinitely.
Spark has no built-in timeout for individual S3 write operations beyond what is configured in Hadoop and the AWS SDK.
✅ If all retries for a task fail, Spark fails the stage — but it will not retry the stage beyond per-task retries.
Before long, non-Spark jobs relying on Hadoop to upload data to S3 began failing in the same way. Spark was no longer a suspect. We experimented with increasing timeouts and tuning retry and backoff configurations, but the behavior remained unchanged. After exhausting every reasonable permutation, the Time Theory had to be dismissed.
📤 The Multiupload Theory
We were told that only large updates to S3 were failing while small ones continued to work. The first stacktrace we analyzed explicitly referenced multipart copy logic, which immediately pointed us toward multipart behavior as a likely factor.
In Hadoop AWS, the S3A multipart feature efficiently handles large uploads and copies by splitting objects into multiple parts, each uploaded or copied independently. Here’s a detailed breakdown:
Multipart Upload (PUT): When a file exceeds fs.s3a.multipart.threshold (default 128 MB), S3A splits it into parts of fs.s3a.multipart.size (default 32 MB). Each part is uploaded in parallel via the AWS SDK’s async client, and once all parts succeed, a CompleteMultipartUpload request finalizes the object.
Multipart Copy (renames / commit): Large objects cannot be copied in a single request. S3A performs a multipart copy using CopyObject for each part, reading from the source and writing to the destination independently. The AWS SDK orchestrates these parts asynchronously, but the entire operation must complete within the configured apiCallTimeout or requestTimeout. Failures in any part trigger retries according to fs.s3a.attempts.maximum.
Configuration-sensitive behavior:
fs.s3a.multipart.threshold: minimum size to trigger multipart upload/copyfs.s3a.multipart.size: part size for splitting objectsfs.s3a.fast.upload: can speed up small uploads by bypassing multipartfs.s3a.connection.timeout/fs.s3a.connection.request.timeout: critical for multipart copy, as each part is an independent request
Integration with Hadoop committers: During task commit, temporary files are renamed to final output paths. For large files, this rename is implemented as multipart copy + delete. Inadequate timeouts or poorly tuned part sizes can cause large-file commits to fail, while small files succeed without issue.
📦 The Mismatched Versions Theory
🎲 The Random Theory
shortlived urandom
🧵 The Thread Theory
🚧 The Mischievous Proxy Theory
Occam’s razor
software.amazon.awssdk.http.apache.addProxyConfigspark.executorEnv.NO_PROXY
🛠️ Do It Yourself
/etc/hosts no entiende de comodines, así que tenemos que poner los dominios completos s3.us-east-1.amazonaws.com
Me está poniendo endpoint aunque no se lo indico: Aunque hadoop-aws establece a “” la propiedad fs.s3a.endpoint cuando no viene veteada, Spark le pone por defecto s3.amazonaws.com
// In Hadoop 3.3.1, AWS region handling with the default "" endpoint only works
// in EC2 deployments or when the AWS CLI is installed.
// The workaround is to set the name of the S3 endpoint explicitly,
// if not already set. See HADOOP-17771.
if (hadoopConf.get("fs.s3a.endpoint", "").isEmpty &&
hadoopConf.get("fs.s3a.endpoint.region") == null) {
// set to US central endpoint which can also connect to buckets
// in other regions at the expense of a HEAD request during fs creation
hadoopConf.set("fs.s3a.endpoint", "s3.amazonaws.com", setBySpark)
}1️⃣ Endpoint resolution is delegated to the AWS SDK
If fs.s3a.endpoint is unset, S3A calls the AWS SDK to resolve it:
These endpoints are:
- Generated internally by the SDK
- HTTPS-only
- Not configurable via Hadoop properties
2️⃣ fs.s3a.connection.ssl.enabled is not consulted
When no endpoint is set:
- Hadoop does not construct the URI
- The SDK does
- The SDK always uses HTTPS
So the flag becomes irrelevant.
3️⃣ There is no override hook
There is:
- No Hadoop property
- No AWS SDK property
- No environment variable
that says:
“Use HTTP for auto-resolved AWS S3 endpoints”
AWS intentionally does not allow this.
When Hadoop S3A (via the AWS SDK) is not explicitly configured with fs.s3a.endpoint, it assumes it is talking to real AWS S3 and automatically generates virtual-host–style endpoints like https://<bucket>.s3.<region>.amazonaws.com; redirecting those hostnames to 127.0.0.1 (via DNS, /etc/hosts, or NAT) only changes the IP, not the logical endpoint, so the SDK still sends requests signed for AWS S3 and performs TLS hostname verification against <bucket>.s3.<region>.amazonaws.com, which will never match a MinIO certificate and causes TLS handshake failures. Because endpoint selection drives hostname construction, request signing, and routing, S3A cannot be “tricked” into talking to MinIO by redirection alone; the only correct approach is to explicitly configure MinIO as a non-AWS endpoint using fs.s3a.endpoint=https://<host>:<port> together with fs.s3a.path.style.access=true, which forces path-style requests (https://<host>:<port>/<bucket>/…) and bypasses AWS endpoint resolution entirely.
hacemos prueba con servicios parados
nginx+lua
🙏 Acknowledgments
“If we don’t quit, yeah
If we don’t stop, yeah
Lots of theories will pop
Until we get the cause of the connections drop
Yeah, now jobs don’t stop
’Cause it was 60 seconds on an undercover block
Yeah, now jobs don’t stop
’Cause it was 60 seconds on an undercover block”
(adapted from “Deep Cover” by Dr. Dre & Snoop Doggy Dog)