The Night Of The Living Lakebases

🪦 The Rule Of The Grave

Scale-to-zero sounds like a button. In Lakebase, it is closer to a ritual.

You enable it, choose how long compute can sit idle, and then Lakebase waits. If nothing touches it during that window, no queries, no connections, no tiny “are you there?” taps, the compute is suspended.

The data stays. The endpoint stays. The connection string still works. Only the running compute goes quiet, until the next real query wakes it up.

That is the beautiful part, and the trap. Idle doesn’t mean “I stopped using the app.” It means the database stopped hearing from everyone.

Humans may have left the room. The architecture may not have.

---

🕵️‍♂️ The Investigation Board

Once we understood the rule, the investigation became simple:

💡 Every process with a Lakebase connection string was now a suspect. Not only the app. Everything around it.

Suspect #1: The Healthy Health Check

The classic innocent-looking killer:

@app.get("/healthz")  
def healthz():  
    db.execute("SELECT 1")  
    return {"ok": True}

The platform checked the app. The app checked Lakebase. Lakebase heard activity. The grave stayed open.

🛠️ So we split the checks: /healthz became app-only, and the real database check moved to a rarer /readyz or /deepz.

Suspect #2: The Pool With A Pulse

A connection pool is useful when the app is busy. But in a sleepy demo branch, this can become a tiny life-support machine:

minimum pool size > 0  
idle connections kept forever  
validation queries  
keepalive pings  
eager pool startup

🛠️ So we made the pool lazy: min_size = 0, shorter idle lifetime, smaller max size, and reconnect on the first real request.

Suspect #3: The Dashboard That Loved Us Too Much

The leaderboard looked harmless. But auto-refresh every few seconds is still traffic. A forgotten browser tab is still traffic. A BI panel polling Lakebase is still traffic.

🛠️ So we changed the demo dashboards to manual refresh, increased refresh intervals where needed, paused polling when the tab was inactive, and moved historical views away from the operational endpoint.

Suspect #4: The Notebook In The Attic

The notebook wasn’t haunted. The open connection inside it was.

A running cell, scheduled notebook, JDBC read, or forgotten polling loop can keep the database awake long after the demo is over.

🛠️ So we stopped treating notebooks like harmless scratchpads. Connections had to be opened late, closed early, and wrapped properly.

Suspect #5: The Cron Goblins

The worst traffic is the traffic nobody thinks of as traffic:

leaderboard refresher  
queue poller  
metrics exporter  
cleanup job  
simulation generator  
temporary script

🛠️ So we made a list of every process with a connection string, disabled unnecessary workers in sleepy environments, and replaced some polling with less noisy patterns.

Suspect #6: Branches And Replicas

A branch is a parallel timeline.

The production branch may be quiet while the demo branch is undead. A read replica may stay warm because a dashboard is hitting it. Yesterday’s notebook may still point to yesterday’s endpoint.

🛠️ So we created an endpoint inventory: branch, compute, owner, app, dashboard, notebook, scale-to-zero policy, and delete-after date.

Suspect #7: The Compute That Cannot Die

Some computes are not supposed to sleep.

If high availability is enabled, availability wins over scale-to-zero. That’s not a bug. That’s the job.

🛠️ So we stopped expecting every environment to behave the same way: HA for production, scale-to-zero for dev, test, demo, and experiment branches.

Suspect #8: The Resurrection

When Lakebase finally sleeps, the first request has to wake it.

That means stale pool connections, missing session state, broken prepared statements, or a first-user 500 can appear if the app is not ready.

🛠️ So we made wake-up part of the design: retries with jitter, reconnect on stale connections, idempotency keys, short timeouts, and no critical state hidden inside the session.

---

📝 Takeaways From The Graveyard

The main lesson was not “turn on scale-to-zero and forget about it.”
The lesson was that scale-to-zero is an architectural contract.

💡 Lakebase can sleep, but only if the rest of the system allows it.

In our demo, that meant treating health checks, pools, dashboards, notebooks, workers, and branch endpoints as part of the cost model.

Once we removed the zombie traffic, the dev branch finally behaved like a dev branch: awake during the fraud-investigation demo, asleep after the show. In our PoC math, that meant around 70% fewer active compute-hours across nights, weekends, and abandoned demo sessions.

But sometimes you don’t want the database to be asleep when the user arrives.

For planned sessions, we added a small prewarming step: before the demo, workshop, or analyst shift, we triggered a tiny real query to wake Lakebase up.

💡 That became the operating model: sleep when nobody cares, wake up before someone does.

Production may stay warm. HA workloads may not be candidates. Dev branches should sleep aggressively. Demo branches should sleep most of the time, then prewarm before showtime.

Lakebase gave us the grave. We had to stop bringing the candles every thirty seconds.

---

When your database refuses to scale to zero because /healthz keeps whispering “SELECT 1” from the grave.