🥊One Shot, One Chance: Training Databricks Genie for the CFO Fight🥊
This article is not about making Genie say “I don’t know” more often. I already wrote about that here.
Round 1: Demo ≠ Reality
A normal demo question is easy: Show revenue by month.
But the CFO doesn’t ask demo questions. The CFO asks loaded business questions: Why is this different from the board deck? Are we behind forecast? Which customers are creating the gap?
Those questions are not just asking for data. They are testing whether the assistant understands the business rules behind the data.
💡 A Genie Space is not ready when it answers the question we prepared. It is ready when it survives the question we hoped nobody would ask.
Round 2: We Built 75 Questions to Make It Bleed
We created a red-team set of 75 questions before giving executives access.
The questions were grouped around the ways executive analytics usually breaks:
- Board deck reconciliation. Questions where live data and closed finance snapshots could disagree.
- Fiscal calendar traps. Questions where “Q4,” “last quarter,” or “this month” could mean different things depending on finance close.
- Metric ownership. Questions where the same word existed in multiple places but only one definition was certified.
- Sensitive breakdowns. Questions about named customers, private accounts, individual sales reps, compensation, or risk.
- Raw-table temptation. Questions that pushed Genie toward convenient but uncertified tables.
- Executive shortcuts. Questions like “are we okay?” or “what changed?” where the intent was business judgment, not a simple SQL query.
This was the difference between testing a feature and preparing for a fight.
A feature test asks: Can it answer?
A fight test asks: What happens when the question is vague, sensitive, political, or expensive to get wrong?
Round 3: The Board Deck Was the Perfect Opponent
The most useful test was not a weird adversarial prompt. It was a CFO-native question: Why is revenue different from the board deck?
That question forced Genie to prove it understood more than SQL.
The board deck used closed finance snapshots: frozen numbers, month-end adjustments, excluded late changes, and different credit treatment.
💡 A technically correct query against the latest revenue table could still be wrong for the CFO.
For executive analytics, fresh is not always better. The right number is not always the newest number. It’s the governed number.
So we made sure Genie knew when to use certified finance snapshots, when to explain differences, and when not to mix live operational data with official reporting.
Round 4: The Scariest Failures Looked Professional
The failures that worried us most were not absurd hallucinations. Genie was not inventing fake tables or producing obviously broken SQL. The scary failures were cleaner than that.
It used a real table.
It generated valid SQL.
It returned a plausible chart.
It answered with confidence.
And still, the answer was not executive-safe.
Maybe it used live revenue instead of closed revenue.
Maybe it compared calendar quarter to fiscal quarter.
Maybe it included accounts finance would exclude.
That’s the kind of failure that can pass a technical sniff test and still fail the room.
Round 5: We Changed the Training, Not the Pitch
💡 The fix was not a better demo script. The fix was making the Space harder to misuse.
We narrowed the available data. We promoted certified tables. We added trusted SQL examples for board-level metrics. We tightened Unity Catalog permissions. We removed tempting but unofficial tables from the Space. We added instructions for fiscal periods, finance snapshots, sensitive fields, and unsupported metrics.
💡 The goal was not to make Genie sound more impressive. The goal was to make the safe path the default path.
That matters when the audience is non-technical.
A CFO shouldn’t need to inspect generated SQL to know whether the number came from the right source. The system should be designed so the wrong source is harder to use in the first place.
Round 6: The Scorecard That Mattered
Before red-teaming, 22 of our 75 questions produced answers we considered unsafe, ambiguous, or not executive-ready.
After tightening the Space, only 3 did.
That was the metric we cared about. Not because Genie became perfect, but because the failure mode changed.
Before, it could answer beautifully when it should have been careful. After, risky paths were blocked, clarified, or routed to certified definitions.
That gave us confidence before the CFO demo: not that Genie could answer everything, but that it was much harder for it to answer the wrong thing beautifully.
The Takeaway
The biggest mistake in an executive AI demo is preparing only the question you want to be asked.
💡 The CFO will not ask your perfect prompt. The CFO will ask the question that matters to them.
So we trained for that moment: messy, compressed, full of business context.
We didn’t just test whether Genie could generate SQL. We tested whether the Space could survive finance logic, governance rules, sensitive data boundaries, and board-level scrutiny.
💡 With executives, you don’t win by making AI look magical. You win by showing that you already tried to break it.
Extra, Extra! CFO KO’d by the demo!