Apache Spark WTF??? 🤖 Prompting In The Name 🧠

“Some of those that push codeforces
 Are the same that trust Genie sources
 Some of those that push codeforces
 Are the same that ship Genie sources”
(adapted from “Killing in the name” by Rage Against the Machine)

---

🤖 What Genie Code Is

An AI Agent for Data Work

For years, most “AI for developers” tools have focused on one thing: helping us write code faster. Autocomplete a function, suggest a SQL clause, maybe refactor a block of Python. Useful? Absolutely. Revolutionary? Not really.

Genie Code is Databricks’ attempt to push that idea further. Instead of behaving like a smarter autocomplete engine, it’s designed to act more like an AI agent that can assist with real data work. That means tasks like writing queries, creating transformations, debugging pipelines, exploring datasets, or helping build machine learning workflows.

The key difference is the scope of the problem it tries to solve. Traditional coding assistants operate within the boundaries of a single file. Data engineering problems rarely do. They involve tables, schemas, pipelines, dashboards, metadata, and often several layers of infrastructure. Genie Code tries to navigate that environment by leveraging context from the Databricks workspace — things like catalog metadata, table structures, and existing workflows.

The result is a shift in how the assistant is supposed to be used. Instead of prompting it with something like “write a Spark join between these two tables,” the idea is that we can ask higher-level questions such as “build a pipeline that cleans this dataset and produces daily aggregates.” The AI then helps plan the steps, generate the logic, and iterate on the solution.

Why Generic Coding Assistants Struggle With Data

Most AI coding assistants were built with software engineering workflows in mind. They analyze the code in our editor, look at surrounding files, and suggest the next lines based on patterns learned during training.

For traditional application development, that works surprisingly well. But data engineering is a very different beast.

In a data platform, the real context isn’t just the code we’re writing — it’s everything around it. The meaning of a query depends on table schemas, column definitions, data lineage, governance rules, and how datasets are actually used across the organization. A simple SQL join might technically run but still produce completely wrong results if it ignores business logic hidden in the metadata.

That’s where generic assistants tend to struggle. They can generate syntactically correct SQL or Spark code, but they often lack awareness of theactual data environment. They don’t know which tables are authoritative, how datasets relate to each other, or which transformations already exist somewhere else in the platform.

Databricks’ approach with Genie Code is to plug the assistant directly into the data ecosystem itself. By leveraging context from Unity Catalog — schemas, tables, lineage, and metadata — the system can reason about the structure of the data platform, not just the script in front of it.

The Secret Sauce: Unity Catalog Context

What really makes Genie Code different from a generic AI coding assistant is its connection to Unity Catalog, the governance and metadata layer of the Databricks platform.

In most AI coding tools, the model only sees what’s inside our editor. It doesn’t know what tables exist in our environment, what the schemas look like, or how datasets are related. So when we ask it to generate a query, it often has to guess the structure of the data.

Genie Code, on the other hand, can access context from Unity Catalog. That means it can see things like tables, columns, schemas, data lineage, and metadata descriptions, as well as the governance rules associated with those datasets. Instead of inventing table names or assuming column structures, the assistant can reason about the actual data assets available in our workspace.

This context dramatically changes how queries and workflows can be generated. If we ask something like “calculate monthly revenue by customer segment,” Genie Code can inspect the relevant tables, identify the appropriate columns, and build a query that aligns with the real schema of our data platform.

It also means the assistant operates within the same security and governance boundaries as the user. If we don’t have access to a dataset, Genie Code won’t magically expose it either.

🤔 But What Happens if our Unity Catalog Looks Less Like a Well-governed Data Catalog and More Like the Wild West of Tables and Metadata?

In that case, Genie Code becomes a lot less like a knowledgeable coworker and a lot more like a very confident intern.

The whole strength of Genie Code comes from its ability to leverage metadata, schema information, lineage, and descriptions from Unity Catalog to understand how datasets relate to each other. If that context is incomplete or poorly maintained — missing column descriptions, unclear table purposes, inconsistent naming conventions — the AI simply has less signal to reason with.

Practically speaking, that means the assistant may still generate syntactically correct SQL or Spark code, but it will struggle with semantic correctness. It may join the wrong tables, misinterpret the meaning of columns, or miss business logic that normally lives in documentation or metadata.

This isn’t really a limitation of Genie Code specifically — it’s a broader truth about AI applied to data platforms:

AI works best when our data ecosystem is well organized.

In other words, Genie Code doesn’t magically fix a messy data platform. If anything, it tends to amplify the quality of the underlying metadata. Well-governed environments make the assistant look brilliant. Poorly documented ones make it guess. Or put more bluntly:

If our Unity Catalog is chaos, the AI will do its best … but it will be guessing with confidence*.*

Under the Hood: An Agentic System, Not a Single Model

When people hear “AI assistant,” they often imagine a single large language model sitting behind the scenes, answering prompts like a very smart chatbot. But systems like Genie Code are a bit more complex than that.

Rather than relying on one model to do everything, Genie Code is designed as an agentic system. In practice, that means it combines multiple components — language models, retrieval mechanisms, and platform tools — to complete a task. Instead of simply generating text, the system can plan steps, gather context, interact with the Databricks environment, and iterate on the result.

For example, if we ask Genie Code to investigate a dataset or build a transformation pipeline, the system doesn’t just generate a block of code and hope for the best. It may first retrieve metadata from Unity Catalog, inspect relevant tables, analyze schemas, and then generate the appropriate Spark or SQL logic. If the task requires additional context — such as logs from a failed job or information about a pipeline — it can query those resources before producing an answer.

This approach is often described as tool-augmented AI. The language model handles reasoning and interpretation, while specialized tools provide access to the actual data platform — tables, metadata, APIs, job logs, and so on.

The result is an assistant that can move beyond static text generation and participate in multi-step workflows. Instead of just answering questions, it can analyze problems, propose solutions, and interact with the platform to help execute them.

Of course, the word agentic has become something of a buzzword in the AI world. But in this case, the core idea is fairly practical: combine reasoning models with access to the data platform itself so the assistant can do more than just talk about the work — it can help perform it.

---

🛠️ Use Cases and Real Examples

Talking about AI agents could be fun (for some), but data engineers usually ask a much simpler question: okay … but what can it actually do? Let’s look at a few practical examples.

From Natural Language to SQL (and Insights)

One of the most immediate uses of Genie Code is the ability to translate plain-English questions into working SQL queries. Instead of manually inspecting tables, figuring out joins, and writing the query from scratch, a user can simply describe the analysis they want to perform.

For example, imagine an analyst wants to understand how user activity changed after a new product feature was launched. Instead of opening the SQL editor and building the query step by step, they might ask Genie Code something like:

Analyze daily active users before and after the new onboarding flow launched   
on March 1. Break the results down by user segment.

Using context from Unity Catalog, Genie Code identifies the necessary relationships. A key advantage of its “Agentic” design is its ability to automatically detect Primary Key and Foreign Key (PK/FK) constraints defined in the catalog to build accurate joins.

SELECT   
  DATE(event_timestamp) AS event_date,  
  user_segment,  
  COUNT(DISTINCT user_id) AS daily_active_users  
FROM user_events  
JOIN users USING (user_id)  
WHERE event_timestamp >= '2025-02-01'  
GROUP BY event_date, user_segment  
ORDER BY event_date;

Building Data Pipelines From a Prompt

Another interesting use case for Genie Code is generating data pipelines directly from a natural language description. Instead of manually stitching together ingestion logic, transformations, and output tables, a user can describe the desired workflow and let the assistant draft the pipeline.

With the launch of Lakeflow on March 11, 2026, Genie Code is now specifically optimized to generate Lakeflow Pipelines and Delta Live Tables (DLT) code, rather than just isolated Spark scripts. Imagine a data engineer who needs to process daily sales data. They could start with a prompt like this:

Create a Lakeflow pipeline that ingests daily sales data from sales_bronze,   
removes records with null customer IDs, and writes the results into a   
sales_silver table.

Using context from the workspace, Genie Code generates the integrated pipeline logic:

import dlt  
from pyspark.sql.functions import col, sum  
  
@dlt.table(name="sales_silver")  
def sales_silver():  
    return (  
        spark.readStream.table("sales_bronze")  
            .filter(col("customer_id").isNotNull())  
            .groupBy("order_date")  
            .agg(sum("revenue").alias("daily_revenue"))  
    )

The real advantage here is speed. Much of data engineering involves writing repetitive scaffolding. By generating the Lakeflow YAML configurations and DLT decorators automatically, Genie Code allows engineers to focus more on business logic and optimization rather than boilerplate orchestration.

Debugging Pipelines and Jobs

As any data engineer will tell you, debugging broken pipelines is part of the job. But while the investigation can be interesting, a big part of pipeline debugging involves repetitive tasks: searching for stack traces, checking recent schema changes, and verifying table dependencies.

This is where Genie Code can potentially help. Because it has access to the workspace context — jobs, pipelines, and System Tables — it can assist with analyzing failures much faster than manually digging through raw text files.

In “Agent Mode,” Genie doesn’t just “read logs” like a text-based chatbot. It queries structured metadata within the System Catalog (such as system.query.history and system.lineage), allowing it to correlate a failure with upstream changes or specific compute errors. For example, if a nightly ETL pipeline fails, an engineer might ask:

Analyze the failure of the job run in the daily_orders_pipeline  
and identify the most likely cause.

The assistant can inspect the system records, detect the error, and suggest a fix:

The pipeline failed due to a schema mismatch in the orders_raw table. The column order_total changed from INTEGER to DECIMAL in the latest ingestion batch.

By surfacing these insights directly from the system’s own metadata, Genie Code significantly reduces the time spent on initial diagnosis — the most tedious part of troubleshooting.

Data Science Workflows

Genie Code isn’t only useful for data engineering tasks — it can also assist with common data science workflows inside notebooks. From feature engineering to model training and evaluation, the assistant can help speed up many of the repetitive steps that come with building machine learning models.

Imagine a data scientist working on a churn prediction model. Instead of manually exploring the dataset, writing aggregation logic for features, and setting up the training pipeline, they can start with a simple prompt describing the goal:

Create features for a churn prediction model using the user_activity table.  
Include weekly usage frequency, average session duration, and total sessions per user.

Using the available tables and schema information, Genie Code can generate feature engineering logic such as:

from pyspark.sql.functions import count, avg  
  
features = spark.table("user_activity") \  
    .groupBy("user_id") \  
    .agg(  
        count("session_id").alias("session_count"),  
        avg("session_duration").alias("avg_session_duration")  
    )  
  
display(features)

From there, the assistant can help by generating boilerplate code for model training and instrumenting MLflow tracking. It can draft the syntax needed to log parameters and metrics, ensuring our experiment is recorded correctly from the start.

The real advantage here is workflow acceleration. Data scientists often spend a large portion of their time writing repetitive setup code — data preparation, feature extraction, and experiment configuration — before they even get to the interesting part of modeling.

With Genie Code acting as a kind of virtual ML engineer embedded in the notebook, these boilerplate steps can be generated quickly, allowing practitioners to focus more on experimentation and interpreting results.

Of course, just like with data pipelines, the generated code should always be reviewed carefully. While Genie Code can handle the instrumentation and execution logic, the actual comparison of performance and final model selection still require human expertise and domain understanding. But as a productivity boost within the notebook workflow, this kind of assistance can remove a lot of the friction from everyday machine learning tasks.

Exploratory Analysis and Business Insights

One of the more interesting promises of Genie Code is that it can help with exploratory analysis, not just code generation. Analysts or data teams can ask a business question and let the assistant handle the heavy lifting: figuring out which queries to run and where the signal might be hiding.

For example, imagine a product manager asks:

Investigate the revenue decline in Q3.  
Break it down by region, product category, and customer segment,  
and identify the main drivers.

Instead of returning a single SQL query, Genie Code approaches this as an Agentic Workflow. It runs multiple diagnostic queries across different dimensions, detects anomalies via Unity Catalog metadata, and then summarizes the findings.

A typical answer could look something like this:

The Q3 revenue decline was primarily driven by a 22% drop in the APAC region. The strongest impact came from the electronics category.

This is where the feature becomes especially interesting. The value is not only in generating SQL, but in connecting analysis steps into a reasoning flow. However, it is important to remember that Genie provides descriptive analysis (what happened); the causal interpretation (why it happened) still requires human judgment and domain context.

---

✨ Where Genie Code Really Shines

After looking at some practical examples, the next logical question is: where does Genie Code actually stand out compared to generic AI assistants?

While many tools can generate code or SQL, the real advantage of Genie Code comes from how deeply it is integrated into the Databricks Intelligence Platform. That integration gives the assistant access to context that most general-purpose coding tools simply don’t have.

Deep Context Awareness

The biggest strength of Genie Code is its ability to leverage Semantic Context from the data platform itself. Through integrations with Unity Catalog and workspace metadata, the assistant understands more than just table names — it understands lineage relationships, Primary/Foreign Key constraints, and Query History patterns.

This context changes the quality of the output significantly. Instead of guessing table names or inventing column structures — as many generic assistants do — Genie Code generates queries and transformations based on the actual structure and usage patterns of the organization’s data landscape.

For example, if we ask the assistant to analyze customer revenue by segment, it can inspect the relevant datasets, identify the correct PK/FK joins, and produce SQL that aligns with the real schema and respects existing governance rules. That doesn’t guarantee perfect results, but it dramatically reduces the hallucinated joins and invalid column references that often occur when AI tools operate without access to a live data catalog.

End-to-End Workflow Assistance

Another area where Genie Code stands out is the breadth of the workflows it can support. Most AI coding assistants are limited to a narrow scope: helping us write functions or refactor code inside an editor. Genie Code, however, is designed as a unified agentic layer across the entire data lifecycle.

That means it can assist with a wide range of tasks: generating SQL for exploratory analysis, drafting Lakeflow pipelines, helping debug failed jobs via system tables, or even supporting the creation of Lakeview dashboards. Instead of focusing on a single step in the workflow, the assistant participates in multiple stages of the data process.

This matters because data work rarely happens in isolation. A typical workflow might start with exploring a dataset, evolve into a production pipeline, and eventually feed dashboards or machine learning models. Traditionally, these steps involve switching tools and manually connecting the pieces with boilerplate code.

By acting as an ambient assistant within the Databricks Intelligence Platform, Genie Code aims to provide consistency across those stages. Whether we’re querying data, building pipelines, or monitoring jobs, the assistant provides the initial implementation and orchestration logic.

Of course, it operates as a Human-in-the-Loop agent — engineers and analysts remain responsible for high-level architecture decisions, final validation, and performance optimization. But as a productivity layer, Genie Code moves closer to the idea of an assistant that helps manage the full lifecycle of data work — not just the coding part.

Built-in Governance and Security

One of the biggest concerns with AI assistants in enterprise environments is data access. If an AI tool can see everything, it can potentially expose sensitive information; if it can’t see enough, it becomes useless.

Genie Code addresses this by operating as a native component of the Databricks Intelligence Platform, inheriting the exact authentication and permission model of the logged-in user. In practice, this means the assistant respects all policies defined in Unity Catalog, including table-level permissions, row-level filters, and column masks. If a dataset is restricted, the assistant won’t surface it — even with a clever prompt — because it is architecturally bound by the same Fine-Grained Access Control (FGAC) as the user.

For enterprise adoption, this is a critical distinction. Many organizations are cautious about external AI tools that require data to be sent outside their security perimeter. By acting as a governance-aware extension of the workspace, Genie Code ensures that its reasoning and “agentic” actions stay within the guardrails already protecting the data.

Continuous Learning and “Institutional Knowledge”

Another key aspect of Genie Code is its ability to improve over time as it interacts with users and the workspace. Unlike generic LLMs that restart with every prompt, the assistant uses Persistent Memory to adapt to the specific usage patterns and coding styles of our team.

As teams use the tool, Genie Code observes which datasets are frequently accessed, how queries are typically written, and which internal libraries are preferred. Over time, this allows the assistant to surface “institutional knowledge” that would otherwise be trapped in senior engineers’ heads. It starts recommending the authoritative tables, common joins, and specific coding conventions that reflect how our organization actually works.

To ensure this learning doesn’t go off the rails, Databricks recently acquired Quotient AI to embed a continuous reinforcement learning loop. This helps Genie Code analyze its own “agent traces” to identify why a specific plan failed or where a hallucination occurred, tuning its internal instructions based on past interactions.

While this doesn’t replace the need for clear documentation, it turns Genie Code into a hands-on development partner that grows smarter the more it’s used. Instead of every new hire learning the quirks of our data through trial and error, the AI can proactively guide them toward the patterns that already define our platform’s “source of truth.”

🤔 Does Context Overload Cause the Assistant to “Drift”?

Yes, that risk definitely exists. As a data platform grows, the sheer volume of assets can become overwhelming. Even with high-end reasoning, the assistant must perform Semantic Pruning to decide which pieces of metadata are relevant to our task. If the environment is a “dark data” swamp, the agent may struggle to distinguish between production tables and legacy backups.

This leads to three specific challenges in large environments:

1. Context Overload: Too many potential candidates for a join, forcing the AI to guess between similar schemas.
2. Dataset Ambiguity: Multiple tables with overlapping purposes that lack ‘Certified’ tags or clear ownership in Unity Catalog.
3. Metadata Dilution: Inconsistent descriptions that provide a “weak signal,” making it harder for the AI to understand business intent.

The way to mitigate this is not by limiting the size of the platform, but by treating Unity Catalog as the “LLM Training Manual.” By using Governed Tags, documenting Primary/Foreign Key relationships, and organizing assets into domain-specific Genie Spaces, we provide the “Agentic” system with a high-resolution map of our data.

In short: Genie Code doesn’t fix a messy data platform — it amplifies the quality of our governance. A well-governed catalog turns that complexity into usable context; a poorly governed one turns it into noise.

---

⚠️ Pitfalls, Drawbacks, and Shortfalls

While Genie Code introduces some genuinely interesting capabilities, it’s important to keep expectations grounded. Its effectiveness is fundamentally bottlenecked by the Semantic Density of our Unity Catalog. Beyond the obvious risks, there are practical shortfalls that teams should be aware of.

Garbage Metadata In, Garbage AI Out

The most significant limitation of Genie Code is its total reliance on catalog context. If tables lack descriptions, column meanings are unclear, or governance practices are inconsistent, the assistant has very little reliable signal to work with.

In those situations, Genie Code may still produce Syntactically Valid but Semantically Dead logic. It might generate a query that executes without error but joins the wrong datasets or ignores Certified Gold assets in favor of “shadow data” found in unmanaged schemas.

The shortfall here is simple: Genie Code doesn’t magically fix a messy data platform. If anything, it acts as a technical debt amplifier — exposing weaknesses in metadata management that an organization may have been ignoring for years. Without a well-governed foundation, the assistant transitions from a “Senior Engineer” to a “Confident Hallucinator.”

Over-Trusting AI-Generated Data Logic

Another risk is the temptation to treat AI-generated code as inherently “correct.” Genie Code can generate queries, transformations, and analysis steps in seconds — but speed is not a proxy for accuracy.

Subtle mistakes can still appear in generated SQL, especially around:

• Join Conditions: Joining on columns with similar names but different grains (e.g., user_id vs. account_id).
• Aggregation Logic: Incorrectly handling nulls or using COUNT instead of COUNT(DISTINCT).
• Filtering Assumptions: Missing “soft delete” flags or active/inactive record filters.
• Business Metric Definitions: Defining “Revenue” or “Churn” in a way that contradicts our internal fiscal reporting.

These errors are particularly dangerous in a Lakehouse environment where one flawed query can propagate into a “Gold” table and lead to misleading dashboards.

The shortfall here is cultural as much as technical. Organizations must treat Genie Code as a co-pilot, not an autopilot. Success requires rigorous Human-in-the-Loop (HITL) validation, ensuring that AI suggestions are subjected to the same unit testing and peer review as code written by a human.

The “Agent Hype” Problem

The concept of AI agents has become the defining buzzword of the 2026 AI landscape. Nearly every platform now claims “agentic” capabilities that can reason, plan, and execute tasks autonomously.

However, the industry is quickly learning that building an agent framework is relatively easy compared to making those agents reliable in production. The real challenge isn’t creating an AI that can generate a pipeline in a polished demo — it’s maintaining high Retrieval-Augmented Generation (RAG) precision when the agent has to navigate petabyte-scale metadata and messy, real-world schemas.

This is the “Demo-to-Production” gap. Agent-based systems often look flawless in a controlled environment but can struggle when they encounter undocumented legacy systems or complex edge cases. Databricks’ attempt to solve this via the Quotient AI acquisition — focusing on rigorous evaluation and “agent tracing”— is a step toward fixing this, but the burden of proof still lies in the day-to-day operations.

Until these systems can consistently handle the “messy middle” of data engineering, the “Agentic” label will remain a mix of high potential and high marketing.

Operational Complexity and Compute Costs

Even though Genie Code is built into the Databricks platform, the work it performs isn’t “free.” Because it acts as an Agent, its ability to autonomously plan, run, and iterate on code means it can trigger significant compute consumption behind the scenes.

Every time Genie Code tests a query, validates a schema, or retries a failed Spark job, it consumes Databricks Units (DBUs). In Agent Mode, the assistant often follows a “Plan-Execute-Verify” loop. If the task is complex, the agent might run multiple trial queries to “feel out” the data before presenting us with a final result. While this improves accuracy, it also increases cluster utilization compared to a human writing a single, manual query.

To manage this, organizations in 2026 are increasingly using System Tables (specifically system.billing.usage) to attribute costs directly to AI-driven activities.

• Serverless SQL Warehouses: Genie Code often defaults to Serverless compute to ensure fast “scale-to-zero” startup, but heavy experimentation can lead to unexpected bills if not monitored via Budget Policies.
• Attribution Tags: Many teams now apply specific Usage Tags to Genie-generated jobs to distinguish between human-initiated production runs and AI-driven exploratory tasks.

The shortfall isn’t the tool itself, but the frictionless nature of experimentation. When it becomes this easy to “ask and execute,” the volume of compute-intensive jobs can spike. Organizations must shift from monitoring “who is writing code” to monitoring “what the agents are spending,” using Databricks’ built-in observability dashboards to ensure the ROI of an AI-built pipeline doesn’t get eaten by the cost of its own creation.

The Human Expertise Still Required

Despite the promise of autonomous agents, Genie Code does not replace the need for experienced data professionals. Instead, it shifts the role of the data engineer from code-writer to system-orchestrator.

AI assistants can generate code and speed up experimentation, but the high-stakes responsibilities — data modeling, architecture design, governance policies, cost-efficiency auditing, and agent-tracing validation — remain firmly human domains. We still need an engineer to decide if a star schema is better than a flat table for a specific use case, even if the AI writes the DDL.

In other words, Genie Code is best viewed as a Human-in-the-Loop (HITL) force multiplier, not an autonomous data engineer.

The shortfall here is one of expectations: teams that expect AI to fully automate the “messy middle” of data engineering will likely be disappointed. However, teams that treat Genie Code as a collaborative assistant — letting the AI handle the boilerplate while humans handle the strategic logic and validation — will see the most significant ROI.

---

🧞 Putting the Genie to the Test

I decided to make the Genie sweat a little.

To start, I asked ChatGPT to generate a challenging SQL query using the TPC-DS data model (the industry-standard benchmark available in any Databricks account).

WITH combined_sales AS (  
    SELECT  
        ss_customer_sk AS customer_sk,  
        ss_net_paid AS sales_amt  
    FROM samples.tpcds_sf1000.store_sales  
    WHERE ss_customer_sk IS NOT NULL  
      AND ss_net_paid IS NOT NULL  
  
    UNION ALL  
  
    SELECT  
        cs_bill_customer_sk AS customer_sk,  
        cs_net_paid AS sales_amt  
    FROM samples.tpcds_sf1000.catalog_sales  
    WHERE cs_bill_customer_sk IS NOT NULL  
      AND cs_net_paid IS NOT NULL  
  
    UNION ALL  
  
    SELECT  
        ws_bill_customer_sk AS customer_sk,  
        ws_net_paid AS sales_amt  
    FROM samples.tpcds_sf1000.web_sales  
    WHERE ws_bill_customer_sk IS NOT NULL  
      AND ws_net_paid IS NOT NULL  
),  
sales_by_customer AS (  
    SELECT  
        customer_sk,  
        SUM(sales_amt) AS gross_revenue  
    FROM combined_sales  
    GROUP BY customer_sk  
),  
combined_returns AS (  
    SELECT  
        sr_customer_sk AS customer_sk,  
        sr_return_amt AS return_amt  
    FROM samples.tpcds_sf1000.store_returns  
    WHERE sr_customer_sk IS NOT NULL  
      AND sr_return_amt IS NOT NULL  
  
    UNION ALL  
  
    SELECT  
        cr_refunded_customer_sk AS customer_sk,  
        cr_return_amount AS return_amt  
    FROM samples.tpcds_sf1000.catalog_returns  
    WHERE cr_refunded_customer_sk IS NOT NULL  
      AND cr_return_amount IS NOT NULL  
  
    UNION ALL  
  
    SELECT  
        wr_refunded_customer_sk AS customer_sk,  
        wr_return_amt AS return_amt  
    FROM samples.tpcds_sf1000.web_returns  
    WHERE wr_refunded_customer_sk IS NOT NULL  
      AND wr_return_amt IS NOT NULL  
),  
returns_by_customer AS (  
    SELECT  
        customer_sk,  
        SUM(return_amt) AS total_returns  
    FROM combined_returns  
    GROUP BY customer_sk  
),  
customer_net_revenue AS (  
    SELECT  
        s.customer_sk,  
        s.gross_revenue,  
        COALESCE(r.total_returns, 0) AS total_returns,  
        s.gross_revenue - COALESCE(r.total_returns, 0) AS net_revenue  
    FROM sales_by_customer s  
    LEFT JOIN returns_by_customer r  
        ON s.customer_sk = r.customer_sk  
),  
ranked_customers AS (  
    SELECT  
        customer_sk,  
        net_revenue,  
        ROW_NUMBER() OVER (ORDER BY net_revenue DESC, customer_sk) AS rn  
    FROM customer_net_revenue  
)  
SELECT  
    SUM(net_revenue) AS top_100_omnichannel_net_revenue  
FROM ranked_customers  
WHERE rn <= 100;

The query required a complex Omnichannel Net Revenue calculation across store, web, and catalog channels, involving six different tables and multiple CTEs.

The goal? See if Genie could produce this same logic from a natural-language prompt.

The First Fail

I fed Genie a business description of the requirement.

The result was completely wrong. It wasn’t just a syntax error; the logic was semantically flawed.

I refined the prompt to be clearer, but Genie entered a hallucination loop, confidently producing an incorrect implementation of the returns logic.

The “Agent-to-Agent” Solution

I tried a different approach. I asked a colleague to use Claude Code to draft a “perfect” technical prompt for Genie based on my target SQL.

When I used the Claude-optimized prompt, the Genie finally “woke up.” It performed the Schema Mapping correctly, identified the disparate sales and returns tables, and produced a query that matched the expected output perfectly.

🤔 Why the Third Prompt Worked

The third prompt succeeded because it shifted Genie from Inferential Mode to Instructional Mode. It stopped asking the agent to guess the logic and instead provided a strict execution plan.

The first two prompts were “Business-Intent” heavy:

Compute omnichannel revenue, combine channels, rank the top 100.

To a human, that’s clear. To an AI agent, it’s an invitation to hallucinate. Genie produced queries that sounded right but were structurally broken — performing aggregations at the wrong grain or unioning results that were already partially collapsed. This is a common failure: the Business Story is preserved, but the Relational Algebra is not.

🤔 What the Third Prompt Changed

The successful prompt (optimized via Claude Code) removed all “Relational Ambiguity” by defining the exact Logical Plan:

• The Granularity: Exact columns and customer keys to join on.
• The Operator Order: Strict sequence (UNION ALL ➜ Global Aggregation ➜ Left Join).
• The Determinism: Forcing ROW_NUMBER with a unique customer_sk tie-breaker to ensure a stable Top 100 result.

In SQL, the two things that matter most are Grain and Operator Order. Once those were explicit, Genie stopped guessing and started executing perfectly. The first prompts described what the metric meant; the third prompt described how to build it. In the world of Agentic Data Work, that distinction is the difference between a dashboard we can trust and a confident lie.

🤔 But isn’t Specifying the Exact Execution Plan Like Cheating?

No — it’s not cheating, but it fundamentally shifts the evaluation from Autonomous Reasoning to Syntactic Translation.

When we provide the full plan, we are no longer testing Genie’s ability to infer the correct business logic from scratch. Instead, we are testing whether it can accurately translate a well-defined technical blueprint into valid Spark SQL.

There are two distinct levels of evaluation here:

1. Zero-Shot Semantic Reasoning: Prompt: “Compute omnichannel net revenue per customer and return the top 100.” Goal: Genie must autonomously infer grain, joins, and aggregations. (This is where the agent currently struggles with complex TPC-DS logic).
2. Plan-to-Code Translation: Prompt: “UNION sales ➜ aggregate by customer ➜ UNION returns ➜ aggregate ➜ LEFT JOIN ➜ ROW_NUMBER ➜ top 100.” Goal: Genie acts as a translator, converting a manual execution plan into optimized code. (This is where Genie excels).

Our third prompt succeeded because it narrowed the agent’s Search Space to a single path, leaving no room for logic drift. It’s not cheating — it just turns a high-risk reasoning problem into a low-risk specification-following problem.

In the real world, this is how we’ll likely use these tools: using high-level “Reasoning Agents” (like Claude) to define the plan, and “Execution Agents” (like Genie) to build it inside the governed environment of the Lakehouse.

---

🤜 Genie Code vs Claude Code + Databricks MCP 🤛

Databricks’ new Genie Code isn’t the only way to bring AI agents into our data. A second, parallel path is decoupled agency, where external tools like Claude Code or Cursor connect via Databricks Managed MCP Servers (launched in Public Preview on March 10, 2026).

The two approaches solve different architectural problems:

• Genie Code (The Native Specialist): This is a high-context, inward-facing agent. Because it’s built directly into the platform, it has “native-speed” access to the Unity Catalog, Lakeflow pipelines, and system logs. It’s optimized for the deep data engineering and ML tasks happening inside the Lakehouse.
• Claude + MCP (The Ecosystem Generalist): This is an open, outward-facing agent. By exposing Databricks tools (SQL, UC functions, Vector Search) via the Model Context Protocol, we allow Claude to treat our data platform as just one tool in its belt. This enables Claude to orchestrate workflows that span multiple systems — like reading a schema in Databricks to generate a Python script that updates a specialized tool in AWS or a record in a CRM.

💡 The Strategic Trend: Bi-Directional AI

Databricks is building in both directions simultaneously to avoid “Agent Lock-in”:

1. Genie Code: For teams who want a “Senior Data Engineer” living inside their workspace.
2. Managed MCP Servers: For teams who have already standardized on an external AI (like Claude or Cursor) and want it to have “governed hands” inside the Databricks environment.

If both mature, the future looks like a hub-and-spoke model: Genie becomes the native interface for deep data work, while MCP ensures Databricks remains the “Engine Room” for the entire world of autonomous AI agents.

---

🔮 What Comes Next: The Agentic Data Platform

Genie Code is not just another feature release — it’s a signal that Databricks is moving toward the Agentic Data Platform. The company has pivoted from a world where AI simply helps humans write code to a world where AI agents are responsible for the planning and operation of the entire data ecosystem.

The Rise of the “Agentic Data Platform”

Genie Code marks the transition from AI-assistance to Agentic Data Work, where the platform moves from autocomplete to autonomous execution. In this vision, the Agentic Lakehouse becomes a self-optimizing system where specialized agents assist with:

• Proactive Maintenance: Agents monitor Lakeflow pipelines and AI models in the background to triage failures and fix anomalies before a human is paged.
• Continuous Evaluation: Using the newly acquired Quotient AI framework, the platform embeds a continuous reinforcement loop to automatically score agent accuracy and fix hallucinations.
• Autonomous Engineering: Agents that don’t just suggest code, but design production-grade workflows for Change Data Capture (CDC) and data quality.

Multi-Agent Data Workflows

Rather than a single assistant, the future involves Agent Bricks — a swarm of specialized agents working together.

• Discovery Agents: Use Unity Catalog to find relevant datasets and lineage.
• Engineering Agents: Draft and deploy transformations via Lakeflow.
• Governance Agents: Enforce access policies and audit logs.
• Supervisor Agents: Orchestrate the handoffs between specialists, ensuring a coherent end-to-end result.

AI-Native Data Teams

As these capabilities mature, the role of data professionals will evolve from code-writers to system-orchestrators. Humans will focus on:

• Defining Goals: Setting the business intent and high-level strategy.
• Agent-Tracing Validation: Inspecting “agent traces” to verify how an AI reached a conclusion.
• Governance & Ethics: Setting the guardrails that the agents must follow.

The Beginning of the Agentic Lakehouse

Genie Code is the first chapter in the evolution toward an Agentic Lakehouse. On real-world data science tasks, Databricks has already seen success rates jump from 32.1% to 77.1% when moving to this agent-driven model.

However, the fundamentals remain: our AI is only as good as our data. A well-governed Unity Catalog is no longer just a “nice-to-have” — it is the literal training manual for our autonomous agents.

---

🏮 Final Thoughts: The High Cost of “Fast”

As I finish this article, two headlines from the last hours have me reconsidering what “productivity” actually means in 2026.

First, Rahul Patil (CTO at Anthropic) officially launched Code Review for Claude Code. His logic is hard to argue with: AI is helping us write code so fast that the human review process has become the ultimate bottleneck. Anthropic’s own internal data shows that by letting a swarm of agents “pre-review” PRs, they’ve boosted substantive feedback from 16% to 54%.

But there’s a catch. If we need AI to review the code that AI just wrote because humans can no longer keep up, are we actually building better systems, or just accelerating the rate at which we accumulate AI-generated technical debt?

Second, while Databricks is leaning into the Model Context Protocol to connect these agents, we’re seeing the first major cracks in that alliance. Denis Yarats (CTO at Perplexity) just announced at Ask 2026 that they are moving away from MCP in favor of traditional APIs and CLIs. His reason? MCP’s context overhead and authentication friction make it too “heavy” for high-scale agents.

Where does this leave us?

We are entering an era of “Agentic Exhaustion.” We have Genie building pipelines in the Lakehouse, Claude reviewing the PRs in GitHub, and a growing war over the protocols that connect them.

I’ll be honest: I’m not sure I like where this is headed. We’re reaching a point where the “human-in-the-loop” feels less like a pilot and more like an overwhelmed safety inspector trying to watch a thousand screens at once. Genie Code is a brilliant piece of engineering, but it’s also a reminder that as the “Genie” comes out of the bottle, the responsibility for the mess it might make stays entirely with us.

The future is faster, yes. But is it better? I guess we’ll find out in the next sprint.

---

P.S. If you’re wondering — yes, I acted as the “human-in-the-loop” for this article. While ChatGPT and Gemini did the heavy lifting, I spent hours curating the output to ensure my style and technical standards stayed intact. It’s my way of proving that while the “Genie” is out of the bottle, it still — and I hope always will — need a boss. At the end of the day, an agent can follow a plan, but only a human can decide if that plan is worth following.

---

“Prompt you? I won’t trust what you tell me (no, you’re not under control … yet)”
(adapted from “Killing in the name” by Rage Against the Machine)

---