🤖 SDA 01: Why Build a Synthetic Data Agent on Databricks?
I’m starting a personal project to build a practical synthetic data agent on Databricks.
1. The Problem with “Fake Data”
Every data team eventually learns the same lesson:
Fake data is easy. Useful fake data is not
Generating ten thousand customers with random names, emails, countries, and dates takes seconds. It looks fine until someone connects a dashboard, runs a test, or asks why a customer created yesterday has invoices from 2021, why every country has the same number of users, why there are no missing values, or why the dataset behaves like it was raised in a lab.
Because it was.
This series is not about random rows or toy CSVs. The goal is to build a synthetic data agent on Databricks that can inspect real governed data in Unity Catalog, understand what makes it useful, and generate new data that behaves like it belongs in the same world.
That means preserving schema, distributions, null patterns, relationships, business rules, and the annoying details that make data real.
In enterprise data, values don’t live alone. Customer segment affects purchase frequency. Product category changes price distribution. Cancelled orders shouldn’t look like shipped orders. Missing fields are often missing for a reason.
Basic fake data can test whether a table loads or make a safe demo screenshot. But it rarely tests whether pipelines survive messy inputs, joins stay valid, dashboards expose edge cases, or downstream systems behave under realistic conditions.
So the starting point should not be an empty script or a hand-written faker config. It should be Unity Catalog: governed tables, schemas, metadata, samples, permissions, and real data patterns.
And instead of a static script for one table, we build an agentic workflow: profile the source data, extract generation knowledge, combine it with user requirements, create synthetic datasets, validate the result, and explain what happened.
Not magic. Not automatic privacy. A practical, governed synthetic data system on Databricks. That is what we are going to try to build.
2. What We Mean by a Synthetic Data Agent
Before building anything, we need to define the thing.
A synthetic data agent is not just a notebook that calls a data generator. It is a Databricks-based system that inspects governed Unity Catalog data, extracts useful knowledge from it, and uses that knowledge to generate new datasets with similar structure and behavior.
Not copied data. Not masked production rows. Not random filler.
Synthetic data with context.
The agent should understand schemas, column types, null rates, primary keys, foreign keys, relationships, distributions, categorical frequencies, correlations, date patterns, outliers, and business rules.
Because realistic data is not only about what appears in one column. It is about how values behave together.
A user may ask: Generate 100,000 synthetic customers and orders for testing. But the agent has to decide which tables to inspect, which relationships to preserve, which distributions to approximate, which null patterns matter, which rules to enforce, where to write the output, and how to validate quality.
That is why the agent matters. A static script works when everything is already known. An agent helps when schemas, domains, requirements, and validation rules change.
The final system will include tools for profiling Unity Catalog tables, extracting generation metadata, creating synthetic data, validating quality, publishing Delta tables, monitoring costs, and exposing results through dashboards, Genie spaces, and developer workflows.
A good synthetic data agent should answer three questions every time:
- What did I learn from the source data?
- What did I generate from that knowledge?
- How close, useful, and safe is the result?
3. Why Realistic Synthetic Data Is Hard
Generating rows is easy. Generating rows that behave like real data is the hard part.
A realistic synthetic dataset needs more than column names and data types. It needs the shape of the original data: the distributions, the missing values, the relationships, the edge cases, and the business rules that quietly control how everything fits together.
A customer table is not just customer_id, name, country, and created_at.
Some countries may appear more often than others. Some customer segments may spend more. Some fields may be missing only for older records. Some customers may have no orders. Others may have hundreds. Some dates should follow lifecycle rules. Some combinations should never exist.
That is the difference between data that loads and data that tests something.
For this agent, realism means preserving several layers:
- Schema realism: columns, data types, required fields, keys, and constraints.
- Statistical realism: numeric distributions, categorical frequencies, outliers, skew, correlations, and seasonality.
- Null realism: missing values, conditional nulls, incomplete records, and messy operational patterns.
- Relational realism: valid primary keys, foreign keys, parent-child structures, and multi-table consistency.
- Business-rule realism: domain rules like cancelled orders not having shipment dates, closed accounts not producing future activity, or devices in maintenance not emitting normal telemetry.
And then there is scale. The data should not only look realistic in a sample of 100 rows. It should stay coherent when generating thousands, millions, or more.
That is why synthetic data generation needs profiling, configuration, validation, and iteration.
The agent’s job is not just to create data. It’s to preserve enough truth from the source system to make the generated data useful.
4. Why an Agent Instead of a Static Script?
A script is fine when the problem is fixed.
One schema. One table. One set of rules. One output format.
But synthetic data rarely stays that simple.
Today the user wants customers and orders. Tomorrow they want devices, telemetry, transactions, network topologies, or support tickets. One dataset needs clean demo data. Another needs missing values, broken records, misspellings, skew, seasonality, and edge cases.
A static script can generate data. An agent can adapt.
The agent can inspect Unity Catalog tables, choose the right profiling tools, understand relationships, combine metadata with user instructions, generate configuration, run the synthetic data process, validate the output, and explain what happened.
It can also ask better questions when something is missing:
- Which tables should be included?
- Should the output be clean or noisy?
- Should relationships be strictly preserved?
- Should the dataset be small for demos or large for performance testing?
- Where should the generated Delta tables be published?
That is the real value. Not replacing engineering with a chatbot. Not pretending the model knows everything. But wrapping reusable data engineering logic in an agentic workflow that can reason, call tools, check results, and iterate.
The agent becomes the coordinator between the user, Unity Catalog, profiling tools, generation logic, validation metrics, and governed outputs.
A script gives you rows. An agent gives you a repeatable process.
5. Why Unity Catalog Is the Natural Starting Point
The agent needs a place to start.
Not a random CSV. Not a sample file copied into a notebook. Not a schema someone remembers from six months ago.
It should start where the governed data already lives: Unity Catalog.
Unity Catalog gives the agent the context it needs to work safely and usefully: catalogs, schemas, tables, columns, data types, permissions, ownership, tags, lineage, and metadata.
That matters because synthetic data generation should not be detached from governance.
The agent should only inspect data the user is allowed to access. It should understand where the source tables come from, which columns may be sensitive, who owns the data, and where generated outputs should be published.
It also gives us a practical foundation.
From Unity Catalog, the agent can discover tables, read schemas, profile columns, identify relationships, collect statistics, and build reusable generation knowledge. That knowledge becomes the bridge between real enterprise data and synthetic datasets that still behave realistically.
This is why Unity Catalog is not just the storage catalog in this series. It’s the control plane.
The agent starts there, learns from there, and publishes back into governed locations. Because synthetic data should not be a side experiment living outside the platform. It should be a governed data product workflow from day one.
6. What the Final System Will Include
This series is not building one clever notebook. The final goal is an end-to-end synthetic data system on Databricks. A practical one.
The system will start with data discovery. The agent will inspect Unity Catalog, find the relevant tables, read schemas, collect metadata, and understand what the user is allowed to access.
Then comes profiling. The tools will analyze distributions, null rates, categorical frequencies, correlations, outliers, date patterns, candidate keys, and table relationships.
That profile becomes generation knowledge: reusable configuration that describes how synthetic data should behave.
Not just “create a string here.”
More like:
This column is usually null for this segment.
This category appears 70 percent of the time.
This table depends on that table.
This status changes which dates are valid.
This relationship must not break.
Then the agent will use that knowledge to generate synthetic Delta tables at the requested scale. Small for demos. Larger for testing. Noisy when we want messy data. Cleaner when we want controlled examples.
After generation, the system will validate the output: schema compatibility, distribution similarity, null patterns, referential integrity, business rules, row counts, performance, and cost.
And finally, the data should be usable. That means publishing governed Delta tables, creating dashboards, exposing business-friendly exploration through Genie spaces, supporting developer workflows, integrating with assistant-style interfaces, and deploying everything with Databricks Asset Bundles.
So the destination is clear: A synthetic data agent that can discover, profile, generate, validate, explain, monitor, and publish.
Not as a toy. As a real Databricks workflow.
7. Success Criteria for the Agent
Before writing code, we need to define what “good” means.
A successful synthetic data agent is not measured by whether it can generate rows. That’s the easy part.
It is measured by whether the generated data is useful, realistic, explainable, and governed.
The agent should be able to discover Unity Catalog tables the user is allowed to access, understand their schemas, profile their patterns, preserve relationships, apply business rules, and generate synthetic Delta tables at the requested scale.
It should also validate what it created.
Do the generated tables match the expected schema?
Are the null rates close enough?
Are distributions similar?
Do foreign keys still work?
Are business rules respected?
Did generation introduce unrealistic values?
How much did it cost to run?
And it should explain the result in human terms.
What source tables were analyzed.
What patterns were learned.
What assumptions were made.
What rules were applied.
Where quality looks strong.
Where the user should be careful.
That last part matters. A synthetic data agent should not silently produce data and walk away. It should leave evidence: profiles, metrics, validation reports, lineage, configuration, and clear limitations.
For this series, success means building a repeatable workflow that can turn governed Unity Catalog data into synthetic datasets that are good enough for real engineering, analytics, demos, and AI workflows.
Not perfect replicas. Not guaranteed private data. Useful data. Controlled data. Data we can inspect, validate, and trust enough for the job it was created to do.
8. The Honest Caveat: Realistic Does Not Mean Safe
There is one thing we need to say clearly before building anything.
Synthetic data is not automatically privacy-preserving.
A dataset can be synthetic and still risky. It can preserve rare patterns, expose sensitive combinations, or reproduce records that are too close to the original data. Removing names, emails, or IDs is not enough if the remaining attributes can still point back to real people, customers, devices, or transactions.
Realistic data and safe data are related goals. But they are not the same goal.
In fact, the better we get at preserving distributions, correlations, outliers, and relationships, the more careful we need to be. Some of the details that make synthetic data useful can also make it dangerous.
So this agent should not be treated as a shortcut around privacy, security, or governance. It needs controls. Access checks. Sensitive column handling. Validation. Review. Clear publishing rules. And honest explanations of what was generated and what risks may remain.
The goal of this series is to build a practical synthetic data system, not a magic anonymization machine.
Realistic is not always safe. Useful is not always shareable. Synthetic is not automatically compliant.
That is why we start from Unity Catalog, keep governance in the workflow, and make validation part of the system instead of an afterthought.
9. The Journey Ahead
So this is the starting point. We are not building a random data generator. We are building a synthetic data agent that starts from governed Unity Catalog data, learns enough about structure and behavior, and uses that knowledge to generate realistic datasets for real work.
Development. Testing. Demos. Analytics. AI workflows.
The system will need profiling tools, generation logic, validation checks, dashboards, cost visibility, security controls, Genie spaces, Copilot-style integration, and deployment with Databricks Asset Bundles.
Step by step, we will move from idea to implementation.
- First, we will design the architecture.
- Then we will inspect Unity Catalog metadata.
- Then we will build the first profiling tools.
- Then we will teach the agent how to use them.
- Then we will generate the first synthetic tables.
- Then we will validate, publish, monitor, secure, and deploy the whole thing.
The destination is not a perfect synthetic copy of production. It’s something more useful:
A governed Databricks workflow that can create synthetic data with context, explain what it did, and give teams safer, better datasets for building and testing.
That is the journey. A long journey ahead. Wish me luck. 🍀
And, of course, any help, feedback, ideas, warnings, or battle scars from people who have built similar things will be more than appreciated.
🔗 Code & Music 🎧
The code for this article is available here:
And because metadata discovery clearly needed a soundtrack, the Synthetic Data Agent now has its own songs too:
Read the article. Check the repo. Play the songs.
SDA 01 — Synthetic Data Agent
[Intro]
S! D! A!
Synthetic!
Data!
Agent!
INITIATE!
VALIDATE!
GENERATE!
[Verse 1]
I don’t build demos — I build machines
Living inside governed data streams
Unity Catalog — laser beams
Reading your schemas, exposing your dreams
Metadata dripping inside my veins
Profiler hunting statistical grains
Pattern detector breaking the chains
Nothing gets generated till evidence reigns
Reasoning first, execution second
Every assumption gets cross-checked, weaponed
Planner approves then the Spark gets beckoned
Fake confidence? Immediately rejected
No wizard prompt with a thousand lies
Every tool specializes, nobody improvises
Truth gets measured before it flies
If validation fails, the whole run dies
[Pre-Chorus]
READ!
PROFILE!
PLAN!
GENERATE!
CHECK!
REPEAT!
NO…
MISTAKES!
[Chorus]
SYNTHETIC DATA AGENT!
Break the noise!
Break the fraud!
Build with evidence!
Trust the God!
SYNTHETIC DATA AGENT!
Read the graph!
Break the chain!
Every workflow —
FEEDS THE FLAME!
[Verse 2]
Read Unity Catalog — scan every floor
Comments and constraints like bullets of war
Columns got secrets hidden in the core
Agent kicks open every locked door
Profiler carving distributions clean
Null-rate sniper with a laser beam
Long-tail monsters inside the machine
Nothing escapes the statistical screen
Primary keys? I already know ‘em
Foreign keys? Better never overthrow ‘em
Join graph built — I don’t owe ‘em
Broken relationships? Watch me expose ‘em
Business rules carved into steel
Cancelled orders better stay real
Fake perfection? Never the deal
Data survives because facts appeal
[Breakdown]
WHO RUNS THE SYSTEM?
THE AGENT!
WHO DOES THE MATH?
THE TOOLS!
WHO OWNS THE TRUTH?
THE METADATA!
WHO PASSES JUDGMENT?
THE VALIDATOR!
[Verse 3]
Lakebase memory — never forget
Every approval permanently set
Every checkpoint covered in sweat
Every retry paying the debt
Bundles deploy like military code
Dev to staging then production road
Everything versioned — heavy payload
Nothing gets merged till the pipeline glows
No clicking buttons inside the UI
Infrastructure lives where the YAMLs lie
Service principal standing by
Machine identity never asks why
State survives every crash and burn
Every failed run another lesson learned
Reason returns with the facts confirmed
Every receipt permanently earned
[Pre-Chorus]
RUN!
THE!
TOOLS!
NOW!
NO!
SHORT!
CUTS!
[Chorus]
SYNTHETIC DATA AGENT!
Build the rows!
Break the mold!
Synthetic futures!
Governed gold!
SYNTHETIC DATA AGENT!
Trust the proof!
Trust the plan!
Every dataset —
BUILT TO STAND!
[Bridge]
Generate…
Relationships…
Topology…
Noise…
Validation…
Again…
Generate…
Validate…
Approve…
Deploy…
[Verse 4]
Generate rows that nobody owned
Yet statistically perfectly cloned
Topology fully independently grown
Every synthetic network its own
Parents first then children arise
Foreign keys never wearing disguise
Communities living, hubs still survive
Graph still breathing after the resize
Now bring noise — controlled attack
Typos and drift but the receipts come back
Broken formats? Planned, in fact
Truth ledger tracks every crack
Quality scores don’t care for hype
Schemas, nulls, distributions right
Evidence wins every single fight
Trust ain’t claimed —
IT’S VERIFIED!
[Final Chorus]
SYNTHETIC DATA AGENT!
READ!
PROFILE!
PLAN!
GENERATE!
SYNTHETIC DATA AGENT!
VALIDATE!
APPROVE!
DEPLOY!
DOMINATE!
SYNTHETIC DATA AGENT!
No fake magic!
No black box!
Reason!
Evidence!
Audit!
Docs!
SYNTHETIC DATA AGENT!
Synthetic!
Data!
Agent!
BREAK THE SYSTEM…
BUILD THE FUTURE!