🤖 SDA 04: Reading Unity Catalog Metadata — The Agent’s First Tool

The Agent Opens Its Eyes

From Deployment to Awareness

A deployable project is useful, but it’s still just infrastructure.

After the previous installment, the Synthetic Data Agent has structure, separated environments, and deployable Databricks resources through Declarative Automation Bundles. Bundles let us define project resources and configuration as code, then validate, deploy, and run workflows through the Databricks CLI.

Very civilized. Almost suspiciously mature. But deployment is not understanding.

The agent may now live in a cleaner project, but it still doesn’t know what data exists, where it lives, what it means, who owns it, or which parts need careful handling. That’s the next problem.

The First Real Tool is not a Generator

Before the agent can generate realistic synthetic data, it needs a map of the territory.

It needs to discover catalogs, schemas, tables, views, columns, comments, tags, owners, and declared constraints. It also needs to know which objects are visible to its execution identity, which tables look like useful source candidates, and which fields already carry signs of sensitivity or business meaning.

This is why the first concrete tool is not a synthetic data generator. It’s a metadata reader.

uc_metadata_reader gives the agent its first governed view of the data estate by reading Unity Catalog (UC) metadata. UC is Databricks’ governance layer for data and AI assets, with capabilities such as centralized access control, discovery, lineage, and auditing across workspaces.

The agent shouldn’t discover production data through hardcoded table names, random notebooks, or someone’s memory of where “the real customer table” lives. It should start from governed metadata.

Evidence, not Prophecy

The goal of uc_metadata_reader is not to make the agent magically smart. The goal is to give it evidence.

A table comment might explain business meaning. A column tag might identify a sensitive field. An owner might point to the team responsible for validation. A declared primary or foreign key might suggest an important relationship. But suggest is doing important work here.

In Databricks, primary key, foreign key, and unique constraints are informational rather than enforced. They can describe intended relationships, but they don’t guarantee that the underlying data actually behaves correctly. Those relationships still need to be validated later during profiling. The same logic applies to comments and tags.

A missing sensitivity tag doesn’t mean a column is safe. An old comment does not deserve blind trust. An owner field may identify who created the table, not who currently owns the business meaning.

So the metadata reader shouldn’t dump raw catalog output into a prompt and ask the LLM to “figure it out.”

It should collect, normalize, and summarize metadata into something compact, traceable, and useful for the next steps of the agent workflow.

Because synthetic data generation doesn’t begin with generation. It begins with discovery.

---

Why Metadata Comes Before Data

Synthetic Data Needs Context, not Just Rows

A synthetic data agent shouldn’t start by sampling tables blindly. That’s how you end up with fake data that technically exists but behaves like it was raised by a spreadsheet and abandoned by statistics.

Before reading actual values, the agent needs structural context: which tables exist, what they appear to represent, how columns are named, which fields are documented, which columns might be identifiers, which objects are tagged as sensitive, which tables look like dimensions or facts, and which relationships have been declared.

Metadata doesn’t solve the synthetic data problem. But it tells the agent where to look, what to avoid, and what questions to ask next.

That’s why Unity Catalog is the right starting point. UC organizes governed data and AI assets through a hierarchical model. Data assets live in a three-level namespace such as catalog.schema.table; schemas can contain tables, views, volumes, functions, models, and other governed objects. That hierarchy also provides the foundation for access control.

For this agent, discovery should begin with the governed shape of the estate, not with a hardcoded list of tables copied from a Slack thread six months ago.

Metadata Reduces Randomness

The goal of the first tool is not to be clever. The goal is to be boring, repeatable, and useful.

The agent shouldn’t infer that customer_email is sensitive because the model “feels it deeply in its transformer soul.” It should first check the available evidence: column tags, table tags, comments, classification metadata, naming patterns, ownership information, catalog context, and declared constraints.

Databricks supports Unity Catalog tags as key-value attributes for organizing and categorizing securable objects, exposes column-level tag metadata through INFORMATION_SCHEMA.COLUMN_TAGS, and offers Data Classification capabilities that can automatically classify and tag sensitive data in UC tables. Those are useful signals when deciding which fields may need careful handling.

Useful, however, doesn’t mean complete.

• A missing tag doesn’t mean a field is safe.
• A comment doesn’t mean the documentation is current.
• A declared relationship doesn’t mean the data actually behaves correctly.

Metadata reduces randomness, but it doesn’t remove the need for validation.

LLMs are good at interpreting messy signals. They’re not that good at being the only source of truth.

So uc_metadata_reader gives the agent evidence before it gives it freedom.

---

What Unity Catalog Gives Us

A Governed Inventory, not a Random List of Tables

For uc_metadata_reader, Unity Catalog becomes the inventory source: catalogs, schemas, tables, views, columns, comments, tags, owners, privileges, and declared constraints.

The important shift is from location hunting to object discovery.

The tool shouldn’t depend on a hardcoded table reference like:

source_table = "prod.customer.final_v2_do_not_delete"

Instead, it should resolve the inventory from governed metadata:

Which objects are visible?
Which ones are in the approved scope?
Which tables look like candidate sources?
Which columns carry useful descriptions, sensitivity tags, ownership, or relationship hints?

Information Schema as the First Interface

The most practical starting point is Unity Catalog’s INFORMATION_SCHEMA.

Databricks describes information schema as a SQL-based, privilege-aware, self-describing API for accessing catalog metadata. That makes it a clean first interface for uc_metadata_reader: standard SQL, predictable relations, and no need to invent a custom metadata crawler on day one.

For this first version, the useful views include CATALOGS, SCHEMATA, TABLES, COLUMNS, TABLE_TAGS, COLUMN_TAGS, TABLE_CONSTRAINTS, KEY_COLUMN_USAGE, and related constraint views. Together, they let the tool collect the basic inventory the agent needs before it touches any actual data values.

Information schema filters results based on the querying user’s privileges. For example, INFORMATION_SCHEMA.TABLES returns only relations the user is privileged to interact with, and COLUMN_TAGS only shows column tagging metadata the user is allowed to access.

That’s exactly what we want. The metadata reader should help the agent discover the estate, not become a governance bypass wearing a friendly robot costume.

So the first implementation starts here: not with data sampling, not with profiling, and definitely not with LLM vibes. It starts with governed metadata.

---

The Tool We Are Building: uc_metadata_reader

Tool Responsibility

The first tool in the project is deliberately humble.

uc_metadata_reader doesn’t profile data. It doesn’t generate synthetic rows. It doesn’t call an LLM to hallucinate foreign keys because two columns both contain the word id.

Its responsibility is to create a clean, structured inventory of candidate source tables from governed Unity Catalog metadata. Databricks exposes this metadata through INFORMATION_SCHEMA, a SQL-based, privilege-aware API for accessing catalog metadata.

At a minimum, the tool should collect catalogs, schemas, tables, views, table types, owners, comments, columns, data types, nullability, column comments, tags, constraints, and relationship hints.

Some of this comes directly from INFORMATION_SCHEMA.TABLES, which contains object-level metadata for tables and views, and INFORMATION_SCHEMA.COLUMNS, which describes columns, positions, nullability, and data types. Tags and constraints come from related views such as TABLE_TAGS, COLUMN_TAGS, TABLE_CONSTRAINTS, KEY_COLUMN_USAGE, and REFERENTIAL_CONSTRAINTS.

That inventory becomes the first version of the agent’s working context.

Not memory as in: I remember your favorite pizza.
Memory as in: I have a structured representation of what this data estate contains, and I can use it without re-querying everything every five seconds like an anxious intern.

Inputs and Outputs

The tool should accept a small set of explicit inputs: a catalog allowlist, optional schema filters, optional table name patterns, and a maximum number of objects to inspect in one run.

Metadata discovery can become surprisingly large in enterprise environments. Since INFORMATION_SCHEMA automatically filters results according to Unity Catalog privileges, broad access can still produce a broad inventory. The first implementation should filter early by catalog, schema, and table patterns instead of scanning the entire visible universe just because it can.

The output should be boring JSON. Beautifully boring JSON.

{  
  "catalog": "main",  
  "schema": "sales",  
  "object_name": "customers",  
  "object_type": "BASE TABLE",  
  "owner": "data_engineering",  
  "comment": "Customer master table",  
  "columns": [  
    {  
      "name": "customer_id",  
      "type": "BIGINT",  
      "nullable": false,  
      "comment": "Unique customer identifier",  
      "tags": []  
    },  
    {  
      "name": "email",  
      "type": "STRING",  
      "nullable": true,  
      "comment": "Customer email address",  
      "tags": ["pii"]  
    }  
  ],  
  "constraints": [],  
  "relationship_hints": [],  
  "sensitivity_signals": ["email_column_name", "pii_tag"],  
  "metadata_warnings": ["no_declared_constraints"],  
  "agent_summary": "Customer master table with a likely identifier and email field. Sensitive signals were found on email. No declared constraints were found."  
}

The exact JSON shape can evolve. The principle should not.

Structured fields are for deterministic processing. The agent_summary is for reasoning. The warnings are for honesty.

That honesty matters because metadata is not verified data behavior. Databricks enforces NOT NULL and CHECK constraints, but primary key, foreign key, and unique constraints are informational and not enforced. So if the reader finds a relationship, that relationship is a useful hint, not proof.

---

Reading The Unity Catalog Hierarchy

Catalogs and Schemas

The first step is listing the catalogs and schemas visible to the agent’s execution identity. This is where discipline starts.

The agent shouldn’t automatically inspect every visible catalog. It should work from an allowlist, because visibility doesn’t always mean relevance.

In Unity Catalog, catalogs sit at the top of the object hierarchy, schemas sit inside catalogs, and schemas contain assets such as tables, views, volumes, functions, and models. This hierarchy is also the basis for UC’s three-level namespace: catalog.schema.table.

For example, dev, staging, prod, sandbox, and samples may all exist in the same environment.

Reading all of them without intent is how you accidentally mix serious production sources with someone’s abandoned Friday demo called test_customers_real_final.

A good first implementation should read visible catalogs, filter them through a configured allowlist, read schemas inside those catalogs, exclude known system, temporary, or irrelevant schemas where appropriate, and record what was skipped.

Skipping is not failure. Silent skipping is failure.

If the agent ignores a catalog because it is outside the allowlist, the output should say so. If a schema is excluded because it matches a sandbox pattern, that decision should be visible.

The answer should not be because vibes. It should be traceable configuration.

Tables, Views, and Columns

Once catalogs and schemas are selected, the tool reads tables, views, and columns.

INFORMATION_SCHEMA.TABLES contains object-level metadata for tables and views, while INFORMATION_SCHEMA.COLUMNS describes the columns of those objects. Databricks information schema is privilege-aware, and the returned metadata is filtered based on what the querying user is allowed to access.

This is the first place where the agent starts to understand shape.

A table with columns like customer_id, first_name, last_name, email, and signup_date tells a very different story from one with transaction_id, customer_id, amount, currency, and event_timestamp.

The metadata reader shouldn’t fully interpret that story yet. But it should preserve enough context so the next tools can.

That means keeping object names, column order, data types, nullability, comments, and table type together in a structured output. It also means distinguishing between base tables and views, because a view may already contain business logic, filters, joins, or masking behavior that the agent should not casually ignore.

At this stage, the agent is not profiling values. It’s reading the map. And maps are useful precisely because they are not the territory.

---

Reading Comments, Tags, And Owners

Comments are Helpful, When they Exist

Comments are one of the nicest forms of metadata because they contain human intent. They are also one of the most dangerous forms of metadata because they may contain human intent from 2021.

Unity Catalog supports comments on securable objects such as catalogs, schemas, tables, volumes, AI models, and more. Databricks also supports comments on table columns, including AI-assisted comment generation in Catalog Explorer. That makes comments valuable for the agent.

A column named status_cd becomes easier to interpret if the comment says “Customer lifecycle status: active, paused, closed.”

But comments should never be treated as guaranteed truth. They may be missing. They may be stale. They may be wrong. They may say “temporary field” on a column that has been powering executive dashboards for four years. Enterprise archaeology is not for the weak.

So uc_metadata_reader should collect comments, preserve them, and use them as interpretation signals. It should also flag missing or suspiciously thin documentation, because an empty comment is still useful information. It tells the agent not to pretend it understands too much.

Tags are Signals, not Magic

Tags are another key source of context. Unity Catalog tags are key-value attributes used to organize and categorize securable objects. Databricks exposes tag metadata through information schema views, including TABLE_TAGS for tables and COLUMN_TAGS for columns.

For this project, tags are especially useful for sensitivity detection.

• A column tagged as pii=email, classification=confidential, or domain=customer gives the agent a strong signal about how to handle it.
• A table tagged as source=crm or domain=sales can also help establish business context before deeper profiling begins.

But again: signal, not truth.

A missing PII tag doesn’t mean a field is safe. It may just mean nobody tagged it. Which is not a data governance strategy. It’s a lifestyle choice.

So the metadata reader should combine tags with other evidence: column names, comments, table names, schema names, known sensitive patterns, and later, actual profiling results. Tags should raise confidence.

They should not replace judgment.

Owners Help Route Responsibility

Owners matter because synthetic data generation is not just a technical exercise.

Sooner or later, the agent will need to say: I found this table. I think these fields are sensitive. I think these relationships matter. Someone should confirm this before I generate anything useful enough to be dangerous.

Owner metadata helps route that conversation. Unity Catalog has an ownership model for securable objects, and INFORMATION_SCHEMA.TABLES exposes owner information for relations. That means uc_metadata_reader can preserve the owner of each table or view alongside the rest of the inventory.

The first version doesn’t need a full approval workflow. But it should preserve ownership information so later parts of the system can build toward review, validation, and governance.

Because when synthetic data starts looking realistic enough to be useful, “who owns this?” stops being a metadata detail. It becomes a control point.

---

Reading Constraints And Relationship Hints

Declared Keys are Valuable

Synthetic data gets much harder when tables are related. Single-table fake data is a toy problem. Relational synthetic data is where things start getting spicy.

If orders.customer_id references customers.customer_id, the generator must preserve that relationship. Otherwise, it creates orphan records, broken joins, and dashboards that look fine until someone asks one basic question.

That’s why uc_metadata_reader should collect declared constraints when they exist.

Databricks exposes constraint metadata through Unity Catalog information schema views such as

• TABLE_CONSTRAINTS
• KEY_COLUMN_USAGE
• CONSTRAINT_COLUMN_USAGE
• CONSTRAINT_TABLE_USAGE
• REFERENTIAL_CONSTRAINTS

These views describe primary keys, foreign keys, unique constraints, and referential relationships, depending on what has been defined and what the querying principal is allowed to access.

That metadata becomes first-class evidence for the agent.

• A declared primary key can suggest the likely grain of a table.
• A declared foreign key can reveal parent-child relationships.
• A unique constraint can suggest identifier-like behavior. Referential constraints can help the agent understand how tables are expected to connect.

Expected is the key word. At this stage, the agent is not proving the relationship. It’s collecting the claim. And claims are useful, as long as nobody mistakes them for reality.

But Databricks Keys are Informational

In Databricks, primary key, foreign key, and unique constraints are informational only and are not enforced. They can describe intended relationships and support modeling or query optimization, but they don’t prevent invalid data from being written. Enforced constraints exist separately, such as NOT NULL and CHECK constraints.

That doesn’t make declared keys useless. It makes them honest.

A declared foreign key is a strong hint that two tables are supposed to be related. But the next profiling tool still has to validate whether the actual data behaves that way.

Because there is a big difference between: This column is declared as a foreign key.
And: This column actually behaves like a foreign key in production data.

The first one is metadata. The second one is evidence. We will need both.

So uc_metadata_reader should collect constraints, relationship hints, and key declarations, but mark them as unvalidated. Later, the profiling layer can check orphan rates, uniqueness, null behavior, cardinality, and join coverage.

That’s the right division of labor. Metadata tells the agent what the data model claims. Profiling tells the agent what the data actually does. And realistic synthetic data needs both.

---

Identifying Sensitive Fields

Use Every Signal Available

The metadata reader should flag possible sensitive fields early. Not because it can solve privacy. Because it can prevent the agent from being reckless.

The first implementation should combine every useful signal it can collect: column tags, table tags, column names, comments, table names, schema names, known classification labels, and Databricks Data Classification results when available.

Databricks Data Classification can automatically classify and tag sensitive data in Unity Catalog. It assigns system tags to matching columns for supported sensitive data types, and those tags can support governance workflows such as attribute-based access control.

That’s useful. But the metadata reader should still remain conservative. If a column is named email, ssn, passport_number, iban, phone, address, birth_date, national_id, or anything equally suspicious, the agent should flag it even if no formal tag exists.

Because “no tag found” is not the same as safe. It may mean the field is safe. It may also mean nobody tagged it, classification has not run, permissions hide the tag, naming is inconsistent, or the governance process is held together with dashboards, hope, and one heroic data steward.

So uc_metadata_reader shouldn’t produce a binary answer like sensitive: true and walk away proudly. It should produce sensitivity signals: pii_tag, email_column_name, passport_comment_match, customer_table_context, classification_tag, or missing_classification_metadata.

That gives the downstream agent something more useful than blind confidence. It gives it reasons.

Separate Detection from Policy

The tool shouldn’t decide final privacy behavior. It should detect sensitivity signals and pass them forward.

Later parts of the system can decide whether a field should be masked, excluded, tokenized, generalized, redacted, generated statistically, generated with format-preserving patterns, or blocked entirely.

The metadata reader answers: What do we know about this field?
It shouldn’t answer: Are we legally allowed to generate synthetic replacements for this field in this exact regulatory context?

That’s not a metadata reader. That’s a governance process. With lawyers nearby. And probably a meeting.

So the output of uc_metadata_reader should stay factual and traceable. It can say that a column has a sensitive classification tag, that its name matches a PII pattern, that its comment mentions personal data, or that no sensitivity metadata was found.

But it shouldn’t pretend that detection equals permission. That decision belongs to policy, approval, and governance layers built later in the system.

For now, the tool’s job is simpler: spot risk early, explain why it was flagged, and make sure the agent does not confuse missing metadata with safety.

---

Producing An Agent-readable Summary

The Agent does not Need Raw Metadata Soup

The output of uc_metadata_reader should be useful for software and for the LLM.

That means two layers: a structured metadata payload for deterministic processing, and a compact natural-language summary for reasoning.

The structured payload keeps precision. The natural-language summary gives the agent a digestible description of the table.

For example:

Table main.sales.customers appears to be a customer master table.  
It contains a likely primary identifier customer_id, personal attributes,  
contact fields, and lifecycle dates. Sensitive signals were detected on  
email, phone_number, and birth_date. No declared foreign keys were found.  
Comments are incomplete for 8 of 14 columns.

That’s much better than feeding the model hundreds of raw rows from INFORMATION_SCHEMA and asking it to “figure it out.”

Databricks exposes Unity Catalog metadata through information schema views, which is exactly what we want for collection. But raw metadata is an interface for querying, not a finished reasoning context for an agent. uc_metadata_reader should convert it into something smaller, cleaner, and easier to use downstream.

That’s not hiding detail. It’s packaging detail.

The structured payload should still contain table, column, tag, owner, comment, constraint, and sensitivity information. The summary is just the readable layer on top: what the object appears to be, which fields look important, what risks were detected, and what is missing.

In other words, not metadata soup. Metadata mise en place. Still nerdy. Much more useful.

Keep Summaries Compact and Traceable

Every generated summary should remain traceable back to the metadata that produced it.

If the agent says “this table may contain sensitive fields,” the payload should show why: tag match, column-name match, comment match, classification tag, declared constraint, or table context.

That traceability matters because Unity Catalog metadata can include many different signals. Databricks supports tags for organizing and categorizing securable objects, exposes metadata through information schema, and can classify and tag sensitive data through Data Classification when enabled. Those are valuable inputs, but the summary should never turn them into unexplained conclusions.

A bad summary says:

This table contains PII.

A better summary says:

This table may contain PII because email was detected by column-name pattern,  
phone_number has a sensitivity tag, and birth_date has a classification signal.

That difference is the difference between an agent that sounds confident and an agent that can be audited.

A good agent should be able to explain its metadata interpretation. Not poetically. Operationally.

So the summary should stay short, but the payload should keep the receipts. That way, later tools can profile the data, validate relationships, assess risk, and challenge assumptions instead of blindly trusting a polished paragraph generated by a model with excellent grammar and no legal accountability.

---

Designing The Implementation

Keep SQL Deterministic

The implementation should use deterministic SQL queries against Unity Catalog metadata.

The first version can be a Python module or notebook-backed job that runs parameterized SQL against INFORMATION_SCHEMA, normalizes the result, and writes a compact Delta table or JSON artifact for downstream use.

Databricks describes INFORMATION_SCHEMA as a SQL-based, privilege-aware API for accessing catalog metadata. It also automatically filters results to objects the querying user has Unity Catalog privileges to access, which makes it a good first interface for this tool.

The important design principle is repeatability.

Same inputs ✚ Same permissions ✚ Same metadata state ➔ Same output

That makes the tool testable, schedulable, deployable, and suitable for the bundle structure created in SDA 03. Declarative Automation Bundles, define Databricks projects as source-controlled files, including resources such as jobs, pipelines, dashboards, serving endpoints, experiments, and tests.

So the metadata reader should behave like infrastructure-friendly software, not like an exploratory notebook that only works when the author is emotionally available.

No hidden widgets. No manual table names buried in cell 17. No “run this part first unless it is Tuesday.”

Just explicit parameters, deterministic SQL, normalized output, and logs clear enough that future-you does not start negotiating with the universe.

Suggested Project Placement

Inside the project, the tool should live as a small package component rather than a heroic notebook.

A practical layout could be:

src/  
  sda/  
    tools/  
      uc_metadata_reader.py  
    models/  
      metadata.py  
    utils/  
      sql.py  
  
resources/  
  jobs/  
    uc_metadata_reader_job.yml  
  
tests/  
  test_uc_metadata_reader.py

This keeps the metadata reader close to the rest of the agent tools while still allowing it to be deployed as a Databricks job. Bundles support Databricks resource definitions as configuration, so the reader can be packaged and executed through the same deployment approach used by the rest of the project.

The notebook can exist if it helps with exploration, demos, or debugging. But the logic shouldn’t live only in the notebook.

The reusable part belongs in code. The job definition belongs in bundle resources. The tests belong in the test suite. The configuration belongs outside the implementation, where it can change without rewriting the tool.

That separation keeps the first tool small, boring, and useful. Which is exactly the point. We are trying to build an agent platform, not a museum of cells.

---

What The First Version Should Not Do

No Profiling Yet

The first version shouldn’t profile actual data values.

No distributions. No null percentages. No min/max dates. No cardinality analysis. No relationship validation.

That comes next. This tool only reads metadata.

That boundary keeps the first deliverable small, testable, and easier to secure. Databricks INFORMATION_SCHEMA is designed as a SQL-based, privilege-aware API for catalog metadata, and its results are filtered to objects the querying user has Unity Catalog permissions to access. That makes it a good fit for discovery, but discovery is not the same as reading table contents.

The agent shouldn’t blur that line casually. Reading metadata can tell us that a table exists, that a column is called email, that a comment says “customer contact address,” or that a relationship has been declared.

It shouldn’t calculate how many distinct emails exist. That belongs to the profiling tool, where permissions, sampling strategy, cost, privacy handling, and validation rules can be designed deliberately instead of smuggled into the metadata reader like a raccoon in a trench coat.

No Automatic Trust

The tool should also avoid pretending metadata is complete. This is the honest caveat of the whole article: metadata is usually incomplete.

Sometimes comments are missing. Sometimes tags are wrong. Sometimes owners are outdated. Sometimes constraints exist only in PowerPoint.

The agent must treat metadata as helpful evidence, not absolute truth.

This is especially important for relationships. Databricks supports primary key, foreign key, and unique constraints as informational metadata, but they are not enforced. They can describe intended relationships and may help with modeling or optimization, but they do not guarantee that the data actually respects those relationships. Enforced constraints exist separately, such as NOT NULL and CHECK.

So the output should include confidence indicators or warning flags: missing comments, no tags found, no constraints found, possible sensitive fields without tags, declared keys requiring validation, and tables with unclear business meaning.

Because a useful agent is not one that always sounds confident. A useful agent knows when to say: I found something, but someone should probably check this before we build synthetic reality on top of it.

That is not weakness. That is the difference between an agent and a very expensive autocomplete with access to production metadata.

---

Deliverable: uc_metadata_reader

What “DONE” Means

By the end of this article, the project has its first real tool: uc_metadata_reader.

Not a generator. Not a profiler. Not an LLM wearing a data engineer costume. A metadata reader.

The first version should be able to read allowed catalogs and schemas, inventory tables and views, collect column metadata, extract comments and tags, capture owner information, read declared constraints, flag possible sensitive fields, and produce both structured JSON and compact agent-readable summaries.

The definition of done is deliberately practical: the tool runs with explicit configuration, respects the execution identity’s permissions, records what it inspected and skipped, separates raw metadata from interpreted signals, keeps sensitivity and relationship findings traceable, and does not pretend metadata is more complete than it really is.

The output of uc_metadata_reader is not the final truth about the data estate. It’s the first map.

Some roads may be missing. Some labels may be wrong. Some bridges may exist only in someone’s architecture diagram from 2022.

Still useful. Still necessary. Still much better than asking the agent to discover production data through vibes and heroic confidence.

At this point, there’s still no synthetic data. Still no profiling. Still no fake customers called Test McTestface. But now the agent can see the shape of the estate. And that changes everything.

---

From metadata to reality

At this point, the Synthetic Data Agent still has not generated a single row.

Good. That’s not a failure. That’s restraint.

We now have a deployable project, a first concrete tool, and a metadata inventory that gives the agent a governed view of the data estate. Unity Catalog provides the governance layer for Databricks data and AI assets, including access control, lineage, discovery, and auditing, while INFORMATION_SCHEMA gives us a privilege-aware way to read catalog metadata.

The agent can now see catalogs, schemas, tables, columns, comments, tags, owners, constraints, and sensitivity signals before touching actual data.

The next step is where things get more interesting: profiling the source data itself.

That means moving from: What does Unity Catalog say this data is?
To: How does this data actually behave?

Because metadata tells us where to look. Profiling tells us what is real. And realistic synthetic data needs both.

---

UC metadata gives the agent its first governed map of the data estate before profiling or generation begins.

---

🔗 Code & Music 🎧

The code for this article is available here:

🔗 synthetic-data-agent

And because metadata discovery clearly needed a soundtrack, the Synthetic Data Agent now has its own songs too:

🎧 YouTube playlist

Read the article. Check the repo. Play the songs.

---

SDA 04 — Smart From The Start

[Verse 1]

I came up where the schemas split
Metadata stacked like a blueprint kit
Cold code, sharp mind, got the whole stack lit
One tool call deep, make your whole flow quit

Catalog maps on a governed block
Build truth from stats, never guess the clock
Every scan got weight, every graph got shock
When the validator speaks, all the shortcuts stop

They want random? Nah, I want structure
Want fake? Nah, I preserve the texture
Agent runs clean with a real dispatcher
Reason first, tools do the pressure

[Pre-Chorus]

Hear that workflow start
Watch the engines spark
One more scan, then chart
Build it smart from the start

[Chorus]

Synthetic Data Agent
Synthetic Data Agent
Build it right, let the metrics paint it
Synthetic Data Agent (S-D-A)

Synthetic Data Agent
Synthetic Data Agent
Hands up when the tools orchestrate it
Synthetic Data Agent

[Verse 2]

Read the Catalog, every schema mapped
Columns tell stories where the comments sat
Primary hints but the facts get checked
Validation first, earn every respect

No wizard tricks, every tool got purpose
Profiler digging underneath the surface
Relationship graph when the joins emerge
Generation plans, then the engines surge

Bundle deploy, every target aligned
Dev to prod with a disciplined mind
State remembers what the last run found
Every checkpoint leaves another round

[Pre-Chorus]

Hear that workflow start
Watch the engines spark
One more scan, then chart
Build it smart from the start

[Chorus]

Synthetic Data Agent
Synthetic Data Agent
Build it right, let the metrics paint it
Synthetic Data Agent (S-D-A)

Synthetic Data Agent
Synthetic Data Agent
Hands up when the tools orchestrate it
Synthetic Data Agent

[Bridge]

[Scratch Break]

Read first, read first
Profile, profile
No guess, all proof
Real workflows, real truth

Route tools, validate
Govern, generate
Reason, calculate
Never fabricate

[Final Chorus]

Synthetic Data Agent
Synthetic Data Agent
Built from facts, no one can fake it
Synthetic Data Agent (S-D-A)

Synthetic Data Agent
Synthetic Data Agent
From Unity Catalog to validated data
Synthetic Data Agent