🤖 SDA 05: Profiling Source Data — Distributions, Nulls, Formats, and Outliers

What Profiling Adds That Metadata Cannot

From Structure to Behavior

uc_metadata_reader gave the agent a governed map of the data estate: tables, columns, declared types, nullability, comments, tags, and constraints. Unity Catalog’s information schema exposes this structural metadata, but it describes the data model—not the values currently stored inside it.

• A column declared as nullable may contain no nulls at all — or almost nothing else.
• A STRING may hold country codes, emails, JSON, or several formats invented during separate migrations.
• A DECIMAL type defines what can be stored, but not whether the values are concentrated around zero, heavily skewed, or dominated by a long tail.

Profiling replaces those assumptions with measurements: actual null rates, distributions, categorical frequencies, string formats, temporal coverage, freshness, and unusual values.

📌 Schema realism tells us what values are legal. Statistical realism tells us what values are plausible.

A Preview is not a Profile

A twenty-row preview is useful for orientation. It’s useless as statistical evidence.

• Rare categories may not appear.
• Long tails may be invisible.
• Null rates can look completely different from the full table.
• Date ranges, skew, and category weights cannot be inferred reliably by scrolling through a handful of attractive rows and nodding thoughtfully.

Those facts must be calculated through deterministic Spark or SQL aggregations. Spark provides operations for grouped frequencies, approximate distinct counts, and configurable approximate percentiles, allowing the profiler to measure large tables efficiently while recording which results are estimated.

The result should be a structured, reproducible profile — not an LLM impression of a preview. The boundary remains simple:

📌 Code calculates what the data does. The agent reasons about what those results mean.

---

Defining The table_profiler Contract

Inputs: Explicit Scope, no Surprise Archaeology

table_profiler should accept a small, validated request rather than treating every visible table as an invitation.

Its inputs should include:

• The fully qualified Unity Catalog name: catalog.schema.table
• An optional column allowlist or denylist
• A profiling mode such as quick or full
• A sampling percentage and reproducible seed
• Limits for retained categorical values
• Configurable outlier thresholds
• An optional business-event timestamp
• Optional segmentation columns for conditional-null analysis

Unity Catalog uses a three-level namespace, so resolving the complete table name avoids accidentally profiling an identically named object in the wrong environment.

The tool should reuse the inventory produced by uc_metadata_reader: column names, declared types, nullability, tags, comments, and sensitivity signals. It can then assign the appropriate strategy to each column—numeric, categorical, textual, temporal, or unsupported—without rediscovering the schema.

Scope must remain intentional. Visibility is not authorization to inspect everything, and sensitivity signals may require columns to be excluded, summarized differently, or blocked by policy. The profiler should process only the requested table and approved columns, then record anything it skipped.

Outputs: Structured Evidence with Receipts

The result should be a versioned, machine-readable profile containing:

• The source table and, where supported, its snapshot or table version
• Profiling time, mode, and configuration
• Source row and column counts
• Metrics for every supported column
• Whether each metric is exact, approximate, sampled, or metadata-derived
• Sample percentage and seed
• Freshness indicators
• Warnings, skipped columns, and unsupported types
• A compact summary for the agent

For Delta tables, recording the table version provides a reproducible reference to the data that was profiled. Databricks table history and time travel support querying versions recorded in the transaction log, subject to the configured retention period.

Persist the canonical profile in governed Delta tables so it can be queried, versioned, compared, and reused by later jobs. A compact JSON representation can be returned to the agent or passed between workflow tasks.

The profile should store measurements, not an accidental copy of production data. Sensitive and high-cardinality strings should normally be represented through lengths, format signatures, cardinalities, frequency bands, or protected values — not unrestricted value lists or raw samples.

📌 Realistic statistics are useful. A convenient collection of production emails is rather less charming.

Explicit Boundaries

table_profiler measures how one table behaves. It doesn’t infer primary or foreign keys, validate joins across tables, discover broad cross-column dependencies, declare business rules, decide privacy policy, or generate synthetic records.

It may surface useful clues — a nearly unique column, a conditional-null pattern, or two fields that deserve joint analysis — but those remain evidence for later tools, not conclusions.

That boundary is deliberate. SDA 06 handles keys and relationships. SDA 07 handles cross-column patterns and business rules.

📌 table_profiler should give both of them reliable evidence without quietly expanding beyond its job.

---

Profile According To Data Type

There should be no universal profiling query applied blindly to every column. A transaction amount, status code, customer identifier, and event timestamp may share a storage type while representing completely different behavior.

table_profiler should therefore classify each column and dispatch a type-specific strategy. Declared type is the starting point, not the final decision: an integer with nearly one distinct value per row is probably an identifier, while an integer containing only 0, 1, and 2 is probably categorical.

Numerical Distributions

For genuinely numerical columns — integers, floating-point values, and decimals — the profile should capture:

• Non-null count and null rate
• Minimum, maximum, mean, and standard deviation
• Percentiles such as p01, p05, p25, p50, p75, p95, and p99
• Skewness
• Zero, negative, and positive rates where meaningful
• Concentration around repeated values
• Histogram or quantile-bin counts
• NaN and positive or negative infinity counts for floating-point columns
• Candidate outlier counts and the method used to detect them

Percentiles matter because averages hide shape. Two columns can have the same mean while one is tightly concentrated and the other has a long, expensive tail. Spark’s approximate percentile functions accept a configurable accuracy parameter, trading additional memory for greater precision, so the profile must record that the result is approximate and which configuration produced it.

Minimum and maximum alone are generation instructions written by chaos. Uniformly sampling between them would flatten skew, remove peaks, and give every long-tailed amount column the suspicious emotional stability of demo data.

For example, transactions.amount might have:

• A median near 42
• A p95 near 310
• A small tail above 5,000
• A strong concentration of prices ending in .99
• A few negative adjustments

That’s reusable generation knowledge.

“Decimal between -10,000 and 10,000” is merely a type with ambition.

Categorical Frequencies

Booleans, low-cardinality strings, and low-cardinality numeric columns should be treated as categorical. The profiler should collect:

• Exact or approximate cardinality
• The most frequent categories and their counts and percentages
• The share represented by the dominant value
• Cumulative frequency covered by the retained categories
• Null, blank, and whitespace-only rates
• The count and combined weight of rare categories
• Original versus normalized category counts when trimming or case normalization is useful

Spark provides exact distinct counting and approx_count_distinct, which estimates cardinality with a configurable relative standard deviation. Approximate values should be labelled as estimates rather than quietly presented as facts carved into stone.

High-cardinality columns require different treatment. Retaining every value from a million-row customer identifier is expensive, unsafe, and not particularly enlightening. For those columns, store cardinality, uniqueness ratio, duplication concentration, and perhaps protected pattern information. Then classify the field as identifier-like or free-form rather than pretending it is a useful category.

📌 Preserving the category list is not enough. We need its weights, dominant values, and long tail.

String Lengths and Formats

String profiling should describe structure without turning the profile store into a convenient replica of production text.

Useful measurements include:

• Minimum, maximum, average, and percentile lengths
• Null, empty, and whitespace-only rates
• Leading or trailing whitespace
• Uppercase, lowercase, title-case, and mixed-case frequencies
• Digit-only, alphabetic, alphanumeric, and punctuation-heavy rates
• Common character-class signatures
• Safe prefixes or suffixes
• Parsing success for expected formats
• Common format patterns rather than unrestricted real values

A value can be transformed into a normalized signature such as:

ABC-482193  ➜  AAA-999999  
555-12-9876 ➜  999-99-9999

Email-like values might be summarized as lowercase local text followed by @ and a domain-shaped suffix. Internal codes might be described as two letters, four digits, and a final check character.

These signatures allow the future generator to reproduce structural realism without copying real emails, phone numbers, account codes, or identifiers.

Format discovery remains heuristic. The profiler may report that 96% of values match a signature. It should not declare that signature a business rule, because the remaining 4% may represent valid exceptions, legacy formats, corruption, or the combined creativity of three migrations and an Excel upload.

Dates and Timestamps

Temporal profiling must capture behavior, not just the earliest and latest values.

For date and timestamp columns, collect:

• Minimum, maximum, and null rate
• Values outside configurable reasonable boundaries
• Future-date rate
• Counts by year, month, weekday, and hour where relevant
• Activity in recent time windows
• Long gaps and inactive periods
• Concentration at midnight or other default times
• Common intervals or lags where meaningful
• The declared timestamp type and profiling session timezone

In Databricks, TIMESTAMP—also referred to as TIMESTAMP_LTZ—uses session-local timezone semantics. Extracted components such as year or hour therefore depend on the session timezone. TIMESTAMP_NTZ, by contrast, represents a local date and time without timezone semantics. Recording both the source type and the session timezone is necessary if hourly or daily patterns are expected to be reproducible.

The resulting profile might reveal that:

• Most transactions occur during business hours
• Weekend activity is much lower
• Volume spikes near month end
• Thousands of records use exactly 00:00:00
• Recent activity stopped three weeks ago
• Twelve events apparently occurred in 2099, because the future has already started invoicing us

Those findings matter later when generating seasonality, realistic operating windows, stale periods, and controlled temporal anomalies.

Complex and Unsupported Types

Arrays, maps, structs, variants, binary values, and large free-text fields should not be flattened into strings and pushed through the primitive-column profiler.

Databricks supports nested and complex types including ARRAY, MAP, STRUCT, and VARIANT, but each requires its own traversal and aggregation strategy.

For version one, the profiler should:

• Fully profile supported primitive columns
• Report null rates and basic size metrics for arrays and maps
• Optionally profile explicitly selected nested fields
• Record schema depth and element types
• Flag binary, large-text, and unsupported structures clearly
• Never silently cast complex values to strings

This keeps the first implementation useful without pretending that every semi-structured enterprise payload can be understood by one heroic aggregation query.

---

Nulls Are Data, Too

Missing does not Always Mean NULL

A useful profiler should not reduce missingness to value IS NULL.

Depending on the column, absence may appear as:

• SQL NULL
• Empty or whitespace-only strings
• NaN in floating-point data
• Sentinel values such as -1, UNKNOWN, N/A, or 1900-01-01
• Values that fail an expected parse

These aren’t equivalent. A sentinel may be a documented business value, a legacy default, or a data-quality defect. NaN is also distinct from SQL NULL, while an empty string is still a string.

table_profiler should therefore report each pattern separately, along with the rule used to detect it. Interpretation comes later. Otherwise, an innocent -1 category may be cleaned into a null—or a fake date from 1900 may be treated as meaningful history.

Overall Null Rates can Hide the Real Pattern

A table-wide null percentage is useful, but often incomplete.

cancellation_timestamp may be almost always null for active orders and usually populated for cancelled ones. closed_at may be empty while an account remains open. A tax identifier may be missing mostly for certain countries or customer types.

The first version of table_profiler should therefore support limited conditional-null summaries using explicitly configured or low-cardinality segmentation columns.

For example: cancellation_timestamp is null in 99.8% of non-cancelled rows and 2.1% of cancelled rows.

That is evidence of a dependency, not yet a business rule. The profiler measures the pattern; later tools determine whether it is stable, meaningful, and suitable for generation.

📌 The boundary remains simple: Profiling tells us where values are missing.
Pattern detection tells us why.

---

Outliers Are Not Automatically Mistakes

Use more than one Signal

An outlier is not a fact until the method is defined.

table_profiler should combine several signals:

• Percentile thresholds
• Interquartile-range fences
• Robust measures such as median absolute deviation
• Configured domain limits
• Unusual string lengths
• Impossible or future dates
• Extremely rare categories

Mean plus three standard deviations can be useful for roughly symmetric distributions, but it performs poorly on skewed or heavy-tailed data and is itself influenced by extreme values. Robust methods and domain checks usually provide better evidence.

Every finding should therefore record the method, threshold, and affected-row count. Outlier without methodology is just a judgmental label attached to a value.

Classify, do not Delete

The profiler should identify unusual values, not repair them.

An extreme observation may represent legitimate tail behavior, a rare business event, a technical default, a data-entry error, or a value requiring human review. Those cases should not be treated as equivalent.

This distinction matters later. Removing legitimate tails produces synthetic data that is clean but unrealistic. Reproducing every anomaly can preserve defects, sensitive combinations, or meaningless defaults.

table_profiler provides the evidence. The generation plan later decides whether each pattern should be preserved, capped, transformed, deliberately injected as noise, or excluded.

---

Data Freshness Has Two Clocks

Storage Freshness

For Delta Lake and supported Apache Iceberg tables, each modification creates a new table version. DESCRIBE HISTORY returns recent operations, timestamps, and versions in reverse chronological order.

The profiler should distinguish the latest commit from the latest data-changing write. Maintenance operations such as optimization may create newer history entries without adding newer business records.

Record:

• Latest table commit
• Latest data-changing operation where identifiable
• Source version
• Profile execution time
• Elapsed time since the relevant write

Do not use INFORMATION_SCHEMA.TABLES.LAST_ALTERED as a data-freshness measure. It records when the relation definition was last changed, not when its rows or business events were last updated.

Business-event Freshness

The second clock comes from the data itself: usually the maximum value of a configured event-time column.

A table may have been rewritten this morning while containing transactions only through last month. Conversely, an old table definition may receive new events continuously.

The profiler should capture:

• Maximum business timestamp
• Lag from profiling time
• Row counts in recent windows
• Unexpected future events
• Empty or unusually quiet periods

📌 Table freshness is not one timestamp. One clock describes when storage changed. The other describes how recent the business data is.

Both are useful, but neither is perfect: late-arriving records, backfills, and incorrect timestamps can distort the picture. Still, ignoring freshness entirely is worse — a generator trained on stale behavior will produce beautifully realistic history.

---

Profiling Without Scanning The Budget Into Another Dimension

Profiling every row, column, and category on every run is thorough in the same way that setting money on fire is thorough. table_profiler needs two execution modes.

Quick and Full Profiles

-A quick profile supports exploration and agent planning. It uses a repeatable percentage sample, approximate cardinalities and percentiles, capped category frequencies, and basic null, length, range, and freshness checks under strict runtime limits.

A full profile creates the baseline used for approved generation. It scans the complete table where required, examines tails and rare categories more deeply, calculates conditional-null summaries, performs complete freshness checks, and persists a versioned result. “Full” does not require every calculation to be exact; scalable approximations may still be appropriate, provided they are clearly identified.

Databricks TABLESAMPLE supports percentage, row-count, and bucket sampling, with REPEATABLE(seed) for reproducible results. For statistically random sampling, percentage sampling should be preferred: row-count sampling is implemented using LIMIT, not as a simple random sample. Sampling can also miss rare categories and extreme values, so its configuration must be stored and its results must never be presented as full-table facts.

Exact, Approximate, and Sampled are Different Claims

Every metric should record how it was produced:

• exact: calculated from all relevant rows
• approximate: calculated across the full input with an approximation algorithm
• sampled: calculated from a subset of rows
• metadata-derived: reused from table or catalog metadata

This prevents the agent from treating a sampled null rate or estimated cardinality as equivalent to an exact baseline. Spark provides approximate distinct-count and percentile functions specifically for scalable aggregation, and their accuracy parameters should be retained with the profile.

Reuse Work Instead of Profiling Repeatedly

The profiler should minimize scans by combining compatible aggregations, avoiding one Spark action per column, limiting retained categories, and using approximate distinct counts and percentiles where appropriate. Profiles should be versioned and reused until the source version or profiling configuration changes, with quick and full results stored separately.

Existing optimizer statistics may also help. ANALYZE TABLE … COMPUTE STATISTICS collects table row count and size, while column analysis can collect values such as minimum, maximum, null count, distinct count, and average or maximum column length. These statistics are designed for query optimization, however—not synthetic-data modelling. They do not replace percentiles, format signatures, conditional nulls, freshness, or explicit outlier evidence.

📌 Optimizer statistics help Databricks plan queries. table_profiler helps the agent model reality.

---

Building table_profiler Into The Project

Keep Profiling Logic out of the Notebook

table_profiler should extend the package structure created in SDA 03, not become another notebook-shaped subsystem. Core profiling logic belongs in importable Python modules; the workflow task should only validate parameters, invoke the package, persist the result, and report status. That keeps the implementation testable without requiring a live interactive notebook.

A practical layout is:

src/  
└── sda/  
    ├── tools/  
    │   ├── uc_metadata_reader.py  
    │   └── table_profiler.py  
    ├── profiling/  
    │   ├── numeric.py  
    │   ├── categorical.py  
    │   ├── strings.py  
    │   ├── temporal.py  
    │   └── missingness.py  
    └── models/  
        └── profile.py  
bundle/  
└── resources/  
    └── table_profiler_job.yml  
notebooks/  
└── run_table_profiler.py  
tests/  
├── test_numeric_profiler.py  
├── test_categorical_profiler.py  
├── test_string_profiler.py  
├── test_temporal_profiler.py  
└── test_missingness_profiler.py

The exact resource folder is a project convention — the bundle include configuration determines which YAML files are loaded—but it should remain consistent with the repository established earlier.

table_profiler.py coordinates the profiling modules. Each module owns one family of calculations, while profile.py defines the versioned output contract. The notebook remains a thin execution wrapper.

📌 Notebooks call the profiler. They do not contain the profiler.

Execution Flow

The implementation should follow a predictable pipeline:

Read metadata inventory  
        ↓  
Validate table, permissions, and columns  
        ↓  
Classify columns by profiling strategy  
        ↓  
Select quick or full mode  
        ↓  
Run table-level and type-specific aggregations  
        ↓  
Attach computation methods, warnings, and provenance  
        ↓  
Persist the versioned profile  
        ↓  
Return a compact summary to the agent

The profiler should begin with the output of uc_metadata_reader, reusing declared types, tags, comments, and sensitivity signals instead of rediscovering them.

Validation happens before data access: confirm the fully qualified table, approved columns, supported types, sampling configuration, and output destination. After execution, every metric should retain enough provenance to explain how it was produced — source snapshot, mode, sample seed, thresholds, and whether the result was exact, approximate, or sampled.

The agent remains outside the calculations. Its role is to select the source, request an allowed profiling mode, review warnings, and decide which tool runs next. Deterministic tools calculate facts; the agent interprets them.

Test the Awkward Cases

The profiler should be tested with small, deterministic fixtures containing:

• Skewed and long-tailed numerical values
• Nulls, NaN, and sentinel values
• Dominant and rare categories
• Blank and whitespace-only strings
• Multiple string formats
• Stale and future timestamps
• Conditional-null patterns
• Empty tables and all-null columns
• Arrays, maps, structs, and unsupported types

Tests should verify both the metrics and the warnings. An unsupported struct that produces a clear warning is correct behavior. A struct silently cast to text is not.

Reproducibility also needs an explicit test. The same source snapshot, configuration, thresholds, and sampling method should produce the same profile. A fixed random seed helps, but strict reproducibility may still depend on stable input and execution behavior. When repeatable row selection matters, deterministic hash-based sampling over a stable key is safer than relying only on random sampling.

The goal is not merely to prove that the profiler runs. It is to prove that it produces predictable evidence — even when the source table is empty, malformed, skewed, or apparently designed by someone with strong feelings about NULL.

---

What The Profile Becomes

The profile is not just a report. It becomes reusable generation knowledge.

Later tools can use it to sample numerical values from realistic regions, assign categories with observed weights, reproduce null behavior, generate strings from format signatures, preserve temporal ranges and seasonality, and decide whether tail behavior should be retained, capped, or excluded. The same profile also becomes the baseline against which synthetic output is validated.

That persistence matters beyond generation. Future dashboards and Genie Spaces can compare source and synthetic percentiles, null rates, category weights, formats, freshness, and warnings only because the original measurements were stored in a structured form.

📌 The profile therefore serves two purposes: It tells the generator what to reproduce. It tells the validator what to verify.

---

What The Profile Becomes

The output of table_profiler is not a one-off report that someone opens, nods at, and forgets in a workspace folder.

It becomes a reusable description of how the source table behaves.

That distinction matters. The profile is the bridge between observation, generation, and validation. It records what the source looked like at a specific point in time, how those measurements were calculated, and which patterns later tools should preserve or challenge.

Generation Knowledge

The future generator should not inspect the source table again every time it needs to create data. It should consume the approved profile.

For numerical columns, it can use percentiles, histogram bins, repeated-value concentrations, and tail measurements to generate values from realistic regions instead of sampling uniformly between min and max.

For categorical fields, it can reproduce dominant values, rare categories, and their observed weights. A status column that is 92% ACTIVE, 6% SUSPENDED, and 2% CLOSED should not become three equally likely options simply because all three values exist.

For strings, the generator can use length distributions and format signatures to produce values that look structurally valid without copying source content. Customer codes can retain their prefix and digit pattern. Emails can preserve casing and domain structure. Phone numbers can follow observed masks.

Temporal profiles provide similar guidance. The generator can preserve date ranges, business-hour activity, weekday patterns, month-end spikes, and periods of inactivity. Null profiles tell it not only how often values are missing, but, where measured, under which segments that missingness occurs.

Outlier evidence also becomes part of the plan. Legitimate tails may need to be preserved. Suspicious defaults may need to be excluded. Rare anomalies may be reserved for a later noisy-data mode.

The profile does not make those policy decisions itself. It gives the generation planner enough evidence to make them explicitly.

A Baseline For Validation

The same measurements used to guide generation should also be used to judge the result.

After synthetic data is created, the validator can compare it with the source profile:

Did the null rate remain within tolerance?
Are categorical weights still similar?
Did the median and upper percentiles drift?
Are string lengths and format signatures preserved?
Does temporal activity still follow the expected shape?
Were rare values removed, exaggerated, or reproduced as planned?

Without a persisted source profile, validation becomes subjective. Someone looks at a histogram, says it seems reasonable, and the dataset is promoted through the ancient enterprise quality gate known as “looks fine to me.”

A versioned profile gives the validator a fixed baseline. It also makes results reproducible: the system can show which source version was measured, which profiling mode was used, and which thresholds were later applied.

A Reusable Platform Asset

Persisting profiles also prevents repeated scans of the same source tables.

The agent can check whether a valid profile already exists for the current source version and configuration. If it does, later planning, generation, and validation runs can reuse it. If the source changes, the system can create a new profile version and compare it with the previous one to detect drift.

That history will later support dashboards and Genie exploration. Users will be able to inspect how source distributions changed, which synthetic runs used which profile, where null rates drifted, or why a validation check failed.

The dashboards and Genie Spaces come later. The important step now is storing the evidence they will depend on.

📌 The profile tells the generator what to reproduce. It tells the validator what to verify. And it tells the platform what changed.

---

Deliverable: table_profiler

By the end of this article, the project should have its second deterministic tool: table_profiler.

It consumes the inventory created by uc_metadata_reader, validates the requested Unity Catalog table and approved columns, and dispatches the correct profiling strategy for numerical, categorical, textual, and temporal data. It measures distributions, frequencies, formats, missingness, outliers, and freshness through quick or full execution modes.

Each result should include a profile_id, schema version, source reference, configuration, and computation method. Metrics must be labelled as exact, approximate, sampled, or metadata-derived so later tools know how much confidence to place in them.

The canonical profile should be persisted as governed Delta data, with a compact JSON representation returned to the agent. Delta tables create a new version when written, but the profile should still carry its own explicit version and source snapshot reference rather than relying only on storage history.

By default, the artifact should contain measurements — not raw sensitive values. High-cardinality or sensitive strings should be represented through counts, length distributions, format signatures, and protected summaries. Unsupported types, skipped columns, suspicious patterns, and incomplete calculations should produce visible warnings rather than disappearing quietly.

Finally, the profiler must run through the existing bundle-managed job workflow. Declarative Automation Bundles can define, validate, deploy, and execute Databricks jobs, so this tool should follow the same deployment path established for the rest of the project.

The definition of done is simple:

• One approved table can be profiled reproducibly
• Its behavior is stored as reusable, versioned evidence
• The agent receives a compact explanation and explicit warnings
• No raw-data preview is mistaken for statistical truth

Still no generator. But now the system has something far more useful than random values: a measurable description of reality.

---

Honest Caveat: Good Columns Can Still Make A Bad Table

Column-level profiling is necessary, but it is not enough. A synthetic table can match every individual distribution and still make no sense. Cancelled orders may lack cancellation timestamps. Premium customers may behave exactly like basic customers. Transaction amounts may ignore product type. Lifecycle dates may occur in impossible sequences. Child records may reference parents that do not exist.

The problem is that table_profiler measures mostly marginal distributions: how each column behaves in isolation. It can describe the distribution of amount, the frequency of status, and the null rate of cancelled_at, but not fully explain how those fields depend on one another.

Realism has several separate layers:

• Statistical realism: individual distributions, frequencies, null rates, and formats.
• Cross-column realism: conditional behavior and correlations within a row.
• Relational realism: keys, joins, cardinality, and parent-child integrity.
• Temporal realism: valid event order and lifecycle dependencies.
• Business realism: rules that may not be discoverable from statistics alone.

This is why a collection of perfectly profiled columns does not automatically produce a realistic dataset. The profile gives the system the statistical shape of reality. Later tools must recover the structure connecting it.

The agent now knows how each column behaves on its own. It understands ranges, frequencies, nulls, formats, tails, and timestamps.

What it still does not know is how the tables fit together. A perfectly profiled customer_id is not useful if half the synthetic orders point to customers who do not exist.

So next, we move from statistical shape to relational structure: declared keys, inferred joins, uniqueness, value overlap, cardinality, and parent-child generation order.

📌 Because realistic columns are useful. Realistic data systems need relationships.

---

From Columns to Connections

The Synthetic Data Agent still has not generated a single row. Good.

It now has something more useful than a random-data script: a versioned, measurable profile of how a source table actually behaves. table_profiler turns Unity Catalog tables into structured evidence about distributions, frequencies, nulls, formats, outliers, timestamps, and freshness—without asking the model to calculate statistics from a handful of rows and optimism.

That profile becomes the baseline for everything that follows. The generator will use it to reproduce realistic behavior. The validator will use it to measure drift. Later dashboards and Genie Spaces will use it to explain what changed and whether the synthetic output remains credible.

But column-level realism is only one layer. The agent still does not know which fields behave as keys, which tables depend on others, how joins behave, or whether parent-child relationships are valid.

That is the next step. In SDA 06, we move from profiling individual columns to detecting primary keys, foreign keys, value overlap, join cardinality, and generation order.

Because realistic values are useful. Realistic data systems need connections.

---

table_profiler finally teaches the agent what production data actually looks like: skewed, nullable, oddly formatted, occasionally from 2099, and full of surprises. Still no synthetic rows—first measure the chaos, then fake it properly.

---

🔗 Code & Music 🎧

The code for this article is available here:

🔗 synthetic-data-agent

And because metadata discovery clearly needed a soundtrack, the Synthetic Data Agent has its own songs too:

🎧 YouTube playlist

Read the article. Check the repo. Play the songs.