🤖 SDA 03: Bootstrapping with Declarative Automation Bundles

Why Bootstrap Before Building the Agent?

With the goal from SDA 01 and the architecture from SDA 02 in place, we can give the project a proper engineering home. Before giving the agent a brain, we need a skeleton: boring, structured, and version-controlled.

The agent will eventually inspect Unity Catalog tables, profile data, infer distributions, understand relationships, generate synthetic datasets, and validate the results. None of that belongs in a random notebook titled synthetic_data_agent_DO_NOT_DELETE_PLEASE_PLEASE_PLEASE.

That’s not a project; that’s a crime scene.

Notebooks Are a Sandbox, Not a Home

Notebooks are fantastic for exploration, but terrible for running a production system. They hide state, encourage manual execution, and make it far too easy to forget which cell or magical cluster setting made things work.

When you’re building a data product that handles enterprise metadata, manages schemas, and generates outputs that other teams will trust, structure isn’t optional.

📌 The goal: transition from notebook-shaped chaos to project-shaped engineering.

Environmental Boundaries

The system shouldn’t behave the same way across its lifecycle:

• Development: Total freedom to break things, test profiling logic, and discover that a production column contains seven different date formats.
• Staging: Pure validation. Testing whether the workflow deploys, the jobs run, and configurations match expectations before anyone touches real data.
• Production: Total control. Restricted permissions, locked-down service principals, and zero “I thought this was dev” surprises.

Separate bundle targets enforce these boundaries.

Same project, completely different guardrails.

Making Deployments Boring

A deployment shouldn’t feel like a high-stakes ritual involving ten manual steps, three UI clicks, and a prayer to the demo gods. Everything — jobs, resources, paths, variables, and permissions — should be defined cleanly as code.

That’s where Declarative Automation Bundles come in. They give the project a deployable, repeatable shape. If the deployment process feels entirely boring, that’s the ultimate compliment.

Permissions Aren’t Parsley

Security isn’t something you sprinkle on top at the end, like parsley on a suspicious pasta. 🍝

Even in an early version, we have to establish who can deploy, who can execute, and which identity runs the workflow. We don’t need the final, ultra-granular security model today, but we do need to acknowledge that permissions are a foundational requirement, not a post-launch afterthought.

The First Job Is a Contract

The initial workflow doesn’t need to be impressive. In fact, it shouldn’t be. Right now, it’s just a contract between the repository and the workspace proving that the project can be deployed, configured, and observed.

Before the platform can generate realistic data, it needs to prove it can execute a basic, unglamorous task reliably. If a boring pipeline fails, adding an LLM won’t fix the architecture — it will just fail with a much better vocabulary.

---

What We Are Bootstrapping

This step is about creating the first deployable shape of the project.

We aren’t building the profiling engine, the generation logic, or the validation framework yet. The objective is to establish a rock-solid skeleton: a clear repository structure, a declarative bundle configuration, environment-specific deployment targets, an initial permissions model, and a first executable workflow.

This foundation is non-negotiable. Because the agent will eventually interact with governed Unity Catalog assets, inspect schemas, and generate derived datasets, its operational blueprint cannot live in scattered notebooks or manual workspace configurations. It needs absolute control from day one.

Repository Structure

To prevent code from dissolving into chaos, we must separate concerns cleanly. Source code, job notebooks, bundle configurations, and tests should never be tossed into a single directory.

A lean, production-ready layout looks like this:

├── databricks.yml          # Main bundle configuration definition  
├── bundle/                 # Targets and deployment resources  
│   ├── resources.yml       # Workflows, jobs, and pipeline definitions  
│   └── targets/            # Environment specific configurations (dev, staging, prod)  
├── src/                    # Reusable Python packaging modules  
│   ├── profiling/          # Statistical metadata extraction  
│   ├── generation/         # Synthesis orchestration engine  
│   └── validation/         # Quality and constraint checking  
├── notebooks/              # Slim entry point notebooks called by Databricks jobs  
├── tests/                  # Unit and integration tests  
└── README.md

This structural separation ensures that our metadata extraction, relationship discovery, and generation engines can evolve at completely different speeds without breaking each other.

Bundle Configuration

The bundle is our deployment contract. At the root of the project, the databricks.yml file allows us to manage resources programmatically. Databricks reads this file to validate, deploy, and run workspace assets automatically.

By relying on Declarative Automation Bundles, we eliminate manual UI clicking entirely. Every cluster setting, environment variable, database path, and execution workflow is tracked as code. This gives us a single source of truth that we will reuse and expand throughout the rest of this build series.

Deployment Targets

The bundle configuration explicitly isolates our environment lifecycles:

• Development (dev): The sandbox where developers have the freedom to test new profiling logic, generate experimental samples, and break things against isolated schemas.
• Staging (staging): The integration gate used to validate deployment configurations, verify permissions, and smoke-test workflows before release.
• Production (prod): The locked-down environment where the system runs on automated schedules, requiring strict access controls and audited executions.

Code moves across these environments seamlessly via parameters, not manual copy-pasting. A data architecture that cannot cleanly distinguish development from production isn’t a platform — it’s an infrastructure incident waiting for a calendar invite.

The First Workflow

Our initial workflow is intentionally tiny. It doesn’t need to profile a massive 100-column table or synthesize rows ; its only job is to prove that our deployment pipeline functions correctly.

[Deploy Bundle]  
       ⤷ [Resolve Variables]  
                 ⤷ [Verify Target Environment]  
                                 ⤷ [Log UC Discovery Status]

A practical first task simply runs a minimal execution check: it resolves bundle variables, confirms the target environment context, and ensures the workspace connection is active. Once this delivery path is proven, we can confidently swap out the dummy tasks for our actual profiling, relationship detection, and generation stages.

Permissions and Run Identity

As we established during the design phase, security is not parsley you sprinkle on top of a suspicious pasta at the end of a project.

DABs allow us to declare access control lists (ACLs) directly inside the configuration files. Crucially, they let us define the run_as identity boundary:

# Example target snippet  
targets:  
  dev:  
    # Runs interactively using the deploying developer's credentials  
  prod:  
    run_as:  
      service_principal_name: "da-SDA-prod-sp" # Executed securely under a machine identity

Because this project will eventually possess the power to read production schemas and generate enterprise data products, setting strict parameters around who can deploy, who can execute, and what identity handles the data compute is a Day 1 requirement.

---

How Declarative Automation Bundles Work

A bundle isn’t just a packaging format; it is the project contract. It provides a declarative, end-to-end blueprint describing the code files, environment configurations, variables, permissions, and orchestration jobs that Databricks needs to construct your workload reliably. Without this contract, a system quickly splinters into fragmented assets scattered across the workspace interface.

The Root: databricks.yml

Every bundle centers around a single root file named databricks.yml. To protect this file from ballooning into an unreadable mess of configuration, the best engineering pattern is to keep it thin and pull in modular component files via the include block.

bundle:  
  name: synthetic-data-agent  
include:  
  - bundle/resources.yml  
  - bundle/targets/*.yml

As our modules for table profiling, relationship graph discovery, data synthesis, and quality auditing grow, they can live in distinct files rather than battling for space in one massive YAML file.

Decoupling Files, Artifacts, and Resources

A major pitfall when adopting bundles is treating everything as code. A clean repository layout separates three completely different layers:

• Files (Source Code): These are your notebooks, Python modules, and configurations that are synchronized directly to the workspace during deployment. This is where your core algorithmic logic and task wrappers live.
• Artifacts (Built Packages): These are compiled outputs produced during deployment — like a Python wheel (.whl) packaging our validator—that get attached directly to cluster executions when running jobs.
• Resources (Databricks Assets): These are platform entities managed entirely via REST API schemas. This includes your actual multitask jobs, Delta tables, pipelines, and schemas.

📌 Why this matters: A notebook is not a job, and business logic should not be entangled with structural deployment definitions. Keeping these distinct prevents circular dependency headaches down the road.

Multi-Environment Targets & Substitutions

Targets dictate where and how the bundle deploys. Rather than copying codebases for different deployment contexts, target parameters override environments gracefully using variables.

variables:  
  catalog_name:  
    description: "Target catalog for agent execution"  
targets:  
  dev:  
    mode: development # Automatically tags and user-prefixes assets for safety  
    variables:  
      catalog_name: "sda_sandbox_dev"  
  prod:  
    mode: production  # Locks down resource changes and restricts direct edits  
    variables:  
      catalog_name: "sda_secure_prod"

To bind these variables seamlessly into your resource definitions, invoke target-aware substitutions:

resources:  
  jobs:  
    profiling_run:  
      name: "[${bundle.target}] Profiling Run"  
      tasks:  
        - task_key: run_profiler  
          notebook_task:  
            notebook_path: ../notebooks/profile_trigger.py  
            base_parameters:  
              catalog: ${var.catalog_name}

The Infrastructure Lifecycle

Moving code from your repository into an active workspace follows three primary command phases:

[databricks bundle validate]   
              ⤷ [databricks bundle deploy]  
                             ⤷ [databricks bundle run]

• validate: Audits your configuration parameters against exact object schemas to guarantee deployability before reaching the cloud.
• deploy: Syncs code files, packages binary artifacts, and automatically updates the remote workspace infrastructure to match your local state.
• run: Remotely kicks off an active instance of a specific resource workflow for rapid end-to-end iteration.

If you are onboarding existing workspace infrastructure into source control, you can use databricks bundle generate and databricks bundle deployment bind to reverse-engineer and link existing assets into your bundle structure.

Golden Rules & Fatal Mistakes

💡 The Bundle Playbook

• Isolate Production Roots: Do not deploy production bundles into /Shared. Keep them tucked inside highly restricted workspace path targets managed exclusively by your deployment identity.
• Decouple Logic: Keep heavy business routines inside tested Python modules. Treat workspace notebooks purely as un-cluttered task entry-points.
• Enforce Machine Identities: Always explicitly assign service principals via the run_as blocks for staging and production targets.

❌ Architectural Traps

• Relying on UI State: Using the Databricks UI to tweak production job schedules or access controls manually. The bundle must remain your absolute source of truth.
• Hardcoded Environments: Embedding direct workspace parameters, specific warehouse IDs, or schema string names deep within notebooks.
• Treating Targets Equally: Forgetting that dev and prod targets require fundamentally separate security parameters, retention rules, and computing contexts.

---

Designing the Repository Structure

Putting an entire agentic system into a single folder isn’t simplicity; it’s deferred suffering. Our system isn’t a single notebook; it will eventually contain profiling engines, metadata extraction routines, relationship graphs, generation pipelines, and validation suites.

The repository layout determines where responsibilities live, how the solution scales, and how painful future updates will be. We need a clean structure from day one.

synthetic-data-agent/  
├── databricks.yml            # Project entry point and environment routing  
├── bundle/  
│   ├── resources.yml         # Job and workflow declarations  
│   └── targets/              # Environment target overrides  
├── src/  
│   └── sda/                  # Reusable, importable Python package modules  
│       ├── __init__.py  
│       ├── config.py  
│       ├── uc_discovery.py  
│       └── logging.py  
├── notebooks/  
│   └── 00_bootstrap_check.py # Thin workflow entry point  
├── tests/  
│   └── test_config.py        # Unit testing suites  
├── docs/  
│   └── architecture.md       # Architecture decision records  
├── pyproject.toml            # Python packaging metadata  
└── README.md

To keep the root databricks.yml clean and prevent configuration sprawl as we add downstream capabilities, we immediately delegate environment and asset details to our subfolders using the include block:

bundle:  
  name: synthetic-data-agent  
include:  
  - bundle/resources.yml  
  - bundle/targets/*.yml

bundle/resources.yml: Assets as Code

All Databricks workspace objects must be explicitly declared within the bundle configurations, never manually clicking through the UI.

resources:  
  jobs:  
    bootstrap_verification:  
      name: "[${bundle.target}] Bootstrap Verification"  
      tasks:  
        - task_key: run_check  
          notebook_task:  
            notebook_path: ../notebooks/00_bootstrap_check.py

Under the hood, these resource declarations map directly to live platform infrastructure, tracking our end-to-end orchestration pipeline.

src/: Reusable Python Modules

The src/ directory houses importable, unit-testable Python packages.

📌 Notebooks may orchestrate, but core logic must live in Python modules.

As the architecture evolves, these modules will handle distinct algorithmic steps: statistical profiling, structural relationships, generation models, and validation routines. Decoupling logic from execution environments ensures that the code remains modular, easily testable outside a live cluster, and ready to be exposed via APIs or Databricks Apps down the road.

notebooks/: Entry Points, Not the System

Notebooks are highly convenient, but convenience can quietly warp into fragile architecture.

📌 Notebooks call the system; they do not contain the system.

Our bootstrap notebook (00_bootstrap_check.py) does only the bare minimum:

• Resolves the active bundle environment variables.
• Validates read access to the targeted Unity Catalog schema.
• Executes a quick dummy query to prove the runtime path is operational.

Tools for Reliability: tests/, pyproject.toml, and docs/

• tests/: Synthetic data relies heavily on statistical accuracy and structural validity. Validating schema transformations and null boundaries cannot happen by simply glancing at data and assuming it looks fine. Testing habits start here.
• pyproject.toml: Establishes proper packaging discipline from the start. Designing the project as a formal Python package ensures clean versioning, consistent local dependency management, and predictable builds before deploying to clusters.
• docs/: Human memory is a volatile database. We use architectural decision records (architecture.md) to clearly document exactly which identities execute tasks, where data lands, and where deterministic logic hands off to LLM components.

Avoiding the Template Trap

While databricks bundle init provides useful default layouts like default-minimal or default-python, we are building this repository structure intentionally. A generic starter template cannot anticipate the governance parameters, relationship graph logic, or multi-environment verification steps required for an end-to-end synthetic data framework.

By deliberately crafting this environment, we bypass common architectural pitfalls: core code isn’t trapped in notebook cells, variables aren’t hardcoded, and the UI isn’t our source of truth. The result is an asset built for stability.

---

Creating the First Bundle Configuration

With the repository structure mapped out, we need to build the first deployable bundle. The objective is completely unglamorous: create a minimal configuration that proves our project can be validated, deployed, and executed within Databricks without a single piece of the core architecture failing.

The Root: databricks.yml

Our root file acts as the project entry point. It sets the global configuration parameters, pins the CLI version to ensure environment consistency, and maps our deployment targets.

bundle:  
  name: synthetic-data-agent  
  databricks_cli_version: ">= 0.218.0"  
  
include:  
  - resources/*.yml  
  
variables:  
  catalog_name:  
    description: "Unity Catalog destination for outputs"  
  schema_name:  
    description: "Target schema for early outputs"  
  source_schema_name:  
    description: "Source schema inspected by discovery task"  
  
targets:  
  dev:  
    mode: development  
    default: true  
    workspace:  
      host: https://<dev-workspace-url>  
    variables:  
      catalog_name: sda_dev  
      schema_name: sandbox  
      source_schema_name: sample_source  
  
  staging:  
    mode: production  
    workspace:  
      host: https://<staging-workspace-url>  
      root_path: /Workspace/Users/<deployment-identity>/.bundle/${bundle.name}/${bundle.target}  
    variables:  
      catalog_name: sda_staging  
      schema_name: validation  
      source_schema_name: controlled_source  
  
  prod:  
    mode: production  
    workspace:  
      host: https://<prod-workspace-url>  
      root_path: /Workspace/Users/<deployment-identity>/.bundle/${bundle.name}/${bundle.target}  
    variables:  
      catalog_name: sda_prod  
      schema_name: synthetic  
      source_schema_name: governed_source

Targets Are Built-In Guardrails

The mode property dictates native platform guardrails:

• development mode: Optimized for rapid iteration. It automatically prefixes deployed assets with your username, tags workflows, pauses active crons/triggers, and enables concurrent development runs.
• production mode: Locks the workspace down. It enforces strict deployment path validation, restricts direct UI code updates, and ensures that resources run consistently under stable identities.

Using targets entirely avoids the high-risk pre-release ritual of opening a notebook and manually switching environment strings. That isn’t promotion; it’s gambling with syntax highlighting.

Deployment Variables vs. Runtime Parameters

There is a critical boundary between deployment-time configurations and runtime task inputs:

• Bundle Variables (variables): Evaluated strictly when you deploy the infrastructure. They handle static cloud topology like catalog names, storage volume locations, and warehouse IDs. Changing a bundle variable after deployment will not alter a running job without a redeploy.
• Task Parameters (base_parameters): Dynamic values passed directly into the compute process at runtime (e.g., scale factor settings, specific tables to profile).

Keep them strictly separated. Use bundle variables to configure your target workspace, and use task parameters to instruct the engine on what to compute.

The First Job Resource

Now we declare our verification task in resources/jobs.yml. This workflow does not generate data; it validates our entire infrastructure loop.

resources:  
  jobs:  
    bootstrap_check:  
      name: sda-${bundle.target}-bootstrap-check  
      tasks:  
        - task_key: bootstrap_check  
          notebook_task:  
            notebook_path: ../notebooks/00_bootstrap_check.py  
            base_parameters:  
              catalog_name: ${var.catalog_name}  
              schema_name: ${var.schema_name}  
              source_schema_name: ${var.source_schema_name}

This boring workflow instantly answers our foundational questions: Can the bundle deploy? Can the job execute? Can the worker parse configuration variables, and does the running identity have permission to view the targeted Unity Catalog metadata?

Critical Foundations: Paths, Compute, and Identity

• Lock Down Your Paths: In staging and prod, explicitly configure your root_path. 
  ⚠️ Do not deploy production bundles into /Shared. The /Shared workspace directory is often broadly writable by everyone in the organization, meaning unauthorized users could modify production code or orchestration assets.
• Abstract Your Compute: Compute configurations (serverless job settings or specific policy-driven job clusters) belong strictly in the bundle configuration or target definitions — never hardcoded inside a python script or notebook cell.
• Decouple Run Identity: While running under your user profile works in dev, production targets must include a run_as block mapped to a dedicated service principal. Employees shift teams, but service principals don’t resign on a Friday afternoon.

The Command Lifecycle

With your configurations written, run through the standard development execution sequence using the Databricks CLI:

# 1. Audit configuration variables against platform schemas  
databricks bundle validate -t dev  
  
# 2. Sync local assets and compile infrastructure resources  
databricks bundle deploy -t dev  
  
# 3. Remote trigger the deployed verification job  
databricks bundle run bootstrap_check -t dev

The delivery pipeline is now fully functional. The repository maps our system, the bundle deploys it as infrastructure-as-code, and the environment changes seamlessly via targets. Now, we can write the code that connects this skeleton directly to Unity Catalog metadata.

---

Building the First Bootstrap Workflow

The first workflow should not attempt to generate synthetic data. That would be starting on third base before learning how to run. Before the agent can infer statistical distributions, map relational graphs, or synthesize datasets, it must pass a structural platform validation: can it deploy, receive its runtime configurations, resolve its target environments, and safely read catalog metadata?

Enter the bootstrap notebook (00_bootstrap_check.py). It isn’t flashy, but it serves as our critical first operational test line.

The Integration Check Chain

We want this notebook to thoroughly validate the entire deployment loop:

• Parameter Hand-off: Verifying that parameters declared in the bundle configuration pass directly into the runtime context.
• Target Identification: Ensuring the script accurately recognizes which environment catalog and schema target it has landed on.
• Privilege Verification: Proving that the executing identity has active read access over your targeted metadata.
• Asset Discovery: Compiling a baseline inventory of visible tables and columns before executing deep data scans.

Parameters Come from the Job

During our bundle layout design, we mapped target-specific bundle variables to the job resource configuration. Inside our notebook, we extract these deployment-time assets using the standard dbutils.widgets.get framework.

Because these configurations control our database boundary routing, parse them using defensive input checking:

import re  
from pyspark.sql import functions as F  
  
_IDENTIFIER = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")  
  
def require_identifier(value: str, name: str) -> str:  
    if not value:  
        raise ValueError(f"Missing required parameter: {name}")  
    if not _IDENTIFIER.match(value):  
        raise ValueError(  
            f"Invalid {name}: {value!r}. "  
            "Use simple Unity Catalog identifiers only."  
        )  
    return value  
  
catalog_name = require_identifier(dbutils.widgets.get("catalog_name"), "catalog_name")  
schema_name = require_identifier(dbutils.widgets.get("schema_name"), "schema_name")  
source_schema_name = require_identifier(  
    dbutils.widgets.get("source_schema_name"),  
    "source_schema_name",  
)  
  
print(f"Target Catalog: {catalog_name}")  
print(f"Output Schema:  {schema_name}")  
print(f"Source Schema:  {source_schema_name}")

Treating target catalogs and schema strings as validated SQL identifiers blocks accidental formatting quirks or environment injection path bugs immediately.

Querying the Catalog, Not Guessing

To locate target schemas, query the platform’s native metadata architecture instead of manually guessing workspace tables. Databricks exposes privilege-aware metadata layers via the system.information_schema views. If our execution identity lacks adequate permission to interact with a catalog or schema, the discovery query will safely catch the barrier right here.

First, verify that our target catalog exists and is visible to the workspace:

catalogs_df = spark.table("system.information_schema.catalogs")  
catalog_exists = (  
    catalogs_df  
    .where(F.col("catalog_name") == catalog_name)  
    .limit(1)  
    .count()  
)  
  
if catalog_exists == 0:  
    raise RuntimeError(  
        f"Catalog {catalog_name!r} is not visible to the current execution identity."  
    )

Next, cross-check our expected database boundaries:

schemata_df = spark.table("system.information_schema.schemata")  
required_schemas = [schema_name, source_schema_name]  
visible_schemas = {  
    row.schema_name  
    for row in (  
        schemata_df  
        .where(F.col("catalog_name") == catalog_name)  
        .where(F.col("schema_name").isin(required_schemas))  
        .select("schema_name")  
        .collect()  
    )  
}  
  
missing_schemas = sorted(set(required_schemas) - visible_schemas)  
if missing_schemas:  
    raise RuntimeError(  
        f"Missing or inaccessible schemas in catalog {catalog_name!r}: "  
        f"{', '.join(missing_schemas)}"  
    )

Discovering Tables and Columns

Once access is cleared, we pull a structured data map from the environment. Remember the core design rule: discovery is not profiling. Discovery stays entirely within the high-level metadata schema, making it incredibly fast, secure, and cost-effective.

To isolate available tables, pull from the tables view mapping:

tables_df = (  
    spark.table("system.information_schema.tables")  
    .where(F.col("table_catalog") == catalog_name)  
    .where(F.col("table_schema") == source_schema_name)  
    .select(  
        "table_catalog",  
        "table_schema",  
        "table_name",  
        "table_type",  
        "data_source_format",  
        "created",  
        "last_altered",  
    )  
    .orderBy("table_name")  
)  
display(tables_df)

Execute the identical pattern to extract your core structural column fields:

columns_df = (  
    spark.table("system.information_schema.columns")  
    .where(F.col("table_catalog") == catalog_name)  
    .where(F.col("table_schema") == source_schema_name)  
    .select(  
        "table_catalog",  
        "table_schema",  
        "table_name",  
        "column_name",  
        "ordinal_position",  
        "data_type",  
        "is_nullable",  
        "comment",  
    )  
    .orderBy("table_name", "ordinal_position")  
)  
display(columns_df)

Exiting with a Structured Handoff

A clean workflow task should finish with an explicit summary object rather than just unparsed cell print logs:

summary = {  
    "catalog_name": catalog_name,  
    "source_schema_name": source_schema_name,  
    "output_schema_name": schema_name,  
    "visible_tables": tables_df.count(),  
    "visible_columns": columns_df.count(),  
}  
print(summary)

When this pipeline expands into a multi-task workflow, this metadata summary can be passed downstream to subsequent tasks (such as the profiler or generation planner) using the dbutils.jobs.taskValues framework.

Architectural Traps to Avoid

• The Notebook Monolith: Keep this verification notebook strictly scoped. It should never sample table arrays, calculate statistical medians, or build rows. If the code isn’t an execution wrapper, move it straight into our reusable src/sda/ modules.
• Metadata vs. Data Access: Seeing structural metadata in your information schema does not mean your run identity automatically possesses data access privileges to read the underlying database rows. Discovery proves identity boundaries; actual table profiling scans come later.
• Silent Defaults: Never let a parameter error gracefully resolve into a baseline variable. If a credential mapping or target catalog fails, crash loudly and early. Silent failures are exactly how production pipelines end up writing over development sandboxes with absolute, automated confidence.

---

Validating, Deploying, and Running the Bootstrap

A bundle definition and a notebook entry point look great on paper, but they don’t mean a thing until they survive a real delivery loop. To transform this codebase into a dependable data product, it must move seamlessly through a repeatable engineering lifecycle: validation, planning, stateful deployment, execution, and post-run auditing.

1. Validate Before You Deploy

Command: databricks bundle validate -t dev

Bundle validation parses your local configurations against strict platform object schemas before anything touches your workspace. It is your cheapest failure point: it catches missing target variables, invalid YAML syntax, or broken file path definitions while the project is still entirely local. Make validation a hard habit; when CI/CD pipelines come into play later, this command will be your first automated gatekeeper.

2. Inspect the Deployment Plan

Command: databricks bundle plan -t dev

Never deploy blindly. The plan command acts as a structural dry-run, generating a clear diff of exactly which assets will be created, modified, or permanently removed from your workspace. Modern bundle engines support isolating specific assets using the --select flag during rapid development. However, staging and production promotions should ignore selective shortcuts in favor of a complete, immutable state deployment.

3. Sync State to Development

Command: databricks bundle deploy -t dev

Deploying synchronizes your files, builds your binary wheel packages, and instantiates the defined resources inside the workspace. Crucially, the platform tracks resources using unique IDs stored as deployment state files, not by resource string names. Treat your core bundle mapping options — the bundle name, target environment, and the root_path—as stable infrastructure choices. Changing them casually on a Thursday afternoon will break resource associations, causing the engine to lose track of previously managed workspace assets.

4. Execute the Bootstrap Run

Command: databricks bundle run bootstrap_check -t dev

The moment of truth. This kicks off an active instance of our verification workflow remotely. This run handles the ultimate plumbing test: it confirms that parameters pass from the bundle into the execution workers, and that the run identity possesses the active permissions required to query Unity Catalog’s information schema. If there’s an authentication error or a configuration mapping mismatch, we want the pipeline to crash loudly right here.

5. Audit the Infrastructure Summary

Command: databricks bundle summary -t dev

Instead of hunting through the Databricks UI like a lost soul to find your freshly built assets, leverage the summary command. It outputs your exact bundle identity footprint alongside deep, direct links to every managed resource in the workspace. As the project scales from a single verification script to multiple pipelines, Delta Live Tables, and serverless compute schemas, this programmatic map keeps the workspace organized and easily auditable.

Red Flags & Fatal Pitfalls

• Manual UI Tweak Drifts: Fixing a broken scheduler or adjusting an access control list directly in the workspace browser UI is an architectural trap. The moment you run your next local bundle deployment, the engine will ruthlessly overwrite those ad-hoc hotfixes to match the repository’s source of truth.
• Treating Staging like a Playground: Staging isn’t just dev in a slightly cleaner shirt. It must thoroughly validate production-like path controls, isolated catalog permissions, and strict service-principal execution behaviors. Production should never be the first place that forces you to discover a misconfigured identity credential.
• Skipping Explicit Target Flags: Relying blindly on default target mappings in your terminal is a recipe for disaster. Get comfortable passing -t dev or -t staging explicitly in every execution command to guarantee you know exactly where your compute is landing.

---

Adding the First Permission Boundaries

The bootstrap workflow is officially running. That is great news, but it doesn’t mean the system is safe. A production-ready architecture requires more than just functional code blocks and successful notebook runs; it needs strict access controls.

This isn’t the complete, granular security model for the agent — that deserves its own dedicated post later in the series. However, we must map out the security boundaries today, because access controls added as an afterthought usually arrive as emergency data archaeology.

The Three Distinct Permission Layers

The most common security mistake is treating permissions as a single, monolithic concept. For an automated project, you must decouple three entirely separate layers:

• The Deployment Identity: The user or automated CI/CD principal executing commands like databricks bundle deploy.
• The Run Identity: The active identity used by the Databricks Job cluster when the workflow notebook actually executes.
• The Unity Catalog Privileges: The governance layer dictating whether that specific run identity has permission to see schemas, read metadata, or write rows.

A developer might have permission to deploy a job without having the right to see production data. Conversely, a machine identity might have data-read privileges but no right to alter the bundle infrastructure. Keeping these concerns separated is fundamental to our architecture.

Target-Level Bundle Permissions

Declarative Automation Bundles allow you to define access control lists (ACLs) globally at the target level, or granularly within individual resources. Defining them at the target level is the cleanest approach for managing consistent environment access:

targets:  
  dev:  
    mode: development  
    permissions:  
      - group_name: sda-developers  
        level: CAN_MANAGE  
  staging:  
    mode: production  
    permissions:  
      - group_name: sda-platform-admins  
        level: CAN_MANAGE  
      - group_name: sda-data-engineers  
        level: CAN_RUN  
  prod:  
    mode: production  
    permissions:  
      - group_name: sda-platform-admins  
        level: CAN_MANAGE  
      - group_name: sda-operators  
        level: CAN_RUN

📌 Crucial Distinction: This block only dictates who can view, run, or alter the Databricks Job asset itself. It does not grant access to the underlying table data. Actual database security remains isolated within Unity Catalog.

While resource-level blocks (directly under the job definition) can handle exceptions, avoid overlapping them with top-level target permissions for the same principal group. The platform will not handle conflicting ACL definitions gracefully. Keep it simple: use target-level blocks by default and keep resource overrides highly visible.

The run_as Identity Boundary

The run_as configuration block defines exactly who executes the data processing compute. While executing jobs under your personal user profile is perfectly fine during local development, production tasks must be bound to an isolated service principal:

targets:  
  prod:  
    mode: production  
    run_as:  
      service_principal_name: ${var.runtime_service_principal}

The agent will eventually execute highly sensitive routines: scanning metadata, profiling enterprise tables, and generating derived data products. These actions cannot depend on a single developer’s personal workspace permissions. People change teams, go on holiday, or leave companies; your core workflows should never collapse because a human user account was deactivated.

📌 Note: When your deployment identity differs from the run_as block identity, the bundle engine natively restricts execution support to automated workloads like Jobs and Delta Live Tables pipelines. Plan for this limitation early before attempting to bind interactive resources to machine identities.

Unity Catalog Privileges: The Real Guardrail

Bundle ACLs govern the job orchestration, but Unity Catalog secures the data. To allow the bootstrap workflow to run a successful discovery scan, its execution identity must be explicitly granted the correct securable object privileges down the hierarchy:

-- Minimum setup requirements for the runtime Service Principal  
GRANT USE CATALOG ON CATALOG sda_dev TO `sda-runtime-sp`;  
GRANT USE SCHEMA ON SCHEMA sda_dev.sample_source TO `sda-runtime-sp`;  
GRANT SELECT ON SCHEMA sda_dev.sample_source TO `sda-runtime-sp`;  
  
GRANT USE SCHEMA ON SCHEMA sda_dev.sandbox TO `sda-runtime-sp`;  
GRANT CREATE TABLE ON SCHEMA sda_dev.sandbox TO `sda-runtime-sp`;  
GRANT MODIFY ON SCHEMA sda_dev.sandbox TO `sda-runtime-sp`;

To read metadata or rows, the identity requires USE CATALOG, USE SCHEMA, and SELECT. To publish synthetic tables later, it requires CREATE TABLE and MODIFY on the destination. If our bootstrap notebook hits a privilege error here, it is designed to crash loudly. A permissions failure isn’t a code bug—it’s a critical governance signal.

Golden Rules & Fatal Pitfalls

💡 Governance Best Practices

• Keep Privileges Transparent: Maintain your Unity Catalog grants inside your infrastructure provisioning layer (like Terraform or dedicated SQL setup scripts). Never let a workflow job attempt to secretly grant itself database access at runtime. That isn’t a bootstrap; it’s a privilege escalation exploit with a YAML garnish.
• Automate Deployment Identity: Transition to an unattended model as soon as possible. Developers handle local deployments to dev, but headless CI/CD service principals using OAuth M2M authentication should handle all promotions to staging and prod.

❌ Access Control Traps

• The Admin Account Crutch: Running every environment under your personal admin identity. You won’t know if the system actually works because the configuration is correct, or if it only succeeded because your account happens to have god-mode permissions across the workspace.
• Ignoring Parent Dependencies: Granting explicit SELECT rights on a table while forgetting to declare USE CATALOG or USE SCHEMA on its parent containers. Unity Catalog will block execution immediately.
• Ad-Hoc Workspace Tweaks: Making quiet permission adjustments inside the Databricks UI without updating your repository configurations. This invites immediate configuration drift — and drift is just production chaos masquerading as convenience.

---

Architecture Checkpoint

At this stage, the system still doesn’t generate a single row of synthetic data. That is entirely intentional. This milestone was never about the intelligence layer; it was about building the unglamorous delivery layer that everything else will depend on.

We now have an active, end-to-end operational pathway:

Git Repository  
      ⤷ Declarative Bundle   
        (Targets, Workflows, Configurations, Access Controls)  
                   ⤷ Databricks Workspace   
                     (Files, Jobs, Entry Notebook)  
                               ⤷ Unity Catalog Metadata   
                                          ⤷ Structured Discovery Result

This foundational path establishes five critical guarantees:

• The Project Is Fully Deployable: The system is no longer a collection of local scripts or experimental workspace cells. It lives under a version-controlled, repeatable lifecycle. Every future asset — profilers, planners, and validators — will enter the platform through this exact pipeline. One delivery mechanism, zero ad-hoc chaos.
• Environments Are Strictly Isolated: Development, staging, and production are now hard boundaries defined in target configurations, not just vague intentions left in a developer’s head. The same codebase transitions seamlessly across environments while automatically overriding paths, database scopes, and runtime identities.
• Workflows Have a Stable Entry Point: Our initial job is tiny, but it cements our execution pattern. As the project matures, this wrapper notebook easily expands into a comprehensive, multi-task orchestration chain:

Discover Metadata   
        ⤷ Profile Source Tables  
                    ⤷ Infer Relationships  
                                ⤷ Generate Data   
                                        ⤷ Validate Quality & Privacy   
                                                        ⤷ Publish Assets

• Unity Catalog Secures the Starting Line: We didn’t wait until production to think about governance; it’s hardcoded into our bootstrap loops. The workflow handles metadata discovery through a privilege-aware layer. The system operates strictly within what its active execution identity is permitted to see — no wandering blindly through the metastore.
• Permissions Are Highly Visible: The bundle forces a clean split between deployment actions and data compute execution. Developers author the routines, CI/CD engines handle promotions, and isolated runtime service principals execute the analytics.

What We Aren’t Solving Yet

This checkpoint lets us see exactly what’s left to tackle. The system doesn’t profile raw column statistics, map distribution curves, discover implicit foreign-key joins, or generate multi-table outputs yet.

Good. Those are complex computational engineering problems, and they finally have a secure, structured home to live in. The entire point of bootstrapping is to make sure day two doesn’t start with architectural regret.

Where This Leaves Us

Generating fake data is incredibly easy. Generating useful, high-fidelity synthetic data that behaves exactly like real production tables requires cold, hard statistical evidence.

We now have a stable platform to stand on. Our next goal is Profiling: transforming high-level Unity Catalog metadata and deep source-table statistics into a persistent, structured knowledge layer that our tools can actively reason against before manufacturing a single block of data.

The framework still can’t fake reality yet. But at least now, it has a safe place to stand before trying. Let’s roll.

---

Still no synthetic Data. That’s the point. Build the skeleton. Deploy the system . Earn the brain.

---

🔗 Code & Music 🎧

The code for this article is available here:

🔗 synthetic-data-agent

And because metadata discovery clearly needed a soundtrack, the Synthetic Data Agent now has its own songs too:

🎧 YouTube playlist

Read the article. Check the repo. Play the songs.

---

SDA 03 — Build The Skeleton

[Verse]

Everybody wanna be the king with a prompt and a prayer,
Whole repo lookin’ like somebody threw spaghetti in the air,
Notebook after notebook…
Copy…
Paste…
Repair…

Then they wonder why production caught fire over there.

I don’t build hope — 
I build systems.

Every single folder got a purpose and a rhythm.

Every workflow versioned,
Every permission written,
Every target isolated,
Every path algorithmic.

You click buttons.

I commit.

You freestyle.

I submit.

You call it magic.

I call bullshit.

’Cause infrastructure never cared about your confidence.

No random notebook.

No mystery code.

No “works on my machine.”

No cowboy mode.

Bundle validate.

Bundle deploy.

Bundle run.

Destroy the noise.

Development…
Break it.

Staging…
Shake it.

Production…
You don’t touch it unless the pipeline say it.

Service principal.

Machine precision.

Humans leave.

Bundles never miss ‘em.

You built a project?

Nah…

You built a crime scene.

“final_v7_real_FINAL”

Know what I mean?

Folders full of corpses.

Broken dependencies.

Somebody changed a widget…

Killed three entities.

Git knows.

Bundle knows.

CLI knows.

Everybody knows.

The only thing your notebook knows
Is how to die alone.

Repository clean…

Python mean…

Every module laser-beam.

Notebook just the trigger finger.

Logic lives behind the screen.

Validate…

Deploy…

Run…

Repeat.

That’s the rhythm.

Every environment got different ammunition.

Dev got freedom.

Prod got prison.

Wrong identity?

Denied permission.

I don’t pray deployments work…

I prove they work.

I don’t trust green checks…

I read the dirt.

Because the smallest bootstrap failure
Shows the biggest architectural curse.

Build…

The…

Skeleton…

Before…

You…

Build…

The…

Monster.