In Part 1 we set up the project structure and deployed the three schemas and the landing volume to our dev workspace. Now we need to fill that volume with data.

This post covers everything required to pull your LinkedIn analytics via API: getting access, authenticating, calling the right endpoints, handling a few quirks the documentation doesn’t warn you about, and writing the results to the landing volume we created in Part 1.

LinkedIn API access

LinkedIn’s API is not a developer-friendly, open ecosystem. In order to use the APIs you need to own a LinkedIn Page for an organization, create a developer application, and apply for access to the Community Management API. Once that’s in place, you can fetch analytics data for your personal profile and your organization. In this series we focus on the personal profile.

Here’s the full process.

If you ask yourself why I did not use some Agent for automation, this is a valid question. My intention was to create an Enterprise grade process that uses reliable APIs with minimal cost. You can as well set up an Agent that pull the data from LinkedIn. If you have done that, feel free to contact me. I would be interested in how you achieved this.

Create a LinkedIn Page for an Organization

You need an organization page to back your developer application. In my case, I have an organization page for my consulting company SNOWGLOBE anyway. So I went with this, but I think you can create a page for your purpose, but note that later you have to provide details for your organization like homepage and address that will be reviewed by LinkedIn before approval. If you want to use your company's existing page, make sure you know the page administrator, who will need to approve your developer application.

Turns out, this is the trickiest part. LinkedIn can deny API access for your organization without providing a reason. You could get the impression that they don't want individuals to use their API.

Create a Developer Application

The LinkedIn Developer Portal allows you to create a developer application. Once created, the page has to be verified by an administrator of the organization page. In the Settings tab of your newly created application, click on the Verifiy button. You will get a URL you can send to an admin to verify your application.

Request Access to the Community Management API

In the Products tab of your application, request access to the Community Management API. This covers all the analytics endpoints we need, for both your personal profile and your organization.

LinkedIn will ask you to verify your business email and fill out a Development Tier Access Form with details about your use case and organization. Once approved, the API endpoints are unlocked.

Create an Access Token

In order to use the API, you have to create an access token in the Developer Portal. In the Auth tab of your application, click on the OAuth 2.0 tools link in order to generate a Token. For this pipeline, the required scopes are:

• r_member_postAnalytics

• r_member_profileAnalytics

Storing the Token securely

The access token should not live in the code or notebook. Ever. Not hardcoded, not in a widget default value, not in a comment. A notebook that gets committed to source control with a token in it is a security incident waiting to happen.

The production pattern here is:

1. Store the token in Azure Key Vault

2. Back the Key Vault with a Databricks secret scope

3. Retrieve it at runtime with dbutils.secrets.get()

access_token = dbutils.secrets.get(”secret-scope-key-vault-adb”, “linkedin-token”)

This means the token never appears in logs, never gets printed to a notebook output cell, and never touches source control. Databricks redacts secret values automatically in cell outputs. if you accidentally print the token, you’ll see [REDACTED] instead of the actual value.

If you're running in Databricks Free Edition without Key Vault, you can use a Databricks-backed secret scope as an alternative.

In the following sections, I outline how to get the data from the API. Only excerpts of the code is shown. Please visit my GitHub for the full ingestion notebook.

The five endpoints that matter

For a personal LinkedIn analytics pipeline, there are exactly five calls you need using the Community Management member analytics APIs. All of them sit under api.linkedin.com/rest/.

The full code is available in the accompanying GitHub Repo. Click the Button below to view the repo.

GitHub Repo

Follower delta by date (1)

This gives you the net change in followers for each day within a date range. The response comes back as a list of elements, one per day, each with a DateRange object and a memberFollowersCount field.

url = f”https://api.linkedin.com/rest/memberFollowersCount?q=dateRange&dateRange=(start:({start}),end:({end}))”

Total follower snapshot (2)

This gives you your current total follower count as a snapshot at the moment of the API call. There’s no date range parameter here, it’s just “right now.” I’ll come back to why this creates an interesting data modeling problem.

url = “https://api.linkedin.com/rest/memberFollowersCount?q=me”

Post analytics (3, 4, 5)

The memberCreatorPostAnalytics endpoint serves impressions, reactions, and comments depending on the queryType parameter. Each is a separate API call with aggregation=DAILY and a date range.

# Impressions
url = f”https://api.linkedin.com/rest/memberCreatorPostAnalytics?q=me&queryType=IMPRESSION&aggregation=DAILY&dateRange=(start:({start}),end:({end}))”

# Reactions
url = f”https://api.linkedin.com/rest/memberCreatorPostAnalytics?q=me&queryType=REACTION&aggregation=DAILY&dateRange=(start:({start}),end:({end}))”

# Comments
url = f”https://api.linkedin.com/rest/memberCreatorPostAnalytics?q=me&queryType=COMMENT&aggregation=DAILY&dateRange=(start:({start}),end:({end}))”

Initially, I wanted to pull the metrics for each individual post as well. Unfortunately, I was not able to accomplish that although I followed the documentation. If you have done it successfully please reach out to me and share your solution.

The date format LinkedIn expects

LinkedIn’s API does not use ISO 8601 date strings. It uses a custom struct format that looks like this:

(start:(year:2024,month:3,day:1),end:(year:2024,month:3,day:31))

This is LinkedIn’s Restli protocol format. Perfectly parseable, but unusual enough that it trips people up the first time they look at the API reference. The Python to generate it looks like this:

import datetime as dt

end_dt = dt.datetime.now() - dt.timedelta(days=1)
end = f”year:{end_dt.year},month:{end_dt.month},day:{end_dt.day}”

start_dt = end_dt - dt.timedelta(days=60)
start = f”year:{start_dt.year},month:{start_dt.month},day:{start_dt.day}”

Note the minus one day on end_dt. LinkedIn’s data for “today” isn’t fully settled yet when you’re running a 6am job, so I always pull up through yesterday to avoid partial-day numbers showing up in the pipeline.

The Required Headers

Every request needs three headers beyond the authorization token:

headers = {
  ‘Authorization’: f’Bearer {access_token}’,
  ‘Content-Type’: ‘application/json’,
  ‘X-Restli-Protocol-Version’: ‘2.0.0’,
  ‘Linkedin-Version’: ‘202506’
}

The X-Restli-Protocol-Version: 2.0.0 header is required for the Restli query syntax to work. Without it, LinkedIn either rejects the request or returns an unexpected response shape.

The Linkedin-Version header pins you to a specific API version. LinkedIn’s REST API is versioned by date (year + month), and they do deprecate old versions. Pinning this explicitly in code means your pipeline doesn’t silently break when LinkedIn releases a new API version with a different response schema.

The silent failure to watch out for

This one cost me time. LinkedIn’s API returns HTTP 200 with an error payload when the token expires. It looks like a success to your HTTP client, but the body contains something like:

{
  “code”: "EXPIRED_ACCESS_TOKEN",
  “message”: “The token used in the request has expired”
}

A naive response.raise_for_status() call will not catch this. The request returned 200, so as far as requests is concerned, everything is fine. Your pipeline runs, writes a file with an error body to storage, and you won’t notice until you wonder why your follower count is null.

The fix is a second check after deserializing the response:

def get_linkedin_data(url, access_token):
    response = requests.get(url, headers=headers)
    response.raise_for_status()
    data = response.json()

# LinkedIn can return error details with HTTP 200
if “status” in data and data[”status”] >= 400:
  raise RuntimeError(f”LinkedIn API error {data[’status’]}: {data.get(’message’, data)}”)

return data

This pattern, checking for an error status in the response body after already checking the HTTP status code, is worth keeping in any LinkedIn API integration. It’s explicitly handled in the bronze transformation layer too, which I’ll cover in Part 3.

Storing the data: landing in a Databricks Volume

Once you have the API responses, you need somewhere to put them. I’m writing raw JSON directly to a Databricks Unity Catalog Volume. Each API call produces one timestamped file:

def write_data_to_json_file(data, path, file_prefix):
    timestamp = dt.datetime.now().strftime(”%Y%m%d%H%M%S”)
    file_path = f”{path}{file_prefix}{timestamp}.json”
    dbutils.fs.put(file_path, json.dumps(data), overwrite=True)

The directory structure in the volume mirrors the data type:

/Volumes/<catalog>/bronze_linkedin/landing/
├── followers_delta/    followers_delta_20260315060112.json
├── followers_count/    followers_count_20260315060114.json
├── impressions/        impressions_20260315060118.json
├── reactions/          reactions_20260315060121.json
└── comments/           comments_20260315060124.json

The timestamp in the filename turns out to be more than just cosmetic. The followers_count endpoint (the q=me one that returns your current total) doesn’t include a date in its response. It’s a point-in-time snapshot. In the transformation layer, I derive the date of that snapshot from the filename timestamp itself. More on that in Part 2.

Initial load vs. delta

My example notebook accepts a run_type parameter with two values: initial and delta.

Initial load pulls everything from January 2020 to yesterday. You run this once when you first set up the pipeline.

Delta load pulls a rolling 60-day window ending yesterday. This is what runs every morning. The 60-day window is intentionally wider than one day to handle late-arriving data and to make re-runs safe. If a day’s job fails and you re-run it the next day, you’ll pick up yesterday’s data within the delta window without needing a manual backfill.

if run_type == “delta”:
    start_dt = end_dt - dt.timedelta(days=60)
    start = f”year:{start_dt.year},month:{start_dt.month},day:{start_dt.day}”
else:
    start = f”year:2020,month:1,day:1”

Why 60 days and not 7, or 30? LinkedIn’s analytics data can be revised retroactively for a few days after the fact. A wider window means the pipeline naturally self-corrects if LinkedIn updates historical numbers. The deduplication logic in the silver layer (covered in Part 3) handles the overlapping records cleanly using CDC upserts keyed by date.

The full code is available in the accompanying GitHub Repo. Click the Button below to view the repo.

GitHub Repo

Adding the notebook to the bundle

Now we wire the notebook into the bundle as a job task. Create bundles/projects/linkedin/resources/jobs.yml:

resources:
  jobs:
    linkedin_ingest:
      name: linkedin-medallion

      parameters:
        - name: catalog
          default: ${var.catalog}
        - name: schema
          default: ${resources.schemas.bronze_linkedin.name}
        - name: volume
          default: ${resources.volumes.landing.name}
        - name: run_type
          default: ${var.run_type}

      run_as:
        service_principal_name: ${var.service_principal_app_id}

      email_notifications:
        on_success:
          - <your-email@domain.com>
        on_failure:
          - <your-email@domain.com>

      tasks:
        - task_key: ingest
          notebook_task:
            notebook_path: ../notebooks/ingest.ipynb
          environment_key: serverless

      environments:
        - environment_key: serverless
          spec:
            client: "4"

The job passes the catalog, schema, volume, and run_type down to the notebook as parameters, and the notebook reads them via dbutils.widgets. The service principal identity runs the job, and email notifications alert you on both success and failure.

Note there’s only one task here for now. In Part 3 we’ll add the pipeline task as a second step with an explicit dependency on this ingest task.

Deploy the updated bundle:

databricks bundle deploy --target dev-user

Run it manually and verify

Trigger a manual run from the Databricks UI (or via the CLI: databricks bundle run linkedin_ingest --target dev-user). Set run_type to initial for the first run to pull the full historical data.

When it completes, navigate to your catalog in the workspace, open the bronze_linkedin schema, click on the landingvolume, and browse the folder structure. You should see JSON files in each of the five subdirectories, timestamped from this run.

That’s the data. In Part 3 we’ll build the Declarative Pipeline that reads these files, transforms them through bronze and silver, and produces a clean daily_metrics gold table.

What’s next

Part 3 covers the transformation layer: streaming tables for incremental file ingestion, CDC upserts for idempotent daily records, and a materialized view to join all five metrics into a single queryable table. We'll also add the pipeline as the second task in the job so that ingestion and transformation run end-to-end automatically.

Part 3 will be published 2026/05/28.

Ever since I started posting regularly on LinkedIn, I wanted to have my metrics (follower count, impressions, comments, reactions) not just in the LinkedIn UI, but in my own Databricks Lakehouse. Something I own, can query freely, and that accumulates history indefinitely rather than disappearing behind a rolling API window.

It turns out this is also a near-perfect project for learning how to build a proper Lakehouse-style data pipeline. It has everything: data ingestion via API, processing through the medallion layers (Bronze, Silver, Gold), and insights delivered via a dashboard as the end result. Not a toy example, not an enterprise-scale complexity monster, just a real use case at a manageable size, with all the patterns that matter.

Across four posts, we’ll build this end to end. By the time we’re done you’ll have a pipeline that runs automatically every morning, a clean daily metrics table in Unity Catalog, and a Lakeview dashboard on top of it, all deployed from version-controlled code.

Before writing a single line of pipeline code, I want to set up the project properly. That means infrastructure as code from the start: schemas, volumes, jobs, pipelines, all defined in version-controlled YAML and deployable to any environment with a single command.

The tool that makes this possible on Databricks is Declarative Automation Bundles. If you set this up first, everything that follows in Parts 2, 3, and 4 slots in cleanly. If you skip it and create resources manually, you’ll eventually want to come back and do it properly anyway.

So let’s start here.

What are Declarative Automation Bundles?

A Declarative Automation Bundle (DAB, formerly known as Databricks Asset Bundles) is a project format that packages your Databricks code and resource definitions together. You define everything (jobs, pipelines, schemas, volumes, dashboards) as YAML files alongside your SQL and Python code, commit it all to source control, and use the Databricks CLI to deploy to any target workspace.

The mental model is similar to Terraform, but Databricks-native. The CLI understands the relationships between resources, handles variable substitution across environments, and deploys notebooks and SQL files alongside the infrastructure that runs them.

Prerequisites

To follow along you’ll need:

• A Databricks workspace with Unity Catalog enabled

• A catalog where you have permission to create schemas (or an existing catalog you can use)

• The Databricks CLI installed (official docs)

• A terminal and a code editor like VS Code

• A service principal added to your Databricks workspace. Create one either via Azure Entra ID and add it to your workspace or create a Databricks scoped SP directly in the Databricks UI. Note down its application (client) ID, you’ll need it shortly.

Authenticate the CLI against your workspace. The recommended approach with the newer Databricks CLI (v0.200+) is OAuth, which opens a browser for interactive login, no token to create or manage:

databricks auth login --host https://<your-workspace-url>

Alternatively, if you prefer a personal access token, use:

databricks configure --token

Project structure

Now we can start creating our resource definitions. In order to that, we need a repo with a proper project structure. I will go with a structure that separates shared infrastructure from project-specific resources. Here’s the full folder layout we’ll build across this series:

bundles/
├── databricks.yml # Root bundle config
├── resources/
│   └── sql-warehouses.yml # Shared SQL warehouse
└── projects/
    └── linkedin/
        ├── resources/
        │   ├── jobs.yml # The daily ingest job
        │   ├── pipelines.yml # The Declarative Pipeline
        │   ├── schemas.yml # Bronze, silver, gold schemas
        │   └── volumes.yml # Landing volume
        └── src/
            ├── dashboards/
            │    └── linkedin.lvdash.json # Dashboard
            ├── notebooks/
            │   └── ingest.ipynb # LinkedIn API ingest notebook
            └── pipelines/
                 ├── load_bronze.sql
                 ├── load_silver.sql
                 └── load_gold.sql
            

In this post we’ll create the bundle root, the shared warehouse, and the LinkedIn project’s schemas and volume. By the end you’ll have a working databricks bundle deploy that creates real resources in your workspace.

The structure is built for growth. You can easily add more projects like linkedin and deploy them in one bundle. However, when things get really big and you have several completely separated data products, I recommend to go with multiple bundles. But for this use case we are good to go with this structure with some room for further development.

The full code is available in the accompanying GitHub Repo. Click the Button below to view the repo.

GitHub Repo

The root bundle configuration

Create bundles/databricks.yml. This is the entry point for everything:

bundle:
  name: my-lakehouse

include:
  - resources/*.yml
  - projects/*/resources/*.yml

variables:
  catalog:
    description: "Unity Catalog catalog name for this target"
  service_principal_app_id:
    description: "Application (client) ID of the service principal used to run jobs"

targets:

  dev-user:
    mode: development
    workspace:
      host: https://<your-workspace-url>.azuredatabricks.net
    variables:
      catalog: <your-dev-catalog>
      service_principal_app_id: <your-sp-app-id>

  dev:
    mode: production
    workspace:
      host: https://<your-workspace-url>.azuredatabricks.net
      root_path: /Users/${var.service_principal_app_id}/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog: <your-dev-catalog>
      service_principal_app_id: <your-sp-app-id>

  prod:
    mode: production
    workspace:
      host: https://<your-prod-workspace-url>.azuredatabricks.net
      root_path: /Users/${var.service_principal_app_id}/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog: <your-prod-catalog>
      service_principal_app_id: <your-prod-sp-app-id>

A few things to understand here:

include tells the bundle which YAML files contain resource definitions. The glob patterns mean any new project you add under projects/ is automatically picked up without touching the root config.

variables defines the parameters that differ between environments, primarily the catalog name and the service principal identity. These are referenced throughout all resource files as ${var.catalog} and ${var.service_principal_app_id}.

targets defines three environments, and the distinction between dev-user and dev is worth explaining.

dev-user is for active development on your local machine. It uses mode: development, which automatically prefixes all deployed resource names with your username (e.g. martin_linkedin-medallion instead of linkedin-medallion). This means multiple developers can deploy their own version of the project to the same shared workspace simultaneously without stepping on each other.

dev is an integration environment. It uses mode: production (no name prefixing) and deploys to a fixed path owned by the service principal. The purpose is to have a stable, always-running version of the pipeline in the dev catalog that you can validate against before deploying to prod. It’s what you’d point a shared team dashboard at, for example.

prod is a completely separate workspace with its own catalog and service principal, isolated from dev by design.

If you’re working alone or just getting started, you can simplify to just dev-user and prod pointing at the same workspace with different catalogs. The key is that prod should always use a separate catalog from dev.

Shared resources: the SQL warehouse

Create bundles/resources/sql-warehouses.yml:

resources:
  sql_warehouses:
    serverless_warehouse:
      name: "serverless-warehouse"
      cluster_size: "2X-Small"
      warehouse_type: PRO
      enable_serverless_compute: true
      auto_stop_mins: 10
      channel:
        name: CHANNEL_NAME_CURRENT

This warehouse will be used by the Lakeview dashboard we add in Part 4. It’s defined at the bundle level (not inside the LinkedIn project folder) because it’s shared infrastructure, and other projects in the same bundle can reference it.

enable_serverless_compute: true means the warehouse uses serverless compute rather than classic clusters. auto_stop_mins: 10keeps costs low by shutting down after 10 minutes of inactivity.

If you are running this on Databricks Free Edition, make sure to remove the SQL Warehouse by commenting out the resources/*.yml include in the databricks.yml. There is only one Warehouse allowed in the Free Edition and you will run into a long running (30 mins) deployment and finally into an error. You can use the Warehouse provided by Databricks Free Edition later on.

Project resources: schemas and volumes

Create bundles/projects/linkedin/resources/schemas.yml:

resources:
  schemas:
    bronze_linkedin:
      name: bronze_linkedin
      catalog_name: ${var.catalog}
      comment: "Raw ingested data from LinkedIn"
      grants:
        - principal: <your-user@email.com>
          privileges:
            - ALL_PRIVILEGES
        - principal: ${var.service_principal_app_id}
          privileges:
            - ALL_PRIVILEGES

    silver_linkedin:
      name: silver_linkedin
      catalog_name: ${var.catalog}
      comment: "Cleaned and deduplicated LinkedIn data"
      grants:
        - principal: <your-user@email.com>
          privileges:
            - ALL_PRIVILEGES
        - principal: ${var.service_principal_app_id}
          privileges:
            - ALL_PRIVILEGES

    gold_linkedin:
      name: gold_linkedin
      catalog_name: ${var.catalog}
      comment: "Aggregated LinkedIn metrics for reporting"
      grants:
        - principal: <your-user@email.com>
          privileges:
            - ALL_PRIVILEGES
        - principal: ${var.service_principal_app_id}
          privileges:
            - ALL_PRIVILEGES

Three schemas, one per medallion layer. Each has grants defined inline, meaning your user account and the service principal both get full access. When you deploy to a new environment, the grants are applied automatically. No manual permission management in the UI.

Now create the landing volume in bundles/projects/linkedin/resources/volumes.yml:

resources:
  volumes:
    landing:
      name: landing
      catalog_name: ${var.catalog}
      schema_name: ${resources.schemas.bronze_linkedin.name}
      volume_type: MANAGED
      comment: "Landing zone for raw LinkedIn API data before loading into Delta tables"

The volume lives inside bronze_linkedin and uses MANAGED storage, meaning Databricks controls where the files are stored. The ${resources.schemas.bronze_linkedin.name} reference means the bundle knows the schema must exist before the volume can be created, and the bundle handles the dependency ordering automatically.

The full code is available in the accompanying GitHub Repo. Click the Button below to view the repo.

GitHub Repo

First deploy

With the root config, the warehouse, the schemas, and the volume in place, you can do your first deploy via terminal in VS Code (being in the bundles root folder):

databricks bundle deploy --target dev-user

The CLI will validate your configuration, resolve all variable substitutions, and create the resources in your workspace. The validation can be done without deploying to check if everything is in order. For that, just use validate instead of deploy:

databricks bundle validate --target dev-user

During deployment, you should see output like:

Open your Databricks workspace, navigate to the catalog, and you’ll see the three schemas and the landing volume already there, exactly as defined.

The prefixed shared Serverless SQL Warehouse should appear in the SQL Warehouses section.

Nothing runs yet (although the Serverless SQL Warehouse is started after deployment, but will shut down after 10 minutes if you not terminate it manually). There’s no job, no pipeline, no notebook. But the scaffolding is in place and version-controlled. Everything we add in Parts 2, 3, and 4 will slot into this structure.

If you want to start from scratch, you can destroy all deployed resources like this:

databricks bundle destroy --target dev-user

You get prompted before anything critical like Schemas are destroyed (if they contain data).

What’s next

In Part 2 we’ll connect to the LinkedIn API, walking through the API access setup, building the ingest notebook step by step, and wiring it into the bundle as the first job task. By the end of Part 2 you’ll have raw JSON files landing in the Volume you just created.

Part 2 will be published 2026/04/30.

How to use Query Caching

The SQL query cache in Databricks SQL Warehouses1 can be a powerful tool. Once a query is sent to the Warehouse, its results are cached and reused in subsequent runs.

Query caching is automatically activated in Databricks SQL. You can check if the cache was used by looking at the statement execution details.

If a query is run for the first time, it is fully executed and stored in the cache.

-- First invocation, populates cache
SELECT 
  * 
FROM 
  samples.bakehouse.sales_customers; 

Subsequent queries will use the cached result. All these queries will utilize the cached results from the first query:

-- Reads from cache 
SELECT 
  * 
FROM 
  samples.bakehouse.sales_customers; 

-- Reads from cache, too
select 
  * 
from 
  SAMPLES.BAKEHOUSE.SALES_CUSTOMERS; 

-- Still reads from cache :-)
SELECT 
  * 
FROM 
  samples.bakehouse.sales_customers sc; 

Note that the cache is not case-sensitive, SELECT * FROM table_name yields the same result as select * from TABLE_NAME. This is a thing that Snowflake handles differently2, because the cache there is case-sensitive.

Getting Query Details

To examine query execution details, use the system tables for query metadata. The following example shows how to retrieve caching information for specific queries from the system.query.history table:

SELECT 
  statement_text,
  total_duration_ms,
  read_rows,
  produced_rows,
  from_result_cache,
  cache_origin_statement_id
FROM
  system.query.history
WHERE 
  statement_id in ("put", "your", "statement_ids", "here")

This example shows four query executions—two distinct queries that were each run twice. Notice that when from_result_cache = true, the read_rows value is 0 and total_duration_ms is significantly lower. The cache_origin_statement_id field provides a helpful reference back to the original query that populated the cache.

System tables like these are invaluable for administrators monitoring cache utilization and identifying opportunities to optimize queries for better cache performance.

Technical Details

There is not only one, but four layers of caches involved. They are checked in the following order:

1. UI Cache: a user-scoped cache in the workspace file system

2. Local Cache: the in-memory cache of the SQL Warehouse that is lost when the Warehouse is terminated

3. Remote Cache: this applies to Serverless Warehouses only and is shared by Warehouses across the Workspace

4. Disk Cache: the last resort for retrieving query results

Here are some more useful facts about caching:

• The cache is invalidated after 24 hours or when the underlying tables get updated.

• If you want to disable caching, you can add SET use_cached_result = false to the top of your script. This could be useful for true “cold” benchmarking of query performance.

• Because of the remote cache, switching from one Serverless Warehouse to another will utilize the cache.

Limitations

• The remote cache is only available in Serverless SQL Warehouses. In practice, this means that the Warehouse can be terminated and started again, and the cache is still used. This will not work with Classic and Pro Warehouses.

• Query caching does not work with dynamic views or functions like current_timestamp(). Using this will lead to cache invalidation and execution of the query.

• Users have no control over the cache lifecycle.

• The user cannot interact with the cache entries such as listing all cached queries or deleting.

Key Takeaways

Query caching in Databricks SQL can dramatically improve performance for analytical workloads with repeated queries. To make the most of it, design your queries to be cache-friendly by avoiding unnecessary dynamic functions, and consider using Serverless Warehouses when cache persistence across warehouse restarts is important for your use case.

Use the system.query.history table to monitor cache utilization and verify performance gains. The from_result_cache and cache_origin_statement_id fields provide valuable insights into which queries benefit from caching, helping you identify optimization opportunities and understand your workload patterns.

• Part 1 is about redirecting the Driver logs from Databricks Clusters to Volume in order to permanently store and analyze them.

• In Part 2 we are creating a custom logger object in Python that can be used in the data processing framework of the Data Lakehouse.

• Part 3 is on how the resulting cluster log files can be analyzed using Databricks onboard tools.

In this third and final part we will make use of the log entries we created. The log files could be analyzed with specific tools like Grafana or Azure Monitoring (Log Analytics), but we will stick to an all Databricks approach. The log entries will be loaded into delta tables and prepared for further analytics. Finally, we will build a simple dashboard to visualize the log information.

Create some (synthetic) Log Entries

For this tutorial, I have used the Logger to create some artificial log entries. I simply created a Notebook and ran the following code multiple times and with different names for the Logger.

import random
import time

from databrickslogger import DatabricksLogger

# Define log levels and their weights (higher weight = more frequent)
log_levels = [
    (logging.INFO, 40),    # INFO: 40% chance
    (logging.DEBUG, 30),   # DEBUG: 30% chance
    (logging.WARNING, 20), # WARNING: 20% chance
    (logging.ERROR, 10)    # ERROR: 10% chance
]

# Generate artificial logs
def generate_logs(num_logs=200):
    for _ in range(num_logs):
        # Choose a random log level based on weights
        level = random.choices(
            [l[0] for l in log_levels],
            weights=[l[1] for l in log_levels],
            k=1
        )[0]

        # Generate a log message
        if level == logging.ERROR:
            message = f"Failed due to {random.choice(['timeout', 'permission denied', 'corrupt data'])}"
        elif level == logging.WARNING:
            message = f"{random.choice(['Slow response', 'Partial data loaded', 'Retrying...'])}"
        elif level == logging.DEBUG:
            message = f"{random.choice(['Debug info: row count=1000', 'Debug info: schema validated', 'Debug info: connection established'])}"
        else:  # INFO
            message = f"{random.choice(['Success', 'In progress', 'No issues detected'])}"

        # Log the message
        logger.log(level, message)

        # Simulate delay between logs
        time.sleep(random.uniform(0.1, 0.5))

# Create Logger Object
logger = DatabricksLogger('DataLakehouseLogger')

# Run the log generator
generate_logs()

This generates log entries with the configured distribution of log level with some artificial messages. To test the loading mechanism of log files, this is sufficient.

Read the Log with a Lakeflow Declarative Pipeline

Assuming the DatabricksLogger was used to create some log entries (either real or synthetic like described above), we now proceed to read the log files, process them and write them into some tables. This will be a two-step process: First, we will read all log data from the driver stdout into a table. Second, we will extract only the entries created by the DatabricksLogger and write them to another table.

We will use Spark Declarative Pipelines and SQL to create the data processing pipeline.

Create a new pipeline like this:

• Go to Pipelines & Jobs

• Select ETL pipeline

• Give it a name like Process Log Files

• Select Start with an empty file

• Select SQL as language for the first file

• On the on the right side next to the pipeline, configure your default catalog and schema for this pipeline. If no further name spaces are used in the code, this will be used.

Rename the my_transformation.sql to driver_stdout_raw.sql in the transformations folder and add the following content.

CREATE OR REFRESH STREAMING TABLE 
  driver_stdout_raw
COMMENT "Raw Driver Log Data."

AS 

SELECT 
  value, 
  current_timestamp AS _load_timestamp, 
  _metadata.file_path AS _file_path
FROM STREAM 
  read_files(
    '/Volumes/<catalog-name>/system/logging/*/driver/stdout*',
    format => "text"
  )

This will read all driver log stdout data from all clusters that wrote their log data into the volume. The table is defined as a STREMING TABLE reading from a Volume. That means the Databricks Autoloader is utilized and only new files will be processed. If no updates are found, no data is loaded.

Note: In order to not only catch new files but also changed files, enable file events for an external location.

Next, we extract the DatabricksLogger data and stored them in a MATERIALIZED VIEW. We split the value from the log file using the logger pattern defined in our custom DatabricksLogger. Add a logger_data.sql to the transformations folder with this code.

CREATE OR REFRESH MATERIALIZED VIEW 
  logger_data
COMMENT "Data From Databricks Logger."

AS

SELECT 
  SPLIT(value,"\\|")[0] AS app,
  TO_TIMESTAMP(
         REGEXP_REPLACE(SPLIT(value,"\\|")[1], ',', '.'),
         'yyyy-MM-dd HH:mm:ss.SSS'
       ) AS timestamp,
  SPLIT(value,"\\|")[2] AS level,
  SPLIT(value,"\\|")[3] AS message,
  _file_path AS file_path
FROM 
  driver_stdout_raw
  
WHERE SPLIT(value,"\\|")[3] IS NOT NULL

Now you can run the Pipeline and see if everything works without error. In this case it should look something like this.

Finally, create a schedule to run the job periodically to refresh the tables.

I recommend turning the pipeline into a Databricks Asset Bundle and deploy it to your workspace.

Visualize Data on a Dashboard

The data can now be analyzed using SQL. We will go a step further and create a simple Databricks Dashboard.

• go to Dashboards

• click Create dashboard

• give the dashboard a name like Databricks Logger

• on the data tab, click Add data source and select the logger_data materialized view

• go to the Untitled page tab and rename it into Overview

You can now create visualizations for your data. I did the following:

• added a filter for App

• added a counter for the total number of log entries

• added a bar chart to show number of logs per App

• added a bar chart to show number of logs per Log-Level

• added a table with the full log data

The result looks like this.

You should publish your Dashboard (I recommend Individual Data Permissions) and share it with others.

Wrap Up

In this comprehensive end-2-end tutorial I presented my approach on how to create a custom Python logger for Databricks. I focused on using Databricks built in features like redirecting the cluster logs into a Volume (Part 1), creating a custom logger object from the standard Python logging module (Part 2) and finally ingesting the log data into tables to enable log analytics (Part 3).

Thank you for reading and following along. If you liked this tutorial consider subscribing to my Substack.

• Part 1 is about redirecting the Driver logs from Databricks Clusters to Volume in order to permanently store and analyze them.

• In Part 2 we are creating a custom logger object in Python that can be used in the data processing framework of the Data Lakehouse.

• Part 3 is on how the resulting cluster log files can be analyzed using Databricks onboard tools.

In the first part of this tutorial, we made sure that the log files are permantly stored in a Unity Catalog Volume. The second part is about proper logging inside Python that will show up in the cluster logs.

Create and Test a Custom Python Logger Class

Note: All things described in this tutorial will work without completing Part 1: Deliver Cluster Logs to a Volume. However, in order to properly analyze the logged events this should be taken care of.

In order to quickly create and test a custom Python logger class, go to your Databricks Workspace and create a Notebook. In the first cell copy and paste the following code.

import logging
import sys

class DatabricksLogger(logging.Logger):
    """
    A custom logger class that extends logging.Logger to facilitate    
    logging with configurable log levels using string inputs. 

    The logs will be directed to the stdout destination. Make sure to
    redirect cluster logs of the Databricks Cluster to a Volume for      
    later use.

    This class allows users to set the logging level using strings, 
    abstracting the need to import and use the logging module directly.

    Parameters
    ----------
    name : str
        The name of the logger.
    level : str, optional
        The logging level as a string (e.g., 'DEBUG', 'INFO'). 
        Default is 'DEBUG'.

    Attributes
    ----------
    _LOG_LEVELS : dict
        A dictionary mapping string representations of log levels to 
        their corresponding logging level constants.
    """

    # Define a mapping of level names to logging levels
    _LOG_LEVELS = {
        'DEBUG': logging.DEBUG,
        'INFO': logging.INFO,
        'WARNING': logging.WARNING,
        'ERROR': logging.ERROR,
        'CRITICAL': logging.CRITICAL
    }

    def __init__(self, name, level='DEBUG'):
        """
        Initialize the DatabricksLogger with a specified name and
        logging level.

        Parameters
        ----------
        name : str
            The name of the logger.
        level : str, optional
            The logging level as a string (e.g., 'DEBUG', 'INFO'). 
            Default is 'DEBUG'.
        """
        # Map the string level to the corresponding logging level 
        # constant
        log_level = self._LOG_LEVELS.get(level.upper(), logging.DEBUG)
        super().__init__(name, log_level)
        self._setup_logging(log_level)

    def _setup_logging(self, level):
        """
        Set up the logging configuration with the specified level.

        Parameters
        ----------
        level : int
            The logging level constant from the logging module.
        """
        handler = logging.StreamHandler(sys.stdout)
        formatter = (
            logging
                .Formatter(
                    '%(name)s|%(asctime)s|%(levelname)s|%(message)s'
                )
        )
        handler.setFormatter(formatter)
        # prevent duplicate handlers
        if not any(
            isinstance(h, logging.StreamHandler) 
            and h.stream == sys.stdout
            for h in self.handlers
        ):
            self.addHandler(handler)

        self.setLevel(level)

Our DatabricksLogger directly inherits from the build-in Python logging facility.1 Basically, we are only changing three things:

1. We create a Mapping from strings to logging objects for the log levels. This increases usability since we only have to pass the log level as a string (I love string parameters, although I know some would like to kill me for that).

2. We add a StreamHandler that redirects the logs into the stdout destination, which in Databricks are the Cluster logs. This way, we don’t have to take care of the file handling of the log files since it is already implemented by Databricks.

3. The formatting of the log string ist defined as the name of the Logger, the timestamp, the log level and the actual message. They are all divided by a pipe so we can comfortably parse them later on.

To test the logger, run the cell. The compute must not be a serverless instance since log file delivery is not available yet for this type of compute.

Next, create another cell and an instance of our logger. With this logger object, anything might be logged within Notebooks or custom Python packages using this logger.

logger = DatabricksLogger("DataLakehouseLogger")

logger.info("Logger is working")

This results in something like this in the console:

DataLakehouseLogger|2025-08-13 10:23:20,065|INFO|Logger is working

When inspecting the driver logs of the cluster this was running on, you will see something like this.

Creating a Python Package

Now that we have tested our logger, it is time to create a Python package that can be used in our Databricks Workspace.

Setup the Repo Structure

Note: Instead of creating this Repo from scratch like described here, you can download it from my GitHub.

GitHub Repo

Go to your favorite version control platform and create a Git-Repository. Check it out to your local computer. I use Azure DevOps or GitHub in combination with VS Code most of the time.

We will create a very basic Python package with only the absolutely necessary files. For this, create the following folder structure with these files in it.

DatabricksLogger/
│
├── src/
│   │
│   └── databrickslogger
│       ├── __init__.py
│       └── logger.py
│
├── .gitignore
└── setup.py

Copy & paste the code from our very first cell from the Notebook which defines the custom logger class into logger.py. Next, add the following code to the __init__.py file:

from .logger import DatabricksLogger

Note: This will make sure we will later be able to directly import the logger like this:

from databrickslogger import DatabricksLogger

If we wouldn’t do this, we would have to import it like this, which is a bit ugly:

 from databrickslogger.logger import DatabricksLogger

In the root of your repository, add to the setup.py file the following content:

from setuptools import setup, find_packages

setup(
    name="DatabricksLogger",
    version="0.1",
    author="Martin Debus",
    author_email="martin.debus@snowglobe.ai",
    description="A simple Databricks Logger",
    python_requires=">=3.9",
    classifiers=[
        "Programming Language :: Python :: 3",
        "Operating System :: OS Independent",
    ],
    license="MIT",
    packages=find_packages(),
)

You may change the personal information or apply your own versioning scheme.

Finally, add the following lines to the .gitignore file:

dist
DatabricksLogger.egg-info

This keeps your repository clean from the files created during the build process of the wheel package file when committing changes.

Create a Wheel

With all the necessary files in place, we will proceed to create a wheel file. In this tutorial, I will describe how to do that using VS Code on Mac. The process is quite similar on Windows.

1. Make sure you have checked out the repository and the files to your local machine.

2. In VS, open the terminal and navigate to the root of the repository.

3. Run: python setup.py bdist_wheel

This creates a wheel file in the dist folder (this folder is automatically created).

This wheel might now be used in any Package manager. We will skip this step and will directly upload into Databricks. This is not ideal for production environments, but since we do not intend to change the logger frequently, for this use case this is the most straightforward approach.

Use the Logger in Databricks

To use the Logger in Databricks we have to upload it into a Volume and add it to the cluster libraries. Then we can use it in our code.

Upload to Volume

We will now upload the wheel file to a Volume in Unity Catalog where any Cluster might be able to access it. In Part 1 of this tutorial we created a system schema and within this a Volume named packages. Upload the wheel file into this Volume.

Install on Cluster

Next, go to your Cluster and do the following:

• go to the libraries section

• click install

• select Volume

• browse to your wheel file and select it

It will now be installed on your cluster every time you start it.

Note: For any job cluster, this must be defined as a library in the job configs in order to use it.

Usage in code

In any Notebook or Python package you can now use the logger to log events that will end up in the log files in the logging Volume like this:

from databrickslogger import DatabricksLogger

logger = DatabricksLogger("DataLakehouseLogger")

logger.info("Logger is working")

Wrap Up

This concludes the second part of the series Simple Python Logger Framework for Databricks. We created a custom Python logger object, packaged it to a wheel file and deployed it to our Databricks Workspace. Using this logger writes the log messages to the console as well as to the Cluster logs.

The last part of the series will dive into how to use the cluster logs and analyze them: Part 3: Analyze Cluster Log Files in Databricks SQL.

• Part 1 is about redirecting the Driver logs from Databricks Clusters to Volume in order to permanently store and analyze them.

• In Part 2 we are creating a custom logger object in Python that can be used in the data processing framework of the Data Lakehouse.

• Part 3 is on how the resulting cluster log files can be analyzed using Databricks onboard tools.

Why Logging Matters

When building a Data Lakehouse from scratch, most start by adding value through the integration of data sources. I believe this is the right approach because it allows you to deliver business value as quickly as possible. However, there are some downsides to this method. After integrating the second data source, you may realize that some kind of framework to automate processes would be very helpful. Otherwise, you risk ending up with a growing collection of ingestion and data transformation scripts that become difficult to manage.

As you progress, you might find that information about your processing is needed to debug or audit your data pipeline runs. More often than not, this leads to the use of print commands. While this approach is valid for debugging and development purposes, it is not suitable for a production environment.

Therefore, implementing a lightweight logging framework would be beneficial to keep things organized and structured. Here are some common use cases for logging in Data Lakehouses:

• Good coding practice: Avoid using print statements in code; instead, use Python logging.

• Keep logs for audit purposes.

• Manage numerous job runs efficiently without manually searching through them; have all logs in one place.

• Utilize log analytics frameworks like Azure Log Analytics.

Note: All technical details explained in this article apply for Azure Databricks in Azure region Western Europe. They were verified and tested by me in August 2025. They may be subject to changes by Databricks in the future. Feature enablement and technical details may vary for other regions and cloud providers. Please refer the official documentation.

These are the assumptions and prerequisites to implement this solution:

• An Azure Databricks Workspace is in place.

• A catalog was created in Unity Catalog and attached to the Workspace by an Account Admin.

• You are an Admin or member of a special group like developers in the Databricks Workspace.

• (Optional) You have a Service Principal that you are allowed to use in the Workspace and this service principal is Admin or in the special group like developers.

The steps described in the following sections will enable you to recreate it manually step-by-step. Creating a CI/CD process that can easily be deployed to several environments is explicitly out of scope for this tutorial. However, we take care that the resulting artifacts can be integrated into some CI/CD strategy (e.g. Azure DevOps, Databricks Asset Bundles, GitHub Actions) with low effort.

Driver Logs in Databricks

Every Compute type in Databricks creates log files. There are several log files for the driver, the executor, event logs and even init_script logs. We will focus on the driver logs here. They can be accessed in the Driver Logs section of a Cluster in the UI.

There are three types of driver logs delivered:

• stdout (Standard Output): This log captures the standard output of the driver node. It typically includes the results of print statements, logs from user applications, and other output generated by the execution of code on the driver. It's useful for tracking the progress and output of your applications.

• stderr (Standard Error): This log captures error messages and stack traces that are output to the standard error stream. It is particularly useful for debugging issues as it contains error messages and exceptions that occur during the execution of your code on the driver node.

• log4j: This log contains messages generated by the Log4j logging framework, which is used by Databricks and many other Java-based applications. It includes detailed log messages from various components of the Databricks environment and your applications, allowing for comprehensive logging and monitoring.

By default, Databricks retains cluster logs (that includes the driver logs) for 30 days. They may also be manually be purged from the Workspace.

Redirect the Cluster Logs to a Volume

To permanently store the Cluster logs they can by redirected into a Volume. This feature was introduced by Databricks in March 20251. This is currently (as of August 2025) not available for Serverless Compute. Therefore this will not work with the Databricks Free Edition.

To set it all up, we will need to do these two steps:

• Setup a Schema and Volume as a destination for the logs

• Configure your clusters to write the logs to this destination

Create Schema and Volume

If you already have a Schema and Volume where system related data is stored, you can use them. If not, create a Schema and/or Volume as a destination for the logs.

I recommend creating there using code to apply via your CI/CD in all environments in the same manner. I assume you have a catalog tied to your workspace in Unity Catalog and you will be using this. I my case this the catalog is called snowglobe_demo_dev_we since it is for demo purposes on my dev environment in Western Europe.

In Databricks, create a new Notebook named something like init-system-schema. Put the following code in this Notebook:

# Create a Widget to get the catalog and schema name
dbutils.widgets.text("catalog", "snowglobe_demo_dev", "Catalog")
dbutils.widgets.text("schema", "system", "Schema")

# Get the catalog and schema name
catalog = dbutils.widgets.get("catalog")
schema = dbutils.widgets.get("schema")

# Create a managed Schema system
spark.sql(f“CREATE SCHEMA IF NOT EXISTS {catalog}.{schema}”)

# Grant access for developers group (including SP)
spark.sql(f"GRANT ALL PRIVILEGES ON SCHEMA {catalog}.{schema} TO `developers`")

# Create a managed Volume logging
spark.sql(f"CREATE VOLUME IF NOT EXISTS {catalog}.{schema}.logging")
# Create a managed Volume package
spark.sql(f"CREATE VOLUME IF NOT EXISTS {catalog}.{schema}.package")

The catalog and schema name can be dynamically set by a Widget in order to work in different environments and may be injected via a Job Parameter. I am going with the name system for the Schema and logging and package for the Volume to describe their purpose.

Note that the developers group gets all privileges on the Schema. You may want to restrict that in a production environment.

Run this Notebook in interactive mode or create a Job with one Task running this Notebook. I would recommend running it as a Job in the name of the Service Principal.

Now, we have set up the location where the log should be delivered. In the next step we will make sure the log are delivered to this location.

Configure Clusters

There are several ways to configure the clusters to deliver the cluster logs to the Volume. You can use one or any of them to fit your needs.

Databricks Web UI

The most basic way ist to simply use the Databricks UI to configure a cluster to deliver the log to the Volume.

1. Go to the Cluster and click Edit

2. Go to Advanced Options

3. Select the Logging tab

4. Select Volume as Destination

5. Browse your Catalog to the Volume and select it as path or simply paste the path into the field

To quickly go ahead and try things out, this is definitely the way to go. However, this is not very convenient for multiple clusters and environments, so it might be advisable to automate this using Terraform or configuration files. We will briefly cover those in the next sections.

Terraform

If you are deploying your Databricks resources like clusters and/or jobs via Terraform, simple add this to the databricks_cluster resource2:

cluster_log_conf {
  volumes {
    destination = "/Volumes/${local.catalog_name}/system/logging/"
  }
}

This assumes that the catalog name is set in the Terraform locals to keep it dynamic.

Job Cluster Configuration

If you manage your jobs using yml-config files, add the following to the job configuration:

cluster_log_conf:
  volumes:
    destination: /Volumes/{catalog}/system/logging

Cluster Policies

Cluster Policies are also a convenient way to configure cluster log delivery. You can either to it by hand and overwrite the policies or use the Databricks SDK to automate it and apply dynamic resolution of environment specific variables. The code looks quite similar to the job cluster configuration:

cluster_log_conf.path:
  type: fixed
  value: /Volumes/{catalog}/system/files/cluster_logs

Detour: Resolve Variables in Configuration Files

In order to dynamically resolve variables like the catalog that might vary between environments, you can apply some custom Python functions to the configuration files.

First, add a function that recursively formats values in a dictionary or list. The format_values dictionary contains the mapping of the actual variables to values like “catalog” : “snowglobe_demo_dev_we”.

def format_dict_values(
    dictionary: dict, format_values: dict = {}
) -> dict:

    def recursive_format(obj):
        if isinstance(obj, str):
            return obj.format(**format_values)
        elif isinstance(obj, dict):
            return {k: recursive_format(v) for k, v in obj.items()}
        elif isinstance(obj, list):
            return [recursive_format(item) for item in obj]
        else:
            return obj

    return recursive_format(dictionary)

Second, when reading the configuration, e.g. the jobs.yml where jobs are described, create a mapping dictionary and apply the format_dict_values function to the configuration. In our case, the implementation of the get_catalog_name() function dynamically returns the currently used catalog (e.g. the dev or prod catalog). When applied to the configuration file, the variables {catalog} will be replaced with the actual value during runtime.

import yaml 

with open(f"{path}/jobs.yml", 'r') as file:
  jobs_config = yaml.safe_load(file)

mapping = {"catalog": get_catalog_name()}

jobs_config_mapped = format_dict_values(jobs_config, mapping)

Wrap Up

After the cluster log destination is configured, the cluster logs are delivered every five minutes to the Volume based on a best effort method. That means:

• If the Volume does not (yet) exists, the logs are not delivered. This is done silently, so no error occurs.

• The user running the cluster must have write access to the Volume. If not, the logs are not delivered.

The logs show up in folders named with the corresponding cluster id.

This concludes the first part of the series Simple Python Logger Framework for Databricks. The next parts of this series are:

• Part 2: Create a Python DatabricksLogger

• Part 3: Analyze Cluster Log Files in Databricks SQL

This changed when I first encountered PySpark.

That was back in the days when we ran a Hadoop Cluster on-premise and had Spark installed on top. I began exploring PySpark, because we had a ton of data we could not handle with Pandas. While it often required more code to achieve the same goals as Pandas, it felt more natural to me. And over time, I began to understand why.

One of the first things that clicked was how PySpark encourages chaining operations. I love that style: transforming data in a series of clearly separated steps, each returning a new DataFrame. No surprises. No accidental overwriting. No mysterious inplace=True flags that behave inconsistently. In PySpark, every transformation is explicit and composable. The whole pipeline feels clear and easy to follow, especially when revisiting your code months later.

With Pandas, I often found myself juggling side effects and debugging strange behaviors. Something about the way Pandas allows (and sometimes encourages) mutation of data structures in place, combined with its rich but sometimes inconsistent syntax, just didn’t sit well with me.

I could say that Pandas is just not my style, but I tried to dig a bit deeper.

Pandas and PySpark in Action

To illustrate the difference let's take the simple operation of adding a column to a Dataframe. We will look at the Pandas way and at the PySpark way.

We create a simple dataset and add a new column by multiplying an existing one with the factor 2.

Pandas

import pandas as pd  
df = pd.DataFrame(
  {
     "id": [1, 2, 3, 4], 
     "value": [10, 20, None, 40] 
  }
)  

# Create a new column
df['value_doubled'] = df['value'] * 2

print(df)

This modifies df directly. The original version of df does not exist any longer.

PySpark

import pyspark.sql.functions as F

df = spark.createDataFrame(
  [(1, 10),(2, 20),(3, None), (4, 40)], 
  ["id", "value"]
)  

# Transformations always return a new DataFrame 
df_new = df.withColumn('value_doubled', F.col('value') * 2)

With PySpark, the original df is preserved. You always assign the result of a transformation to a new variable. Even more, PySpark encourages chaining of commands, which I especially like. Say you want to add another column, you can simple adjust the code to this:

df_new = (
  df
    .withColumn('value_doubled', F.col('value') * 2) # first new column
    .withColumn('department', F.lit('Marketing')) # second new column
)

The dot syntax allows for clean chaining of commands. This is also possible in Pandas, but not always smart. To find out why, we have to take a little look under the hood.

The technical difference

The core difference between Pandas and PySpark is how they run operations and where the data lives during processing.

Pandas runs eagerly. Every time you write a line like df['value'] * 2, the operation is executed right away. The result is held in memory. If your dataset is small, this is fast and efficient. But as data grows, so does memory usage. And unless you explicitly copy your DataFrame, you're often modifying it in place. That can be risky. You might overwrite something by accident or struggle to track what changed and when.

PySpark works the other way around. It uses lazy execution. When you write withColumn or filter, nothing happens immediately. You're not processing data yet. You're building a plan. Spark collects all the steps you've described and waits until you trigger an action. That could be something like .show(), .collect(), or writing the data out. Only then it starts doing the actual work.

This is a big difference. Because Spark waits, it can look at the full plan and find the most efficient way to run it. It can skip steps that don’t matter. It can reorder operations. It can push filters down to the data source so less data needs to be loaded in the first place.

Another important point is mutability. Pandas allows you to change data in place. That can save memory, but it also opens the door to side effects. PySpark takes a different approach. Each transformation creates a new DataFrame. The original stays untouched. This leads to cleaner code and fewer surprises, especially in larger projects or shared environments.

So the technical difference isn’t just about memory or performance. It’s also about how they execute. Pandas is immediate and flexible, great for quick work. PySpark is deliberate and structured, built for scale and clarity.

Wrap Up

Technically, Pandas and PySpark are very different, of course. Pandas runs in-memory on a single machine and is great for small to medium-sized datasets. PySpark was designed for distributed computing from the beginning. It can handle massive datasets by spreading the work across a cluster. That adds complexity, but also enables scalability.

Pandas is working in memory and for this reason is very fast. But this comes with limits, too, because if Pandas runs out of memory, the processing will fail. So, one strategy to avoid this, is to change objects in place without replicating them. From a memory point of view, this is the smartest thing you can do.

PySpark brings a more functional mindset. You don’t mutate, you transform. And although the syntax can be more verbose, it enforces a discipline that leads to cleaner code structure. That creates a predictable data flow and allows easier debugging. At least for me.

Today, I always use PySpark. Not just because of the scale it offers, but because I genuinely prefer the structure, clarity, and coding style. Even when my data would fit into memory, I find myself reaching for PySpark. It just feels right.

Waiting for a Databricks cluster to start up is frustrating—especially when you just want to test something or run a small piece of code against your data.

Let’s assume you start a compute cluster 2–3 times a day and wait 4–5 minutes for it to be ready. Replace that with a 15-second startup time, and you’ll save over 40 hours of waiting throughout the year. That’s based on an average of 2.5 cluster starts per day, 200 workdays, and a 5-minute waiting period.

There are still some limitations, like custom Python environments, but Databricks is addressing those. Meanwhile, we can enjoy the time saved running (and debugging) our code.

So, why not test serverless while you wait for your cluster to start?

Who I am

I am Martin, a Data Professional since 2008. I started out as a Data Scientist in the publishing industry, learning the basics using SAS. After three years I moved on to grocery retail and build a team analyzing customer behaviour and building data products around the loyalty program. Over time, my role changed more in the direction on how to architecture the next generation of our data platform. Still being on SAS in 2015, I began exploring Python and ended up with Databricks in 2018 when it was first launched on Azure, which was the cloud provider of our choice back then. Since then, I am designing and building Data Solutions with Databricks.

In 2022, I launched a boutique consultancy called SNOWGLOBE, based in Hamburg, Germany. We focus on enabling our clients to get the most out of their data with Azure Databricks. We already built custom data solution for the retail, construction and financial industry. Our clients range from mid-sized to large scale businesses.

What to Expect

In “The Lakehouse Path” I share what I am learning building data lakehouses and using Databricks. These are the main content categories you will see on this blog:

• Databricks Bits & Pieces: On LinkedIn, I share Bit & Pieces about best practises using Databricks. From time to time, I will compile them into longer and more comprehensive posts in this blog.

• Deep-Dive Tutorials: Occasional in-depth guides and tutorials that explore complex topics and advanced techniques in Databricks.

Thank you for joining me on "The Lakehouse Path."

Jetzt abonnieren