The month in short
A personal highlight for me this month was that I have been selected as Databricks MVP. The message came in when I was on vacation at the danish baltic coast. I had to celebrate by jumping in the air with my Omnigent shirt on.
In the current batch of MVPs were a total of 20 professionals across the globe with different subject matter areas of expertise. Check out the whole list of Databricks MVPs (at the time of this writing the list was not updated yet with the new MVPs).
Release radar
The speed of development of the Databricks Platform is incredible. I was three weeks away and when I came back there were 47 new entries in the platform release notes. Here is what caught my eye.
Unity Catalog Secrets
What shipped: Secrets in Unity Catalog went GA on August, 3. Instead of using a secret scope, secrets can now additionally stored in Unity Catalog. They are stored on the schema level. Secret values can be retrieved in the code pretty much the same way as from secret scopes, the only difference is the three layer namespace in the call.
Why it matters: Secrets are better discoverable since they are displayed in Unity Catalog and can be found using the search mechanism. Also, Unity Catalog permissions apply to the secrets.
Watch out: Some limitations apply like no support for SQL Warehouse, only Databricks Runtime 17.3 LTS and above and Quotas of 1,000 secrets per metastore and 100 per schema. The question is: migrate from secret scope to Unity Catalog or not? And how is a proper migration path when there are 100+ users on the platform already?
Link: https://learn.microsoft.com/en-us/azure/databricks/security/secrets/unity-catalog-secrets
Unity AI Gateway GA
What shipped: Unity AI Gateway went GA as well on the beginning of August. This way users have a better way to manage the use of AI models within Databricks.
Why it matters: That AI is a big thing is obvious and Databricks does its part to deliver governance for models inside Databricks. This covers permissions, traffic routing, MCP server governance and general cost control and lineage. Part of that is also the budget feature, which I covered in a recent blog post.
Watch out: Some features like agent policies and service policies remain in Beta.
Tag Automation (Beta)
What shipped: Tags can be applied or removed automatically to Unity Catalog objects by a set of business rules defined by the user.
Why it matters: It can be a tedious task to assign and manage tags to Databricks resources. But they have huge benefits for cost monitoring, rule based access control and providing general context in Unity Catalog. With this feature, tag application can be automated. E.g. when a table meets certain criteria like it was not updated in the last month a tag “deprecated” could be applied.
Watch out: The feature is currently only in Beta.
Link: https://learn.microsoft.com/en-us/azure/databricks/admin/governed-tags/automate-tag-assignment
Genie One Free usage extended through end of January 2027
What shipped: Genie One free usage is extended until 31 January, 2027.
Why it matters: Databricks is heavily leaning on the Genie lineup and is pushing Genie One as a means to simplify analytics for business users. So this is a chance to try it for free.
Watch out: With all the Genie products it can be confusing what is covered and what is not. It is for Genie One (excluding service principal usage) and not for Genie Code. Genie Code pricing remains as described in the previous newsletter, but gets a 25% off on the cost once the initial free 150 DBUs are consumed.
The full list of release notes for August 2026 has some more highlights like Role-based access control went GA.
Feature of the month: serverless compute access control
This is the one feature I’ve been waiting for ever since interactive serverless was introduced in Databricks.
Over the past year, I got more calls than I can count from people claiming their code wasn’t running properly. They swear it ran yesterday. Most of the time, it turned out they’d accidentally run their code on serverless, which wasn’t properly configured in their organization for several reasons (e.g. missing network configuration for serverless in private networks). Switching to a classic compute cluster they used yesterday solved the problem, but many people had already sunk a lot of time into it. The usual question was: “Can’t we turn off serverless at all if it doesn’t work for us?” The answer was always: “No”.
That changed recently with the newly introduced serverless compute access control. Now you can remove the permission to use serverless — controlled separately for interactive and automated compute — from all users, and grant it only to selected groups.
This will save a lot of time and frustration for many users.
This little video shows how the access to all workspace user can be removed and specific groups are allowed to use serverless compute.
Worth reading and watching
Omnigent Policies
Zoë Van Noppen has created another video on Omnigent. This time she gives a deep dive into how Omnigent can enforce policies on three levels. She explains the concept in a visual way without using slides or digital help. Curious how she did that? Just watch the video!
Data Contracts in Practice
This piece from Peter Baumann is from May, but I just stumbled upon it this month.
The useful line Peter draws is between a contract that lives in Confluence and one the pipeline actually enforces at runtime. I see the first kind constantly on client work, signed off in a governance meeting and never checked again. The second kind is where Unity Catalog and policy as code start earning their keep, and it’s the harder thing to build. The bit worth your time is the reframe that a contract is infrastructure you version and monitor like an API, not a document you file. If you work with federated data across domain teams, that’s the section to read.
Rebricked
Last month Databricks renamed Genie Spaces into Genie Agents. It seems Databricks is renaming things as much as they throw out new features. To keep up with the changes Maksim Pachkouski and Ilya Aniskovets launched a website called REBRICKED. You can even contribute on GitHub.
It is nicely designed like the Databricks workspace and you can get a list of features from the release notes. They keep track on the renaming of products and even provide an AI generated name for the next renaming of the feature.
Check it out, it is fun!
From my desk
As usual, I keep posting every week about Databricks on LinkedIn, check out my posts if you missed them:
Declarative Asset Bundles lookup feature: Learn how to lookup IDs from SQL Warehouses and other resources by their name in DABs.
SQL dynamic parameters: Learn how to use dynamic parameters in SQL based on a query.
Serverless Compute access control: As shown above in the feature of the month.
Lakebase is available in four more regions: I‘ve waited some time for region Germany West Central to be supported, now I can start using it for my clients.
Normally, I prepare my Posts and send them out manually. But since I was on vacation, I tried Buffer (no affiliate in any way) and it worked really well. The free version can handle up to 10 scheduled posts.
Finally I got to get Omnigent working on my machine and played around a little bit. But I am still a beginner, so nothing to share here.
One question
How is your migration path for secrets into Unity Catalog?