Because the Information Platform group at Databricks, we leverage our personal platform to supply an intuitive, composable, and complete Information and AI platform to inside information practitioners in order that they’ll safely analyze utilization and enhance our product and enterprise operations. As our firm matures, we’re particularly motivated to determine information governance to allow safe, compliant and cost-effective information operations. With hundreds of workers and a whole lot of groups analyzing information, we now have to border and implement constant requirements to attain information governance at scale and continued compliance. We recognized Unity Catalog (UC), typically obtainable as of August 2022, as the inspiration for establishing commonplace governance practices and thus migrating 100% of our inside lakehouse to Unity Catalog turned a prime firm precedence.
Why migrate to Unity Catalog to attain Information Governance?
Information migrations are HARD – and costly. So we requested ourselves: Can we obtain our governance targets with out migrating all the info to Unity Catalog?
We had been utilizing the default Hive Metastore (HMS) in Databricks to handle all of our tables. Constructing our personal information governance options from scratch on prime of HMS could be a wasteful endeavor, setting us again a number of quarters. Unity Catalog, alternatively, offered large worth out of the field:
- Any information on HMS was readable by anyone. UC securely helps fine-grained entry.
- HMS doesn’t present lineage or audit logs. Lineage assist is essential to understanding information flows and empowering efficient information lifecycle administration. Together with audit logs, this supplies observability about information modifications and propagation.
- With higher integration with the in-product search function, UC permits a greater expertise for customers to annotate and uncover high-quality information.
- Delta Sharing, question federation and catalog binding present efficient choices to create cross-region information meshes with out creating safety or compliance dangers.
Unity Catalog migration begins with a governance technique
At a excessive degree, we may go down one in every of two paths:
- Elevate-and-shift: Copy all of the schemas and tables as is from legacy HMS to a UC catalog whereas giving everyone learn entry to all information. This path is low degree of effort within the brief time period. Nevertheless, we threat bringing alongside outdated datasets and incoherent/dangerous practices motivated by HMS or natural progress. The chance of needing a number of massive subsequent migrations to wash in place could be excessive.
- Transformational: Selectively migrate datasets whereas establishing a core construction for information group in Unity Catalog. Whereas this path requires extra effort within the brief time period, it supplies an important course-correction alternative. Subsequent rounds of incremental (smaller) clean-up could also be crucial.
We selected the latter. It allowed us to put the groundwork to introduce future governance coverage whereas offering the requisite skeleton to construct round. We constructed infrastructure to allow paved paths that ensured clear information possession, naming conventions and intentional entry, versus opening entry to all workers by default.
One such instance is the catalog group technique we selected upfront:
| Catalog | Objective | Governance |
|---|---|---|
| Customers | Particular person person areas (schemas) |
|
| Crew | Collaborative areas for customers who work collectively |
|
| Integration | House for particular integration initiatives throughout groups |
|
| Primary | Manufacturing setting. |
|
Challenges
Our inside information lake had develop into extra of a “information swamp” over time, as a result of beforehand highlighted lack of lineage and entry controls in HMS. We didn’t have solutions to three primary questions vital to any migration:
- Who owns desk foo?
- Are all of the tables upstream of foo already migrated to the brand new location?
- Who’re all of the downstream clients of desk foo that should be up to date?
Now think about that lack of visibility on the scale of our information lake:
Now think about a four-person engineering group pulling this off with none devoted program administration assist in 10 months.
Our Strategy
The migration can virtually be damaged down into 4 phases.
Part 1: Make a Plan, by Unlocking Lineage for HMS
We collaborated with the Unity Catalog and Discovery groups to construct information a lineage pipeline for HMS on inside Databricks workspaces. This allowed us to determine the next:
A. Who up to date a desk and when?
B. Who reads from a desk and when?
C. Whether or not the info was consumed through a dashboard, a question, a job or a pocket book?
A allowed us to deduce the almost definitely house owners of the tables. B and C helped set up the “blast radius” of an imminent migration i.e., who’re all of the downstream shoppers to inform and which of them are mission vital? Moreover, B allowed us to estimate how a lot “stale” information was mendacity round within the information lake that might be merely ignored (and ultimately deleted) to simplify the migration.
This observability was vital in estimating the general migration effort, creating a practical timeline for the corporate and informing what tooling, automation and governance insurance policies our group wanted to spend money on.
After proving its utility internally, we now present our clients a path to allow HMS Lineage for a restricted time frame to help with the migration to Unity Catalog. Discuss to your account consultant to allow it.
Part 2: Cease the Bleeding, by Imposing Information Retention
Lineage observability revealed two vital insights:
- There have been a ton of “stale” tables within the information lake, that had not been consumed shortly, and had been in all probability not value migrating
- The brand new desk creation charge on HMS was pretty excessive. This needed to be introduced down considerably (virtually 0) for us to efficiently cutover to Unity Catalog ultimately and have a shot at a profitable migration.
These insights led us to spend money on information retention infrastructure upfront and roll out the next insurance policies, which turbo-charged our effort.
- Rubbish-Acquire Stale Information: This coverage, shipped proper out of the gates, deleted any HMS desk that wasn’t up to date for 30 days. We offered groups with a grace interval to register exemptions. This drastically diminished the dimensions of the “haystack” and allowed information practitioners to deal with information that really mattered.
- No New Tables in HMS: 1 / 4 after the migration was underway and there was company-wide consciousness, we rolled out a coverage to forestall the creation of any new HMS tables. Whereas holding the legacy metastore in test, this measure successfully positioned a moratorium on information pipelines nonetheless on HMS as they might not be prolonged or modified to supply new tables.
With these in place, we had been not chasing a transferring goal.
Part 3: Distribute the work, utilizing Self-Serve Monitoring Instruments
Most organizations within the firm have a unique cadence for planning, completely different processes for monitoring execution and completely different priorities and constraints. As a small information platform group, our aim was to reduce coordination and empower groups to confidently estimate, coordinate, and observe their OWN dataset migration efforts. To this finish, we turned the lineage observability information into executive-level dashboards, the place every group may perceive the excellent work on their plate, each as information producers and shoppers, ordered by significance. These allowed additional drill-downs to the supervisor and particular person contributor ranges. These had been up to date on a day by day cadence for progress-tracking functions.
Moreover, the info was aggregated right into a leaderboard, permitting management to have visibility and apply strain when required. The worldwide monitoring dashboard additionally served the twin function of a lookup desk the place shoppers may discover the places of recent tables migrated to Unity Catalog.
The emphasis on managing the folks and course of dynamics of the Databricks group was an important success driver. Each group is completely different and tailoring your method to your organization is vital to your success.
Part 4: Sort out the Lengthy Tail, utilizing Automation
Successfully herding the lengthy tail is make or break for a migration with 2.5K information shoppers and over 50K consuming entities throughout each group of the corporate. Counting on information producers or our small platform group to trace and chase down all these shoppers to do their half by the deadline was a non-starter.
Underneath the moniker “Migration Wizard”, we constructed an information platform that allowed information producers to register the tables to be deleted or migrated to a catalog in Unity Catalog. Together with the desk paths (new and outdated), producers offered operational metadata just like the end-of-life (EOL) date for the legacy desk and contact with questions or considerations.
The Migration Wizard would then:
- Leverage lineage to detect consumption and notify downstream groups. This focused method allowed groups to not must repeatedly inundate everyone with information deprecation messages
- On EOL day, render a “delicate deletion” through lack of entry and purge the info every week later
- Auto-update DBSQL queries relying on the legacy information to learn from the brand new location
Thus with a number of traces of config, the info producer was successfully and confidently decoupled from the migration effort with out having to fret about downstream impression. Automation continued notifying clients and in addition offered a swift repair for question breakage found after the deprecation set off was pulled.
Subsequently, the power to auto-update DBSQL and pocket book queries from legacy HMS tables to new UC alternate options has been added to the product to help our clients of their journey to Unity Catalog.
Sticking the Touchdown
In February 2024, we eliminated entry to Hive Metastore and began deleting all remaining legacy information. Given the quantity of communication and coordination, this doubtlessly disruptive change turned out to be easy. Our modifications didn’t set off any incidents, and we had been capable of declare “Success” quickly after.
We noticed rapid value advantages as unowned jobs that failed as a result of modifications may now be turned off. Dashboards silently deprecated now failed whereas incurring marginal compute value and might be equally sunsetted.
A vital goal was to establish options to make migration to Unity Catalog simpler for Databricks clients. The Unity Catalog and different product groups gathered in depth actionable suggestions for product enhancements. The Information Platform group prototyped, proposed and architected numerous options that will likely be rolling out to clients shortly.
The Journey Continues
The transfer to Unity Catalog unshackled information practitioners, considerably decreasing information sprawl and unlocking new options. For instance, the Advertising and marketing Analytics group noticed a 10x discount in tables managed through a lineage-enabled identification (and deletion) of deprecated datasets. Entry administration enhancements and lineage, alternatively, have enabled highly effective one-click entry obtainment paths and entry discount automation.
For extra on this, take a look at our discuss on unified governance @ Information + AI Summit 2024. In future blogs on this collection, we may even dive deeper into governance choices. Keep tuned for extra about our journey to Information Governance!
We want to thank Vinod Marur, Sam Shah and Bruce Wong for his or her management and assist and Product Engineering @ Databricks—particularly Unity Catalog and Information Discovery—for his or her continued partnership on this journey.