A data lakehouse keeps data in open file formats on object storage and adds warehouse features, such as ACID transactions, schema enforcement and SQL access, through an open table format. It fits when several engines must share the same data, when you hold large volumes of varied data, or when SQL and machine learning need one governed copy. A managed warehouse is simpler when the work is mostly SQL and BI and you want the vendor to run storage, security and tuning.
This guide is part of our platform modernization research.
What is a data lakehouse, and what do Delta Lake, Iceberg and Hudi add?
A lakehouse stores data in open formats such as Parquet and uses a table format to add transactions, schema control and time travel on top of the files. The pattern was set out in a 2021 paper by Databricks researchers, which described open direct-access formats, first-class support for machine learning, and strong performance. Because its authors worked at Databricks, treat its benchmark results as vendor-reported.
Delta Lake, Apache Iceberg and Apache Hudi are the three open table formats.
| Format | Stewardship | What it adds (per project docs) | Engines |
|---|---|---|---|
| Delta Lake | Linux Foundation project since 2019 | ACID transactions, scalable metadata, schema enforcement, time travel, upserts and deletes, unified batch and streaming | Spark, with connectors for Flink, Hive, Trino and Athena |
| Apache Iceberg | Apache project | Schema evolution without rewriting tables, hidden partitioning, time travel and rollback, compaction | Spark, Trino, Flink, Presto, Hive, Impala |
| Apache Hudi | Apache project | Upserts and deletes, indexes, clustering and compaction, incremental pipelines; copy-on-write and merge-on-read table types | Spark, Flink, Presto, Trino, Hive |
How well do the main platforms support open table formats?
All five platforms covered in our TCO guide document Iceberg support, and Delta Lake is native on Databricks and Fabric. The limits matter more than the headline support, so check them against your requirements. This table reflects vendor documentation as of October 2026.
| Platform | Open table format support | Limits stated in the docs |
|---|---|---|
| Databricks | Delta Lake natively; managed Iceberg tables (spec versions 1 to 3) in Unity Catalog, which exposes an Iceberg REST catalog API; UniForm lets Iceberg clients read Delta tables | Iceberg tables from other catalogs are read-only in Databricks; external Iceberg engines cannot read Unity Catalog views; UniForm is read-only for Iceberg clients and does not work with deletion vectors |
| Snowflake | Iceberg tables on Snowflake-managed or customer-managed storage; catalog can be Snowflake, AWS Glue or Snowflake Open Catalog | No Fail-safe for tables on customer-managed storage, so you own protection and recovery; cross-region storage adds egress cost |
| BigQuery | Iceberg tables managed by the Lakehouse runtime catalog (Iceberg REST) or by BigQuery | Row-level security, column-level security and data masking are listed as unsupported on catalog-managed Iceberg tables; BigQuery DML and automatic table management are in Preview |
| Microsoft Fabric | Delta Lake in OneLake; Iceberg through metadata virtualization that presents Iceberg tables as Delta and the reverse | Applies to Iceberg tables with Parquet data files |
| AWS (S3 Tables) | Iceberg table buckets, including spec version 3, queried by Athena, Redshift and Spark | Compaction, snapshot management and unreferenced-file removal run automatically, with settings you can change per table |
Catalogs deserve a separate decision, because the catalog controls access and tells each engine where the current table metadata lives. Apache Polaris, an open catalog implementing the Iceberg REST API, graduated to an Apache top-level project in 2026. Vendor catalogs include Unity Catalog, the AWS Glue Data Catalog and Google's Lakehouse runtime catalog.
When does a lakehouse fit, and when is a warehouse simpler?
A lakehouse fits when more than one engine must work on the same tables, when data is varied, or when you need to hold storage and policy in your own accounts. A warehouse is simpler when one SQL engine serves nearly all users and you want the vendor to manage storage, backup and tuning.
| Question | Leans lakehouse | Leans warehouse |
|---|---|---|
| How many engines touch the data? | Several, such as Spark, Flink, Trino and a SQL warehouse | One SQL engine serves nearly all users |
| What kinds of data? | Semi-structured or unstructured data and ML features next to tables | Mostly structured, relational data |
| How does data arrive? | Continuous streams or change data capture (CDC) with upserts at scale | Scheduled batch loads and standard ELT |
| Who holds the storage? | You want it in your own account for policy or cost reasons | You want the vendor to manage it, including backup features such as Snowflake's Fail-safe |
| Who runs maintenance? | You have engineers for compaction, snapshot expiry and catalog operations, or a managed service that automates them | You do not, and prefer a service that tunes itself |
| What governance do you need? | Catalog-level policies across engines | Warehouse-native row-level security and masking on every table |
Storage price rarely decides the question, because US list rates for warehouse-managed storage and object storage are close to each other (see the TCO guide). The two also combine: Snowflake, BigQuery, Redshift and Fabric all work with Iceberg or Delta tables, so you can keep a warehouse for BI and place shared data in open formats. For migrating an existing warehouse, see our legacy data warehouse modernization guide.
What are the implementation phases?
Build the lakehouse in this order: use cases, format and catalog, foundation, ingestion, curated layers, table maintenance, governance, then migration and operations. Plan maintenance and delete handling before go-live, because the table formats leave both to you or to a managed service you must choose.
- Define use cases and success measures. Separate BI dashboards, streaming feeds and ML workloads, and note the latency and concurrency each needs.
- Choose format, catalog and engines using the tables above. Decide the catalog first: the Iceberg specification treats the catalog as its first architectural layer, and it determines how access control reaches every engine.
- Build the foundation: storage accounts or buckets, encryption keys, identity and access, network paths and infrastructure as code, with separate environments for development and production.
- Ingest into raw (bronze) tables from batch files, CDC and streams. Databricks recommends keeping most bronze fields as string, VARIANT or binary so unexpected schema changes do not drop data.
- Build cleaned (silver) and business (gold) layers. The medallion pattern is a recommended practice rather than a requirement, according to Databricks. Quarantine invalid records at the silver step and test the gold models that feed BI.
- Plan table maintenance before go-live. Iceberg's docs recommend expiring snapshots regularly and removing orphan files, and note that tables with frequent commits, such as streaming targets, need metadata cleanup. On Delta tables, Databricks' VACUUM keeps files for 7 days by default, and Databricks recommends liquid clustering for all new tables instead of partitioning. Managed options automate part of this: Databricks predictive optimization, S3 Tables maintenance, and Google's automatic table management (in Preview).
- Set governance and security: one policy model, credential handling for external engines, and a check of the feature gaps in the table above. Deletes need care. Databricks notes that VACUUM permanently removes files that may hold modified or deleted records, and that tables using deletion vectors need a purge step to rewrite files.
- Test serving: run concurrent BI queries and your heaviest transformations, and record cost per workload against the model in the TCO guide.
- Migrate workloads in waves and write runbooks for maintenance, incidents and cost reviews.
What skills should you require from an implementation partner?
Require people who can run the table layer, not only the pipelines. Ask for evidence of each skill below, and for named engineers rather than a firm's general credentials.
| Skill | Evidence to ask for |
|---|---|
| Table format internals | An explanation of commits, snapshots and compaction in your chosen format, and a past case of small-file or metadata growth and how it was fixed |
| Catalog and access control | A design for your catalog (Unity Catalog, Polaris, Glue or Google's), how credentials reach external engines, and which feature gaps affect you |
| Layout and engine tuning | Before-and-after scan metrics for clustering or partitioning decisions; file-size targets |
| CDC and streaming ingestion | Handling of late data, deduplication and schema evolution |
| Modeling and quality | Medallion layers, automated tests, a quarantine process for bad records |
| Deletes and retention | How an erasure request becomes physical deletion (VACUUM, snapshot expiry, deletion-vector purge) |
| Cost engineering | Storage lifecycle, compute sizing, serverless versus classic choices, and cost reports per workload |
| DevOps | Infrastructure as code, CI/CD for pipelines, promotion between environments, rollback of table changes |
| Migration | Experience converting and validating legacy warehouse workloads, as covered in the legacy warehouse guide linked above |
Vendor programs help with screening. Databricks also runs its own services team, which its site now calls Forward Deployed Engineering (Databricks Professional Services in our directory), with Jumpstart, Migration Assurance and Lakehouse Build-out offerings. Databricks' partner directory tags consulting partners with regions and solution areas; for example, it tags Accenture, Cognizant and Infosys with a Lakeflow solution area (checked October 2026). See our Databricks consulting partners guide for more, and use the RFP template to turn this table into questions. DCF Research has not audited any firm's client work.
Frequently asked questions
Should we choose Delta Lake or Apache Iceberg?
Choose by the engines and catalog you will use, not by feature lists. Delta is native on Databricks and Fabric and has connectors for other engines. Iceberg is documented by Databricks, Snowflake, Google, AWS and Microsoft, which makes it a natural fit when engines from several vendors must share tables. UniForm and OneLake's metadata virtualization reduce the cost of choosing wrongly, but they have the limits listed above.
What is the small-file problem, and who fixes it?
Frequent small writes leave many small data files and many metadata versions. The formats provide the cleanup: Iceberg's compaction and snapshot expiry, Delta's OPTIMIZE and VACUUM, and Hudi's clustering and compaction. Managed services automate part of it, so ask who owns the schedule in your design.
How much does a lakehouse implementation cost?
It depends on the number of sources, data volume, pipelines, governance needs and migration scope, so we do not give a single figure. Use the cost components and worked estimate in the data platform TCO guide, and ask partners to price against the same lines.
Sources
- Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics — Armbrust, Ghodsi, Xin and Zaharia, CIDR (2021). https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf
- Delta Lake documentation — Delta Lake project (2026). https://docs.delta.io/latest/index.html
- The Delta Lake Project Turns to Linux Foundation to Become the Open Standard for Data Lakes — Linux Foundation (2019). https://www.linuxfoundation.org/press/press-release/the-delta-lake-project-turns-to-linux-foundation-to-become-the-open-standard-for-data-lakes
- Universal Format (UniForm) — Delta Lake documentation (2026). https://docs.delta.io/delta-uniform/
- Apache Iceberg — Apache Software Foundation (2026). https://iceberg.apache.org/
- Maintenance — Apache Iceberg documentation (2026). https://iceberg.apache.org/docs/latest/maintenance/
- Apache Hudi overview — Apache Software Foundation (2026). https://hudi.apache.org/docs/overview
- The Apache Software Foundation graduates two open source projects from Incubator — The ASF (2026). https://news.apache.org/foundation/entry/the-apache-software-foundation-graduates-two-open-source-projects-from-incubator
- What is Apache Iceberg in Databricks? — Databricks Documentation (2026). https://docs.databricks.com/aws/en/iceberg/
- What is the medallion lakehouse architecture? — Databricks Documentation (2026). https://docs.databricks.com/aws/en/lakehouse/medallion
- Remove unused data files with vacuum — Databricks Documentation (2026). https://docs.databricks.com/aws/en/delta/vacuum
- Use liquid clustering for tables — Databricks Documentation (2026). https://docs.databricks.com/aws/en/delta/clustering
- Databricks Forward Deployed Engineering — Databricks (2026). https://www.databricks.com/professional-services
- Databricks partner directory — Databricks (checked October 2026). https://www.databricks.com/partners/partner-directory
- Apache Iceberg tables — Snowflake Documentation (2026). https://docs.snowflake.com/en/user-guide/tables-iceberg
- Interact with borderless Lakehouse data in BigQuery — Google Cloud Documentation (2026). https://docs.cloud.google.com/bigquery/docs/lakehouse-in-bigquery
- Use Iceberg tables with OneLake — Microsoft Learn (2026). https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables
- Working with Amazon S3 Tables and table buckets — AWS Documentation (2026). https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html