Where Data Lives: Warehouses, Lakes, Lakehouses, and the Stack Around Them

I spent a lot of time working through data tools and came away mostly overwhelmed. Dozens of products, most promising to solve the same problem in slightly different ways, and no obvious way to tell which ones actually belonged in a given stack.

These two posts were my attempt to sort it out for myself: one on where data physically lives, one on everything built around it.

---

Warehouses

Born in the nineties, and still the neat freaks of data storage. Everything gets cleaned, formatted, and validated before it’s allowed in. Schema-on-write, with a bouncer at the door.

They’re built for BI. A retail company pushing sales into Snowflake to watch daily revenue is exactly the job. Complex SQL, aggregation, consistent high-performance answers.

The costs climb fast, and unstructured data doesn’t apply. The rigid schema that makes them reliable also makes them awkward for data science, ML, and streaming.

Modern cloud warehouses at least separated storage from compute, which is why scaling one no longer requires selling an organ.

Lakes

The 2010s answer, with considerably fewer rules. A lake will hold anything: JSON logs, images, video, files nobody has opened since ingest.

Schema-on-read, meaning the data stays raw until something queries it. The mental model is a garage you’ll organize later, and you know how that goes.

The use case is real, though. A healthcare provider dropping scans and records into S3 for research that hasn’t been designed yet is doing something sensible. Storage is cheap and the format is open, which is the whole point. A directory of Parquet files is a data lake.

Without governance, a lake becomes a swamp. Metadata goes in and never comes out.

Lakehouses

The promise is both at once: lake economics and open formats underneath, warehouse reliability on top.

What you get is ACID transactions, open formats you can query with SQL and also train models against, and one place for both BI and ML. The fintech pattern is the usual illustration: real-time transactions in a silver layer, dashboards served off gold.

The catch is metadata. A lakehouse needs serious catalog tooling or it turns back into a swamp with better marketing.

	Warehouse	Lake	Lakehouse
Data types	Structured	All	All
Schema	On write	On read	Optional
Transactions	ACID	None	ACID
Use case	BI, reporting	ML, raw storage	Both
Cost	High	Low	Moderate

---

The open table formats

Delta Lake, Hudi, and Iceberg have the same mission, which is dragging a data lake back from chaos. They go about it differently.

Delta Lake came out of Databricks and is the reliable one. ACID transactions so you don’t end up with half-written Parquet. Time travel, because everyone eventually needs to roll back. Schema enforcement that refuses data which doesn’t match. Best if you’re already on Spark or Databricks, and the medallion pattern of bronze to silver to gold is its native idiom.

Apache Hudi came out of Uber and is built for freshness. Upserts, so you can update and delete without rewriting a table. Incremental processing, so you only touch new data. Automatic compaction of small files. The right pick for real-time pipelines, and for the GDPR deletion requests that make full rewrites unbearable. Its core choice is copy-on-write versus merge-on-read.

Apache Iceberg came out of Netflix, specifically out of hidden partition problems. Partition evolution, so you can change your partitioning without breaking existing queries. Schema evolution without rewriting terabytes. Engine agnostic, which is the real draw: Spark, Trino, Flink, and increasingly everything else. Hidden partitioning means you query by column and don’t think about physical layout.

	Delta Lake	Hudi	Iceberg
Primary use	Batch + streaming	Real-time	Multi-engine
ACID	Yes	Yes	Yes
Time travel	Yes	Yes	Yes
Schema evolution	With enforcement	Yes	Without rewrites
Partition flexibility	Limited	Moderate	Hidden partitioning
Ecosystem	Spark-centric	Spark, Flink	Engine-agnostic

The choice isn’t really about features. Delta if you’re married to Databricks, Hudi if the workload is real-time, Iceberg if you care about not being locked in.

---

The stack around the storage

Storage is one layer. The rest of it:

Sources. Structured from relational systems, semi-structured as JSON, XML, CSV, and unstructured text, images, video, sensor output. Cloud, on-premises, and devices.

Ingestion. Getting all of it into one place. Kafka, Fivetran, Airbyte, Glue. What matters is batch and streaming support, and connector coverage for the sources you actually have.

Transformation. Turning raw into usable. dbt and Dataform push this into the warehouse itself, which is the shift that mattered most in this layer. Clean, enrich, reshape, and make it analysis-ready.

Analytics. Power BI, Tableau, Looker. Visualization, predictive work, and the actual decisions.

Orchestration, governance, monitoring. Airflow and Prefect for coordinating workflows across systems. Governance as the policy layer for quality, security, and privacy, with GDPR and CCPA as the forcing function. Monitoring for pipeline health, so you find out before your stakeholders do.

Reverse ETL. Pushing clean warehouse data back out into the tools people work in. Hightouch and Census are the usual names. The example that makes it click: an account manager in Salesforce seeing real-time product usage next to the lead, without switching applications.

---

Where it gets hard

Governance is the one that decides whether any of this works. Without catalog and metadata tooling you get silos, and a lakehouse without governance is just an expensive swamp.

Skills are the second. Moving a SQL-centric team to Spark and Python workflows is a retraining program, and pretending otherwise is how migrations stall.

Most enterprises end up hybrid anyway. The warehouse stays for mission-critical BI while ML moves to the lakehouse, and that’s a reasonable outcome rather than a failure to commit.

---

A note from 2026

The architecture in here held up. The vendor market did not.

Iceberg became the default interchange format faster than this post expected. When I wrote it, picking a table format still looked like a genuine three-way choice tied to your compute engine. It stopped looking that way fairly quickly once the warehouse vendors started supporting Iceberg directly, which undercut the main reason to pick anything else.

The bigger thing this misses is what sits on top now. Both of these posts assume a human writes SQL or opens a dashboard at the end of the pipeline. That assumption held for about a year after I wrote the second one.

Which makes the line about swamps truer than I meant it. I wrote that a lake without governance becomes a swamp where metadata goes to die, and treated that as an organizational embarrassment. Slow discovery, frustrated analysts, the same dataset rebuilt four times.

The cost is different now. A swamp used to be a place people avoided, and the avoidance was itself a form of safety. Nobody builds a board metric on a table they can’t identify. Remove the person from that loop and the swamp becomes indistinguishable from the lake, because whatever is querying it will drink happily and report back that it tastes fine.

The architecture choices in this post are still the right ones. What I’d change is the order. Governance isn’t the chapter after storage. It’s the thing that decides whether your storage choice was worth making.