What’s in today’s newsletter:

Databricks buys Electric AI to accelerate AI workflows ⚡️🤝

Oracle-AWS partnership enhances multi-cloud database integration 🌐🤖

Google’s Gemini automates PostgreSQL database migrations 🚀🌐

Also, check out the weekly Deep Dive - A Look Into Modern Database Storage Architectures

Over 4 million people have had the same lightbulb moment.

Morning Brew is a free daily newsletter that breaks down what's happening in business, finance, and tech — clearly, quickly, and with enough personality to make it the best email in your inbox.

No yelling. No filler. Just the news, finally making sense.

Try it for free

DEEP DIVE

A look into Database Storage

How eight platforms physically store bytes — and why every architecture decision you make downstream inherits those choices.

For some reason, I had database storage on my mind in the last several days. Particularly the notion of micro-partitions in the Snowflake world. ClickHouse, Databricks, and for some reason, MongoDB were on my mind as well.

I think my preoccupation was/is due to all of the news and noise around Agentic AI. I am still trying to figure out what the role will be for modern databases in the months and years to come in relation to Agentic AI.

Perhaps I have some resentment that databases are becoming some sort of forgotten storage closet that AI agents can just spin up on a whim.

But when you really start to look into how data is stored within these modern database platforms, the thought and execution put into it is remarkable and very innovative.

Every platform in this newsletter markets itself on query speed. Almost none of them market themselves on the thing that produces that speed: the physical layout of bytes on disk, and the metadata that lets the engine avoid reading most of them.

That layout is not an implementation detail. It determines whether your update pattern is cheap or ruinous, whether your clustering key is worth maintaining, and whether the platform you picked for a batch warehouse will survive being asked to serve an agent doing thousands of small point lookups. Here is what is actually happening underneath eight of them.

Snowflake has no indexes. That surprises people coming from SQL Server or Oracle, and it is the single most important fact about how it works.

When you load data, Snowflake writes it into micro-partitions — immutable columnar files of roughly 50–500 MB of uncompressed data, typically landing at 16 MB or so compressed. They are created automatically in insertion order. You do not size them, name them, or manage them.

For each micro-partition, Snowflake records metadata in its cloud services layer: the range of values for every column, the number of distinct values, and null counts. That metadata is the query engine. When you filter on a column, Snowflake consults the metadata, discards every micro-partition whose min/max range cannot contain your predicate, and reads only what remains. This is pruning, and it is doing the job that a B-tree index does elsewhere.

Two consequences follow directly, and both show up in real workloads:

Insertion order is your physical design. If your data arrives roughly ordered by date and you filter by date, pruning is excellent for free. If you filter on a column uncorrelated with load order — customer ID, say — every micro-partition's min/max range spans nearly the whole domain, nothing prunes, and you full-scan. That is what clustering keys and the automatic clustering service exist to fix, by rewriting micro-partitions in the background so ranges get tight. Clustering depth is the metric to watch. It is also a continuous background cost, which is why clustering everything is a real way to burn credits.

Updates are copy-on-write at file granularity. Micro-partitions are immutable. Changing one row means writing a new micro-partition containing the whole file's worth of rows with your change applied, and repointing metadata. A single-row UPDATE can rewrite 16 MB. This is why Snowflake is superb at bulk loads and analytical scans and genuinely poor at high-frequency small mutations — and why teams migrating an OLTP workload into it get an unpleasant bill.

The upside of that same immutability is Time Travel and zero-copy cloning. Old micro-partitions still exist; a clone is a new set of metadata pointers to the same files. Nothing is copied. The elegance is real, and it falls straight out of the storage design.

ClickHouse is the most physically intricate system in this list, and the intricacy is the point.

Data in a MergeTree table lives in parts — immutable directories on disk. Each part contains one file pair per column (.bin for compressed data, .mrk for marks) plus index files. Within a part, rows are stored sorted by the table's ORDER BY key. That sort order is the foundation of everything: it produces the compression ratios ClickHouse is famous for, because sorted adjacent values compress far better than random ones.

The index is sparse. By default, index_granularity is 8192 — ClickHouse stores one primary index entry per 8,192 rows, not per row. Those 8,192-row blocks are granules, and a granule is the smallest unit ClickHouse will read. The entire primary index for a billion-row table is therefore small enough to sit permanently in memory. A query binary-searches the sparse index to find candidate granules, uses the marks files to jump to the exact compressed block offsets, and decompresses only those blocks.

Writes create new parts. A background merge process continuously combines small parts into larger sorted parts, which is where the name comes from. This is LSM-adjacent thinking, though not a textbook LSM tree — merges are sort-merges of already-sorted parts rather than level compaction. Parts are also grouped by partition key, conventionally by month, and merges never cross partitions.

Beyond that baseline, ClickHouse gives you an unusual amount of physical control:

The tax: mutations. UPDATE and DELETE are asynchronous background rewrites of entire parts. ClickHouse will let you run them and will not enjoy it. Design so you rarely need to.

MongoDB's storage story is actually two independent stories, and most confusion comes from treating them as one.

Layer one — the storage engine. Underneath every shard sits WiredTiger, which is a fairly conventional B+tree row store. Documents are stored in BSON, indexes are B+trees, concurrency is document-level MVCC with snapshot isolation, durability comes from a write-ahead journal plus checkpoints roughly every 60 seconds. Block compression is Snappy by default, with Zstd available; indexes get prefix compression. This is a well-understood OLTP engine and it behaves like one — row-oriented, good at point reads and small writes, not built for scanning columns.

Layer two — sharding. Horizontal partitioning sits above that. You choose a shard key. MongoDB divides the key space into chunks (default 128 MB logical ranges), and each chunk lives on one shard. Config servers hold the map of which chunk is where; mongos routers consult that map to direct queries. Ranged sharding preserves key order and enables range queries to hit a subset of shards; hashed sharding distributes evenly but makes range queries scatter-gather across all of them. A balancer migrates chunks between shards to even out distribution.

The failure mode is famous and worth restating because people still hit it: a monotonically increasing shard key — a timestamp, an ObjectId, an auto-increment — sends every new write to whichever shard owns the top of the range. You have bought a distributed cluster and are writing to one node. Compound or hashed keys avoid this. Since MongoDB 5.0 you can reshard a live collection, which was previously a migration project.

The architectural point: the shard key is the one decision that is expensive to reverse and that determines whether a query is targeted or scatter-gather. Everything else in MongoDB is tuneable later.

Worth being precise here, because "Databricks is managed Spark" was accurate around 2016 and now obscures more than it explains. Spark is the compute engine and increasingly not even that — Photon, the vectorized C++ engine, executes a growing share of queries and is not Spark at all. The storage layer is where Databricks actually differentiates.

Delta Lake is Parquet files on object storage plus a transaction log. The log (_delta_log) is an ordered sequence of JSON commit files, each describing files added and removed, with periodic Parquet checkpoints so readers do not replay the entire history. Atomicity comes from the atomic creation of the next log file; concurrency is optimistic — writers assume no conflict, then validate at commit and retry if another writer got there first.

Because the log records per-file statistics (min/max for the first 32 columns by default), Delta does the same trick as Snowflake: prune files on metadata before reading any data. The mechanism differs, the principle is identical.

What you control:

BigQuery stores data in Capacitor, a columnar format, on Colossus. There are no indexes and no user-managed storage at all. You get partitioning and clustering as your two levers; BigQuery handles re-clustering in the background. Execution is a Dremel-style tree with a distributed in-memory shuffle between stages. The trade is the purest form of the storage/compute split: almost nothing to tune, almost nothing to get wrong, almost no control when you need it.

Redshift is the most classically "database" of the cloud warehouses. Columnar data lives in 1 MB blocks; zone maps hold min/max per block and drive pruning. You choose a sort key (compound or interleaved) and a distribution style — KEY to co-locate joined rows on the same node, ALL to replicate small dimensions everywhere, EVEN for round-robin. Getting distribution wrong means every join redistributes data across the network. RA3 nodes moved storage to managed S3-backed tiers, but the physical design decisions remain yours in a way they are not on BigQuery.

Apache Iceberg deserves its own paragraph because it is becoming the neutral ground. Where Delta uses a flat sequential log, Iceberg uses a metadata tree: a catalog pointer to a metadata.json, which points to a manifest list, which points to manifest files, which point to data files. This indirection buys hidden partitioning (query on a timestamp, let Iceberg handle the partition transform) and partition evolution (change the partitioning scheme without rewriting the table) — two things that are genuinely hard elsewhere. As covered on Sunday, the "Iceberg alone doesn't solve interoperability" argument is really an argument about catalogs, not file formats.

PostgreSQL is the useful contrast to everything above. Heap tables, 8 KB pages, tuples stored in no particular order, MVCC implemented by keeping old row versions in the heap with xmin/xmax transaction stamps and cleaning them with VACUUM. Large values get pushed to TOAST tables. Indexes are separate B-tree structures pointing at page/offset locations. This is row-oriented storage designed for reading and writing whole records with low latency — the exact inverse of the columnar designs — and it is why "Postgres compatibility" and "Postgres performance characteristics" are two very different claims when a vendor markets a Postgres-compatible analytical engine.

DynamoDB hashes the partition key to place items across partitions of roughly 10 GB, with LSM-structured storage underneath and a sort key providing ordering within a partition. Same lesson as MongoDB, enforced more strictly: the key design is the data model, and a hot partition key is a self-inflicted outage.

Cassandra and ScyllaDB are the textbook LSM implementations — writes to a memtable, flushed to immutable SSTables, reconciled by compaction strategies you choose per table (size-tiered for write-heavy, leveled for read-heavy). Useful as a reference point: ClickHouse's merges and Delta's OPTIMIZE are solving the same problem in different clothes.

1. Immutability won. Snowflake micro-partitions, ClickHouse parts, Delta Parquet files, Cassandra SSTables — none are modified in place. Every modern system writes new files and reconciles later in the background. Object storage made this mandatory, and it is why "why is my simple UPDATE so expensive" is the most common surprise in every migration.

2. Metadata replaced indexes at scale. Snowflake's micro-partition statistics, Delta's log stats, Redshift's zone maps, ClickHouse's sparse index and skip indexes — all are doing approximate elimination rather than exact lookup. They tell the engine which files it can safely not read. When you evaluate a platform, the real question is how good its pruning is on your actual predicates.

3. Copy-on-write versus merge-on-read is the axis that matters. It is the single best predictor of whether a platform will suit your workload. Snowflake rewrites files on update. Delta with deletion vectors marks and skips. ClickHouse defers to background merges. The question to ask a vendor is not "do you support updates" — everyone says yes — but "what does a single-row update physically cost, and when is it reconciled."

4. Sort order is the last physical design decision you actually own. Indexes are gone from most of these systems. What remains is clustering keys, ORDER BY keys, sort keys, Z-order and liquid clustering, shard and partition keys. That is a short list, and it carries nearly all of the performance. Spend your design time there.

5. The small-file problem is universal and permanent. Every object-store-backed platform degrades under many small files, because every file is a network round trip with per-request billing. OPTIMIZE, compaction, merges, automatic clustering — all the same battle. If your streaming ingest writes every thirty seconds, you have a compaction strategy whether you have designed one or not.

If you take one thing into your next architecture review: ask how the platform physically stores a row, and what it does when you change that row. The answer tells you more about whether it fits your workload than any benchmark a vendor will show you.

Gladstone Benjamin

Looking to reach CTOs, CIOs, and enterprise Data Engineers and Data Architects?

Limited sponsorship slots available each month.

👉 Sponsor Cloud Database Insider