Databricks Unveils LTAP|SQL Server Plagued by Legacy Issues|Deleting Data First

Deep Dive: dbt+kafka+Prefect

Author

Gladstone Benjamin
June 22, 2026

What’s in today’s newsletter:

Databricks launches LTAP for unified data processing 🚀

Microsoft balances SQL Server legacy with cloud innovation ☁️

Proactive data deletion enhances cybersecurity and privacy safeguards 🔒

Autonomous AI revolutionizes databases for real-time action 🌐🚀

Also, check out the weekly Deep Dive - dbt+kafka+Prefect

DATABRICKS

TL;DR: Databricks launched LTAP, the first system enabling real-time ACID transactions and analytics on data lakes, simplifying infrastructure, cutting costs, and accelerating unified, agile insights across industries.

  • Databricks launched LTAP, the first system unifying transactional and analytical processing on data lakes.

  • LTAP supports ACID transactions, enabling real-time workloads and analytics without data duplication or latency.

  • The system aims to streamline data infrastructure, reduce costs, and improve performance across industries.

  • LTAP enhances data management, fostering faster insights and more agile, unified decision-making processes.

Why this matters: LTAP represents a breakthrough by merging transactional reliability with analytics on data lakes, eliminating traditional system silos. This advancement enables businesses to process real-time data more efficiently, reduce infrastructure complexity and costs, and accelerate agile decision-making critical for competing in data-driven markets.

SQL SERVER

TL;DR: Microsoft must balance SQL Server's profitable legacy with the need for cloud-native innovation to stay competitive, support Azure growth, and retain enterprise relevance amid shifting database market demands.

  • Microsoft’s SQL Server faces challenges balancing its lucrative legacy with demands for modern cloud-native features.

  • SQL Server remains deeply embedded in enterprises, delivering steady revenue through licensing and support models.

  • Competitors with scalable cloud-native databases pressure Microsoft to innovate beyond incremental SQL Server updates.

  • Microsoft’s strategy to modernize SQL Server will impact Azure’s growth and its position in enterprise database markets.

Why this matters: Microsoft’s ability to modernize SQL Server amid rising cloud-native competition is critical to sustaining revenue and maintaining enterprise relevance. This balance will shape Azure’s growth trajectory and influence broader database market trends as enterprises navigate legacy dependencies versus the need for scalable, flexible data solutions.

🚀 Work With Cloud Database Insider

Looking to reach CTOs, CIOs, and enterprise Data Engineers and Data Architects?

Limited sponsorship slots available each month.

DATA MANAGEMENT

TL;DR: Strategically deleting unnecessary data reduces cyber risks, complements security measures, aids compliance like GDPR, lowers storage costs, and fosters responsible data practices, enhancing privacy and organizational trust.

  • Proactively deleting unnecessary data reduces cyber attack surfaces and complements encryption and access controls.

  • Automated data lifecycle management tools enable timely deletion, enhancing overall organizational security.

  • Prioritizing data deletion supports compliance with regulations like GDPR and lowers data storage risks and costs.

  • This deletion-first approach fosters responsible data practices and builds greater trust in data privacy protections.

Why this matters: Prioritizing data deletion minimizes cyber risks by shrinking attack surfaces and aligns with regulations like GDPR, reducing storage costs and liabilities. This proactive strategy complements encryption, encourages responsible data management, and strengthens privacy trust, fundamentally shifting how organizations secure and govern sensitive information in a data-driven world.

DATA ARCHITECTURE

TL;DR: Autonomous AI is transforming databases into dynamic platforms supporting real-time decisions, flexible data models, and multi-cloud scalability, accelerating innovation and reshaping IT for AI-driven automated business outcomes.

  • Autonomous AI transforms databases from static storage to dynamic platforms enabling real-time decision-making and action.

  • Databases must deliver ultra-low latency, high throughput, and flexibility to support continuous AI model learning and adaptation.

  • Multi-model and multi-cloud capabilities are essential to manage diverse datasets and ensure AI scalability and portability.

  • This revolution accelerates innovation, reduces overhead, and reshapes IT infrastructure for AI-driven business outcomes.

Why this matters: Autonomous AI demands databases that do more than store data—they must enable real-time decisions and seamless AI integration. This drives a shift to dynamic, flexible platforms supporting diverse data across clouds, accelerating innovation and transforming business through faster, smarter automated actions.

EVERYTHING ELSE IN CLOUD DATABASES

DEEP DIVE

dbt+kafka+Prefect

I have been helping a Director as of late ramp up his hiring of Data Engineers and other staff. I think I can state that without divulging any company secrets. I am pretty sure I have some lurkers around here.

Anyways, a recurring stack that I have seen on c.v.’s is kafka, dbt, and Prefect as the orchestrator.

Not tooting my own horn, but I know a vast amount of things when it comes to the cloud database realm. That does not mean that I use certain tools, or even like certain schools of thought, platforms or specific technologies for that matter.

I know and work with a lot of Data Engineers. They are a fine lot, but I have come to realize that at my 27+ years in the database world, I don’t think there will ever be a point that I work as a DE full time.

However, that does not preclude me from learning about and talking about the many DE tools out there. From surveys I have taken, I have a lot of Data Engineers that read the newsletter weekly.

With that, take a look at a quick synopsis of a couple of very popular DE tools and platforms.

The Role of Each Component

Technology

Core Function

Operating Zone

Apache Kafka

Real-time event streaming and ingestion

In-flight / Data Movement

dbt (data build tool)

SQL-based data transformation (T in ELT)

At-rest / Inside the Warehouse

Prefect

Workflow orchestration and observability

The Control Plane / Glue

Kafka is the transport layer — the real-time event backbone. Producers write events to topics (CDC streams off your operational DBs, clickstream, app/service events, IoT, logs), and Kafka durably buffers and fans them out to consumers. It doesn't transform or persist analytically; it moves and decouples.

dbt is the transformation layer, and critically it operates on data at rest inside the warehouse/lakehouse (Snowflake, Databricks, BigQuery, Fabric). It's the "T" in ELT — SQL (and now Python) models that turn raw landed data into staging → intermediate → mart layers, with tests, contracts, and docs. dbt never touches Kafka directly. It can't consume a topic; it runs batch or micro-batch against tables that already exist.

Prefect is the orchestration layer — the conductor. It schedules and triggers the steps, manages dependencies between them, handles retries/backoff, emits state and alerts, and parameterizes runs. prefect-dbt wraps dbt build/run/test as first-class tasks so dbt becomes a node in a larger flow rather than a standalone cron job.

The thing that actually ties Kafka and dbt together is the loader that drains the stream into the warehouse, because there's a genuine impedance mismatch: Kafka is continuous and unbounded, dbt is set-based and bounded. So a typical layout is:

Kafka topic → a sink (Kafka Connect, Snowpipe Streaming, Spark Structured Streaming, or a custom consumer) lands raw events into a staging/raw schema → Prefect detects that landing (schedule, sensor, or completion signal from the loader) → Prefect runs dbt build to model and test the new data → Prefect kicks off downstream tasks (reverse-ETL, refreshes, notifications).

Two patterns dominate in practice. The micro-batch version is a Prefect flow on an interval (every N minutes) that checkpoints a slice of the topic into staging, runs dbt, then moves on — simple, and good enough for most "near-real-time analytics" needs. The decoupled version has the Kafka sink running continuously and independently, with Prefect orchestrating dbt on its own cadence against whatever has landed, so the streaming lane and the batch lane never block each other.

The one boundary worth keeping sharp: if you need true sub-second transformation, that logic lives upstream in the streaming engine (ksqlDB, Flink, Spark Streaming), not in dbt. dbt owns the analytical/batch side; Kafka owns the real-time movement; Prefect owns the choreography and the handoff between the two worlds.

Gladstone Benjamin

🚀 Work With Cloud Database Insider

Looking to reach CTOs, CIOs, and enterprise Data Engineers and Data Architects?

Limited sponsorship slots available each month.