What’s in today’s newsletter:

AWS cloud access disrupted in Middle East conflict zone 🌍

Amazon hires DuckDB engineers, supports open-source innovation 🌟

OpenAI’s Data Agent revolutionizes enterprise data querying 🪄

Perplexity’s CobbleDB cuts database costs massively 💾💰

Also, check out the weekly Deep Dive - The Databricks Data Engineer Associate Exam Domain Breakdown

AWS

TL;DR: AWS cannot restore Bahrain and UAE cloud access due to war-damaged infrastructure, disrupting critical services for GCC clients and highlighting the urgent need for more resilient, diversified cloud infrastructure in conflict zones.

  • AWS is unable to restore access to its Bahrain and UAE cloud data zone due to war-related infrastructure damage.

  • The affected cloud region supports critical data storage and computing for many clients in the Gulf Cooperation Council.

  • AWS is coordinating repairs with local partners, but the restoration timeline is uncertain amid ongoing regional conflict.

  • The outage exposes vulnerabilities in cloud infrastructure, prompting calls for increased resilience and diversification in conflict zones.

Why this matters: The prolonged AWS outage in Bahrain and UAE disrupts crucial business and government operations, exposing the fragility of cloud infrastructure in conflict zones. This stresses the need for diversified, resilient data strategies to safeguard regional economies reliant on uninterrupted digital services amid geopolitical instability.

TL;DR: Amazon hired DuckDB engineers to enhance AWS analytics but left DuckDB open source, balancing proprietary growth with open innovation and fostering ongoing competition in lightweight, in-process SQL analytics.

  • Amazon hired key DuckDB engineers to enhance its cloud analytics without buying the company or codebase.

  • DuckDB remains open source, supporting competition and innovation in lightweight, in-process SQL analytics.

  • The recruited talent aims to strengthen AWS analytical tools by leveraging DuckDB’s efficient architecture.

  • Amazon balances proprietary service growth with open-source support, promoting a collaborative tech ecosystem.

Why this matters: Amazon’s strategic talent acquisition strengthens its cloud analytics while preserving DuckDB’s open-source status, fostering innovation and competition. This balance benefits the broader tech ecosystem by enabling proprietary advancement alongside continued open collaboration in lightweight, efficient SQL analytics solutions.

Work With Cloud Database Insider

Looking to reach CTOs, CIOs, and enterprise Data Engineers and Data Architects?

Limited sponsorship slots available each month.

👉 Sponsor Cloud Database Insider

DATA AGENTS

TL;DR: OpenAI launched Data Agent, an AI-powered tool enabling natural language queries on enterprise data, enhancing real-time analysis, security, and accessibility, driving innovation and competition in the enterprise analytics market.

  • OpenAI's Data Agent uses generative AI to enable natural language querying of large enterprise datasets.

  • It integrates with existing data systems, offering real-time, context-aware analysis via advanced language models.

  • Data Agent prioritizes security and privacy, meeting enterprise standards while providing actionable conversational insights.

  • The tool aims to democratize data access, accelerating innovation and intensifying competition in enterprise AI analytics.

Why this matters: OpenAI’s Data Agent democratizes complex data analysis by letting non-experts interact naturally with enterprise data, accelerating innovation and decision-making. Its real-time, secure integration raises the bar in AI analytics competition, pushing businesses toward more accessible and efficient data-driven workflows essential for maintaining competitive advantage.

NOSQL

TL;DR: Perplexity AI created CobbleDB to replace costly AWS DynamoDB, achieving similar scalability and reliability while saving up to $100 million annually, potentially inspiring broader shifts away from cloud vendor dependence.

  • Perplexity AI developed CobbleDB to replace AWS DynamoDB, aiming to cut database costs drastically.

  • CobbleDB matches DynamoDB’s scalability and reliability but reduces operational expenses significantly.

  • Migrating to CobbleDB enables Perplexity to save up to $100 million annually on database services.

  • Perplexity’s move may inspire other tech firms to build proprietary infrastructure and reduce cloud vendor dependency.

Why this matters: Perplexity’s creation of CobbleDB marks a strategic move to cut soaring cloud costs while maintaining performance, signaling a shift toward in-house infrastructure among AI firms. Saving $100 million annually not only boosts competitiveness but may pressure cloud providers like AWS to reconsider pricing and service models.

EVERYTHING ELSE IN CLOUD DATABASES

DEEP DIVE

The Databricks Data Engineer Associate Exam By Domain

Back on September 4, I passed the exam.

I come with receipts…

I covered it last week.

I just wanted to provide this week, some of the most important things to think about regarding the domains of the exam. I am not going to go into exam preparation tips, or what the exam is like, but what you need to know. Here we go:

1. Databricks Intelligence Platform (6%)

The platform has three layers. Delta Lake is the transactional storage layer: Parquet data files plus a transaction log that records state history. Unity Catalog is the central governance layer, covering metadata, securable objects, privileges, ownership, lineage and fine-grained access control. Compute runs the workloads.

Compute choice is driven by workload type:

Workload

Default

Move to Classic when

Interactive notebooks

Serverless notebooks

Dev needs capabilities serverless lacks (Classic all-purpose)

Production jobs

Serverless jobs compute

Workload needs unsupported serverless features

SQL/BI

Serverless SQL warehouse (instant start)

Custom networking or hybrid connectivity (Pro); unsupported features (Classic)

Serverless takes provisioning, scaling and termination off your hands. Classic gives you control over VNet/VPC, VPN, ExpressRoute and Direct Connect. Other Classic details:

  • Access modes are Standard and Dedicated.

  • Resource profiles are memory-, compute-, storage- or GPU-optimized.

  • Instance pools cut provisioning latency, but they are not interactive compute.

  • Autoscaling is for workloads whose size varies between runs.

  • Automated production jobs never run on all-purpose clusters.

2. Data Ingestion and Loading (21%)

Source

Mechanism

Simple incremental files

COPY INTO (file-level idempotence)

Huge or continuous file arrival

Auto Loader

Supported enterprise databases and SaaS

Lakeflow Connect managed connectors

Code-oriented access via standard connectors

Lakeflow Connect standard connectors

Relational sources with no better path

JDBC batch

Kafka-style streams

Structured Streaming

One-time files

Upload to a UC Volume

Detection mode. Auto Loader has two:

  • Directory listing is simple to set up but resource-heavy; it suits moderate volume.

  • File notifications (cloudFiles.useNotifications = true, Event Grid on Azure) detect new files through storage events, near real time and at high scale. This is the preferred mode.

Checkpointing stores metadata on discovered files and guarantees exactly-once processing. allowOverwrites is false by default, so each path is processed once. Setting it to true re-ingests any file whose timestamp changes. pathGlobFilter restricts which files are picked up.

Schema evolution modes:

  • addNewColumns (default): updates the tracked schema, then stops the stream; processing resumes on restart.

  • rescue: routes unexpected data to the rescued data column.

  • failOnNewColumns: treats any new column as an error.

  • none: never evolves the schema.

3. Data Transformation and Modeling (22%)

Medallion layers:

  • Bronze holds raw data.

  • Silver is validated, cleaned, deduplicated, standardized and enriched.

  • Gold holds business aggregates.

Gold objects:

  • Views are virtual queries.

  • Materialized views persist precomputed results for BI.

  • Streaming tables process append-only sources incrementally.

  • Regular tables stay under your direct control.

Lakeflow Spark Declarative Pipelines expectations:

  • expect() records violations but still writes the rows.

  • expect_or_drop() drops invalid rows.

  • expect_or_fail() fails the update; ON VIOLATION FAIL UPDATE is the SQL form.

  • expect_or_warn does not exist.

Spark semantics:

  • dropna with how="any" drops a row if any field is null; how="all" drops it only if all are.

  • SEMI keeps left rows that have a match; ANTI keeps left rows that don't.

  • EXPLODE unpacks arrays, and STRUCT fields use dot notation.

  • summary() adds quartiles on top of describe().

  • TRY_CAST returns NULL instead of failing.

  • CHECK constraints enforce boolean conditions on a table.

  • CTAS infers the schema and can't take a manual schema declaration.

Data layout:

  • Partitioning suits low-cardinality columns.

  • ZORDER is legacy.

  • Liquid clustering is the modern default.

  • Predictive optimization ("OVA") runs OPTIMIZE, VACUUM and ANALYZE, but not ZORDER.

4. Working with Lakeflow Jobs (16%)

Jobs are task graphs built from fan-out (ingest feeding validate and transform) and fan-in (branches converging on publish).

Run-if conditions:

  • All succeeded (the default)

  • At least one succeeded

  • None failed

  • All done (runs regardless of outcome; suits cleanup)

  • At least one failed (the error-handler pattern)

  • All failed

Triggers are scheduled, file arrival, table update or continuous.

Parameters. Job parameters, task parameters and task values are distinct. Anything that changes per run belongs in job parameters, not bundle variables. Parameters can be overridden at runtime through the Jobs UI.

Other topics:

  • For Each tasks loop over array inputs with a nested task.

  • max_concurrent_runs controls whether runs can overlap.

  • Spot instances suit non-critical batch jobs that can retry.

  • Job clusters bill at lower DBU rates and terminate on completion.

  • Monitor runs in Lakeflow Jobs run history, not Catalog Explorer, Query History or a generic dashboard.

5. Implementing CI/CD (10%)

Declarative Automation Bundles (renamed from Databricks Asset Bundles) are structured as:

  • targets: for environments, not environments: or workspaces:

  • resources: for deployable objects

  • variables: for parameters

Commands:

  • bundle validate

  • deploy -t dev

  • run -t dev <job>

  • deploy -t prod --fail-on-active-runs

generate creates configuration from existing resources, and --bind links it to the live resource. ${var} is baked in at deploy time, while {{job.parameters}} resolves at run time. Variables can be overridden per target, with --var, or through BUNDLE_VAR_<name>.

Git Folders handle branching, staging, commits, pushes, pulls and rebases. Pull requests, code reviews and branch deletion happen in the Git provider.

Authentication. PAT uses DATABRICKS_HOST and DATABRICKS_TOKEN; OAuth and service principals are preferred.

6. Troubleshooting, Monitoring & Optimization (10%)

Diagnosis follows symptom → bottleneck → remedy.

Startup signatures:

  • ClassNotFoundException or NoSuchMethodError means a dependency problem.

  • Java heap errors or lost executors mean memory or skew pressure.

  • Parse errors mean a data or schema problem.

  • VM acquisition failures mean an infrastructure problem.

Runtime bottlenecks:

  • Skew: one straggler task processing GBs.

  • Shuffle: data moved across executors by groupings, joins and repartition().

  • Disk spill: intermediate data doesn't fit in memory.

  • Driver vs. executor OOM: toPandas() and collect() pull all data to the driver, so these are driver problems.

AQE splits skewed partitions using runtime statistics. Broadcasting skewed keys can make things worse.

Tuning levers:

  • shuffle.partitions

  • default.parallelism

  • executor and driver memory

  • autoBroadcastJoinThreshold

7. Governance and Security (15%)

USE CATALOG and USE SCHEMA get you through the door, and SELECT lets you read. "Has SELECT but can't query" almost always means a missing USE grant.

  • REVOKE and REVOKE ALL PRIVILEGES remove explicit grants only.

  • SHOW GRANTS only inspects.

  • Dropping an external table removes its registration but leaves the files; managed tables can be recovered with UNDROP.

  • ALTER TABLE ... SET MANAGED migrates qualifying external tables in place, with rollback and path redirects.

  • Row filters and column masks attach functions to tables or columns.

  • DESCRIBE HISTORY provides the audit trail.

My final notes is that it is a very broad exam, but if you use the Udemy resources and the official book I mentioned last week, you will indeed pass the exam.

Gladstone Benjamin

🎯 Level Up Your Data Career: Database Certification Strategy Call

Struggling to figure out which cloud database certifications actually move the needle for your career and salary?

Skip the guesswork. Book a 1-on-1, 45-minute Database Certification Strategy Call directly with Gladstone Benjamin. Drawing from 27+ years as a Data Architect and DBA and holding multi-cloud certifications across AWS, Azure, GCP, OCI, Snowflake, and Databricks, I’ll help you map out a targeted, high-ROI certification roadmap tailored to your specific background and goals.

Special offer for Cloud Database Insider readers: Save 25% OFF your session!

👉 Book Your Certification Strategy Call Here (Use promo code SEPTCERT2026 at checkout)

Keep Reading

View more
caret-right
© 2026 Cloud Database Insider.
beehiivPowered by beehiiv