This Week in Open Lakehouse

This week, Scott and Lisa cover one of the busiest release weeks yet across the open lakehouse ecosystem, led by the Apache Spark 4.3.0 release. They also discuss Parquet’s new vector proposal, Iceberg V4 spec votes, catalog security hardening, and big releases from DataFusion Comet, NVIDIA’s cuDF plugin for Spark, and Airflow. Along the way, they talk about giving agents downtime, why security fixes keep landing across projects, and why keeping your stack up to date matters more than ever.
Parquet’s FINITE_VECTOR logical type proposal, which uses LIST storage, along with follow-on work in an Iceberg draft spec PR and a Spark SPIP.
Lakekeeper 0.15.0, including an OpenFGA write privilege that doesn’t grant drop, cache invalidation, autoscaling task workers, and scoped vended credentials.
Unity Catalog’s work toward 0.8.0 with stricter privilege checks and new privileges, plus a Polaris fix for table corruption when a commit’s outcome is unknown.
Iceberg V4 votes on default value expressions and stats-only fields, and Iceberg Rust 0.11 with encrypted writes, deletion vectors, and a new catalog auth design.
The move of iceberg-datafusion into the DataFusion project starting with 0.12, and what that means for your Cargo.toml.
hudi-rs 0.6.0 with experimental merge-on-read metadata table compaction, and Paimon’s Mosaic 0.3.0 for tables with 10,000+ columns.
Spark 4.3.0 highlights, including stateful Real-Time Mode, Project Feather for single-node workloads, SCD Type 2 in Spark Declarative Pipelines, new variant functions, nanosecond timestamps (preview), DSv2 updates, and the move to Python 3.11+.
Spark proposals, including the passed External Lookup Join SPIP, a new ADBC data source SPIP, DSv2 join pushdown, and future Java version minimums.
Flink on Ray (FLIP-608), the Flink Agents preview, and geography type discussions, plus Kafka 4.4.0 and a JWT signature validation fix in 4.3.2.
DataFusion Comet 1.2.0, with memory, spill, and shuffle improvements, opt-in native Iceberg v3 writes, Delta Lake scans, Arrow Python UDFs, and official Spark 4.2 support.
NVIDIA’s cuDF plugin for Spark (formerly Spark RAPIDS) 26.10, with GPU acceleration for variant, expanded Delta Lake support, and Iceberg v3 deletion vectors.
Airflow 3.4.0, with task loops (AIP-111), the Dag Importer for TypeScript, Go, Java, and YAML (AIP-85), metadata database improvements, and hardened Docker images, plus the Common AI provider 1.1.0.
Quick mentions of MLflow 3.18 and Omnigent 0.21.

Temporal developer relations engineer Melanie Warrick joins hosts Lisa Cao and Scott Haines to review latest open source model releases and agent orchestration frameworks for the week. The episode covers a wide range of AI news, from Mistral’s massive LeChonk model and GLM-5.3’s cybersecurity evaluation to new standards, inference infrastructure, agent harnesses, and voice-agent tooling. The discussion focuses on how open models, runtime environments, observability, and governance are shaping the next phase of AI development.Mistral’s LeChonk, a trillion-parameter open-weight multimodal model, including its hardware requirements, MIT licensing, gated cyber variant, and potential cybersecurity applications
GLM-5.3’s Regular evaluation, including multi-stage cyberattack scenarios, prompt-injection risks, rule-breaking behavior, and questions around model guardrails
The Linux Foundation’s Agentic AI Foundation additions, including Anthropic’s Agent Skills, OpenAI’s Open Responses, Hugging Face’s reinforcement-learning environments, and Alibaba’s OpenSandbox
LLMD’s graduation within the Cloud Native Computing Foundation, including Kubernetes-native distributed inference, vLLM and SGLang support, endpoint picking, high availability, and multi-tenant flow control
Hermit, Raven, Google’s Harness, and related agent-harness research focused on tools, memory, retries, checks, modular workflows, and harnesses that can evolve around a model
Nano Muse, Deepgram’s acquisition of Daily and PipeCat, GitLab Orbit, NVIDIA OpenShell, Google Cloud’s Agent Compute Engine, and Bubble Agent 2

This week, Scott and Lisa recap a busy week across the open lakehouse ecosystem, from DuckDB 2.0 and Iceberg’s evolving V4 spec to new proposals for Spark, Flink, and Airflow. They also discuss catalog security and project governance, plus updates to Apache Comet, OpenCode, and Omnigent. Together, they highlight the latest developments, security fixes, and community changes shaping the open-source data landscape.
DuckDB 2.0 and DuckLake 1.1, including new lakehouse features, the Quack client-server protocol, asynchronous I/O, and breaking changes to plan for when upgrading.
Unity Catalog and Lakekeeper changes, including stricter privilege checks, distributed cache invalidation, and updates to credential handling.
Paimon’s new Terraform/OpenTofu provider for managing tables, roles, and policies as code.
Iceberg V4 discussions around file types, multimodal data, efficient column updates, single-file commits, and metadata overhead from small writes.
An Arrow security fix for a vulnerability involving KMS URLs embedded in encrypted Parquet files, with a recommendation to upgrade to Arrow 26.0.0.
Apache DataFusion Comet’s top-level project proposal, including its growing contributor community and a proposed limit on concurrent open pull requests.
StreamFusion’s possible move toward becoming a Flink subproject and what shifting project ownership can mean for users and contributors.
Spark proposals for ADBC connectivity and external lookup joins, including early prototype benchmarks showing 3–10x read throughput over JDBC on column-oriented databases.
Airflow’s inbound webhook proposal and how webhooks could support event-triggered workflows.
OpenCode 2.0’s background agents, execution modes, and plugin support, along with Omnigent 0.19.0 and 0.19.1 updates.
New Paimon, Flink, and Arrow committers, along with a new Iceberg PMC member.

Scott Haines and Lisa Cao are back after a flu-delayed skip, just in time for Week 42 — which, yes, is our favorite week - "Hitchhiker’s Guide" week. We tear through a stacked open lakehouse release week, led by Unity Catalog 0.7.0, shipping native Iceberg REST catalog support, with Paimon, Hudi, Arrow, MLflow, and OmniGent close behind.
In this episode
Cold open / Week 42. Scott had the flu, last week vanished into the void, and the “answer to life, the universe, and the open lakehouse” ...
Unity Catalog 0.7.0. Native Iceberg REST catalog support, plus a playground so you can try classic Unity Catalog and the new Iceberg path without assembling a lab from spare parts.
Production hardening. Live, ready, and metrics endpoints; multi-instance setups behind a load balancer; Spark support through 4.1, with 4.2 still waiting on upstream Iceberg.
Apache Paimon 2.1.0. Multimodal and table-format upgrades: scan planning, Arrow reads/writes, blob collections, variant updates, indexes, and full-text search.
Iceberg compatibility, no shortcuts. Paimon stays aligned with Iceberg’s published type model — including the fussy bits, like nanosecond timestamps and variant.
Apache Hudi 1.1.2. Stronger Apache Flink streaming merge-on-read, 19 backported fixes, and a Flink upsert bug fix that keeps writes from quietly going missing.
Apache Kafka 4.2.2. A short stop: mostly a performance-oriented release.
Apache Arrow 26.0.0. The vote passed, the release is official, and Jeffrey Vo joins the PMC.
MLflow 3.17.1 and 3.17.2. Decision API support, custom judges, evaluation lineage, and UI fixes for Claude Code OpenTelemetry traces.
OmniGent 0.18. Bring-your-own-agents, web reading that turns pages into clean markdown, plus GitLab MR and Azure DevOps PR integrations.
Community, the Apache Way, and how not to land a 4,000-line first PR
The back half leaves the changelog and talks about project health: incubator graduation, contributor onboarding, CI cost, release management, and why credibility in a community still beats dropping a giant patch on day one.
Lisa also flags three new ASF top-level projects:
Apache Auron — a native, Rust-powered accelerator for big data query engines
Apache Reqsign — a lightweight signer for cloud storage HTTP requests, without hauling around vendor SDKs
Apache ResilientDB — a modular, permissioned blockchain framework built on Byzantine fault-tolerant consensus

In this episode, Lisa and Scott cover a dense week of open-source data infrastructure updates from Community Over Code and Data and AI Weekend. They explore new release activity across the lakehouse stack, the growing role of AI agents and policy enforcement, and the emerging standards that will shape multimodal and AI workloads. The discussion highlights the differences between hybrid and native table architectures, standardization of decimal precision and vectors, and the critical need for traceability and guardrails in agent systems.
Unity Catalog 0.7, native Iceberg support, Kubernetes-style endpoints, metrics, and batch loading for large schemas
Delta Lake 4.4.1, catalog-managed maintenance operations, metadata cleanup, column mapping removal, and automatic log cleanup
Table-format architectures, native versus hybrid support, and Delta and Iceberg workload management
Apache Arrow 26.0.0, Python compatibility updates, Parquet reader flexibility, DLPack 1.3, and the Flight SQL ODBC driver
Apache Parquet, vector logical types, embedding-native storage, deck float, and decimal precision standards
MLflow 3.17.0, judged feedback, fine-grained permissions, rollups, evaluation lineage, and run restore support
Omnigent 0.17.0, OIDC sign-in, policy engine updates, session-aware controls, and agent sandboxing
Data Fusion Comet 1.10, native Iceberg writes, Spark 4 support, and native shuffle
Ballista improvements, memory management, AQE, higher broadcast thresholds, and handling large I/O-heavy jobs
Agent governance, contextual policy enforcement, traceability, evaluation lineage, and behavioral guardrails
Iceberg REST catalog, user agent support, and identity management
Apache Hudi, vector search support, and semantic storage choices