The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish Giri, Senior Software Engineer and active Apache Airflow contributor focused on security, about three recent contributions that touch nearly every Airflow 3 deployment: hardening connection testing, adding JWT revocation, and splitting execution tokens by scope.

Key Takeaways:

00:00 Introduction.
01:16 Anish's inside-out path into Airflow, starting from source code and docs rather than running pipelines in production.
03:13 Why security engineering flips the usual software question from "what can this do?" to "what can anyone make this to?"
04:49 How connection testing worked before, why it was disabled in Airflow 2.7 after a CVE, and how the redesign moves execution off the API server.
06:26 Running connection tests on the worker in Airflow 3 so credentials never leave the API server boundary.
07:50 What the mailing list and architecture review looked like, including a first-round rejection and eventual consensus.
10:00 The JWT logout problem in Airflow 3 and the revocation table that makes logout an actual logout.
13:12 A production incident with a GPU pipeline task hitting 403 after 15 minutes, and why it required splitting the token in two.
15:41 How the workload token and the auth token now have different lifetimes and scopes, tied to the task queue timeout.
19:53 Advice for new Airflow contributors: start small, don't fear the PR review, and treat feedback as mentorship.

Resources Mentioned:
Annual Airflow survey
Apache Airflow
Airflow on GitHub
Airflow Slack
Anish Giri at Airflow Summit

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Orchestration #Airflow

Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework his Center of Excellence built on top of Airflow, how they baked data quality and custom operators into it, and how a spec-driven AI workflow now lets product and business users contribute to templates without knowing Python.

Key Takeaways:
00:00 Introduction.
01:47 What itti and Grupo Vázquez do, and the Data Engineering Center of Excellence's mandate to build a 360-degree view of the customer across more than 30 companies.
02:56 How Airflow fits in as a central task orchestrator (not a processing engine) across around 300 production DAGs.
04:25 Managing enterprise-scale Airflow: preferring Airflow-as-a-service, plus enabling self-service for non-technical users through YAML.
05:44 Why itti built a second, more opinionated YAML framework after DAG Factory-style customization created a code review bottleneck.
07:14 More than 80% of new DAGs are now created with the new framework because it's simply faster.
07:54 How the framework works end to end: Python DAGs, Jinja templates, YAML configs, and CI/CD compilation.
09:10 A Google Sheets ingestion example that shows how prevalidation, download, and processing tasks are hidden behind a simple YAML config to preserve reliability.
10:14 Building an in-house data quality tool that tests per partition instead of full-scanning tables, triggered via custom Airflow operators.
11:17 Custom operators for dbt, the in-house data quality tool, and AWS services like QuickSight dashboard refreshes, and how OSS Airflow makes them portable across instances.
15:35 Spec-driven development with a fork of GitHub Spec Kit so business users can describe what they want and let agents generate DAGs against certified templates.
17:24 Slack-native error routing: every DAG has an owner team, common errors ship with explanations, and only deep issues escalate to the central team.
19:29 Where they're heading next: agent-triggered pull requests for self-healing pipelines.
20:22 Airflow 3 wishlist: backfill improvements, event-driven orchestration for streaming pipelines, and Human in the Loop for generative AI DAGs.

Resources Mentioned:
Apache Airflow
Apache Spark
DAG Factory
GitHub Spec Kit

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In this episode, Kenten Danas talks with Kumuda Sreenivasa, Founder of Receitly and Senior Data Architect at ATC Drivetrain, about how her team built self-healing Airflow pipelines, where AI fits into the recovery loop, and how they orchestrate AI agents as governed workflow components.

Key Takeaways:
00:00 Introduction.
02:29 How ATC Drivetrain uses Airflow across thousands of DAGs to orchestrate ETL/ELT jobs, data quality checks, and production reporting for a complex automotive remanufacturing environment.
04:08 Defining self-healing: pipelines that identify a known failure, decide whether they can recover safely, execute an approved action, and validate the result, all without paging an engineer.
07:03 The five-layer self-healing architecture: observe, classify, policy, recover, and validate.
10:00 Where AI fits in the recovery loop: classification, context gathering, and recommendations, but never bypassing operational policies or approval steps.
13:00 A concrete before/after: a currency-exchange failure caught overnight by AI-assisted recovery that saved four hours of downtime and roughly 120K.
14:27 Confidence levels and success rates: about 95% of small failure modes recover on their own.
15:34 AI-assisted troubleshooting at scale: how contextual log analysis and recommended actions save engineers from digging through thousands of log lines.
20:13 Orchestrating AI agents through Airflow: treating agents as bounded, governed workflow components with human approval and confidence-based stop conditions.
23:45 What Kumuda wants next from Airflow: stronger AI agent governance, standardized tracking of prompts and tool calls, and more flexible event-driven execution.

Resources Mentioned:
Orchestrate Everything
Apache Airflow
ATC Drivetrain

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3.

Key Takeaways:
(00:00) Introduction.
(02:16) The stack: Snowflake source, MongoDB with a medallion (bronze, silver, gold) layout, Spark for processing, Airflow for orchestration, Elasticsearch for reports.
(05:29) Why standard Airflow options (max active runs, pools, dependency setups) each solved only part of the problem.
(06:40) Data consistency across bronze, silver, and gold layers is what forced strict sequential execution.
(09:11) Scheduled DAGs versus event-driven DAGs triggered at any moment from the application side.
(10:04) The three-layer architecture: trigger DAGs, a single proxy DAG that controls the queue, and main ETL DAGs.
(12:00) The queue is literally another DAG. The proxy DAG allows only one active run and serializes everything behind it.
(15:13) 200-task DAGs generated from nested task groups and YAML configuration files, with DAG versions tied to release numbers.
(17:38) How the layers talk to each other: sensors and TriggerDagRunOperator.
(19:44) Migrating from subdags to task groups without losing the ability to retry a whole group.
(21:49) Airflow 2.9 approach: reset task instance state via the metadata DB, keyed off the task group identifier.
(23:13) Airflow 3 approach: move the retry logic onto the official REST API for stability, security, and maintainability.

Resources Mentioned:
Orchestrate Everything
Apache Airflow
Snowflake
MongoDB
Apache Spark
Elasticsearch

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to talk about how the central data and analytics engineering team runs Airflow for critical financial datasets, tiers pipelines by reliability, and orchestrates LLM-enabled workflows with validation layers and agent-checking-agent patterns.

Key Takeaways:
00:00 Introduction.
01:47 What Wise does and Antonello's role in the central data and analytics engineering team, a hybrid platform-plus-analytics team that owns dbt infrastructure, BI, and Analytics MCPs as a service.
05:53 Three principles that guide Airflow pipeline design at Wise: a clean separation between orchestration and computation logic, computational awareness (offloading memory-intensive tasks to EMR or SageMaker), and standardized deployments.
07:21 Why Wise treats Airflow as a pure orchestration layer and pushes memory-intensive work to external workers.
08:45 Moving to the Python Virtual Environment Operator to standardize Airflow deployments across the org while giving analysts and data scientists per-job Python environments.
10:50 The tiering system for pipelines, how it distinguishes highly controlled, well-documented, well-observed pipelines from newer ones, and how requirements from downstream drive tier promotion.
17:18 Where LLM-enabled workflows differ from standard pipelines: validation layers for specific use cases, plus observability and evaluation platforms that track model performance across executions.
19:08 Using Airflow to orchestrate LLM generation of monthly variance commentary for analysts.
21:10 Handling non-idempotent LLM outputs with multi-layer validation against source-of-truth data, and using a second agent (CI/CD style) to validate the first agent's output.
23:04 How AI-enabled workflow orchestration differs from batch ETL, and why teams should start small before building fully agentic pipelines.
25:25 What Antonello would most like to see from Airflow next: native support for agentic workflows and better local development that mirrors production.

Resources Mentioned:
[Orchestrate Everything](https://astronomer.link/data-flowcast-oe)
[Wise](wise.com)
[Wise Careers](wise.jobs)
[Apache Airflow](airflow.apache.org)
[dbt](getdbt.com)
[Python Virtual Environment Operator](airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html)

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow