The MAD Podcast with Matt Turck, is a series of conversations with leaders from across the Machine Learning, AI, & Data landscape hosted by leading AI & data in...
Beyond Brute Force: Chollet & Knoop on ARC AGI 2, the Benchmark Breaking LLMs and the Search for True Machine Intelligence
In this fascinating episode, we dive deep into the race towards true AI intelligence, AGI benchmarks, test-time adaptation, and program synthesis with star AI researcher (and philosopher) Francois Chollet, creator of Keras and the ARC AGI benchmark, and Mike Knoop, co-founder of Zapier and now co-founder with Francois of both the ARC Prize and the research lab Ndea. With the launch of ARC Prize 2025 and ARC-AGI 2, they explain why existing LLMs fall short on true intelligence tests, how new models like O3 mark a step change in capabilities, and what it will really take to reach AGI.We cover everything from the technical evolution of ARC 1 to ARC 2, the shift toward test-time reasoning, and the role of program synthesis as a foundation for more general intelligence. The conversation also explores the philosophical underpinnings of intelligence, the structure of the ARC Prize, and the motivation behind launching Ndea — a ew AGI research lab that aims to build a "factory for rapid scientific advancement." Whether you're deep in the AI research trenches or just fascinated by where this is all headed, this episode offers clarity and inspiration.NdeaWebsite - https://ndea.comX/Twitter - https://x.com/ndeaARC PrizeWebsite - https://arcprize.orgX/Twitter - https://x.com/arcprizeFrançois CholletLinkedIn - https://www.linkedin.com/in/fcholletX/Twitter - https://x.com/fcholletMike KnoopX/Twitter - https://x.com/mikeknoopFIRSTMARKWebsite - https://firstmark.comX/Twitter - https://twitter.com/FirstMarkCapMatt Turck (Managing Director)LinkedIn - https://www.linkedin.com/in/turck/X/Twitter - https://twitter.com/mattturck(00:00) Intro (01:05) Introduction to ARC Prize 2025 and ARC-AGI 2 (02:07) What is ARC and how it differs from other AI benchmarks (02:54) Why current models struggle with fluid intelligence (03:52) Shift from static LLMs to test-time adaptation (04:19) What ARC measures vs. traditional benchmarks (07:52) Limitations of brute-force scaling in LLMs (13:31) Defining intelligence: adaptation and efficiency (16:19) How O3 achieved a massive leap in ARC performance (20:35) Speculation on O3's architecture and test-time search (22:48) Program synthesis: what it is and why it matters (28:28) Combining LLMs with search and synthesis techniques (34:57) The ARC Prize structure: efficiency track, private vs. public (42:03) Open source as a requirement for progress (44:59) What's new in ARC-AGI 2 and human benchmark testing (48:14) Capabilities ARC-AGI 2 is designed to test (49:21) When will ARC-AGI 2 be saturated? AGI timelines (52:25) Founding of NDEA and why now (54:19) Vision beyond AGI: a factory for scientific advancement (56:40) What NDEA is building and why it's different from LLM labs (58:32) Hiring and remote-first culture at NDEA (59:52) Closing thoughts and the future of AI research
--------
1:00:45
Why This Ex-Meta Leader is Rethinking AI Infrastructure | Lin Qiao, CEO, Fireworks AI
In 2022, Lin Qiao decided to leave Meta, where she was managing several hundred engineers, to start Fireworks AI. In this episode, we sit down with Lin for a deep dive on her work, starting with her leadership on PyTorch, now one of the most influential machine learning frameworks in the industry, powering research and production at scale across the AI industry. Now at the helm of Fireworks AI, Lin is leading a new wave in generative AI infrastructure, simplifying model deployment and optimizing performance to empower all developers building with Gen AI technologies.We dive into the technical core of Fireworks AI, uncovering their innovative strategies for model optimization, Function Calling in agentic development, and low-level breakthroughs at the GPU and CUDA layers.Fireworks AIWebsite - https://fireworks.aiX/Twitter - https://twitter.com/FireworksAI_HQLin QiaoLinkedIn - https://www.linkedin.com/in/lin-qiao-22248b4X/Twitter - https://twitter.com/lqiaoFIRSTMARKWebsite - https://firstmark.comX/Twitter - https://twitter.com/FirstMarkCapMatt Turck (Managing Director)LinkedIn - https://www.linkedin.com/in/turck/X/Twitter - https://twitter.com/mattturck(00:00) Intro(01:20) What is Fireworks AI?(02:47) What is PyTorch?(12:50) Traditional ML vs GenAI(14:54) AI’s enterprise transformation(16:16) From Meta to Fireworks(19:39) Simplifying AI infrastructure(20:41) How Fireworks clients use GenAI(22:02) How many models are powered by Fireworks(30:09) LLM partitioning(34:43) Real-time vs pre-set search(36:56) Reinforcement learning(38:56) Function calling(44:23) Low-level architecture overview(45:47) Cloud GPUs & hardware support(47:16) VPC vs on-prem vs local deployment(49:50) Decreasing inference costs and its business implications(52:46) Fireworks roadmap(55:03) AI future predictions
--------
59:14
Top AI Researcher on GPT 4.5, DeepSeek and Agentic RAG | Douwe Kiela, CEO, Contextual AI
Retrieval-Augmented Generation (RAG) has become a dominant architecture in modern AI deployments, and in this episode, we sit down with Douwe Kiela, who co-authored the original RAG paper in 2020. Douwe is now the founder and CEO of Contextual AI, a startup focusing on helping enterprises deploy RAG as an agentic system. We start the conversation with Douwe's thoughts on the very latest advancements in Generative AI, including GPT 4.5, DeepSeek and the exciting paradigm shift towards test time compute, as well as the US-China rivalry in AI. We then dive into RAG: definition, origin story and core architecture. Douwe explains the evolution of RAG into RAG 2.0 and Agentic RAG, emphasizing the importance of self-learning systems over individual models and the role of synthetic data. We close with the challenges and opportunities of deploying AI in real-world enterprise, discussing the balance between accuracy and the inherent inaccuracies of AI systems.Contextual AIWebsite - https://contextual.aiX/Twitter - https://x.com/ContextualAIDouwe KielaLinkedIn - https://www.linkedin.com/in/douwekielaX/Twitter - https://x.com/douwekielaFIRSTMARKWebsite - https://firstmark.comX/Twitter - https://twitter.com/FirstMarkCapMatt Turck (Managing Director)LinkedIn - https://www.linkedin.com/in/turck/X/Twitter - https://twitter.com/mattturck(00:00) Intro(01:57) Thoughts on the latest AI models: GPT-4.5, Sonnet 3.7, Grok 3(04:50) The test time compute paradigm shift(06:47) Unsupervised learning vs reasoning: a false dichotomy(07:30) The significance of DeepSeek(10:29) USA vs. China: is the AI war overblown?(12:19) Controlling AI hallucinations at the model level(13:51) RAG: definition and origin story(18:46) Why the Transformers paper initially felt underwhelming(20:41) The core architecture of RAG(26:06) RAG vs. fine-tuning vs. long context windows(30:53) RAG 2.0: Thinking in systems and not models(31:28) Data extraction and data curation for RAG(35:59) Contextual Language Models (CLMs)(38:04) Finetuning and alignment techniques: GRIT, KTO, LENS(40:40) Agentic RAG(41:36) General vs. specialized RAG agents(44:35) Synthetic data in AI(45:51) Deploying AI in the enterprise(48:07) How tolerant are enterprises to AI hallucinations?(49:35) The future of Contextual AI
--------
50:44
Empowering Millions of Creators with AI Video Editing | Gaurav Misra, CEO, Captions
In this episode, we dive into how AI is transforming video editing with Gaurav Misra, the CEO of Captions. Launched in New York in 2021, Captions already empowers over 10 million creators worldwide, leveraging AI to make video production as simple as clicking a button.Discover the strategic framework that led to the inception of Captions, and learn how the founders identified societal changes and technological advancements to build a groundbreaking company. We explore the challenges and opportunities of building an AI product for video editing, including how Captions is outpacing traditional content production workflows.Gaurav shares insights into the future of video editing, the role of AI in democratizing video production, and the unique approach Captions takes to differentiate itself from industry giants like Adobe and Capcut. CaptionsWebsite - https://www.captions.aiX/Twitter - https://x.com/getcaptionsappGaurav MisraLinkedIn - https://www.linkedin.com/in/gamisra1X/Twitter - https://x.com/gmharharFIRSTMARKWebsite - https://firstmark.comX/Twitter - https://twitter.com/FirstMarkCapMatt Turck (Managing Director)LinkedIn - https://www.linkedin.com/in/turck/X/Twitter - https://twitter.com/mattturck(00:00) Intro(01:30) What is Captions?(03:43) How did Captions start?(08:25) The strategy behind launching Captions(12:32) How is Captions different from other editing tools?(14:13) How does it compare to CapCut?(18:22) Who is the typical Captions user?(20:13) Why ‘Captions’?(23:47) Captions’ product suite for production and editing(26:37) AI models powering Captions(36:22) AI lipsync(38:49) Personalized fine-tuned models for creators?(39:38) Building models vs. building wrappers(43:09) Cloud AI vs. Local AI(45:19) Optimizing for low latency(48:07) AI/ML stack at Captions(51:10) “Hallucinations are a feature, not a bug”(53:19) Prompt engineering(54:12) Have we passed the uncanny valley for AI avatars?(01:01:47) The impact of deepfakes(01:04:33) CapCut ban and its effects(01:05:05) Evolving from paid to freemium(01:07:42) Building a company on foundation models(01:09:01) Running an AI company in New York
--------
1:13:14
Farewell, Chatbots: AI Agents Are Taking Over Customer Service | Mike Murchison, CEO, Ada
AI customer service agents are quickly replacing the often clunky AI chatbots of years past, and revolutionizing how we all interact with customer service. In this episode, we dive into this rapid transformation with Mike Murchison, CEO of Ada, a fast-growing leader in the space.Mike shares how harnessing the power of several Generative AI models enables Ada to automate up to 83% of customer interactions, providing a seamless and empathetic service that rivals, and will soon surpass, human agents. We explore the challenges and triumphs of deploying AI in customer service in this new era, from the intricacies of model orchestration to the importance of resolution and empathy. Mike also teases the future of agentic AI in the enterprise, where AI agents collaborate across departments to innovate and improve products.AdaWebsite - https://www.ada.cxX/Twitter - https://x.com/ada_cxMike MurchisonLinkedIn - https://www.linkedin.com/in/mikemurchisonX/Twitter - https://x.com/mimurchisonFIRSTMARKWebsite - https://firstmark.comX/Twitter - https://twitter.com/FirstMarkCapMatt Turck (Managing Director)LinkedIn - https://www.linkedin.com/in/turck/X/Twitter - https://twitter.com/mattturck(00:00) Intro(02:27) Why is customer service a perfect use case for AI?(03:36) Why didn’t foundation models replace AI “thin wrappers” out of the box?(05:27) What is Ada?(10:41) Reasoning engine, model orchestration, instruction following, routing(15:45) Hybrid systems, finetuning, customization(18:28) Prompt engineering, observability, self-improvement(22:07) RAG (Retrieval-Augmented Generation) and AI as a judge(23:06) Guardrails and security(24:33) Should we expect perfection from AI?(26:14) Measuring “resolution”(29:29) What actions can Ada AI Agents take?(32:12) Authentication and personalization(35:09) Handoff vs human delegation(38:12) ACX (AI Customer Experience) and the future of customer service professionals(42:13) Leveraging analytics and customer support data(45:54) AI agents for cross-selling and upselling(48:25) Traditional AI chatbots vs the new generation of AI Agents(51:24) Emotion, empathy, personality(54:56) Transparency and AI improvement(57:58) Managing AI: the measure-coach-improve loop(1:00:15) Ada Voice and Email(1:06:25) Future predictions for AI(1:07:56) Multi-agent collaboration
The MAD Podcast with Matt Turck, is a series of conversations with leaders from across the Machine Learning, AI, & Data landscape hosted by leading AI & data investor and Partner at FirstMark Capital, Matt Turck.
New Zealand