Skip to content

Why inference is harder than you think

The Generalist

Hosted by Mario Gabriele

44m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode explores the rapidly expanding and often misunderstood market of AI inference with Tuan Srivastava, co-founder and CEO of Base10. The conversation details how Base10 evolved from early machine learning tooling for data scientists into a specialized infrastructure provider operating across twenty different clouds and handling massive production workloads. The discussion examines why inference represents the definitive future market for AI, the complexities of managing heterogeneous compute during severe supply shortages, and how enterprises are moving toward a portfolio approach utilizing both closed models and customized open-weight models.

Listeners get a behind-the-scenes look at the operational realities of scaling AI infrastructure, including the financial engineering, five-year terms, and thirty percent prepayments required to secure billions of dollars in compute. Srivastava explains the concept of owned intelligence and continuous learning loops where companies capture data from user interactions to retrain proprietary models. The episode stands out for its clear, highly specific breakdown of the multi-step journey organizations take as they transition from basic API routing to dedicated deployments and custom fine-tuning.

  • Anticipate long-term forecasting gaps by planning for rapid growth that can outpace initial projections by severalfold in emerging markets. 7:17
  • Recognize that serving models at scale requires a dedicated, heterogeneous stack rather than traditional cloud or training infrastructure. 17:46
  • Build compounding organizational advantages by establishing continuous learning loops that integrate proprietary data, user feedback, and model retraining. 25:15
  • Navigate severe hardware shortages through architectural flexibility and multi-cloud strategies that stitch together regional compute resources. 29:44

Chapters

  1. 0:00
    Podcast Introduction and Inference Overview

    Hosts introduce the episode, framing inference as the next critical AI challenge.

  2. 0:35
    The Complexity of Inference at Scale

    Discussion of why inference is hard to scale, the need for capacity, and the software challenges involved.

  3. 2:39
    Early Years of Base10 and Product Pivot

    The founders recount starting in 2019, the initial focus on data science tools, and the shift to serving inference as the core product.

  4. 7:38
    Market Dynamics and the Supply Crunch

    Exploration of the rapidly growing inference market, the limited compute supply, and the impact on pricing and growth.

  5. 20:55
    Customer Journey: From Closed to Open Models

    Illustrates how customers move from using closed models to incorporating open‑weight models, routing, and post‑training loops.

  6. 29:44
    Heterogeneous Compute Strategy

    Base10’s approach to using multiple clouds and regions to achieve reliability, low latency, and geographic flexibility.

  7. 31:33
    Financial and Operational Challenges

    Details the complexities of financing large compute spend, project‑finance style deals, and the need for long‑term commitments.

  8. 36:54
    Hiring Philosophy and Customer Success

    Tuan shares what he looks for in talent and how a hospitality mindset drives customer relationships and product excellence.

  9. 42:34
    Future Outlook and Experimentation

    Speculation on future model portfolios, the role of post‑training, and the potential for unlimited compute resources.

  10. 43:42
    Closing Remarks and Call to Action

    Final thanks, podcast subscription reminders, and a brief sign‑off from the host.

Highlights

  • 2:44

    Pivoting From Early ML Toolsstory

    The guest explains how the company originally built scaffolding for data scientists using small models and internal tools, eventually discarding three years of work in late 2022 to focus entirely on high-scale inference serving.

  • 7:17

    Revenue Forecasts Consistently Offdata

    The guest reveals that the company's revenue forecasts for consecutive years have been completely off, including being off by 4x against initial January projections for the year 2025 due to exponential market demand.

  • 17:46

    Inference As The Final Marketinsight

    The discussion frames inference as the final market of the AI economy because it represents the actual running of intelligence and compute cost of goods, projected to reach trillions of dollars as AI scales.

  • 21:26

    The Evolution Of Customer Workflowsstrategy

    The guest outlines the typical customer journey from using closed APIs to adopting open-weight models, implementing dedicated deployments, and finally building custom post-trained models with specialized routing layers.

  • 26:53

    Enterprise Shift To Open Modelsinsight

    The host and guest discuss how companies ranging from AI natives like Sierra to large enterprises like AT&T are increasingly relying on a portfolio of models to optimize costs and protect proprietary data.

  • 34:49

    The Financial Reality Of Computedata

    The guest breaks down the capital intensity of modern inference infrastructure, noting that securing billions in compute typically requires five-year terms and thirty percent prepayments.

  • 37:47

    Unreasonable Hospitality In Softwareinsight

    The conversation connects the company's client relationship philosophy to Danny Meyer's principles of hospitality, highlighting high-touch service and deep integration with customer engineering teams.

Quotes

  • “Inference in a lot of ways, I think, is maybe like one of the most misunderstood markets in the world.”
    Unnamed speaker · 0:08
  • “The only thing left to do will be inference.”
    Unnamed speaker · 0:00
  • “We are critical path. Like if Base10 is down, a lot of our businesses aren't working.”
    Unnamed speaker · 0:25
  • “Inference is the cause of AI.”
    Unnamed speaker · 18:36

About this episode

Tuhin Srivastava https://www.linkedin.com/in/tuhin-srivastava is the co-founder and CEO of Baseten https://www.baseten.co/, a company building infrastructure for training and inference. When an AI model runs in production (responding to a user, powering an app, making a decision), Baseten is one of the companies that makes that happen reliably, quickly, and at scale. The company counts Abridge, Cursor, and OpenEvidence among its customers, spans 20 clouds and 90 regions, and was most recently valued at $13 billion following a $1.5 billion funding round. Tuhin sees inference as AI’s “final market”: as AI spreads through the economy, the cost of running intelligence grows with it. Baseten’s breakout began when it discarded three years of other product work in roughly six weeks to focus on an inference engine it had been giving away. In this conversation, we explore why inference has become one of the most consequential and least understood markets in AI, what it actually takes to run a model in production at scale, and why the compute squeeze may be the single largest constraint on AI growth right now. We also examine the customer journey from closed models to custom fine-tuned deployments, the concept of owned intelligence, and why Tuhin believes luck accounts for roughly 90% of startup success, while insisting that the 10% skill is still indispensable. — Tuhin’s bookshelf When I asked Tuhin what book he’d assign to everyone, he named Breakneck https://www.amazon.com/Breakneck-Chinas-Quest-Engineer-Future-ebook/dp/B0DXQGL4X9. He also pointed to Danny Meyer’s book about Union Square Cafe, Setting the Table https://www.amazon.com/dp/0060742763. — Episode transcript: https://www.generalist.com/i/223344398/transcript — Every timestamp (00:00) Intro (02:06) What Baseten is building and how it began (04:47) Finding balance in a fast-growing startup (06:45) Why inference is the “final market” (08:34) Tuhin’s path to founding Baseten (12:06) What Australia gets right (13:43) The opportunity Baseten found in inference (20:34) How customers use Baseten (29:00) Managing compute across clouds (30:40) The compute crunch (32:11) Financing inference at scale (35:40) Why inference is hard to build in-house (36:41) Who Baseten wants to hire (39:46) Hard work and skill vs. luck (42:20) Final meditations — Resources Baseten: https://www.baseten.co https://www.baseten.co/Sierra: https://sierra.ai https://sierra.ai/Coinbase: https://www.coinbase.com https://www.coinbase.com/No Priors Ep 56 | With Baseten CEO and Co-Founder Tuhin Srivastava: https://www.youtube.com/watch?v=oe3OSlsf4YAEtched: https://www.etched.com https://www.etched.com/Cerebras: https://www.cerebras.ai https://www.cerebras.ai/NVIDIA Groq 3 LPX: https://www.nvidia.com/en-us/data-center/lpxUnreasonable Hospitality: The Remarkable Power of Giving People More Than They Expect: https://www.amazon.com/Unreasonable-Hospitality-Remarkable-Giving-People/dp/0593418573Blackbird VC: https://www.blackbird.vc https://www.blackbird.vc/Greylock: https://greylock.com https://greylock.com/Conviction: https://www.conviction.com https://www.conviction.com/— Where to find Tuhin X: https://x.com/tuhinone LinkedIn: https://www.linkedin.com/in/tuhin-srivastava — Production and marketing by penname.co https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].

Transcript

0:00
44:16

8,839 words · 3 speakers

Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.

Topics

  • Base10
  • Nvidia
  • Greylock
  • Conviction
  • Refusion
  • Sierra
  • Abridge
  • Cursor
  • OpenEvidence
  • Amazon Web Services

Page updated · transcript and summary are machine-generated