
The PhD students who became the judges of the AI industry
- 25m
- Transcript
Summary
Written by a model from the transcript; not yet reviewed.
Rebecca Bellan sits down with Arena co-founders Anastasios Angelopoulos and Wayland Chang, both former PhD students at UC Berkeley who started the platform as a research project in early 2023. Arena, formerly Chatbot Arena and LM Arena, has become the de facto public leaderboard for frontier LLMs, using pairwise human preference votes from millions of users to rank models. The company went from a research prototype to a $1.7 billion valuation in seven months, backed by a16z, OpenAI, Meta, Anthropic, and Google.
Angelopoulos explains why static benchmarks like Humanity's Last Exam suffer from overfitting, while Arena's constant stream of hundreds of thousands of daily conversations produces a leaderboard that converges statistically. He addresses concerns about selection bias (companies submitting optimized variants), fraud (employees flooding votes), and neutrality (taking money from the labs they rank). Wayland Chang details the expansion into agentic evaluations through CoArena, covering tool use, coding, and multi-modal tasks, plus a paid enterprise analytics product.
The episode is useful for understanding how AI evaluation has become its own venture-scale category, how a research project became the industry's North Star for model quality, and what structural safeguards a neutral evaluator needs when its customers are the very companies being judged.
- Static benchmarks suffer from overfitting because once a model has memorized the questions, it can perform well on the test without actually improving at the underlying task. 4:54
- Arena's Style Control methodology factors out stylistic features like response length, markdown headers, and sycophancy level so that rankings reflect true utility rather than presentation. 23:37
- The platform has over five million monthly users across 150 countries generating 60 million conversations per month, creating a data moat that is difficult for competitors to replicate. 18:47
- Arena requires that any model submitted for public leaderboard scoring must be the exact same model released to production, preventing companies from testing optimized variants. 9:31
- CoArena, launched two to three months before the episode, evaluates models on agentic tasks like building web applications, tool calling, and navigating codebases end to end. 20:58
Highlights
- 1:00
PhD students build the AI judgestory
Angelopoulos and Chang met at UC Berkeley three years before the episode, working on how to compare LLMs on real-world user distributions rather than static tests. The project started as a research paper and became a platform.
- 4:27
Why static benchmarks overfitinsight
Angelopoulos argues that once a model has seen benchmark questions, it can memorize them and perform well on the test without actually improving at the underlying task, making static benchmarks unreliable over time.
- 5:47
Task distribution across the platformdata
28% of users do coding, half do software engineering or creative work, 6% do legal tasks, and 6% do medical tasks, giving the leaderboard a broad prosumer base of economically valuable work.
- 9:19
No optimized variants allowedstrategy
Arena requires that any model submitted for public scoring must be the exact same model released to production, forcing providers to make their models genuinely useful for real users.
- 10:21
Structural neutrality over personal integrityinsight
Angelopoulos argues that because users generate the votes and an open-source pipeline converts them to scores, Arena has limited ability to intervene, making neutrality a property of the system rather than the company.
- 13:56
Stopping companies from flooding votesdebate
Bellan asks how Arena prevents AI companies from having thousands of employees vote to inflate their models' rankings. Angelopoulos describes a dedicated team analyzing voting patterns, mouse clicks, and reputation scores.
- 20:58
CoArena evaluates agentic AI capabilitiesstrategy
Wayland Chang describes CoArena as their first agentic evaluation, assessing models on building web applications, tool calling, coding, and navigating codebases end to end in real time.
- 23:32
Style Control removes vibes from rankingsstrategy
Arena's default leaderboard controls for response length, markdown headers, and sycophancy level, similar to controlling for confounds in a social science study, so that rankings reflect true utility rather than presentation.
About this episode
Artificial intelligence models are multiplying fast, and competition is stiff. With so many players crowding the space, which one will be the best — and who decides that? Arena, formerly LM Arena, has emerged as the de facto public leaderboard for frontier LLMs, influencing funding, launches, and PR cycles. In just seven months, the startup went from a UC Berkeley PhD research project to being valued at $1.7 billion. On this episode of TechCrunch's Equity podcast, Rebecca Bellan catches up with Arena co-founders Anastasios Angelopoulos and Wei-Lin Chiang to determine how a team like theirs can build a neutral benchmark when the companies they’re ranking are also their backers. Listen to the full episode to hear: How Arena actually works, and why its founders say you can't game it the way you mighta static benchmark. What "structural neutrality" actually means, and whether taking money from OpenAI, Google, and Anthropic is a conflict of interest. How Arena is moving beyond chat to benchmark agents, coding, and real-world tasks with a new enterprise product. Why Claude is currently winning the expert leaderboard for legal and medical use cases. Arena's bet on what comes after LLMs, and why agents are next on the leaderboard. Subscribe to Equity on YouTube, Apple Podcasts, Overcast, Spotify and all the casts. You also can follow Equity on X and Threads, at @EquityPod. Chapters: 00:00 Intro 03:00 How Arena's leaderboard works, and why it's different from static benchmarks 07:00 Reproducibility concerns and how to scale 08:45 Can Arena stay independent while taking money from the labs it ranks? 11:15 Diversity, fraud prevention, and abuse mitigation 18:15 Arena's "data moat" 19:20 Agent benchmarking and expert leaderboards 21:40 Open sourcing data 22:45 How do Arena's rankings shape AI development? 24:15 Outro Learn more about your ad choices. Visit megaphone.fm/adchoices
Transcript
4,345 words · 4 speakers
Topics
- AI Evaluation
- LLM Benchmarks
- Human Preference Data
- Agentic AI
- Startup Funding
- Model Ranking
- Data Moats
- AI Safety
Page updated · transcript and summary are machine-generated