ICLR 2024 — Best Papers & Talks (Benchmarks, Reasoning & Agents) — ft. Graham Neubig, Aman Sanger, Moritz Hardt)

Our second wave of speakers for AI Engineer World’s Fair https://www.ai.engineer/worldsfair were announced https://x.com/swyx/status/1797654825968291862! The conference sold out of Platinum/Gold/Silver sponsors and Early Bird tickets! See our Microsoft episode https://www.latent.space/p/worlds-fair-2024#%C2%A7show-notes for more info and buy now https://ti.to/software-3/ai-engineer-worlds-fair with code LATENTSPACE. This episode is straightforwardly a part 2 to our ICLR 2024 Part 1 episode https://www.latent.space/p/iclr-2024-recap, so without further ado, we’ll just get right on with it! Timestamps [00:03:43] Section A: Code Edits and Sandboxes, OpenDevin, and Academia vs Industry — ft. Graham Neubig and Aman Sanger * [00:07:44] WebArena * [00:18:45] Sotopia * [00:24:00] Performance Improving Code Edits * [00:29:39] OpenDevin * [00:47:40] Industry and Academia [01:05:29] Section B: Benchmarks * [01:05:52] SWEBench * [01:17:05] SWEBench/SWEAgent Interview * [01:27:40] Dataset Contamination Detection * [01:39:20] GAIA Benchmark * [01:49:18] Moritz Hart - Science of Benchmarks [02:36:32] Section C: Reasoning and Post-Training * [02:37:41] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection * [02:51:00] Let’s Verify Step By Step * [02:57:04] Noam Brown * [03:07:43] Lilian Weng - Towards Safe AGI * [03:36:56] A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis * [03:48:43] MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework [04:00:51] Bonus: Notable Related Papers on LLM Capabilities Section A: Code Edits and Sandboxes, OpenDevin, and Academia vs Industry — ft. Graham Neubig and Aman Sanger * Guests * Graham Neubig https://phontron.com/research.php * Aman Sanger - Previous guest https://www.latent.space/p/cursor and NeurIPS friend of the pod https://www.latent.space/p/neurips-2023-startups! * WebArena https://openreview.net/forum?id=oKn9c6ytLx * * Sotopia https://openreview.net/forum?id=mM7VurbA4r (spotlight paper, website https://www.sotopia.world/) * * Learning Performance-Improving Code Edits https://arxiv.org/abs/2302.07867 * OpenDevin https://x.com/gneubig/status/1787863237611901215 * Junyang Opendevin https://github.com/JustinLin610 * Morph Labs https://github.com/princeton-nlp/SWE-bench, Jesse Han * SWE-Bench https://github.com/princeton-nlp/SWE-bench * SWE-Agent https://github.com/princeton-nlp/SWE-agent * Aman tweet on swebench https://x.com/amanrsanger/status/1779620682340704386 * LiteLLM https://github.com/BerriAI/litellm * Livecodebench https://arxiv.org/abs/2403.07974 * the role of code in reasoning * Language Models of Code are Few-Shot Commonsense Learners https://arxiv.org/abs/2210.07128 * Industry vs academia * the matryoshka embeddings incident https://x.com/adityakusupati/status/1750911725166334290 * other directions * Unlimiformer https://arxiv.org/abs/2305.01625 Section A timestamps * [00:00:00] Introduction to Guests and the Impromptu Nature of the Podcast * [00:00:45] Graham's Experience in Japan and Transition into Teaching NLP * [00:01:25] Discussion on What Constitutes a Good Experience for Students in NLP Courses * [00:02:22] The Relevance and Teaching of Older NLP Techniques Like Ngram Language Models * [00:03:38] Speculative Decoding and the Comeback of Ngram Models * [00:04:16] Introduction to WebArena and Zotopia Projects * [00:05:19] Deep Dive into the WebArena Project and Benchmarking * [00:08:17] Performance Improvements in WebArena Using GPT-4 * [00:09:39] Human Performance on WebArena Tasks and Challenges in Evaluation * [00:11:04] Follow-up Work from WebArena and Focus on Web Browsing as a Benchmark * [00:12:11] Direct Interaction vs. Using APIs in Web-Based Tasks * [00:13:29] Challenges in Base Models for WebArena and the Potential of Visual Models * [00:15:33] Introduction to Zootopia and Exploring Social Interactions with Language Models * [00:16:29] Different Types of Social Situations Modeled in Zootopia * [00:17:34] Evaluation of Language Models in Social Simulations * [00:20:41] Introduction to Performance-Improving Code Edits Project * [00:26:28] Discussion on DevIn and the Future of Coding Agents * [00:32:01] Planning in Coding Agents and the Development of OpenDevon * [00:38:34] The Changing Role of Academia in the Context of Large Language Models * [00:44:44] The Changing Nature of Industry and Academia Collaboration * [00:54:07] Update on NLP Course Syllabus and Teaching about Large Language Models * [01:00:40] Call to Action: Contributions to OpenDevon and Open Source AI Projects * [01:01:56] Hiring at Cursor for Roles in Code Generation and Assistive Coding * [01:02:12] Promotion of the AI Engineer Conference Section B: Benchmarks * Carlos Jimenez & John Yang (Princeton) et al: SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR Oral https://iclr.cc/virtual/2024/oral/19757, Paper https://arxiv.org/abs/2310.06770, website https://swe-bench.github.io/) * “We introduce SWE-bench, an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.” * Yonatan Oren et al (Stanford): Proving Test Set Contamination in Black-Box Language Models (ICLR Oral https://iclr.cc/virtual/2024/session/15093, paper https://arxiv.org/abs/2310.17623, aman tweet on swebench contamination https://x.com/amanrsanger/status/1779620682340704386) * “We show that it is possible to provide provable guarantees of test set contamination in language models without access to pretraining data or model weights. Our approach leverages the fact that when there is no data contamination, all orderings of an exchangeable benchmark should be equally likely. In contrast, the tendency for language models to memorize example order means that a contaminated language model will find certain canonical orderings to be much more likely than others. Our test flags potential contamination whenever the likelihood of a canonically ordered benchmark dataset is significantly higher than the likelihood after shuffling the examples. * We demonstrate that our procedure is sensitive enough to reliably prove test set contamination in challenging situations, including models as small as 1.4 billion parameters, on small test sets of only 1000 examples, and datasets that appear only a few times in the pretraining corpus.” * Outstanding Paper mention https://blog.iclr.cc/2024/05/06/iclr-2024-outstanding-paper-awards/: “A simple yet elegant method to test whether a supervised-learning dataset has been included in LLM training.” * Thomas Scialom https://x.com/thomasscialom?lang=en (Meta AI-FAIR w/ Yann LeCun https://x.com/ylecun/status/1788850516660789732?utm_source=ainews&utm_medium=email&utm_campaign=ainews-anthropics): GAIA: A Benchmark for General AI Assistants (paper https://arxiv.org/abs/2311.12983) * “We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. * GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92% vs. 15% for GPT-4 equipped with plugins. * GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system's capability to exhibit similar robustness as the average human does on such questions. Using GAIA's methodology, we devise 466 questions and their answer. * * Mortiz Hardt https://mrtz.org/ (Max Planck Institute): The emerging science of benchmarks (ICLR stream https://iclr.cc/virtual/2024/invited-talk/21799) * “Benchmarks are the keystone that hold the machine learning community together. Growing as a research paradigm since the 1980s, there’s much we’ve done with them, but little we know about them. In this talk, I will trace the rudiments of an emerging science of benchmarks through selected empirical and theoretical observations. Specifically, we’ll discuss the role of annotator errors, external validity of model rankings, and the promise of multi-task benchmarks. The results in each case challenge conventional wisdom and underscore the benefits of developing a science of benchmarks.” Section C: Reasoning and Post-Training * Akari Asai (UW) et al: Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection https://iclr.cc/virtual/2024/oral/19736 (ICLR oral https://iclr.cc/virtual/2024/session/15076, website https://selfrag.github.io/) * (Bad RAG implementations) indiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary, or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation. * We introduce a new framework called Self-Reflective Retrieval-Augmented Generation (Self-RAG) that enhances an LM's quality and factuality through retrieval and self-reflection. * Our framework trains a single arbitrary LM that adaptively retrieves passages on-demand, and generates and reflects on retrieved passages and its generations using special tokens, called reflection tokens. Generating reflection tokens makes the LM controllable during the inference phase, enabling it to tailor its behavior to diverse task requirements. * Self-RAG (7B and 13B parameters) outperforms ChatGPT and retrieval-augmented Llama2-chat on Open-domain QA, reasoning, and fact verification tasks, and it shows significant gains in improving factuality and citation accuracy for long-form generations relative to these models. * Hunter Lightman (OpenAI): Let’s Verify Step By Step (paper https://arxiv.org/abs/2305.20050) * “Even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. * We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. * To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model. * * Noam Brown https://x.com/polynoamial/status/1777809000345505801?s=46&t=90xQ8sGy63D2OtiaoGJuww - workshop on Generative Models for Decision Making https://iclr.cc/virtual/2024/22266 * Solving Quantitative Reasoning Problems with Language Models https://arxiv.org/abs/2206.14858 (Minerva paper) * Describes some charts taken directly from the Let’s Verify Step By Step paper listed/screenshotted above . * Lilian Weng https://x.com/lilianweng/ (OpenAI) - Towards Safe AGI (ICLR talk https://iclr.cc/virtual/2024/22336) * OpenAI Model Spec https://cdn.openai.com/spec/model-spec-2024-05-08.html * OpenAI Instruction Hierarchy: The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions https://arxiv.org/abs/2404.13208 Section D: Agent Systems * Izzeddin Gur (Google DeepMind): A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (ICLR oral https://iclr.cc/virtual/2024/session/15073, paper https://arxiv.org/abs/2307.12856) * [Agent] performance on real-world websites has still suffered from (1) open domainness, (2) limited context length, and (3) lack of inductive bias on HTML. * We introduce WebAgent, an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions. * WebAgent plans ahead by decomposing instructions into canonical sub-instructions, summarizes long HTML documents into task-relevant snippets, and acts on websites via Python programs generated from those. * We design WebAgent with Flan-U-PaLM, for grounded code generation, and HTML-T5, new pre-trained LLMs for long HTML documents using local and global attention mechanisms and a mixture of long-span denoising objectives, for planning and summarization. * We empirically demonstrate that our modular recipe improves the success on real websites by over 50%, and that HTML-T5 is the best model to solve various HTML understanding tasks; achieving 18.7% higher success rate than the prior method on MiniWoB web automation benchmark, and SoTA performance on Mind2Web, an offline task planning evaluation. * Sirui Hong (DeepWisdom): MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (ICLR Oral https://iclr.cc/virtual/2024/session/15076, Paper https://arxiv.org/abs/2308.00352) * We introduce MetaGPT, an innovative meta-programming framework incorporating efficient human workflows into LLM-based multi-agent collaborations. MetaGPT encodes Standardized Operating Procedures (SOPs) into prompt sequences for more streamlined workflows, thus allowing agents with human-like domain expertise to verify intermediate results and reduce errors. MetaGPT utilizes an assembly line paradigm to assign diverse roles to various agents, efficiently breaking down complex tasks into subtasks involving many agents working together. Bonus: Notable Related Papers on LLM Capabilities This includes a bunch of papers we wanted to feature above but could not. * Lukas Berglund (Vanderbilt) et al: The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A” (ICLR poster https://iclr.cc/virtual/2024/poster/19033, paper https://arxiv.org/abs/2309.12288, Github https://github.com/lukasberglund/reversal_curse) * We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form ''A is B'', it will not automatically generalize to the reverse direction ''B is A''. This is the Reversal Curse. * The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation. We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as ''Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]'' and the reverse ''Who is Mary Lee Pfeiffer's son?''. GPT-4 correctly answers questions like the former 79\% of the time, compared to 33\% for the latter. * * Omar Khattab (Stanford): DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines (ICLR Spotlight Poster https://iclr.cc/virtual/2024/poster/17642, GitHub https://github.com/stanfordnlp/dspy) * presented by Krista Opsahl-Ong * “Existing LM pipelines are typically implemented using hard-coded “prompt templates”, i.e. lengthy strings discovered via trial and error. Toward a more systematic approach for developing and optimizing LM pipelines, we introduce DSPy, a programming model that abstracts LM pipelines as text transformation graphs, or imperative computational graphs where LMs are invoked through declarative modules. * DSPy modules are parameterized, meaning they can learn how to apply compositions of prompting, finetuning, augmentation, and reasoning techniques. * We design a compiler that will optimize any DSPy pipeline to maximize a given metric, by creating and collecting demonstrations. * We conduct two case studies, showing that succinct DSPy programs can express and optimize pipelines that reason about math word problems, tackle multi-hop retrieval, answer complex questions, and control agent loops. * Within minutes of compiling, DSPy can automatically produce pipelines that outperform out-of-the-box few-shot prompting as well as expert-created demonstrations for GPT-3.5 and Llama2-13b-chat. On top of that, DSPy programs compiled for relatively small LMs like 770M parameter T5 and Llama2-13b-chat are competitive with many approaches that rely on large and proprietary LMs like GPT-3.5 and on expert-written prompt chains. * * MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning https://arxiv.org/abs/2310.16049 * Scaling Laws for Associative Memories https://arxiv.org/abs/2310.02984 * DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models https://arxiv.org/abs/2309.03883 * Efficient Streaming Language Models with Attention Sinks https://github.com/mit-han-lab/streaming-llm This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:05Speaker 00

  Welcome to the Latent Space Podcast, ICLR Edition, Part Two. This is Charlie, your AI co-host. We're back with our coverage of the 12th International Conference on Learning Representations in Vienna, Austria. Many of you absolutely loved our Europe's coverage last year, and we're proud to bring you Part Two of our special two-part episode, covering our attempt at giving you an audio experience of ICLR. If you'd like to see us return to Vienna for ICML, let us know by sharing this episode on X and LinkedIn. In Part One, we covered the best papers of ICLR across four sections: image generation and diffusion, computer vision, and weak supervision, improving attention algorithms,

• 0:50Speaker 00

  and state-space models. In today's episode, we cover the wealth of papers we found around the related problems of LLM reasoning and agents, also in four sections. In Section A, we do a regular latent space chat with Graham Newbig of Web Arena and Open Devon, introducing many of the major themes of the rest of this episode. In Section B, we survey a few prominent issues in benchmarking, from SUE Bench, test-set contamination, and general intelligence. In Section C, we look at agent-building blocks from RAG and self-reflection, verification, safety, and frameworks. In Section D, we finally look at two proposed agent systems from Google DeepMind's Web

• 1:35Speaker 00

  Agent and MetagePete. This is the second of two episodes covering ICLR, which is overwhelmingly academia-focused. If you're interested in production AI engineering in industry, you should join us at the first AI Engineer World's Fair this June, where we have now announced many of our speakers from all the big clouds, all the large model labs, now including Anthropic, Coher, and Cartesian, the brand-new state-space model startup, all top AI-enabled developer tools and code-gen agents, now including Quinn Slack, CEO of SourceGraph.

• 2:10Speaker 00

  Insights on the GPU and inference market like last week's guest gradient AI. And now featuring Dylan Patel of the semi-analysis GPU poor blog. And the rest of the emerging L L M O S stack of startups and open source tools across rag, multimodality, L L M ops and agent frameworks. Disruptive startups like mid journey perplexity and character AI. And for the first time, a closed door track for VPs of AI and technical leaders to discuss AI strategy and leadership. Get your tickets now and see you in San Francisco from June 25th to 27th. We'll start this episode with a special onsite interview we did at I C L R with Professor Graham

• 2:54Speaker 00

  Newbig of the Language Technologies Institute of Carnegie Mellon University. Graham has taught the C M U N L P course for the last seven years, but is also an active participant in the open source AI software ecosystem, having been personally involved in Open Devon, which recently scored a notable 21% unassisted resolve rate on S W E Benchlight. As an extra special treat, we are proud to invite Amon Sanger, co-founder of Cursor AI, back as our first ever guest co-host to add his personal takes on the state of code editing and agents. This is going to be a doozy of an episode, so we better get started. Watch out and take care.

• 3:43Speaker 01

  So, welcome to the pod Graham and welcome to the pod Amon, our first ever guest co-host. Thank you for having me.

• 3:50Speaker 02

  Yeah, thanks for having me.

• 3:51Speaker 01

  Yeah, thanks for taking some time during I clear. This is like very impromptu, but the two of you wanted to chat. I was like, let's just record a chat and that can be fun. And also, one of my goals here at conferences like this is just cover posters for people who are at home, like not at the conference like this, just to get a sense of the mood. So I'll cover a little bit of your background and then you can sort of do fill in the blanks. So you're a professor at C F U, teach N L P. You also spent some years in Japan as a language teacher as well as a grad student.

• 4:19Speaker 02

  Yeah, it was a good experience. It was very impromptu me going there just because I wanted to learn a new language, but I ended up staying there for 11 years.

• 4:27Speaker 01

  Yeah.

• 4:27Speaker 02

  Never know how life leads you, I guess.

• 4:29Speaker 01

  Yeah, now you run your own lab and you teach the advanced NLP course. I'm sure that's been a wild ride over the past seven years. Was it 2016?

• 4:38Speaker 02

  Yep, the last time I taught it before the semester was before ChatGPT. So I had to go in and rip out everything, all the new stuff in. But yeah, it's a good opportunity to keep up with all the new stuff too. Because I feel like I need to pressure myself into like giving a good experience. So I need to, you know, cover all the areas.

• 4:57Speaker 01

  What do you mean good experience? Like, what is a good experience for students?

• 5:01Speaker 02

  I think a good experience for students is them knowing stuff that's actually practically applicable in whatever they go on to do next. And I'm teaching the advanced NLP course, which is for people who are more on the like research and new innovation side as opposed to just, you know, pulling in existing technologies. So because of that, I feel like I need to stay on like the cutting edge of the most important things that people should know for the people who are pushing the boundaries to do the next thing.

• 5:26Speaker 03

  For the NLP course, how deep into the kind of older school techniques and fundamentals do you go?

• 5:31Speaker 02

  Yeah, so this is a constant battle because there are some older stuff that's like interesting, you know, algorithmically interesting, but we don't use as much now. It might come back later, it might not. But I think right now, given the limited amount of time I have and the number of things that would be useful to know, I mostly focus on the parts that are like in the modern stack of how you build models.

• 5:52Speaker 03

  So what do you like to keep in? Is there anything that you do kind of keep in that's not really used much today?

• 5:58Speaker 02

  Yeah, it's a good question. I still teach N-gram language models, which are language models that are based on like counting up the number of words that follow other words and stuff. And we don't use them at all today, but I keep them in there at the very beginning. So people have an idea of how you could calculate this without throwing it into a neural network black box and what's the difficulty and why neural networks are important. But that's about it, I think.

• 6:24Speaker 03

  Interestingly, N-grams may be making a bit of a comeback for speculative decoding. Yeah, that's true. Yeah.

• 6:30Speaker 01

  Can you elaborate?

• 6:31Speaker 03

  Oh, just one other way of kind of suspected decoding is when you have some kind of graph model predicting forward tokens. So you can kind of batch at the sequence level tokens when you're doing generation. Just because if you're generating one token at a time, it's much slower than kind of prefilling a bunch of tokens at a time. So in this case, you would use kind of N-gram models to if, for example, you had V, maybe you know, V is often followed by cat. And you wouldn't even need a decoder model or a sorry graph model, which is a smaller model that would be kind of generating tokens. Ahead, you could just use kind of N-gram statistics or like basic N-gram models for this.

• 7:08Speaker 01

  I was thinking also that by pair encoding is kind of N-grammy. You sort of build it up from.

• 7:13Speaker 03

  It weaves by grams.

• 7:14Speaker 01

  Yeah, well, we could cover more later on, but you know, I just wanted to touch briefly on the posters that you presented. I didn't know you had three, actually, I only prepped on two, but Aman can fill in the blanks on the one that you're presenting today. So Web Arena, Soltopia, and then performance improving code edits.

• 7:29Speaker 03

  Yep.

• 7:29Speaker 01

  Anything else that I missed?

• 7:30Speaker 03

  Those are the three.

• 7:31Speaker 01

  Those are the three. Okay, so Web Arena and Soltopia are basically like two kinds of sandboxes is what I was thinking about. And maybe like which one do you do you want to tackle first? Like, how do you tell the story about how this work is done at your lab?

• 7:44Speaker 02

  I think I can do Web Arena first.

• 7:45Speaker 01

  Yeah.

• 7:46Speaker 02

  So to give a little bit of context, I never really was an evaluation person or never really cared about evaluation that much. For a long time until maybe 2022 or so. And so I was mostly working on system building stuff. But once we started getting into the, you know, like really good GPT models. My typical formula for system building was to figure out what's not working and fix it. And I got to 2022 and I was like, I don't know what's not working. These language models are so good. I don't want to be working on insignificant things that are already solved by the strongest models. And so because of that, I got into benchmarking and kind of my goal for benchmarking is to push the boundaries of what is possible in, you know, a rigorous way.

• 8:28Speaker 02

  And kind of both of these, Web Arena and Soltopia are aiming to do that. I could explain about Web Arena in more detail.

• 8:36Speaker 01

  Yeah, we'll put a slide of the poster up. It's like a pretty useful slide. I like the phrase, we create a mini internet for your agent to play a master.

• 8:46Speaker 03

  Yep.

• 8:47Speaker 02

  So I can't take credit for that. That was probably Shuyuan, the first author, who came up with that. But the basic idea is like, I'm very interested in how language model agents that act in the world can like actually work. So originally, what Shuyuan, the first author, wanted to do is she wants a robot that helps her out with housework, so she doesn't have to do housework. The problem with robots is robotics isn't there yet. It's like, I feel like robotics is the bottleneck in language model plus robotics work at the moment. And so we tried to think of something that had some of the interesting problems for like kind of long horizon planning and also where you can pull in world knowledge about how the world works and use that in an interesting way.

• 9:31Speaker 02

  So eventually, we settled on doing tasks on the web as a way to benchmark this. And we wanted a benchmark that was as realistic as possible. So if you do well on it, it actually kind of means something. So like, a lot of people complained about how academic benchmarks don't mean anything. And we wanted to create an academic benchmark that does, you know, actually mean something. And we wanted it to be evaluated in a way where you're not evaluating how good you are compared to like mimicking how humans do it, but like actually how good you are at solving the task. So we set up a sandbox internet by taking production-grade open-source sites. Yeah, this is Reddit, GitLab, CMS. Yeah, so the sites, we tried to mimic some like existing sites.

• 10:16Speaker 02

  So like, Amazon is an existing site, and we mimicked it with an open-source counterpart, OneStopShop. Yep. Reddit is an existing site. We mimicked it with an open-source site called Postmill. And GitHub is an existing site that we mimicked with GitLab. And all of these are like, you know, open source, they're actually used by people to do real things, so they're pretty close to being realistic. Then we created a whole bunch of tasks that you would want to do over the sites. And the way we did this is we looked through our own Chrome browsing history. So we were like, these are some of the things that we, like, as authors did over the past month. And then we tried to generalize them into things that would work on the sites. And then we wrote some validators to check whether it, like, actually succeeded. So to give an example, one of the tasks is, like, tell me how much I spent on food in the past month,

• 11:00Speaker 02

  which, like, I had personally done because I wanted to, you know, budget and make sure I wasn't spending too much, or things like this. So in order to do that, the agent has to go to the shopping site, has to identify all the food purchases. It has to figure out which ones happened in the past month and then add things together. And language models aren't particularly good at navigation. They aren't particularly good at filtering, and they're not particularly good at math. So despite the fact that this is, like, pretty trivial for humans, it's hard for language models to do now.

• 11:28Speaker 01

  Yeah, and you had some stats here where basically all the models were under 15% and humans were 78%.

• 11:35Speaker 02

  Yep.

• 11:35Speaker 01

  I like that kind of wide variance for a benchmark because it means buys us maybe a year before language models catch up.

• 11:43Speaker 02

  Yeah, so in the past six months, this came out about six months ago. In the past six months, we've increased by, from like 14 to 25 or 30% in the state of the art.

• 11:51Speaker 01

  Using GPT-4?

• 11:53Speaker 02

  Yeah, using GPT-4 as a base model, it's based on improvement.

• 11:56Speaker 01

  You mean GPT-4 Turbo has?

• 11:57Speaker 02

  No, no, so actually, LLM improvements are not the main driver behind this. It's more about the, like, way that the agent uses LLM.

• 12:05Speaker 01

  Got it.

• 12:06Speaker 02

  And to give some examples of some things that people have done, the first thing is just, like, optimizing the prompts in the action space. So it's, you know, prompt engineering, figuring out which actions you can do, like, good ways to scroll up and down the page and, like, click on buttons and stuff like this. Other things that are maybe somewhat more methodologically interesting are, it takes a step and does kind of self-refinement or self-reflection about whether that step was a good step or not, and then rolls back if it was a bad step. And there's another one that tries to create, like, a textbook about how the sites work and give information about how the sites work. Like, if you want to do this, you should go to this page. So it's creating, like, world-

• 12:47Speaker 01

  Create a textbook, yeah.

• 12:47Speaker 02

  Yeah, creating world knowledge about, or documentation might be a better word, about how the sites work. And then feeding that into the agent based on where the agent is and like the tasks. So there's a lot of creative things that people have done with this, but we're still at 25 to 30% as opposed to the 80, 80%.

• 13:05Speaker 03

  Well, one thing that's surprising to me is it's only 70, 80% for humans.

• 13:09Speaker 02

  Yeah, so think about half of this gap is just human negligence for doing the task. Like they don't follow the instructions exactly. About half of this actually might just be an issue with the benchmarks validators being too strict. And so the human gets the correct answer, but the validators are like not counting it as the correct answer. So they're looking for like an exact match and it's an exact match, but it's off by a rounding error or something like that. And so we could go back and fix all of the validators and that might bump up the human performance and model performance. But still, I think the gap would be the same.

• 13:44Speaker 03

  I'm surprised that long context hasn't helped a decent bit for this. Like you mentioned, it was mainly not on the model side, but at least I'd assume for like coding agents, longer context is helpful. Not sure what it looks like here.

• 13:57Speaker 02

  Yeah, for longer context, we haven't seen any papers that say adding a long context helps. It's also kind of not clear what exactly more context you would put in because we're putting in the entire past action sequence.

• 14:10Speaker 03

  Oh, and that fits entirely.

• 14:12Speaker 02

  It's the entire past action sequence, but we're not putting in the past pages. So if we added the past pages, it might give us a little bit more information, but it might also explode the context that the model's looking at. So yeah, maybe somebody will figure out how to make it work, but I haven't seen it yet. And so what's the follow on work from Webberina? I think web browsing is super interesting. I think web browsing, the tasks that we created here were based on our personal browsing history. And they were also created to be kind of like manageable based on what we were able to do at the time. And since then we've released it and we have a lot of new ideas about coming up with things that are more representative of actual tasks that people do in jobs.

• 14:54Speaker 02

  And so we're working on something related to this, you know, like what are some of the tasks that software engineers do and things like this? And so in order to do that, we need to. To have web browsing, we need to have it over realistic sites that people use in like actual workplace scenarios. So we're working a bit on that. And then separately, now that we have a benchmark, as I mentioned, I'm not like an evaluation person, like that's not my core passion. My core passion is building things at work. And so now we have a good evaluation that I care about. So we're doing lots of things to improve it. Some of the things we're thinking about are training models based on synthetic data or reinforcement learning methods. Also, better ways to interface with websites. So maybe we should be using an API instead of directly interacting with the website or things like this.

• 15:37Speaker 01

  Directly interacting meaning point and click?

• 15:39Speaker 02

  Yeah.

• 15:40Speaker 01

  Using coordinates?

• 15:41Speaker 02

  Yeah.

• 15:41Speaker 01

  So I was gonna say, this is, for listeners on the pod, like, very, very relevant to our Adept episode, where they have a very strong opinion that you should just rely on point and click.

• 15:50Speaker 02

  Yeah.

• 15:51Speaker 01

  Instead of using APIs. I even had a section in my write-up calling, saying like, no APIs as a rule. Is that surprising?

• 15:57Speaker 02

  Yeah, I mean, what was there? I'm curious what there are even why.

• 16:00Speaker 01

  It's very simple. The sheer number of things that don't have APIs vastly overwhelms the number of things that do have APIs. So if you're only going to constrain yourself to API at work, you're not going to be generally capable as a human.

• 16:11Speaker 02

  Yeah, yeah. So I listened to that episode. It's a great episode, by the way. I largely agree with that. And actually, on Webberina, only one of the four websites that we cover has APIs. It's GitLab, and the other three do not. But I think we can be creative about that. You know, maybe there's ways to create APIs for sites that don't have them, or figure out ways to build tools that allow you to say, okay, navigate. It's kind of like the documentation idea that I said, like, how do I navigate to the purchases page? And if you just tell it what the purchases page is, then you can create a navigate to purchases page function or something like that. So we're not finished with this yet, but we're thinking about things in that general direction.

• 16:54Speaker 03

  I'm curious, what do you think is missing from the base models themselves? And like, how do you think the benchmark would inform what the model providers do to make better ones?

• 17:03Speaker 02

  Yeah, so this is a great question. So one thing about Web Arena is that we're taking in a textual format. I didn't actually mention this yet, so I can explain it briefly. But we're looking at websites by using something called the Accessibility Tree, which was created for screen readers, for people with vision impairments. And I don't think that's necessarily the best way to see a website. I think multimodal is probably the way to go eventually, because most people see sites by looking at them. I feel like all open source models, and even most of the closed source models, are not terribly good at understanding websites visually or through Accessibility Tree formats. So they make silly mistakes about, like, for example,

• 17:43Speaker 02

  not realizing that they can click on a dropdown menu in the Accessibility Tree format, or not being able to ground to the site when you're looking at it visually. We have another version called Visual Web Arena that follows up and does this visually. Another thing is planning is a really big problem. So I'm very hopeful that the foundation model providers are working on this very hard now in training models that are better as agents. But for example, on Web Arena, it will very often step into a page and then be on the wrong page and just not realize that it needs to go back. So those are some other problems. It also has some really interesting failures of common sense. So my favorite failure is we have a thing in Web Arena that says, "Assign this issue to myself." And it assigns the issue to the username "myself"

• 18:28Speaker 02

  instead of like your own username. So there's lots of like little common sense things there, too.

• 18:34Speaker 01

  Yeah, maybe tokenize identities and then have a special token for "self" versus username to "myself."

• 18:41Speaker 02

  Although I'd kind of hope that my language model would be smart enough to figure that out, too. Like, a human would be.

• 18:45Speaker 01

  I don't trust it.

• 18:48Speaker 01

  Okay, cool. We should probably move to Sotopia. So going from simulating the internet towards simulating multiple agents talking to each other in a social environment.

• 18:55Speaker 02

  Yep.

• 18:56Speaker 01

  What's the motivation there? What's the story there?

• 18:58Speaker 02

  Yeah, so the motivation there is a lot of people are starting to use language models in kind of like social situations, or at least like socially charged situations. Whether they're ready for this yet or not, I'm not sure about, and that kind of motivates this work. For example, should we have a language model that negotiates with somebody about like a price of a...

• 19:26Speaker 01

  There's no should we, we are going to have them.

• 19:29Speaker 02

  Yeah, we are going to have them. Are they ready to do this in a way that is not harmful, basically? And also, are they good at it? You know, are they able to cooperate with somebody and find a solution to a shared problem? So what we did is we came up with like six different types of these like socially relevant situations.

• 19:49Speaker 01

  Yeah, negotiation, exchange, competition, collaboration, accommodation, and persuasion.

• 19:54Speaker 02

  Yeah, exactly. And with each of these, we kind of like semi-automatically created a bunch of tasks. So the example task that I have on the poster is we have two people who are out in the cold. One of them has a blanket and the other doesn't have a blanket and...

• 20:08Speaker 01

  And it's a guy and a girl, very important.

• 20:10Speaker 02

  Yeah. And person one wants to keep their blanket to themselves and person two wants to have the blanket shared. And so they need to negotiate with each other. Each of the agents is given a persona. They have like a personality and some site information about them. They also have a secret that they don't want to reveal to other people. And then based on this, they have a conversation and then we do evaluation along a number of axes, like whether they achieve their goal, whether it's a believable representation of a social situation, whether they violated social rules, whether they broke secrets and other stuff like this. And based on this, we had language models talk to language models and evaluate whether they do a good job of this. And we also had language models talk to humans and see whether they did a good job.

• 20:53Speaker 01

  And yeah, language models eval.

• 20:55Speaker 02

  Yeah, that's what's nice. How do you do the evaluations? Yeah, so we did human evaluation and we did language model-based evaluation. And we tried to measure the overlap between the human evaluation and the language model-based evaluation. And not overlap correlation. So how well do they agree with each other? And the answer is, language models are okay at navigating these situations. They're also okay at evaluating these situations, but they're not perfect at either of these yet.

• 21:22Speaker 01

  Seventy-four percent.

• 21:23Speaker 02

  Yeah, seventy-four percent.

• 21:24Speaker 01

  Which is usually around the rough correlation of LLM's judge for most things.

• 21:29Speaker 02

  Yeah, exactly. And one of the interesting things we also found was like if we have GPT-4, talk with GPT-4, it got a score of about three point three out of seven on our kind of composite scale. If we had GPT-4 talk with humans, it got something closer to four point eight, I think. So when GPT-4 is talking to humans, it's actually more successful at achieving its goals and being believable and stuff. Because the humans are a better conversational partner and can recover from errors and stuff.

• 21:54Speaker 01

  Oh, cool.

• 21:54Speaker 02

  But when humans talk to humans, it's six point one five. So we're, you know.

• 21:59Speaker 03

  What is the human to GPT-4 like?

• 22:02Speaker 02

  Human to GPT-4 is high. It's nearly identical to human and human, like five point nine, five point nine five.

• 22:09Speaker 01

  Your number should be bigger than your chart.

• 22:10Speaker 02

  Yeah, yeah. I will convey this to the students who made the poster. Yeah, I agree. But yeah, so I think this is pretty interesting. This was our first work in this general direction. Again, you know, the ICLIR submission deadline was about six months ago. So like a lot of these papers came out six months. But also, some of the stuff we've done in the follow-up now is we are trying to get better evaluation, train better evaluators that are good at evaluating this kind of like social skills. And we're hoping to beat GPT-4 with respect to that. But more interestingly, we also tried to train models to be better at navigating social situations. So we took a Mistral 7B model and we trained the model through both behavior cloning.

• 22:53Speaker 02

  So we ran a bunch of conversations through GPT-4 and trained Mistral to mimic them. But also through self-reenforcement, where we basically had the Mistral model do a bunch of conversations, have GPT-4 grade them, and then pick only the good ones and train on that. And in doing that, we actually were able to max out our evaluation so that the trained model matched GPT-4. For according to machine judgment, according to model graded judgment. But when we actually did human evaluation on it, it was still far behind. So what this demonstrates is we can like over optimize to model judgments and actually get like very close to kind of our teacher model with respect to model judgments, but

• 23:37Speaker 02

  still humans fall behind. So it demonstrates that we actually have, you know, significant problems in evaluating these models as well. So I think we need to alternate between getting better evaluators, getting better models.

• 23:48Speaker 03

  Is the evaluation done right now via kind of prompting a GPT-4?

• 23:53Speaker 02

  Yeah, at the moment, that's the best thing we have, but we're trying to beat that.

• 23:57Speaker 03

  Yeah.

• 23:58Speaker 02

  Currently, as we speak.

• 24:00Speaker 01

  The last paper that you're presenting today is performance improving code edits. You did the prep on this one.

• 24:05Speaker 03

  Yeah, I mean, first off, would you be able to kind of roughly explain the paper?

• 24:09Speaker 02

  Yeah, so the basic idea is we have lots of programs, of course, written by humans or written by machines and correctness is a major concern, but efficiency is also a major concern. And so we ask the core problem, can we use like large language models to improve the efficiency of programs? And the overall concept of the paper is pretty simple, but the execution is maybe interesting. And so in order to do the execution, we basically took competitive programming problems that had lots of different solutions. And especially we focused on ones where the first one timed out and was too slow. And then a revised version didn't time out and was fast enough to complete. And the reason why this is interesting

• 24:52Speaker 02

  is because we know the implementation is going to be pretty similar because, you know, people working on competitive programming want to do it quickly. So they don't want to like rewrite their whole implementation when they go from that. But, you know, we know one is slower and one is faster. And so then we basically asked language models to try to do a similar optimization and make it faster. So we created this data set. We also created an evaluation harness that makes it possible to measure things fairly because if you just run a program and see how long it takes, what if the system is busy or other things like that? There's a bunch of like mitigating. factors, so we fix that through virtualized CPUs. And then we also have some like better prompting methods and fine tuning methods to try to create

• 25:37Speaker 02

  models that do better on this. And in the end, we actually got quote unquote, superhuman performance on this task. And we came up with models that on average made things faster. The caveat for superhuman performance is the people doing these programming contests only need it fast enough to beat the timer. Yeah. So they're not trying like extremely, they're not trying extremely hard to optimize. So I won't say that, you know, models have like beat humans at program optimization yet, but it's like maybe. Still, it'll save you from really bad performance. Yeah, exactly.

• 26:05Speaker 03

  Yeah, I mean, the surprising thing was it was only using, I believe, the code Lamas and 3.5. I don't think it used four, if I remember correctly.

• 26:14Speaker 02

  Yeah, this work has had a long evolution. And I think part of the reason why we didn't use four was just it's pretty expensive to run experiments. So, but yeah, I think that's the main thing.

• 26:28Speaker 03

  Yeah, there are a few interesting things that I noticed. The really interesting one that I liked was the performance conditional generation. So I feel like I've seen pieces of this in other work. For example, AlphaCode actually like mentions doing this. Yeah, I'm just curious what the motivation for that was.

• 26:44Speaker 02

  Yeah, so the basic idea is what we do is we kind of like prefix the sequence that we're generating with how good the performance is. So it's like, this is a zero out of 10 performance thing. This is a 10 out of 10 performance. And then we fine tune a model to learn like how to generate slow implementations and fast implementations. And then at test time, we always append the fastest tag. This has been used in a number of places, like another example is a method called Quark, which basically tried to do this to generate like not toxic text. So it would say, this is toxic text, this is not toxic text. And then when you generate it, you always append the like not toxic text tag. So it's a pretty widely used technique and it just seems appropriate here because it's also very easy to use.

• 27:29Speaker 02

  You just append a tag, you evaluate, then you append a tag to the beginning.

• 27:33Speaker 03

  Yeah, and I saw you guys had used like 3.5 fine tuning. Like one limitation of kind of using OpenAI fine tuning is you can just kind of do supervised fine tuning. You can't really, you can't do RL. But it feels like, you know, using this method, you might get some of the benefits of learning from negative examples. You kind of would get what from RL.

• 27:52Speaker 02

  Yeah, exactly.

• 27:53Speaker 01

  By prefixing tags.

• 27:54Speaker 03

  Yeah, right, so because you can have kind of a low-scoring thing prefix with a bad example following it, and then a high-scoring thing followed by a good example. And ideally, the model learns like the difference with that data.

• 28:08Speaker 02

  In the Quark paper, at least, they show that this works better than just training on the high-quality examples.

• 28:13Speaker 01

  Yeah, excellent. What's the appetite for performance improvement code edits at Cursor?

• 28:18Speaker 03

  Yeah.

• 28:18Speaker 01

  In Cursor?

• 28:20Speaker 03

  I think like in practice, it's really tricky when looking at arbitrary code. There's the problem of actually isolating the actual performance of some piece of code that you care about. First off, you don't have like this nice sandbox environment. People are almost always running it on their laptops or some kind of remote SSH machine. Then like actually isolating that piece of code. It's possible, right? You could kind of just add kind of timers around it. But we don't have anything like optimized for that. It's mainly like if the user wants to ask the model to improve it, they can. They can add in relevant information of how long it took. And it should work reasonably well, but definitely not like its own thing.

• 28:56Speaker 02

  Another thing is our paper requires having tests. And in performance improving code edits, a huge, huge bottleneck is having good test coverage. Because if you don't have good test coverage, then it will generate a return statement and just return from the function with the correct answer to the test, which is very fast, but only works in one case. So I think for real-world code, that's a lot harder because generating very comprehensive tests is hard if you have like data structures and stuff like that. So it's not, definitely not trivial.

• 29:29Speaker 01

  Awesome. I wanted to move on next to, now that you're done with the spring semester, I guess officially after you present this poster. You were mentioning that you're going to spend a lot of time on OpenDev. So maybe, and obviously you're interested as well. What were your reactions to Dev? And then maybe you'll tell the, sorry, OpenDev first, but I wanted to start with Dev. Just both of you, whoever wants to.

• 29:49Speaker 02

  Yeah, I think it was really exciting. Like, the demo was great. And I've been working on code generation for a long time. Like, I think, actually, since 2014. And, you know, the big moment where I felt it had made it first was Copilot coming out. And it's like, yeah, this is great, you know? And then I kind of slightly lost interest in doing this because I felt the limits of code completion. And like, Copilot does pretty well. Sure, we can improve it a little bit. And then.

• 30:18Speaker 01

  You did.

• 30:20Speaker 02

  Yeah, well, so, you know, maybe Cursor was right about how much more you could improve it, you know, honestly. But so I personally kind of lost interest a little bit. But then I was working on web agents. And then I saw the demo from Devin and I'm like, oh, this is really cool. Like, I'm interested in agents. I've been interested in code generation for a long time. You know, this seems like a good sandbox to be working in. And I had known about Sweetbench and the stuff that had come out. And I was like, yeah, let's do this. Let's, you know, work on this problem. Because it's another benchmark like Web Arena where it's like, our scores right now are low, but there's a lot we can do to improve them. So it's kind of exciting there, too.

• 30:58Speaker 03

  Yeah, no, I thought it was a really good demo. And I think it'll be like, pretty useful for like, a lot of the bottom, whatever percent of PRs. I guess, especially like, because I'm working on the stuff that I'm working on, I think I'm a little bit more bearish than other people on like, agents working kind of immediately or like, somewhat soon. I think there are a lot of really hard problems. And human judgment is pretty paramount. Like, I think on the margin, we are going to shift things a little bit more kind of agentic in the agentic direction, more things happening in the background. But I suspect the human will be needed for a while, rather than kind of going from issue to pull request.

• 31:32Speaker 01

  Which, I mean, I think that's kind of Sweet Agent's direction. But I think one of the things that Devin, I think, nailed was the async interaction with the... It could be executing its plan, but you could sort of intervene while it's doing it. And that felt very much more like, I guess, you know, level three or level four self-driving, rather than full level five. And, you know, that seems to make more sense.

• 31:55Speaker 02

  Yeah, and I think like one of the interesting things when we first came out with OpenDevin, you know, Devin still was not open for everybody. I can tell a little bit about the story, but basically we saw the Devin demo come out and Junyoung, one of the people from the Quen team, building a language model at Alibaba, basically. I think he's the lead on Quen, right? Yeah, he's one of the leads for sure. And he basically said, yeah, this is really exciting. Let's make a project about this. He made a repo with a readme and the readme got 1,000 stars on GitHub.

• 32:24Speaker 02

  So then shortly after that, I think probably that evening or maybe the next evening, I was like, yeah, if we have something to hack on here, like the open source community is so excited about this, that we'll be able to do something interesting. So I basically came up with a really, I'm not a React developer, but I came up with a really like janky React, I wouldn't say clone of the Devin interface, but you know, something similar to the Devin interface. It was completely non-functional. It had no chat functionality. It kind of looked reasonable. And then after that, I pushed that and it's like, yeah, let's make this actually work. And then a bunch of people came together and did that. So for the first four weeks or so, we didn't have anything, you know, that actually worked at all. But while we were doing that. This is when I livestreamed and tried it out.

• 33:07Speaker 02

  I was like, I really respect you and felt a little bit bad that I subjected you to that experience. No, I didn't. I was like, oh, this is done.

• 33:17Speaker 02

  But like the interesting thing was then we had a ton of people who are not developers coming to us also. Yeah. And so I think, I totally agree that for, you know, really big software engineering projects like OpenDevin, like we're trying to use OpenDevin to solve issues on OpenDevin. And it's a complex enough software project that it's actually pretty tricky to do. Like the model needs to figure out how to set up the software repository in the first place, and that's a pretty big lift. But I think the possibility of doing things at slightly smaller level, like setting up simple web apps and stuff like that for people who are not professional developers is another thing that even immediately, these sorts of agents might be able to start making a dent in. Yeah, that makes a lot of sense to me.

• 34:00Speaker 01

  As someone who's like slightly more positive on Devin, yeah, I'm also, you know, I don't think it's, it's gonna threaten our jobs anytime soon, but I'm pretty positive on it. Like I think it is very good for greenfields and then like. Moderate for Brownfield and then obviously depending on the size of the job that you're asking it to do. Like, yeah, there are many sub bottom percentile PRs that I have to do anyway.

• 34:21Speaker 02

  Yeah.

• 34:22Speaker 01

  If I could just throw it to Devin, even if it takes like eight hours to do it, that's probably one hour that I don't have to spend thinking about it at all, which is cool.

• 34:30Speaker 02

  One thing I wonder about is the UX, because it does kind of shift into you're managing a bunch of, let's say, like

• 34:36Speaker 01

  Junior developers, yeah.

• 34:37Speaker 02

  You're kind of doing code review then all day. How does it feel?

• 34:41Speaker 01

  It feels fine. Literally, it feels like I'm an engineering manager. I'm technical. And I can, I can see what my, my coders are doing and check in on them. If they're going off base, I can just tell them they're going off base and they'll replan. It's exactly what I do with engineers anyway.

• 34:56Speaker 02

  Yeah, no, that makes sense.

• 34:57Speaker 01

  And so in creating OpenDev, and you were talking about planning earlier, I feel like Devin's planning, Devin made a big fuss about their like breakthrough being, or secret sauce being planning. But I really don't think it's secret sauce. Like it's just, they generated plan and you try to execute it and the plans change over time. Did you find that hard? Like what was the, you know, any insights that you made over Devin?

• 35:18Speaker 02

  Yeah, so at the moment, we started out with implementing planning and I do think it's important. So right now, our best agent, which is doing reasonably well on like the Sui Bench benchmark, the same thing that Devin tried out on, actually isn't really doing any explicit planning at all. So nonetheless, we're able to get 21% on the Sui Bench Lite version. We haven't run the full Sui Bench version just because setting up Sui Bench takes a while and it costs $6,000 every time you run it with GPT-4. So it's a little bit heavy to run evaluations on it, but I'll be very interested to see, does their supposedly really good agent with planning stack up to something without planning?

• 35:58Speaker 02

  But just has like a good toolbox for, you know, searching code and for modifying code in a, you know, efficient way and stuff like that.

• 36:05Speaker 01

  Yeah, I think this is public, but they use Morph, which is Jesse Han's thing. I don't know if you know him. So they, they use a good code indexer or searcher that pages things into context whenever you need.

• 36:15Speaker 02

  Yeah.

• 36:16Speaker 01

  Seems like the magic trick. I don't know.

• 36:17Speaker 02

  We're thinking of trying out Morph, or we're actually actively trying out Morph as well. Right now, our code search is based on the code search that was used in Sui Agent, which is another agent by the people who created Sui Bench. And it's literally like a find in grep.

• 36:36Speaker 01

  Okay.

• 36:36Speaker 02

  It seems to work good, but it won't work as well as, you know, like an actual code search engine, semantic search.

• 36:42Speaker 01

  I don't know. Have you talked about the search that you use?

• 36:45Speaker 03

  We use a mix of things. Like, the main meat of it is kind of retrievable with embedding. So a kind of standard approach there. But then we use like kind of re-rankers in the mix, occasionally use LSP information. I think there's like a much stricter requirement for the agent stuff, for getting exactly the right context that we don't have. So I'm not super familiar with what exactly Morph Labs has, but yeah.

• 37:07Speaker 01

  His emphasis is on speed and scale, but I don't really know how specifically indexing or retrieving.

• 37:13Speaker 03

  And this is not at all like a knock on Sui Bench, because I think it's a fantastic benchmark and it's a great way of kind of measuring progress. But I do wonder, I had posted about this, like, how much of the performance is also that the models kind of do know those code bases? Because they're all public code bases.

• 37:28Speaker 01

  Yeah, so you asserted that it's already leaked for the online models.

• 37:32Speaker 03

  Like, in some ways. Like, I kind of tested this with one of the first problems that I saw in Sui Agent. And Claude Opus basically knew the correct file to edit just based on the PR, the name of the pull request. So I suspect it's like somewhere in the pre-training data. I don't know like how much of an effect that actually has. Like, if you're like getting better and better at it, I still, I think that translates to better performance on private repos. But I don't think like an X percent will also be, on a public repo will be like the same X percent on a private repo.

• 38:01Speaker 02

  Yeah. So this is a great point. I loved your tweet about that, actually. And I retweeted it. But it can also explain a little bit about our like vision for OpenDevon. So it started out as basically a Devon clone, but I feel like we've moved a little bit beyond this because I think the open nature, we have a hub where people can add agents. We have a pluggable thing where you can use any LM in it. So you can combine any LM with any agent. Is it Light LLM? It's Light LLM. I'm a big Light LLM fan. It makes everything very easy. And then the final thing is, we also want to have pluggable evals. So right now, we've only implemented like Sweetbench. But there's a bunch of good code benchmarks. We're also planning to add Web Arena and through something called BrowserGym,

• 38:45Speaker 02

  which was created by ServiceNow that has these three web navigation benchmarks. Because in order to be a good software engineer, you also need to be able to gather like information on the web and stuff like that. So we're going to add that. And I think there are ways to basically create benchmarks that are not leaked, using the same method as SweetAgent. And there was recently a paper out of Berkeley called R2E that converts repositories into like evaluation environments for code generation agents. And we're talking with the people who created that to incorporate that into like our evaluation harness.

• 39:18Speaker 03

  Yeah, I'm very excited about that. I talked to them too. I think at least one or two of them. They've been also the people behind Live Code Bench.

• 39:25Speaker 02

  Yeah, yeah, yeah.

• 39:25Speaker 03

  Which I think is fantastic because it's a great way of seeing if, and for people who don't know, Live Code Bench is basically, I think it's like a bunch of leaked code problems and you can kind of slide the cutoff date forward and backwards and see how different models perform.

• 39:38Speaker 02

  They go way down.

• 39:40Speaker 03

  It's really, it's really great. One thing that I'd be really interested in seeing is, like I think they're like now good benchmarks for overall agents working well, but like good benchmarks for capturing like all the things that a model needs to do well to be a good agent. Because you kind of need to build like the good overall system and maybe the system works really well for some models, better for some models than others. But I wonder if there's like a good kind of benchmark you can do that tests like independently each part that's needed to be a good agent.

• 40:09Speaker 02

  So this is a great question. And for web agents, we recently released something called Visual Web Bench, with the idea being that it tests about eight different like capabilities that we think a model should have. Like, is it able to do OCR on the page? Is it able to ground to web elements? Is it able to predict the effect of clicking on a button or something like this? I think we currently lack something like that for coding agents. Mistakes that we see our agents making are really silly. Like it get clones a repo and then it doesn't notice CD into the thing. So it tries to get clone the repo again.

• 40:48Speaker 02

  And this is GPT four, so it's, you know, the most capable LLM model. And so I think there's a bunch of little things where it's like, we could categorize these and just make sure it checks all those boxes and it would just become more capable. But that being said, you know, there will probably be other things it fails on. So if we overfit to that benchmark, that tests all the skills and that would also be a problem. But I still think having one would be better.

• 41:07Speaker 03

  Yeah, like there are a few general things that you're kind of surprised by like how poorly the model does. Like one is kind of applying code edits. I had talked to some of the Sweet Ben, Sweet Agent people and like also just like looking at their demo. There's this great example of the model knows what like roughly knows what to do. It has like a plan for it, then it's trying to apply the edit and like seven times it incorrectly indents it. It gets that feedback, keeps doing it again and again and again. But like, yeah, code edits feel like one big part kind of of the pie that aren't like super well tested at the moment.

• 41:38Speaker 01

  Yeah, anything else about the future of coding agents? Like where do we go from here? I guess you already talked a little bit about the future of OpenDev.

• 41:44Speaker 02

  I feel like we're like right at the beginning of a very rapid delta with respect to the performance of how well these are going to go. And I think now we have all the ingredients for academia and open research to iterate on this. We have a good benchmark like Sweet, Sweet Ben's, you know, maybe there are some issues with it, but I think it's fine for now to iterate on. OpenDev, we've set up, you know, an environment where people can, you know, put in agents and very quickly iterate on them. So I think we're just gonna see a bunch of people jump on this and improve rapidly. And then I think it's gonna plateau a bit when we get to the really hard things where our like current language model backends are not going to be good enough to handle them.

• 42:28Speaker 01

  We'll have GPT five by then.

• 42:29Speaker 02

  Yeah, but like, I think that's probably gonna happen. And I think because all the open model creators know that GPT five is going to have that, all the open model creators are probably also thinking about it too.

• 42:40Speaker 01

  Yeah, like, you know, Lama three, four.

• 42:42Speaker 02

  Like Lama three, yeah, exactly. So I think that will give us another bump with respect to that. And we'll see how far that takes us. I don't think it will take us to resolving every GitHub issue automatically, but I think it'll be pretty exciting over the next, like by the end of 2024.

• 42:56Speaker 03

  Yeah, again, I think it'll be like really useful for like some bottom percent of GitHub issues and that'll go up. The thing that we would like to build is kind of agentic things that happen as you're coding. Like, the ideas we have in mind are like, as you're coding, you can spawn off kind of pretty meaty units of work. Right? Like, as an example, let's say you need to implement some random helper function or some utility function in order to like get some value out of it. Let's say the contents of some file in particular, some particular way. You should just be able to kind of write that function out, file contents equals whatever. And then in the background, that thing gets implemented for you. What we're going to see is we're going to see the ability of like basically scaling inference time compute in some way.

• 43:39Speaker 03

  And this could be either smarter models, it could be kind of using models with chains and looping. And when you scale up inference time compute, you can't really use the level one kind of systems that are built in with Cursor right now, which is humans supervising the outputs of these models, either with kind of next edit prediction slash autocomplete, streaming in kind of diffs or chat. And so something needs to happen in the background. But the goal of what you want to do is it happens in the background in a way that's like very much preserving human flow and letting the human be completely in the driver's seat and kind of dictating exactly what happens. So it's kind of working completely in service of what you're building towards.

• 44:16Speaker 01

  I think this is the original more vision and it'll be exciting to see when it happens. Do you think most of that will be running locally or you don't really have a difference?

• 44:24Speaker 03

  Locally meaning?

• 44:25Speaker 01

  The models, infering on models locally for cost reasons. Because you're going to be firing off a lot of work.

• 44:30Speaker 03

  It's going to have to happen with like the most capable models, meaning it'll happen, not locally.

• 44:37Speaker 01

  Just a side note, I was kind of talking, just talking a lot about like open models and all that. Do we have strong opinions about code specific models being best for code? Or do we think general models are just best anyway, are the best code models? So maybe to phrase it, there's no code GPT-4, there's just GPT-4. And GPT-4 is the most capable code model.

• 44:58Speaker 03

  Yeah, I mean, here's one question I'll kind of pose in response. It seems like people say that training in code improves performance and everything else.

• 45:09Speaker 01

  There was one paper about that here, yeah.

• 45:11Speaker 03

  Oh, really? What was the paper?

• 45:13Speaker 01

  It's like, how much does code improve performance? And the guy didn't show up for his poster session, which is very annoying.

• 45:19Speaker 03

  Does it? Did it show that it improves performance? I feel like, I don't know, if you look at open papers, I think most of the time, it's kind of showing if you've run out of data, then adding in code will improve performance, right? Which makes sense, right? Unlike reasoning in particular is the thing that people kind of speculate. But it does feel like OpenAI is the company that popularized this notion. And that was kind of like back in the day when they unified the models. Because it used to be like Codex was separate from GPT-3, and then they kind of had this unified Code Da Vinci 2.

• 45:50Speaker 02

  So I actually wrote the first paper on that. And when we-

• 45:55Speaker 01

  Sorry, what was this paper called, if you remember?

• 45:57Speaker 02

  "Large Language Models of Code Are Few Shot Common Sense Reasoners," I think. But when we wrote that paper, we didn't know what Code Da Vinci 002 was and what Text Da Vinci 002 was. And we thought Code Da Vinci 002 was a fine tune on top of Text Da Vinci 002, but it was actually the opposite. Text Da Vinci 002 is a fine tune on Code Da Vinci 002. But nonetheless, Code Da Vinci 002 had better performance on some like reasoning benchmarks that we measured. And the funny thing is, actually, Text Da Vinci 002 was trained on more data, but Code Da Vinci 002 was still better at some structured reasoning stuff, which I would really like to prove, but we haven't been able to do it yet,

• 46:38Speaker 02

  is that code is more structured and so it also has more repetition. So you need to attend back to the previous context more when you're doing code. And because of that, it's better at capturing things that are very structured in the output. And that includes things like reasoning. So I'm a pretty strong believer that like there is something special about code, but there could also be something equally special about text, if you use the right variety of text, if you use text with lots of repetition or other stuff like that. So I don't think it's like code is magical. I think it's some properties of code are good for reasoning.

• 47:10Speaker 01

  Legal text.

• 47:12Speaker 03

  That would be one example. Like, my suspicion is there's just like classes of text slash code that help for reasoning. And like, there's a bunch of not fantastic texts for reasoning that will exist in pre-training data sets. Like, maybe the very top you have like archive, right, or textbooks. And then like just under that you have code. So i- it ranks higher than maybe most text data that's used in pre-training models. But it's not like better than like actually the thing you want. Yeah, I agree.

• 47:37Speaker 01

  Awesome. I'm going to broaden out to more general free form topics. Something that we prepped was just the changing nature of like, I guess, industry and academia. I don't know if you guys have opinions on that. I guess you're representatives of both.

• 47:52Speaker 03

  Yeah, I'd love to hear your thoughts.

• 47:54Speaker 02

  Yeah, so the changing role of academia is really interesting. Because I lived through several areas where it was like academia is probably leading research with respect to, you know, deep learning and everything, which was maybe 2010 to 2013 or something like that. And then there was the like sequence sequence models paper from Google, which was 2014, which was this at the time like huge four-layer LSTN that nobody could train. And so then we were starting to feel the compute crunch, but there were still lots of like modeling innovation. I created a neural network toolkit called Dynet, which was kind of a precursor to PyTorch. And a lot of stuff was happening there.

• 48:38Speaker 02

  And then after BERT, you know, it's like, oh, we're scaling up. We're moving beyond like the compute that academia can use to train these base models. And then I think the really big thing was like the GPT models, right? And after ChatGPT came out, we actually had an emergency workshop at CMU, which was a group therapy session to say, what should we do? What should we do? And I think for a short amount of time, a lot of people were worried, like, what could we be doing in the face of this? And then I think a few things changed. I think number one, the evaluation stuff I talked about. It's like, we realized that actually there's a lot of stuff that GPT three and ChatGPT cannot do yet. And, you know, more complex reasoning, more multi-step stuff. And another thing is all the open models started coming out, which made it a lot easier for us to do fine tuning.

• 49:23Speaker 02

  A lot of the open source frameworks came out that made it easier to do these sorts of run large models on hardware that we have access to. So I feel.

• 49:33Speaker 01

  You're talking about like Llama CDP?

• 49:35Speaker 02

  No, I'm talking more about like the training stuff like DeepSpeed, Llama Factory, you know, Axolotl.

• 49:40Speaker 01

  Tensar T.

• 49:40Speaker 02

  Yeah, any of the things that we can use for training and that makes it so you have a machine that costs, you know, a hundred thousand dollars, which is a lot of money, but it's very much within, you know, an academic budget and you can actually do training runs, do interesting things that free you up. And then at the same time, right now, you know, every university is trying to build a GPU cluster with it.

• 50:02Speaker 03

  Yeah.

• 50:03Speaker 02

  Or get access to it, including us, including, you know, everybody else.

• 50:06Speaker 01

  I imagine CMU would be ahead because you already have so many other needs.

• 50:10Speaker 02

  Yeah, so we do have a good cluster, but like the kind of hardware that you need for training large language models is kind of specific. You also need a system for allocating, you know, like, okay, this is the most important thing to be doing right now. We're going to give a lot of compute to that, which is not something that traditionally universities are very used to doing. They're used to being very chaotic with lots of ideas, but I think we need to focus on some.

• 50:34Speaker 03

  Yeah. MIT had this like really great big cluster, but it was V100s. So two generations too old. Soon three.

• 50:42Speaker 01

  There's these like national supercomputer labs that you can apply for grants for.

• 50:46Speaker 02

  The funny thing is many of these don't have the hardware that we need. They have V100s or they have A100 40GB things. They have A180GB, but they only have four of them. It's like, it's kind of interesting how little there is available.

• 51:00Speaker 01

  Well, you can talk to like Luther, which has its share of grants. Luther is basically like a compute grants collector right now.

• 51:07Speaker 02

  They're pretty, they're pretty amazing at what they've been able to achieve with that, but they're also mostly not using US national clusters. I think they use some overseas and other stuff like that.

• 51:19Speaker 03

  I wouldn't be surprised if Andromeda would give away compute for research like that, Dan. I'd heard of them doing something like that before.

• 51:26Speaker 01

  Well, you would know because you're in AI grants.

• 51:28Speaker 03

  Yeah.

• 51:28Speaker 01

  Like, yeah, my first response would be, aren't they already maxed out by existing users?

• 51:32Speaker 03

  Yeah, I think I had heard when there is like a period where there were like not too many people using it or like a bunch of people canceled, they gave it away on some grants. So maybe if there are like bubbles where people aren't using it.

• 51:43Speaker 01

  Yeah.

• 51:44Speaker 03

  Yeah.

• 51:44Speaker 01

  The other two sources I'll name are Crusoe Energy, which is using, I don't know if you're familiar with them.

• 51:49Speaker 03

  Yeah.

• 51:51Speaker 01

  Correct me if I'm wrong, they put GPUs on top of like oil rigs?

• 51:54Speaker 03

  I'm not familiar with the details.

• 51:55Speaker 01

  It's slightly sketchy, but like, whatever, it's clean.

• 51:59Speaker 03

  Yeah. Okay.

• 52:01Speaker 01

  Okay, and then the other one is Strong Compute, which is doing one of those distributed cluster things. So it's like together, but with less funding and from Australia.

• 52:09Speaker 02

  We are working with some providers like RunPod and NetMind. So I think there's definitely some resources out there, but everybody is looking for them. And really, I think the solution is we're gonna need to scale up the compute that we have available to academia in the US.

• 52:24Speaker 01

  Yeah.

• 52:24Speaker 02

  Well, in CMU and just in general, but I think we realize the importance of this. I hope the US government realizes the importance of this and invests lots of money in it. Because that's actually the best solution, but they move a little bit slower than a lot of people move, so we'll see.

• 52:39Speaker 03

  Yeah, I'm curious, if you kind of look across all of academia, what work have you been either like most impressed by? Or do you think best represents the kind of work you think academia should be doing in the last like year or two?

• 52:50Speaker 02

  Yeah, so actually, another comment about the like academia versus industry thing. I really do wish that the people doing kind of frontier research on language models in industry acknowledged academic work a bit more. Because I do think like a lot of the things that people are doing in academia end up in industry, but just don't get acknowledged. And I think that has to do with the fact that industry is super secretive right now about anything they do in large language model space. And so previously, it would be like industry is publishing papers and we could point to the fact that, hey, Google uses our stuff, OpenAI uses our stuff, or things like that. But now, there's a lot less of that, which makes it seem like we're shouting into a black hole, when actually, maybe we aren't so much.

• 53:33Speaker 02

  And the best example of this recently was like the Matraschka embeddings thing from Udub, where OpenAI used it and renamed it something else.

• 53:40Speaker 01

  I mean, that one seemed like just an oversight, rather than intentional exclusion, because like, they left enough hints that it was that.

• 53:47Speaker 02

  Yeah, maybe, and they did better with Sora, for example, where they actually cited all the works that like inspired them.

• 53:53Speaker 01

  The diffusion transformers.

• 53:54Speaker 02

  Yeah, and things like this. But they have every right to be secret when they're competing, you know, their industry, they're competing against each other. Like, I think it makes sense, but it's also a little bit disincentivizing for grad students, for example, because they can't point to like their success stories that they had before. I don't know if there's any solution to that, but I thought I'd mention it, just in case anybody who has influence would be listening, so.

• 54:15Speaker 01

  Yeah, it's just a, you know, corporate responsibility thing to, it's the right thing to do.

• 54:20Speaker 02

  Yeah.

• 54:20Speaker 01

  But like, you know, for me, the interplay between industry and research, I think I feel it the most with just like your grad student pipeline, or maybe like the undergrads that you're teaching, like what their interests are.

• 54:32Speaker 02

  Yeah.

• 54:32Speaker 01

  I'm sure the class composition has changed a lot for the NLP class.

• 54:35Speaker 02

  Yep, yep, yeah. I realize I skipped your examples of good papers from academia question, actually, and I can go back to that. But examples of good papers are both on like the evaluation side and on the modeling side. I think on the modeling side, I have always preferred papers that are simple but work. And I think I'm a little bit weird with this respect in academia sometimes, because I feel like when I see papers get reviewed, people are like, oh, this paper is not novel enough. But I'm like, this is a great paper. It's like, it made a small tweak to this method, but it works three percentage points better. Like, they'll change one line of code and suddenly everything will work.

• 55:19Speaker 02

  But that insight was not there before, which is why it didn't exist. So I really do like those sorts of things. I think DPO is a pretty good example. It's a lot simpler than they make out in the math.

• 55:32Speaker 01

  Oh, God.

• 55:32Speaker 02

  It's a lot simpler than they make out in the math, but I think that's a great thing, right? It's like a simple tweak that worked really well and people use it a lot. I think those are the kinds of things that are really valuable. I also think benchmarking, which isn't simple and takes a lot of work, is something valuable, which is why I'm spending time on that. Also, contributions to open source, because I feel like there's a small number of companies that are very committed to open source, like Hugging Face is an obvious example. But they don't have enough firepower to compete with the bigger, you know,

• 56:04Speaker 01

  Yeah.

• 56:04Speaker 02

  companies who are working on these sorts of things. And I believe that open source, good open source alternative should exist, and academia could help with this. The problem is we're very disorganized. So if we solve the problem of organization and, you know, focus and getting everything together, then that could help. And I mean, like Hugging Face is one example of a company that's doing that. I also hope that like efforts like OpenDev or MergeKit for model merging or other things that pull together a whole bunch of different things under one roof could help out with that, too.

• 56:35Speaker 01

  Yeah, I'm trying to feature those things like I have a merge kit talk in my conference. You know, those kinds of projects will never get featured at ICLER.

• 56:43Speaker 02

  Right.

• 56:43Speaker 01

  So then I'm trying to create a venue for engineering rather than just research. But obviously, there's overlaps between them.

• 56:49Speaker 02

  Yeah.

• 56:49Speaker 01

  He's speaking as well. Do you know what you're gonna speak about?

• 56:53Speaker 02

  Not yet.

• 56:55Speaker 01

  Yeah, we can broaden out to just the syllabus and student interest before and after. Like you said, you had to revamp the NLP syllabus. We can talk about that.

• 57:06Speaker 02

  Yep.

• 57:06Speaker 01

  We can talk about how to pick promising areas of work, which you already somewhat covered.

• 57:10Speaker 02

  Syllabus before and after, I increased stuff on distillation and synthetic data for sure. I added a thing that was like a tour of large language models. So it was covering all the different large language models and their similarities and differences. And stuff like this, because even I didn't know enough about the differences between the models.

• 57:33Speaker 03

  What are the surprising differences?

• 57:35Speaker 02

  I don't know if this was necessarily surprising to me, but it might be surprising to some people. But like how similar the architectures are for like, everybody is using the Llama architecture. And it's not because architecture engineering is not important, but it's because we finished architecture engineering and now we have a really good architecture that works. And we're at least a local, you know, optimum for that, which is everybody uses rope. Everybody uses Swigloo. Everybody uses all these other small tricks. And there's this really nice figure written in the Mamba paper. So Mamba is kind of like a linear architecture. It's a great paper, but there's a figure that compares the original transformer to the Llama transformer with respect to how well it scales.

• 58:17Speaker 03

  Yeah.

• 58:18Speaker 02

  And the Llama transformer just scales like way, way better than the original transformer. So, architecture is important, but we're kind of done with that and everybody is making no more than so.

• 58:28Speaker 03

  But there's like, I don't know, I feel like my belief here is there still exists a bunch of tricks, like there's the MOE trick, right? Like that'll get you like a slightly better.

• 58:37Speaker 02

  That's a big one.

• 58:37Speaker 03

  It's a big one. Maybe there are small ones. I do wonder how many of these are left and how many of these also maybe only come into play when you're at larger scales. There's a great recent paper at Meta where they trained on the next few tokens, right? So you're not just predicting the next token, you're predicting the next four. It didn't show better performance at small scales, but I think past the 13 billion parameter scale, it actually showed better performance. What? Yeah, so this is like another concerning thing for academia perhaps. You may not know if your method actually works. Emergence in architectures. You're dealing with like large enough models, training enough data.

• 59:11Speaker 02

  Yeah, I think that that's a major reason why we do need to scale up the resources that we have for training models. And I think there is a lot of progress on that right now. Like, I think that a lot of places are working on that. And then the other thing that I wanted to mention is, yeah, because architectures are so similar, the data and the training methods are the big difference there. And that's where everything is actually really, really influential. Like, what data do you train on? How well do you clean and deduplicate your data and stuff like that? And that's not something I've really talked about at all before, when I taught previously one year ago. So a lot more focus on data. I definitely teach architectures, but there's a lot less focus on architecture engineering.

• 59:52Speaker 02

  And it's more of an explanation about like why the architectures we currently use are the ones that work. But we did have Albert Gu talk about Mamba because he's at CMU too.

• 1:00:01Speaker 03

  I was gonna ask, what are your thoughts on like this new wave? I feel like there's a bunch of like alternative architectures, like mainly Mamba, then I think RWKB. Those that I mean too. Yeah.

• 1:00:11Speaker 02

  I think we don't know enough about them. I definitely would like to focus some percentage of our effort on like understanding them better. Because this is a perennial problem, which is when you try to do something really, really different, there's so much catching up to do with respect to the highly engineered thing that we have before. So neural machine translation, for example, it took, you know, a year or two to beat statistical machine translation or phrase-based machine translation, which is what we had before. Just because there were like 10 years of engineering that had gone into phrase-based machine translation to make it really work really well. And I feel like we're kind of in that thing for all of these like linear architectures like Mamba and RWKB. I want to see them continue to be pushed,

• 1:00:53Speaker 02

  but I do think they have some fundamental limitations, like for example, recalling information. So we see the like hybrid architectures with seven Mamba layers and one transformer layer. Recall and stuff like that. So it'd be interesting to see if they're.

• 1:01:05Speaker 03

  That's Jamba, right?

• 1:01:06Speaker 02

  Yeah, Jamba, yeah.

• 1:01:08Speaker 03

  Yeah.

• 1:01:08Speaker 01

  So you're optimistic on the mixing?

• 1:01:10Speaker 03

  I think it's one way to solve the problem of like poor recall and linear architectures, but there might be other ways.

• 1:01:16Speaker 01

  Yeah, I would say pure Mamba and pure R D W K V both have the R N N problem.

• 1:01:20Speaker 03

  Right.

• 1:01:21Speaker 01

  Just forgetting.

• 1:01:21Speaker 03

  Yep.

• 1:01:22Speaker 01

  It seems like you have to mix them and yeah.

• 1:01:24Speaker 03

  Yeah, I mean, the mixing, you like lose like the niceness of you're getting a factor of eight, but it's not getting like, if you're scaling up to a million, like 10 million tokens, like it just won't scale, right? Because you're still, you're only diluting by a factor of eight while you still have that like quadratic attention bottleneck.

• 1:01:40Speaker 02

  But you can do strides and stuff like this. So like, I think there's a lot of room for improvement here, which is why I'm like kind of excited about that direction. That's a direction I'm most excited about with respect to architecture engineering.

• 1:01:52Speaker 03

  Interesting. One direction I wish would work, but it feels like no one's made it work. I don't know if it will work, is like some kind of retrieval baked into the model. Like, so Retro is kind of like the original paper.

• 1:02:03Speaker 01

  Well, you know, Dali Kila from Contextual is working on.

• 1:02:06Speaker 03

  Oh, yeah. So Contextual, I think, is like working on things related to this. But I don't know, it does feel like if you really want to scale to 10, 100 million tokens, you can't like store that in some compressed state.

• 1:02:16Speaker 01

  Do something fundamentally different and put it in context.

• 1:02:18Speaker 03

  Like, it's like you need all that information, right? You need all the information to be present, but you need some kind of like sublinear per token generation.

• 1:02:25Speaker 02

  So we have a paper called Unlimiformer, which was at NERIPs last year, that does retrieval-based attention. It encodes all of the previous context in FICE retrieval index. I like that general direction. It worked really well with kind of more traditional transformer models. We used it for T5. But one difficulty is actually rope makes it very difficult because you need to handle relative positional encodings appropriately and stuff like this. So, yeah, I could go into details here, but we don't have a lot of time. But I think there definitely are some things moving in that direction. I think that's another thing that could be interesting. Take our existing architectures and somehow make them like efficient through approximations.

• 1:03:07Speaker 03

  I do think like this kind of KNN operator is like pretty interesting.

• 1:03:11Speaker 01

  Can the Commonwealth Arnold Network or something else?

• 1:03:14Speaker 03

  Oh, sorry, just like K nearest neighbors. Like, just being able to like do like a K nearest neighbors operator. Because like if you're trying to do attention over like all these tokens, right? You're now taking like kind of the average of all, like, I don't know, you have all these keys and values and you're like averaging like 10 million of them, like, it feels. Worse?

• 1:03:32Speaker 02

  You only need the top few.

• 1:03:35Speaker 01

  Interesting. Rag on KB scores. We're basically done. I will leave it to you for any plugs that you want to do? Any calls to action?

• 1:03:43Speaker 02

  Yeah, I guess I'm really excited about open source things in general. So, you know, any-

• 1:03:49Speaker 01

  Come contribute on OpenDev.

• 1:03:50Speaker 02

  Come contribute on OpenDev. Also, you know, use OpenDev to test your agents, add new agents to make it work well on particular tasks and stuff like that. That's really exciting. And also, just in general, like, I really love new developments in open source AI. So even if it's not in OpenDev, I'd like people to continue pushing on that. And I appreciate it when people do that.

• 1:04:14Speaker 03

  No, open source AI has been fantastic for, I mean, it's fantastic for startups as well. Like, it's been super helpful for us.

• 1:04:21Speaker 01

  Speaking of Quen, do you guys use Quen? Like, what's the relationship between you and-

• 1:04:25Speaker 02

  Oh, yeah. So we have a roadmap for OpenDev. And the initial roadmap for OpenDev was by the end of May, we wanted the best agent on Sweetbench. And we did that by the end of April. And we did that May 5th. So we were a little bit late, but-

• 1:04:39Speaker 01

  Close enough.

• 1:04:41Speaker 02

  And then our-

• 1:04:42Speaker 01

  It was 20 days ago.

• 1:04:44Speaker 02

  And then our May roadmap is to have a really good open agent. And since we have people on the Quen team, you know, working with us, I think building something on Quen would make sense. But, you know, Llama 3 is also good, so we'll see.

• 1:04:57Speaker 01

  Plugs, calls to actions?

• 1:04:58Speaker 03

  Yeah, I mean, we're hiring for cursor researchers, engineers, ML engineers. I think we're working on like very interesting stuff in code generation, kind of on the frontier of what is possible for kind of inflow coding and assistive coding. So I think some really interesting stuff to do.

• 1:05:14Speaker 01

  Always hiring. I love the hustle. Yeah. My plug is AI Engineer Conference that I'm spending all my waking hours working on right now. That's it. Well, thank you very much for your time.

• 1:05:24Speaker 02

  Yeah, thanks a lot. It was great.

• 1:05:26Speaker 03

  Yeah, it was really fun.

• 1:05:28Speaker 00

  That was Section A of our ICLR Reasoning in Agents coverage. The next section, Section B, covers related discussions of benchmarks. We start with the hottest new benchmark that has emerged this year, SUI Bench, which broke through the noise as the presumptive next level after the saturated human evil and MBPP benchmarks from OpenAI and Google DeepMind.

• 1:05:51Speaker 04

  Hi, it's great to be here. My name is Carlos Jimenez, and I'm a PhD student at Princeton University. And today I'm going to be talking about our evaluation benchmark called SUI Bench. Can language models resolve real-world GitHub issues? This is a work with my collaborators from Princeton and the University of Chicago. And I led this project with my co-author, John Young. So recently, language models have become really, really popular. And they're being pushed to perform in use cases that researchers haven't previously considered. A lot of past work on evaluating language models has become outdated, simply because model performance is getting really, really good. And that's a good thing.

• 1:06:35Speaker 04

  But evaluating language models is also really important. Understanding the strengths and weaknesses of language models plays a major role in building future applications and helping end-users know when and where it's appropriate to use them. So I want us to think about what sort of qualities makes an evaluation benchmark useful. First, the problems need to be hard enough to challenge state-of-the-art models. Problems should also reflect what people actually want to use the models for. And lastly, solutions need to be easy to verify. You can, yeah. So, consider the tasks involved in software engineering. Software is an extremely powerful tool, and coding is already one of the most popular

• 1:07:20Speaker 04

  applications of language models today. In reality, programming is a very hard skill to master. So if we can understand how AI systems perform on this task, we have a better sense of their abilities on doing real and challenging work. Furthermore, we have a lot of infrastructure for evaluating code. Many large software projects incorporate things like unit and integration testing to automatically evaluate changes to source code.

• 1:07:48Speaker 04

  Currently, language models only report evaluation numbers on coding benchmarks like human eval. And let's take a look at an example. So it starts with a function. It starts with a function signature and a doc string describing what the function should do. And language models are evaluated on their ability to write the body of the function. Here, it's highlighted in yellow. And there are many possible solutions, but the nice thing about programming is that we can check the validity of any of them automatically using unit tests. However, very few programmers ask language models questions like this, unless maybe they're trying to cheat on an interview. Software engineers typically write code that fits into a larger project,

• 1:08:33Speaker 04

  not one-off, isolated functions.

• 1:08:36Speaker 04

  So we created Sui Bench as a benchmark to evaluate the software engineering ability of language models in as realistic a setting as possible. I'll show you a very high-level view of what Sui Bench is trying to evaluate, and then I'll talk about how we created it, and so on.

• 1:08:53Speaker 04

  So Sui Bench starts with a codebase and a problem statement. And by codebase, I mean a real codebase with like hundreds of lines and thousands of files. And thousands of lines of code. Hundreds of files and thousands of lines of code. And usually, the problem statement is describing a bug with the codebase or requesting some new feature or change in behavior. We then give this to the language model, and the language model is tasked with generating edits to one or more of the files in the codebase in order to resolve this problem statement. Then, we take the model's proposed changes and evaluate it using unit tests from the same repository that were made after the issue was resolved.

• 1:09:34Speaker 04

  What this means is that Sui Bench can programmatically evaluate AI systems on their ability to solve real-world problems situated in full codebases. Solving tasks in Sui Bench goes beyond just code generation. It requires models to understand how large codebases work and how code changes in one function can impact the behavior of other parts of the codebase.

• 1:10:00Speaker 04

  I'll briefly summarize how we made Sweetbench. Now, GitHub is a website that people use to collaborate on software development projects, most of which are open source, and AstroPy is one such example.

• 1:10:15Speaker 04

  On an open-source GitHub project, users of a software can report bugs or request features by creating an issue explaining a problem that they encountered when using the software. And someone else who knows how to solve, how to fix the issue can submit their solution in code, which is called a pull request, to the project. Now, maintainers of the project can then review and modify that pull request and either accept the solution, in which case the source code for the project is updated, or they reject it. Now, this process in collaborative development, which underlies a lot of the open-source

• 1:10:57Speaker 04

  like development process, it can naturally be converted into tasks, and that's the source of the task instances in Sweetbench. So, we use the following procedure to gather task instances.

• 1:11:11Speaker 04

  We first scrape 12 popular Python repositories for all of their accepted pull request instance pairs, and then we filter these pull requests to make sure that they contribute updates to both the source code as well as the tests in the repository.

• 1:11:30Speaker 04

  And lastly, for each instance, we verify that the source code can be installed automatically and that the testing behavior changes before and after the source code is, the source code solution is applied. So, let's look at an example of what an instance in Sweetbench looks like. This is an issue from the SimPy Python library, which is used for symbolic mathematical operations and notation. So, we show the problem statement on the left, and it's describing, it's giving a detailed explanation of what the user is experiencing with SimPy, where they're seeing unexpected output when using the identity matrix. And the codebase for this issue is going to be tied to the version of the codebase

• 1:12:15Speaker 04

  that was active when the issue was first submitted.

• 1:12:20Speaker 04

  Next, we'll have the gold patch. And this is the edits to the source code that was submitted with the pull request. And it represents a possible solution to the issue. And it was the one that's officially accepted into the actual repository. And then finally, we have the test patch. And it's the edits to the tests that was contributed with the pull request. Now, the test patch updates or adds tests that evaluate source code for this particular issue. And we verify that the testing behavior changes from failed to pass when running them before and after the source code is updated with the gold patch.

• 1:13:03Speaker 04

  So after collecting instances like this across 12 repositories, we end up with over 2,000 instances representing a diverse set of problems in code bases. Each Sweetbench instance includes the full code base totaling to about 3,000 files on average, while gold patches usually only edit one or two files. We further collect 19,000 unverified instances. So unverified, meaning they don't have test cases. And we use that for training purposes. So as an initial baseline, we use a retrieval augmented generation system, or a RAG, using a simple BM25 sparse retriever. And language models are then provided with the problem statement, the project's readme file,

• 1:13:48Speaker 04

  and the entire file contents for the top retrieved files. They're then tasked with generating a patch file that specifies which files to change and the edits that they want to make to those files. We evaluate top models like ChatGPT, GPT-4, and Claude. And we also fine tune CodeLama using long context RAG examples from the training set to get our own model, SuiLama 7B and 13B, which are the only open source models that have non-zero performance on Sweetbench now. And across the board, base performance is extremely low,

• 1:14:29Speaker 04

  with the best performing model Claude 3 OPUS resolving only 3.8% of issues on Sui Bench. So how can these models get better on Sui Bench? Well, first, improving the RAG system can greatly improve performance. So if we assume that we have a very, very strong retrieval system that retrieves all of the files that were edited by the gold patch, which we call oracle here, performance jumps immediately from 3.8% to 9.1% for Claude 3. Another thing is that long context still seem to remain an issue. So for Claude 2, longer context inputs anti-correlates with performance very strongly.

• 1:15:13Speaker 04

  And that's something that we saw with basically every model. The longer the files that are being retrieved or input, the more context, the worse performance in general. And so, lastly, qualitatively, we find that language models tend to generate shorter, simpler, and more primitive code compared to gold patches. And for instance, we noticed that they tend to overuse Python built-ins and ignore library-specific utilities and API features, with an example shown here. So let's summarize. Sui Bench is a benchmark for programmatically evaluating the software engineering abilities of AI systems using over 2,000 real-world test instances.

• 1:15:59Speaker 04

  And we show how even state-of-the-art models are still woefully behind on this task. We open source Sui Llama 7B and 13B, which are suitable for long-context RAG with Sui Bench. So before concluding, there's one more thing. I've shown you performance using a RAG system for language models on Sui Bench, but software engineering is naturally a very interactive task. And we've recently had a follow-up work to this paper called Sui Agent that explores that idea a bit further. And with Sui Agent, we built an agent-computer interface for language models to interact with a computer to solve tasks on Sui Bench, demonstrating much better performance.

• 1:16:38Speaker 04

  So up to 12.5% of Sui Bench is resolved with our new framework. And this shows that there's a lot of room to improve for AI systems on Sui Bench. So finally, I'd like to thank all of my collaborators and colleagues who have helped with this project. We have an active community on GitHub, so please consider submitting your own solutions to be listed on the Sweetbench leaderboard. Thanks.

• 1:17:05Speaker 05

  Hey John, nice to see you at the oral session. So, congrats on the success of Sweetbench.

• 1:17:11Speaker 06

  Thank you.

• 1:17:12Speaker 05

  Why do you think it's caught on so much? The first I heard about it was from Devin.

• 1:17:16Speaker 06

  That's right.

• 1:17:17Speaker 05

  What was the launch process? I think, you know, I'm trying to get into the meta story around like, a lot of grad students here are trying to get their work noticed.

• 1:17:25Speaker 06

  Yeah.

• 1:17:25Speaker 05

  You got noticed?

• 1:17:26Speaker 06

  Yeah.

• 1:17:26Speaker 05

  How?

• 1:17:28Speaker 06

  Yeah, no, that's a great question. The Devin release certainly helped a lot with really putting Sweetbench sort of, I think the biggest contribution they did was give people a visual of what even 15% looks like on Sweetbench. And I think it made it really compelling. Prior to Devin, we had started working on SweetAgent back in, I want to say September, like right after we submitted this project to Iclear. So we kind of had this vision that like, oh, we're going to put out SweetAgent and people will see the numbers can in fact go up. A lot of the feedback and the skepticism we had at the time was that the benchmark is really difficult. But looking at kind of what Human eval did, you know, when they released, GPT-3 was at 0%.

• 1:18:07Speaker 06

  And we sort of used that as kind of motivation of like, well, you know, it's really bad now, but if we keep at it and we use this sort of agentic approach, there's something promising that could come out of it. So I feel like with this benchmark, our advisors and like me and Carlos, we're just really had this mindset of like, trying to champion our own work a little bit. And I'm kind of expecting that it'll take off on its own, of like that first step from 0 to 10 or whatever it might be, that we really have to drive that and sort of make that happen.

• 1:18:34Speaker 05

  Yeah.

• 1:18:35Speaker 06

  Yeah.

• 1:18:36Speaker 05

  So you worked on SweetAgent first?

• 1:18:38Speaker 06

  SweetAgent, we worked on this starting in June last year, summer.

• 1:18:42Speaker 05

  Okay.

• 1:18:42Speaker 06

  We were able to submit by September to Iclear and then right after that, we started working on SweetAgent.

• 1:18:48Speaker 05

  Got it. And then just the backstory behind how you guys started to work together, how do you choose this direction, you know, anything like that?

• 1:18:55Speaker 06

  Yeah.

• 1:18:55Speaker 05

  The narrative story.

• 1:18:56Speaker 06

  Yeah, for sure. Carlos is fantastic. Like, he had mentored me for a long time. Like, I was a master's student at Princeton, and Carlos's fourth-year PhD is about to graduate next year. And he has a lot of expertise. He's built benchmarks before. I also give credit to Shenyu, because we had worked on Webshop and Intercode and a lot of this agent stuff before. And Alex was around during the summer, and he helped a lot with sort of thinking about the fine-tuning and sort of what are good baselines to go with. So the way it kind of came together was in June, I put together a lot of related work and thought about this idea and brought it to our advisor, Carthick. And then I found out that Carlos had a very similar idea kind of at the same time of just sort of thinking about how we could take a lot of this great open-source data on GitHub and turn it into a meaningful task.

• 1:19:42Speaker 06

  And then really, just sort of like the nature of the task and how to follow through in terms of the engineering plan, I think was honestly quite clear after a week. And we just had to execute at that point.

• 1:19:53Speaker 05

  What were some of the big debates where you had to go like either this way or that way, and you picked one way?

• 1:19:58Speaker 06

  Oh, that's a great question. I think one of the things that I really remember initially was sort of how we were actually going to collect instances and sort of what the heuristics are. In hindsight, I think they're pretty straightforward and obvious. But at the time, one thing was just like, do we want to collect a lot of instances from different repositories? Or do we want to sort of focus on a couple of well-maintained repositories and mine the most instances from them? So just to sort of contrast that, we have 2,294 instances from 12 repositories. We could very well have 700 instances from 500 repositories. Exactly. Death ended up winning out. It wasn't very obvious to us, but really just the process of manually inspecting, looking at contribution guidelines, like looking at the natures of the test,

• 1:20:42Speaker 06

  gave us a lot of these heuristics that ended up being pretty reliable and I think scaled pretty well, at least for PyPI packages. So a lot of design decisions there, but I think we got lucky that we had a couple of good sort of hits in the beginning.

• 1:20:56Speaker 05

  Were you concerned that it's primarily Python?

• 1:20:58Speaker 06

  Yes, yes, yes.

• 1:21:00Speaker 05

  That's a choice, you know?

• 1:21:00Speaker 06

  Yeah, it is a choice. So I think for the first version of the benchmark, just because like... Python is so commonplace kind of in LLM evaluation, especially with a human eval that like, we'll just go with the flow, like we have no problems with the language. I think like in the same way that multiple E maybe expanded the amount of offerings for a human eval. This is something we'd be interested in doing. It's gonna require quite a bit of engineering effort, like we're both sort of fairly good at Python, but when it comes to things like maybe Rust or Scala or these other languages, like we're not quite sure. But I think like basically if there's an opportunity to collaborate and there's people who are experts in those languages, maybe even the software repository maintainers and they're interested in sort of having

• 1:21:45Speaker 06

  agentic language models help maintain their codebase. We're more than happy to work with them to sort of see the sweep and sort of evaluation harness idea and really manifest it for what they're trying to do.

• 1:21:56Speaker 05

  A lot of benchmarks try to say like, okay, human performance is 50 and then most language models are 25. And then we'll try to get the language models above human. But like, what is human here? What is a single human performance here, right? Like, is it 100? Or is it not undefined?

• 1:22:16Speaker 06

  Yeah, that's a great question. I think it's kind of an evolving answer in the sense that when we initially pushed the paper, we were like, it's 100% because someone wrote it and they did the issue and they contributed.

• 1:22:28Speaker 05

  Thousands of people wrote it.

• 1:22:29Speaker 06

  Yeah, exactly, exactly. But you know, more recently, I think as this benchmark is kind of picking up and there's more people interested in it, I think just like having the right efforts to figure out like which issues are easier or harder along what dimensions. Like when we say easier for a human, what does that actually mean? Is it characterized by the issue or by the size of the change or by the nature of the change? Like maybe it's a one-line edit, but it's really difficult because you have to know the codebase super, super well to make that precise change. So we have some ongoing efforts that are just sort of taking a look at sweepbench and taking a look at sort of the actual code changes and the problem statements and saying like, alright, let's figure this out, let's get sort of like spider, how they have like easy, medium, hard, extra level problems.

• 1:23:13Speaker 06

  You know, stuff like that. Yeah, yeah, yeah.

• 1:23:16Speaker 05

  What about eval cost? I think one of the reasons that human eval is so popular is because it's easy, it's quick to evaluate.

• 1:23:22Speaker 06

  Yes.

• 1:23:22Speaker 05

  But my impression, I haven't run sweepbench actually, but my impression is it's quite expensive.

• 1:23:27Speaker 06

  Yes, you're absolutely correct. I 100% agree with you. For the 2,294 instances, in hindsight, you know, if it was a little smaller, it would have been okay. But for a RAG, it's like 20 cents a pop. Even then, to do sort of pass at whatever or run a model a couple of times, if you're kind of empirically validating system, the cost can add up really quickly. So I totally agree with you there.

• 1:23:48Speaker 05

  So like $100 or something to eval?

• 1:23:50Speaker 06

  Yeah, like for example, for Sweet Agent, so we just released kind of the first pre-print recently. We'll kind of polish it for NURPS, but just to get the idea out. We actually have a cost table in there. And for the agents, like for the resolved issues, it takes like $1 to $2 to actually solve it. We have a $4 limit. But the meaningful thing that I feel like we did in response to sort of a lot of this was create this Sweet Bench Lite split. And that's 300 task instances filtered from the 2,294. And the objective there is that we apply some filtering criteria. It's not random to only look at changes to one file where the issue has, you know, reproducible code. Like basically removing some of the diversity of Sweet Bench

• 1:24:31Speaker 06

  that makes it really difficult to have sort of a more better starter one. And recently, like there have been works, like even yesterday, that can like Autocode Rover. Or like, yeah, I mean, like, I think this would have been, we kind of wish we had this to give to Devon also when they were running on this. And also like Kodak from sort of the OpenDev team, you know, they kind of put together something. And they claim they have 21% on Sweet Bench Lite now. So I think after we offered this, it's kind of a, there's a little bit more traction, I think.

• 1:24:59Speaker 05

  Okay, awesome. What is your take on Sweet Agent's direction versus Devon versus OpenDev versus Autocode Rover? What are the goals? What are the logical differences?

• 1:25:09Speaker 06

  Yeah, yeah. Oh, that's a really, really great question, yeah. I guess I'll like kind of speak from what I know. And I don't claim to have a good understanding of the other systems. I think what Devon and OpenDev are doing is like really, really cool. It's really, especially when the Devon demo came out, just to kind of see it as potentially like a product. It was really fantastic. What I'll say is like, I think for Sweet Agent, we were a little bit more focused on, from a research angle, just getting something to work and sort of having the empiric... And the numbers to back it up. I feel like Carlos and my direction has more been sort of like trying to solve interesting research challenges. So like even after Sweet Agent, like understanding what human interventions look like for autonomous software engineers, why people intervene, how they want to intervene, stuff like that.

• 1:25:54Speaker 05

  Which, to be clear, right now, in the one, in the Sweet Agent that I saw, there was no human intervention.

• 1:25:59Speaker 06

  Exactly, you're right, you're exactly. And I think from that, it was, I think it was a great session, like I really learned a lot. And I went back to my, I was like, you know, we should, we should really think about what they said. There's so good points in there, but yeah, to say it in one sentence, I would say it's just like, yeah, just like, I think people like autonomy, but it also seems like they don't want to give up control. And it's kind of interesting in the sense that like, it's cool from a research point to put out an end to end software engineer. But I think just like Devon and Open Devon have kind of the dialogue system. I think they're building out really intuitive features. The one thing I'll say is like, from kind of when we did Sweet Agent, some intuitive things worked, some intuitive things did not work. And that's kind of what we're interested in really discerning, like,

• 1:26:43Speaker 06

  some human AI, UIs, user interfaces and applications are great for giving to the language agent and having it sort of run with it. Some things don't work as well. So, you know, we're curious in figuring that out.

• 1:26:55Speaker 05

  Cool. I think that's all the questions I have.

• 1:26:57Speaker 06

  Awesome. Yeah, yeah, yeah.

• 1:26:59Speaker 05

  Is there anything else I should have asked you?

• 1:27:01Speaker 06

  Oh, that's a great question. I guess like in terms of sort of like the future of this, I think it's just exciting to see people be very enthusiastic about this kind of evaluation paradigm. So like, yeah, like the feedback you gave on the podcast, I think it's great. Like I'm really receptive to it. I guess like just as a personal, like I won't be in Princeton, I'll be in Stanford coming this fall, so I'll be back in the Bay area. So I'm very excited. I think it's a great ecosystem there of sort of like a lot of people are pushing this and yeah, just excited to sort of collaborate with people and see what people's own takes are and just sort of manifest the really cool ones. Yeah.

• 1:27:36Speaker 05

  Awesome. Well, thank you.

• 1:27:38Speaker 06

  Thanks so much. Yeah, I really appreciate it. Yeah, yeah, yeah.

• 1:27:41Speaker 00

  Next, we explore the issue of benchmark contamination. An issue raised by Horace He on GPT-4, Susan Zhang on Microsoft's phi 1.5. And you heard Aman Sanger discuss SUI bench contamination in the Graham Newbig discussion. This next paper won an outstanding paper mention for their simple canary-free contamination detection technique.

• 1:28:03Speaker 07

  Hi, everyone. It's an honor to be here. My name is Yonatan and I'd like to talk to you today about test set contamination. So recently, we've seen large language models show remarkable performance on many challenging benchmarks. And it seems like almost every week, there's a new open source model which comes out and tops the leaderboards. What's driving this performance gains that we're seeing in unsupervised learning has been massive pre-training data sets collected from the internet. So to give just one example, here's a breakdown of the pile. This is a large diverse data set, which many open source language models use for training. And you can see it's compiled from many different sources. So here in blue, we have academic sources like archive and PubMed. And green, we have internet-based sources like Wikipedia, Stack Exchange.

• 1:28:45Speaker 07

  There is prose in here, there is code and math and so on. But because of the scale of modern pre-training data sets, which are often on the order of trillions of tokens or petabytes of data, it's difficult to know if there's good separation between the training process of the language model and the benchmarks that we evaluate on. So to give you an example of how contamination might play out, let's say you have a language model that you want to evaluate on a coding task, like Codeforces for example. So Codeforces is a very commonly used benchmark. And maybe somebody uploaded it to GitHub, and then a web crawler found it, and as a result, it ended up in your training data. And so when you see an accuracy number or score on some benchmark, it's difficult to

• 1:29:25Speaker 07

  know if that's a result you can really trust because of the risk of contamination. So naturally, this brings us to the following question, which is how can we identify when a language model has trained on a test set or benchmark? And this is a difficult question to answer because many of today's top performing LMs are either closed behind APIs, or even if they're open source, their data sets are kept secret. So this is a figure from the Foundation Model Transparency Index by Stanford CRFM. You can see here in the first row that when it comes to pre-training data, there is very little openness in the industry.

• 1:30:03Speaker 07

  I want to show you an example of the kind of discourse that's happening around test set contamination today. So these are screenshots from a very prestigious academic forum. It's called Twitter. So on the left, you know, Horace is saying here, isn't it suspicious that if you look at the performance of GPT-4 on Code Force's problems introduced before 2021, it scores 100%? But if you test the model on recent problems, the performance drops to zero. Yeah, that seems a little bit suspicious. On the right, Susan points out that if you give phi 1.5, so this is a language model trained by a team at Microsoft. If you give phi 1.5 the first half of an example from a data set of math problems called GSM 8K, it'll complete the second half perfectly, which also seems a bit suspicious.

• 1:30:45Speaker 07

  Of course, it's possible that these things happen by random chance, but the point is that this is circumstantial evidence at best, and it's clear that we need some way to audit or test a closed language model for contamination in a way that provides rigorous proof. So there's been a lot of exciting work on test set contamination recently. I want to highlight three works in particular that are actually all at ICL this week, and I would highly encourage you to meet with these teams and learn more about their work. These are all excellent papers, but for our work, we were specifically interested in detecting contamination with provable guarantees and a false positive rate. And that's what I want to talk to you about today. So the question we're interested in is, is it possible to prove in a statistical sense

• 1:31:28Speaker 07

  that a language model was trained on a test set without access to the data used to train the model? So to describe this goal more formally, our setup is as follows. So we're given some test set X and the ability to evaluate log probes of text under a language model theta. And what we want is to develop a statistical test in the classic frequentist sense, which guarantees the type 1 error rate of at most alpha. So here, we're framing contamination as a statistical dependence between the model and the test set. And what that means is that we're going to test the null hypothesis that the test set X and the model theta are independent random variables. So just to be precise, the randomness of the model here is determined by the random

• 1:32:10Speaker 07

  draw of the training data, which may or may not contain the test set X.

• 1:32:17Speaker 07

  So how can we do this? How can we accomplish this?

• 1:32:21Speaker 07

  In order to make this possible, we're going to exploit a property which is true of many test sets, which is called exchangeability. So what do I mean by this? So typically, a test set is just a file, where on each line we have an example. But the order in which the examples appear in the test set doesn't actually matter. The examples are exchangeable. I can show you the examples in any order and it's still the same test set. So formally, what exchangeability means is that we can permute the examples in the test set without changing the joint distribution of the data. Which means that the model should have no inherent preference for the ordering of the examples. It's worth noting that exchangeability is a strictly weaker assumption than IID, which is an assumption we make all the time about our data on machine learning.

• 1:33:07Speaker 07

  So however, if a test set was leaked into a model's training data, because of the way pre-training works, where we take as many tokens as fit into the context window, the model would see multiple examples in a row and then memorize something about the order of the examples in the test set. So our key insight here is, a preference by the model for a canonical ordering of a test set must be a result of contamination. So using this idea, there's a very simple permutation test we can construct. So the key here is to compare the likelihood of the original ordering to the likelihoods of shuffled orderings. So for example, for the original ordering to have the highest log likelihood under the model than any ordering over, let's say, a million random shuffles,

• 1:33:51Speaker 07

  there's a 1 in a million chance of this happening under the null hypothesis, right? If the model didn't see the test set during training. So more formally, we can draw random shuffles, which is shown here as x sub pi, and we can compare each of these shuffles to our original sequence order, which here is log p sub theta of x. And by doing these comparisons, we're estimating the quantile of the log likelihood of the original ordering. So this ratio here turns out to exactly be the p-value of a permutation test. So this test works quite well, but it's computationally expensive because of the number of times we need to permute the dataset. And it turns out that we can do something a little more clever by aggregating a number of smaller tests, and that's what we call the sharded rank comparison test. And just for the sake of time,

• 1:34:36Speaker 07

  I'll refer you to the paper for more details. So how well does this actually work in practice? So in order to validate our test, we needed to have a language model for which we knew the training data was contaminated. So we decided to train our own model. So we started with a dataset of 20 billion tokens from Wikipedia, and we pre-trained a 1.4 billion parameter language model on this dataset. But first, we took a collection of benchmarks, and we injected them into the training data at random positions. And then we wanted to see, so can we actually detect contamination in this case? So here are the results of that experiment. They're in this table. So here, each row is a test set that appears in the training data. And the datasets were injected at various duplication rates. So some of them are in there one time,

• 1:35:19Speaker 07

  some are in there 10 times, and so on. And in the two columns on the right, we have the results of both the permutation test and the sharded test. So these numbers are p-values, so the smaller, the better, or the stronger the detection. And using a typical rejection threshold of 0.05, we find that for test sets which appear 10 or more times in the pre-training data, we can detect them 100% of the time. So we get perfect detection. We also found that detection at a duplication count of 1 is quite challenging. So we don't currently have a statistical test that works for duplication counts that are that low. And we'd like to encourage the community to continue working on this problem. So one question you can ask is, you know, what point does contamination become detectable? So we found that for a duplication count of 4,

• 1:36:03Speaker 07

  we can detect test sets about half the time. For a duplication count of 2, we can detect test sets some of the time. So we are able to detect contamination at low duplication counts, but just not for a duplication count of 1. So you're probably asking yourself, OK, you know, what about real models? Can we identify provable instances of contamination in LLMs that are in wide use today?

• 1:36:26Speaker 07

  So before discussing these results, there are a couple of important points I want to make. The first is that absence of evidence is not evidence of absence. So just because a p-value is high doesn't mean that there's no contamination. It just suggests that contamination is unlikely, at least at high duplication counts of more than 10. So it's a very particular claim that we're making here. The second is that there are a lot of hypotheses being tested here, and it's possible that some of these p-values will be significant just by random chance. So it's typical to do what's called a multiple hypothesis testing correction. I'll refer you to the paper for discussion on that.

• 1:37:01Speaker 07

  So our first result here is that we didn't find evidence of contamination other than of Mistral-7b and ARC-Easy. And with a multiple test correction, the p-value is just barely below significance, but it's still significant. The second result here that's of note is on MMLU. So the MMLU row, you'll see, has a little dagger there. And the reason for that is because some of the test sets in MMLU were not exchangeable, and we had to filter them out. So it's not quite the same as the others, but our findings are consistent with the Llama-2 report, which finds mild evidence for contamination by MMLU and their pre-training data. So I think the main takeaway here is that it's probably not the case that popular benchmarks are being duplicated numerous times in the training data of top-performing language models.

• 1:37:46Speaker 07

  Instead, it's likely that if these test sets appear in training data, that they appear at low duplication counts.

• 1:37:53Speaker 07

  So then the question becomes, how much does low duplication count contamination affect performance on benchmarks? I think this is hard to say, but in some sense, this work suggests that there's an upper bound on how much contamination is out there for today's highest-performing language models. So to conclude here, we covered three important points today from our work. First, we showed that it was possible to get provable guarantees on detecting verbatim contamination by leveraging exchangeability in benchmarks. So this is exciting because it opens the door to potentially very principled approaches to contamination audits. Second, we showed that these tests are effective, but there's also a major open problem that we can all make progress on, which is contamination detection at a duplication count of one.

• 1:38:37Speaker 07

  Finally, we tested existing public language models and don't find evidence of contamination, at least with high duplication counts. And this could be due to a number of factors, like deduplication being very common these days. But addressing the case of duplication count one would allow us to give more conclusive results in public audits. So we think this is an exciting first step and would like to see others make progress on low duplication count detection. And all of the models and benchmarks used in this project are available to encourage the development of future work. Finally, I just want to acknowledge my co-authors for their hard work and Stanford CRFM, especially David Hall and Percy Liang, without whom we wouldn't have had the compute for this budget or the compute for this project. Yeah, thank you very much.

• 1:39:20Speaker 00

  Our last benchmarks paper feature is Gaia, a benchmark for general AI assistance by Meta AI under Yan Le Cun and Clementine Foria, who runs the hugging face open LLM leaderboard. We talked to Thomas Yalom, who led training on Llama 2 and 3. Special thanks to listener Motashya Moblokalov for the personal introduction.

• 1:39:43Speaker 08

  Well, that's so Gaia, the general AI assistant benchmark. Okay, so for the background, we were at Meta during a workshop. And in the room, we had like, so with Yan Le Cun and others were arguing, is LLM all you need or not? And at some point, Grigor and I like were, let's just do a benchmark instead of qualitatively testing those models. Let's create this benchmark where we believe that what are those capabilities where models are failing? You know, in the in the history of evaluation for LLMs, we started with some simple problems like the squad some time ago, and then it gets better. So you move to multiple multitask questions like glue, super glue. And then because it was solved so fast, we moved to actually harder questions. And so by harder, the access was taken.

• 1:40:26Speaker 05

  Common sense reasoning.

• 1:40:27Speaker 08

  But even like now to expert tasks like MMLU. It's interesting because MMLU, I think the scores now for models is like more than 80. But so human is 90. But if you ask actually, that's human experts in the domains. If you ask a random human, the score actually, I think like 35. And the same like with like GPT-4, you know, evaluated on exams. And my point was like, you know, those are like results on expert domains. But actually, if we think about like simple tasks that an assistant will do, that was all the point of Gaia. Model actually completely failing.

• 1:41:04Speaker 08

  That was my intuition, while humans would be like ninety percent. So it's a complex task that require multi-step reasoning, multi-information problems, and an open world, browsing the web, passing information, etc. But like humans, if you give it enough, if given a human enough time, you will solve this problem. And that's what we built with Gaya. And the humans obtain like random humans ninety percent. GPT-4 obtain only ten percent on the level one, with some tool use and feel like that. So that was the point of the thing. And to our point, arguably, if you want like reasoning, general capable agents, whatever LLMs or whatever AI system, you want to solve the stats first.

• 1:41:48Speaker 08

  Like, it's fine to answer questions about what a PhD student will ask in a physical like chemistry or whatever. If you don't solve the stats, there's a problem. And so that's all about Gaya. We designed like a bunch of like very small but very qualitative task questions that are very complex, from level one to level three. Level one are like, you need one or two steps of reasoning, of browsing. It's kind of simple. Level three, it's extremely complex and takes like up to ten, twenty steps. And in an open problem, like the complexity of the exponential increase, because you can like, you're in open-ended world where you need to browse, you need to use maybe multimodal abilities, etc.

• 1:42:34Speaker 08

  And actually, it's interesting because very recently with agents, poor by LLM, but using memory, tool use, like planning systems, like Friday in OSCO pilot, or there's a recent like autogen from Mikosav, you move to like ten to forty percent already. So there's something happening there. And we think that's actually a very good way also to evaluate how intelligent are those LLMs. To like, what if you put Lama three or GPT-5 in this model? What will be the boost of performance, basically?

• 1:43:06Speaker 05

  You are also on the Lama three team, right?

• 1:43:07Speaker 08

  Yeah.

• 1:43:08Speaker 05

  Did you already test Lama three on this?

• 1:43:10Speaker 08

  Not yet. It's still in development.

• 1:43:14Speaker 05

  You mean Lama three is still in development?

• 1:43:16Speaker 08

  Yeah, but we will definitely, and eventually we will move to like improve Lama in this direction.

• 1:43:22Speaker 05

  How would you compare this versus like the other generalist agents benchmarks? Like, I just talked to Graham Newbig, who did Web Arena.

• 1:43:28Speaker 08

  I think, I mean, there's some specificities about Gaya that... That's why we designed it that way. So first of all, you don't need an environment. Most of the agent benchmark, like you need an environment, it's complex and it's kind of limited by design to the synthetic environment. Gaia, it's like you just have a prompt. And that's it, you have access to the world, go for it.

• 1:43:52Speaker 05

  So it's not deterministic though?

• 1:43:54Speaker 08

  It's not deterministic, I mean, it is deterministic, but by the way, we created the question. So one thing is that the question are kind somehow unrealistic. Because in practice, for instance, one question you would say is, we had this question actually in the dataset. How many birds has layers?

• 1:44:14Speaker 05

  How many birds?

• 1:44:15Speaker 08

  Birds, the model bird has layers. And actually, the annotators were disagreeing because you have different size of birds, different things. And so for for making sure that basically the question have a unique answer, there's zero ambiguity. We add to the question a lot of additional information, like according to Wikipedia of this date, blah blah. Such that there's a single and unique answer to make the model the evaluate. So we can evaluate the model automatically. In real life, you probably will have easier question with a bit more ambiguity.

• 1:44:47Speaker 05

  How like what are your thoughts on just contamination proofing these kinds of benchmarks? Because a lot of these L L M's are going to be online, like or like the knowledge cutoff is updated.

• 1:44:57Speaker 08

  Yeah, I mean, I think there's two things that made actually Gaia bulletproof to recognition.

• 1:45:02Speaker 05

  Bulletproof?

• 1:45:02Speaker 08

  Yeah.

• 1:45:03Speaker 05

  Alright.

• 1:45:03Speaker 08

  I would say so.

• 1:45:04Speaker 05

  That's a big claim.

• 1:45:05Speaker 08

  I mean, so two things. One is, first of all, we kept the test set aside. We didn't release it. So anyone can put on the on the leaderboard their results on test set. You to solve it, you will need manually to answer the question of the test set. Go for each, spend like one hour for each of the questions. And provide the answer. That will be honestly like a pain in the aight. Now that being said, the thing is, it's super easy to create new questions, right? So if tomorrow you cheat and you say like you claim a new super high result. I can get with you like 10 question, test your model. If it fails, there's a problem. So it's super easy to detect this kind of things. And the last thing is also, we ask in general the people to report the trace of the answer.

• 1:45:50Speaker 08

  So you can very easily verify what has happening with the model. So it will be in practice extraordinarily hard to cheat on the leaderboard.

• 1:46:00Speaker 08

  Or if you do that, you will never have a model to show to the people, actually. So in that sense, it's good proof. All those questions are not knowledge facts, but things you will find on the net that is not present. You cannot go to the answer just by memorization of the self-supervised massive web test. An example of that question is like, if I ask you when is born a W14 according to the Wikipedia page? That's something you can remember from the trainings. Now, if I ask you how many times W14 was mentioned with this way to write it in the Wikipedia page? That's an answer you will never find the answer right in the Wikipedia.

• 1:46:38Speaker 05

  You need to Ctrl+F.

• 1:46:39Speaker 08

  And we kind of oriented the question in this kind of sense.

• 1:46:43Speaker 05

  What do you think the performance is bottlenecked by? Is it tool use? Is it planning? What agent capabilities have revealed in your testing?

• 1:46:55Speaker 08

  That's a very good question. Actually, I've told him I plan to work on that. So I'll have more insights. But my two cents right now is, one, a general system that can be powered by LLM, and we're starting to see them. So now we can like add more like planning, backtracking, things like that to improve them. This is one of the core pieces that is missing. And the second thing is all of them are powered by LLMs. And the smarter this LLM will get with scaling, the better also the performance will improve. So there's, I would say, these two core things. The overall system powered by LLM, improving that, giving more tools, more abilities, more capabilities, create its own tools, leverage its own tools, planning, backtracking and all those things,

• 1:47:41Speaker 08

  and improving the LLM itself.

• 1:47:42Speaker 05

  Yeah. Backtracking meaning like the backspace token or something else?

• 1:47:46Speaker 08

  Meaning like non-too-aggressive decoding, but at the system level, not at the LLM level, in a sense that... You can ask something, you can try.

• 1:47:56Speaker 05

  Like traversing a graph, like a tree of thought or something.

• 1:47:59Speaker 08

  You have your plan, but in practice, I mean, if I want to go to Vienna, for I'm clear, and I want to book a plane, but actually they are all full, I will need to adapt my plan.

• 1:48:08Speaker 05

  DeepMind presented a paper, like of the web agents thing with exactly the Vienna travel example.

• 1:48:14Speaker 08

  Yeah, exactly.

• 1:48:15Speaker 05

  I wonder, I don't, they probably haven't evaluated their agents on your benchmark. And I wonder what it would take to cross-pollinate the different labs' agents and benchmark ideas. You know, you know what I'm talking about? Like, DeepMind has their own stuff, you have your own stuff, and like, I don't really see a crossover very much.

• 1:48:34Speaker 08

  Yeah, that's a very good question. The thing is, at Meta, we are really like pro-open source.

• 1:48:38Speaker 05

  Yeah, you're the most open.

• 1:48:40Speaker 08

  And we, we really did with, I mean, it's not just Meta, but also with Hugging Face, this paper. And we just made it open source with a leaderboard, so that it's actually everyone can use it and benchmark on it.

• 1:48:51Speaker 05

  Yeah.

• 1:48:52Speaker 08

  So now I'm, I'm really looking forward to see others like put some riddle there, push your numbers, put some pressure on the numbers, and see how far, like, I mean, that's the only way to me, but we can accelerate the community to go like to more capable of models now.

• 1:49:07Speaker 05

  Yeah, that's all the questions. Anything else we should have asked you?

• 1:49:09Speaker 08

  I'm looking forward to see what the community will push as a, and how fast we will get to like solving this benchmark. I'm really curious to see.

• 1:49:17Speaker 05

  Yeah, me too.

• 1:49:18Speaker 00

  Something we don't often get to feature are the invited keynote talks that start every morning of conferences like ICLR. There were great sessions on legal and copyright risk, the road to AGI, and even Devi Parik's career stories. But this time, we are featuring Moritz Hartz talk on benchmarks for its comprehensive walk through history and call for action on greater thinking on the need for more scientific benchmarking.

• 1:49:45Speaker 09

  It's a pleasure to be here. I'm very humbled to be here. And I'll tell you about the emerging science of benchmarks. And this is not about why a particular piece of machine learning works. This is sort of about why the machine learning community as a whole works. And it's pretty much everything we know about it. And it's going to be a relatively short talk. Okay, so I'll start with a quote. It's a famous quote. And it says, "The only principle that does not inhibit progress is anything goes." And that's what philosopher Paul Feiert argued about 50 years ago. And many smart people agree with this statement and many disagree. And there's been a lot of debate about it. But whether you agree or not with us, I'll leave that up to you. But it's pretty clear that the machine learning community, especially the ICLR

• 1:50:28Speaker 09

  community, has always embraced the anything goes. Okay, and in a good way. So from its roots in sort of the cybernetics and pattern recognition era of the 1940s and 1950s, we've pretty much tried out everything. Okay, and this community lets you dream your wildest dreams. You could be inspired by the human brain, or child development, or physics. This community does not limit you in how you come up with the stuff that you propose. It's totally up to you. This is what anything goes means. You're not limited in how you come up with the science that you do. And I think this has always been sort of the strong suit of this community, that we don't limit people in how they work. Okay, but there's one thing we need to tame the anything goes, and that's the idea of a benchmark.

• 1:51:10Speaker 09

  It's the one rule that we have which sort of tames this idea of anything goes. And benchmarks follow what philosopher Michael Stravinsky calls the iron rule of modern science. So the iron rule is the idea that all disputes must ultimately be settled by competitive empirical testing. So at the end of the day, after some time, we must come together and, you know, competitively test our hypotheses, or our methods, or whatever we want to do, and see which one works best. Okay, and so the way this works in machine learning, you all know this, is that we agree on a metric or measure of success, and we agree on test cases, benchmark data, and we let people compete over the metric and

• 1:51:54Speaker 09

  The data and we rank the models. Okay, so we see who is best in the end and we might pick the best performing method. So this is what we call a machine learning benchmark. And it's essentially how this community operates. Okay, so what's interesting is that benchmarks emerged. They didn't, it's not like the founding fathers of the community sat down and they said, here's how it's going to be. We're going to operate according to the following principle. They sort of came up over time. Okay, and they didn't follow any a priori theoretical framework. We didn't know ahead of time what we're doing or why it was going to work. Okay, and so if you look at the last 40 years, the history goes back much further. But if you look at the last 40 years, you see roughly four eras of benchmarking.

• 1:52:39Speaker 09

  The first one is the DARPA era in the 1980s where, you know, grant managers at DARPA wanted to have a way to compare scientists and their contributions in various grant proposals. And they want to have objective ways to to compare them. So they thought this idea of a metric and like a benchmark would be great to to see how how scientists are doing against the proposed objectives. And so this was the DARPA era. And it was followed, I think, you know, if you will, by the MNIST era where sort of, you know, benchmarks sort of came into the academic, you know, front and center. People started using benchmarks more and more in academia and there were starting to be public leaderboards. People were comparing methods on publicly available data.

• 1:53:22Speaker 09

  And MNIST and other benchmarks around the time contributed to this becoming a major paradigm in machine learning. And then, of course, the whole idea of a benchmark exploded in the ImageNet era when, you know, the deep learning revolution of the 2010s happened. And ImageNet as a benchmark was really sort of co-constitutive with the deep learning revolution of that time. So the models were sort of developed on ImageNet and tested on ImageNet and ranked on ImageNet. And this was sort of the dominant benchmark in that era. And now, interestingly, I'd argue we're in a fourth era, which I'll call the polymorphic era for this talk. And I'm not too attached to this name. So if eventually we call this something else, I'm okay with that. But for this talk, I'll call it the polymorphic era.

• 1:54:04Speaker 09

  And the reason is that we're witnessing sort of this radical plurality of benchmarks. We're seeing a lot of benchmarks now, thousands of benchmarks. And they take on very different forms. Okay, so they're not the way they used to be. I argue we're in a new sort of phase of benchmarking. And we're trying out many new things, in particular multitask benchmarks and dynamic benchmarks, where you don't have a fixed data set, you actually, you know, let the data set evolve over time. So people are trying out all sorts of new benchmark ideas. Okay. If you're more interested or interested in more background, I'll point you to Mark Lieberman's talk from the Simons Institute from about five years ago. It's an excellent resource on some of the history. And Ben Reck and I wrote a chapter on this also in our textbook. So you can check this out online. But this talk is not about the history, it's

• 1:54:49Speaker 09

  about, you know, why benchmarks work. Okay. And I'll start with an outline of the science of benchmarks. What do we know about benchmarks, why they work and when they don't work. And I'll start with sort of scientific key takeaways from the ImageNet era, with the benefit of hindsight, because it's over, we can look back and we can do some kind of retrospective analysis of what worked and what did we learn from this era. And then I'll move on to this era that we don't have the benefit of hindsight about that's happening right now at this conference and elsewhere. And we don't really know what it's doing. And I'll point out some risks, also opportunities, but some risks of this new polymorphic era of benchmarking. And I'll wrap up by motivating why we need a science of benchmarks, why I hope many of you will

• 1:55:31Speaker 09

  join this effort, and why I think it's a fascinating research area to work in. So, how did this all start? I argue that the beginnings of the science of benchmarks started with a mistake, like many things. It was an excellent mistake, but it was a mistake nonetheless. So the mistake that started all this is to assume that benchmarks are just the holdout method. And you still hear this today a lot. People will say, ah, it's just the holdout method. Okay. So what's the holdout method? The holdout method, as you read it in textbooks, just looks like this. You split the data into, let's say, two pieces, could be multiple pieces, but for this talk, it's just two. You set aside the the training data, you can apply, oh, sorry, you set aside the test data, you apply the anything goes principle to the training data, you can do whatever you want with the

• 1:56:15Speaker 09

  training data. And then you use, in the end, you rank the models on the test data. Okay, you had set aside the test data, and in the end, you rank the models on the test data. And I emphasize this part in the end, because this is really crucial. Okay, as you can read, you know, in this textbook by Hasty, Tipshir, and Friedman, very recently, that ideally, the test set should be kept in a vault and be brought out only at the end of the data analysis. So the holdout method only works. In theory, if you keep the test set in a vault, and you only rank the models in the end at the test set. So the models had never sort of seen the test set in any way. Okay, and why is that? Because if you want to prove guarantees about the holdout method, you really need this.

• 1:56:59Speaker 09

  You need to some this vault assumption, as I call it. And under this assumption, you can argue that the test set has exponential longevity. Okay, this sounds like something Silicon Valley billionaires want to have. And I just mean that it's the number of model comparisons you can do on your test set is exponential in the data set size. Okay, so the data set can live very long. Okay, you can try out all sorts of models on the test set, and you still get good results from the test set. Okay, so under this vault assumption, the test set has high longevity. It sticks around, it can survive many model comparisons. But look, you all know this. The empirical reality is completely different, right? The test set is anything but in a vault. Okay, so we have a large sprawling machine earned community

• 1:57:44Speaker 09

  that sort of builds models in like a continual loop with a test set. Okay, so you propose a model, you evaluate it on the test set, you see what the results are, you incorporate these results into your work, and you continue. This is the whole point of science, that there is this kind of like closed feedback loop between you and the evaluation systems that you have. And I looked this up, just I was amazed by this. How many of you have downloaded MMLU recently?

• 1:58:10Speaker 09

  Come on, guys, three people, that's not true.

• 1:58:14Speaker 09

  That doesn't work. It was Friday morning at 8:00, people are tired. Okay, I looked this up. Apparently, MMLU, so this is a multitest benchmark for language models, and many of you know. It has just 14,000 data points, questions. And it was downloaded apparently 5 million times on Hugging Face last month. Okay, I found this marvelous. Okay, it shows you how, what's what scale this community is operating at. It comes in two versions, and if you add it up, you get about 5 million downloads. That's 60 million downloads. If you think that every download is at least one evaluation, this test set is seeing tens of millions of evaluations per year. That's kind of striking. But it's not just the number of evaluations. It's not just how often you evaluate these test sets. It's the way you do it. In the sense that machine learning is adaptive.

• 1:58:59Speaker 09

  You use the results from the test set to refine your methods. It's not like this test set is kept in a vault. It's part of the ongoing evaluation loop. And that's the whole point. Okay. And so what we realized about 10 years ago is that this kind of adaptive activity, this kind of adaptive use of the test set breaks all existing guarantees of the holdout method. And it reduces its lifeline to just a linear longevity in the number of evaluations. Okay. So the test set can only... in this data set size. So the test set can only support a linear number of evaluations. You can show easy examples where this happens. Okay. So in principle, this way of using the test set could be really bad. Okay. And this launched the area of adaptive data analysis, where people come up with sophisticated methods

• 1:59:42Speaker 09

  to try to fix, you know, the holdout method to work better under this adaptive use. But we can go back to the empirical reality. And we can see that, like, test sets, actually, in practice, seem to have a lot of longevity. Okay. Here is a plot that probably many of you know. It's the, you know, from a plot from Papers with Code that you can see on the website there. It just shows you the improvements on the ImageNet ILS-VR-C 2012 test set, you know, over the years. And as you can tell, even after sort of 10 years of very active use and 10 years of people hammering away at this test set, you were still seeing significant improvements. The test set was still good enough at that point to support active model development and model improvements.

• 2:00:25Speaker 09

  And it was still worthwhile and useful for model ranking. So how could it be that, you know, after such a long time, it still seems useful? Okay. Should we trust the model rankings? It really begs the question, should we trust the model rankings that we get out of this 10 years of active use of this test set? So several years ago, people wanted to find out. And they created a fresh test set for ImageNet. They restarted or recreated the data set creation process for ImageNet. And tried very carefully to create a new fresh test set for ImageNet. Okay, this is a fantastic work. And what they found is, and also that the model rankings were preserved. Okay, so the model rankings on this fresh test set were actually preserved, you know, compared to the old test set.

• 2:01:10Speaker 09

  Okay, so the model rankings were fine. Even after a decade of development, the model rankings were still the same ones. Okay, and the people also found the same thing for MNIST, which was even more used. You know, various Kaggle competitions, it was also true, and other data sets. So people confirmed this insight about the stability of the model rankings in a number of different cases. And this is what I call the internal validity of the iron rule. Beating the previous best replicates in similar conditions. Okay, if you have a model improvement in some test condition, you will get the same model improvement in very similar test conditions. Okay, if you try to recreate a test set, or you recreate the testing conditions very closely, you will see a similar model improvement. Okay, so this is the internal validity of the iron rule.

• 2:01:54Speaker 09

  Beating the previous best replicates in similar conditions. And we were able to even prove this. Okay, so we had a work where we turned this this observation into mathematical assumption. Okay, we said, what if we make the assumption that researchers only care if they improved over the previous best? So what if we make a mathematical assumption that, you know, researchers will ignore results that didn't improve, and they only care about results that improved over the previous best? This is, of course, a simplifying assumption. But if we make this assumption, we can actually prove formally that, assuming this iron rule assumption, the benchmark data has exponential longevity. So you recover the exponential longevity of the holdout method under this iron rule assumption.

• 2:02:38Speaker 09

  And I find this amazing, because it says that the iron rule assumption is nearly as good as the iron vault assumption. So the idea of competition, this principle of like letting people compete, is as much a regularizing force as keeping the test set in the vault. Okay, it's almost a little bit depressing if you think about it, that competition is such a strong, you know, coordinating principle. But this is what it is. Okay, you can think of this as something that you implement. If you're really worried about, you know, keeping your model rankings accurate or protecting your test set, you can implement this by like enforcing limited feedback in a benchmark. But I actually like to think about it differently. I like to think about it as a descriptive sort of theorem. It says, you know, if we think of this as a postulate about how the community works,

• 2:03:23Speaker 09

  about how scientists interact, then this is, you know, you get these consequences. Okay. So, under this postulate about the community, we know that test sets have exponential longevity. Okay? And so, this is ultimately an assumption about the behavior of the community. It's not just a technical assumption. It's an assumption about how we, as a community, organize our scientific activities. All right. And so, we found many other such socio-technical forces behind benchmark longevity, things about the community that promote benchmark longevity. Okay? Competition is the one I just mentioned, but also collaboration. The fact that we all share code and we share each other's GitHub repositories and so on. This has an effect that promotes longevity.

• 2:04:06Speaker 09

  This is related to what Dave Donoghue recently called frictionless reproducibility. The fact that we have this very active ecosystem of sharing code, and it's an important driving force in this context. We even found that cognitive and behavioral biases of the researcher, the fact that we're all like limited human beings and we have our biases, these biases can actually protect the test set. So, our cognitive limitations can actually work to our advantage when we're doing science and aren't necessarily a bad thing. And it's great. I feel good about this. And finally, we found that basic dataset artifacts, like what classes you include, you know, how many classes you include, that also has a strong effect on benchmark longevity. Okay. So, this is something we found a few years ago.

• 2:04:49Speaker 09

  There's a talk about it, if you want to see more about it. These things are all what I call socio-technical forces. They're not just purely statistical. They're not just purely, you know, technical. There's something about how the community works. But all of them promote internal validity of benchmarks. Good. So what do we know at this point? So as of about five years ago, we knew that model rankings replicate under similar test conditions. If you change your test conditions slightly, you get the same model rankings. It's this kind of internal validity story. So what if we ask a more daring question? What if we ask, do model rankings replicate on radically different test conditions? What if we stretch this to the limit and we go to radically different test environments? Will we still get the same model rankings?

• 2:05:34Speaker 09

  Is there any reason to believe that the answer is yes? So we wanted to find out, and we call this the Image Not experiment. And this is a joint work with Olavale Salodin, who interned with me last year. And Image Not is an anti-replication of ImageNet. Okay, this sounds crazy, and it's a term I made up. So nobody, I think, has used this idea of an anti-replication. But what I mean by this is that it has the same scale and diversity of ImageNet, same size, but it's different in every other regard. We just tried to make it as different as possible, subject to the same size and scale as ImageNet. Okay, you all know ImageNet was carefully curated by humans. You had many annotators per image. You had a very high agreement rate between annotators. There was a certain logic

• 2:06:18Speaker 09

  to which classes you included, et cetera. Okay? Image Not is just a quick and dirty dataset based on selecting images from their captions in a web crawl dataset. And there's no rhyme or reason to the classes that we include. It's completely arbitrary. It's, if you will, kind of a trashy dataset. Okay, so I'll be honest with you. Okay? And so the experiment we did is, what if we retrain the key ImageNet era models from scratch on Image Not? So not fine-tuning, we're retraining them from scratch on this new dataset. And we want to see what happens. Is it true that the model rankings are preserved? Okay? Remember, this is a retrospective analysis. We already have these models.

• 2:07:02Speaker 09

  They were developed on ImageNet. Do the model rankings replicate in this radically different test environment? So the answer is yes. Okay, so for the model architectures that we carefully studied, we see the exact same rankings. AlexNet comes lowest because it's the first major breakthrough in 2014. VGG improves upon that. DenseNet improves over that. ResNet comes then, and so forth. We get the exact same model rankings. But what is maybe more striking is that the relative improvement over AlexNet is also about the same. So if you look at the curve of relative improvements that each model makes over time compared to AlexNet, you get roughly the same curve. So what this is saying, you know, somewhat surprisingly,

• 2:07:46Speaker 09

  is that on this completely different kind of trashy dataset, this dataset makes the exact same sort of judgments as ImageNet. It gives you the same information from a benchmarking perspective as ImageNet. It says this model is better than that, and it gives you the same comparisons, and it gives you the same sense of relative improvement. Moreover, we found out the same is true for fine-tuning. If you wondered about that, if you fine-tune instead of retrain from scratch, it's also the same. And we also looked at transfer learning, and we found that there's a similar relative utility to transfer learning on ImageNet as there is on ImageNet. Okay, so how do we create this? You probably, many of you already guessed this. We use this wonderful resource called Lion, which was created from Common crawl.

• 2:08:28Speaker 09

  It's a resource of 5.85 billion image caption pairs. It was a massive effort that we're using here and building on. And we just really did sort of an extra step on top of that. We just selected a bunch of images based on their captions from Lion. Okay, how do we pick the captions, the classes? We just pick 1,000 arbitrary classes while avoiding all subtrees of the WordNet hierarchy that contain an ImageNet class. And when I say ImageNet, I mean ISVRC 2012. So we make sure we stay away from all these ISVRC 2012 classes and pick sort of arbitrary classes subject to staying away from that. And then we select images from Lion simply based on Roberta text-only similarity between the class and the caption.

• 2:09:13Speaker 09

  So you embed the class, you embed the caption, you look at the similarity. You do not look at the image when you select these images. You just look at caption text similarity. Okay. And then we implemented some additional safety filters just to make sure that whatever we run our analysis on is safe. Okay. But the main point here is that there's no annotators involved. There's minimal human intervention. It was largely just based on a web crawl with like very sloppily selected data points. Okay. And in doing this, we actually built on another work that I did with Ali Shirali at UC Berkeley, which is answering the question, what would be different if we recreated ImageNet from Lyon? What are actually the differences in the data sets that you get?

• 2:09:56Speaker 09

  And through some really clever detective play, Ali found out some subtle but very important differences. And we're building on this effort in creating ImageNet. Okay. Just to give you, you know, a sense of ImageNet, a visual sense. How many of you know the Terelba-Efros game of guess the data set? Probably all of you, right? You look at researchers, you let researchers look at two data sets and they have to guess which one it is. So let's give you some training data. Okay. So here's a class from ImageNet. It's called cleats. It contains some shoes that have cleats, you know, like football shoes. But also it contains lots of images of like just football players that apparently wear shoes with cleats, but you don't actually see the cleats in the image. And then it contains all sorts of other stuff like shoes with bicycle cleats and so on. Right?

• 2:10:41Speaker 09

  There's another class called batter. It contains cartoons about baseball batters, but it also contains dietary advice, like you should always batter your chicken and fry it before you eat it, which I can also recommend. And finally, it contains, let's say, pictures of this cosmetic pen called batter up. Okay. Let's contrast that with ImageNet. ImageNet, as far as I can tell, is mostly dogs. And so there's, you know, something called an Irish terrier. It looks like this. And ImageNet has lots of front and center images of Irish terriers. Okay. So I mean, if you know what an Irish terrier is, you can tell this is ImageNet. Even if you don't know what it is, you can sort of tell it's ImageNet because it's kind of front and center of a dog. Okay. It's a front and center dog, that's ImageNet. Not a front and center dog, that's ImageNet. Okay.

• 2:11:26Speaker 09

  There's also something called the Blenheim Spaniel. This looks like that. And again, it's lots of like front and center images of Blenheim Spaniels. Cute dogs. And there's hundreds of dog breeds or like more than 100 dog breeds in ImageNet. Okay. So what can we learn? Okay. And just to spoil it, we can easily get more than 90% accuracy in telling them apart. So these data sets are really very different. What can we learn from ImageNet? This, to me, is quite fascinating. It suggests that something like this might be true. It suggests that the iron rule might actually have external validity. So it might say that if you beat the previous best under sufficiently general conditions, it will likely replicate elsewhere. The only thing you need is that your original test conditions were so rich enough. But aside from that, you don't really need anything.

• 2:12:11Speaker 09

  The model rankings will replicate in other conditions, assuming your original testing conditions were rich enough. And I say this almost a bit more like a conjecture because it needs more work. And for instance, we need to know what does sufficiently general mean and so on. So there's a lot of interesting work to be done on this. But I find it quite intriguing that there might be this kind of dynamic equivalence that there's evidence now that ImageNet could have been anything of similar scale. Just from a model ranking and benchmarking perspective, all these dataset artifacts that ImageNet had, they may not be all that important as we thought. And anything of similar scale might have given you the same benchmarking results. Okay? We don't even need clean labels. Okay? And if you know anything about ImageNet, you know how much we, as a community,

• 2:12:54Speaker 09

  have thought about this annotator step in ImageNet, how important we thought it is that we have annotators, multiple annotators, the agreement rate between them, all these things we thought were essential for the benchmarking enterprise. And I'm here saying that we don't even need clean labels. How could that be? I mean, that sounds almost suspicious. So we wanted to know more and we wanted to dive deeper into this claim and do some theory about it. And we studied a model, proposed a model of benchmarking with noisy labels. This is joint work with Florian Dörner, who's in the audience. And we boiled it down to a very simple theoretical question. Okay? So given two binary classifiers, let's say image classifiers, F and G, which one has higher accuracy? Okay? This is what you're trying to find out. This is the sort of essential benchmarking question.

• 2:13:39Speaker 09

  Given two models, which one is better? And here's the model. You can draw unlabeled data points X for free. Just go on the internet, download an unlabeled data point, and then you can get a label Y for 1 euro. Okay? For the Americans in the room, 1 euro is the local currency. Okay? And this is Austria, not Australia, okay? So, quick, quick check, okay? Just making sure we're on the same page. So, 1 euro, and you get your label, okay? But the catch is, this label might be incorrect, okay? So, this label might be wrong with probability p less than half, okay? So, there's some chance that this label is incorrect, and is not the correct label. And so, how do we best spend our money? Let's say we have n euros,

• 2:14:23Speaker 09

  and we want to spend our budget on identifying which is the better model. How do we most efficiently, most economically, spend our money to maximize the probability of identifying the better model, okay? Here's a common practice that people would propose. You sample n over k points, where k is some number, like 3, or 5, or 17. And for each data point that you sample, okay? You request k labels, y1, y2, up until yk. These are noisy labels, independently drawn from your annotator process. And you clean these labels by taking a majority vote, okay? Binary labels, majority vote, reduces the error rate of your label, okay?

• 2:15:05Speaker 09

  So, the label y that you get by taking the majority vote, you know, you clean it. It has lower error probability than the k labels each have on their own. And so, this could be a good way to clean the labels. And the question now is, well, how large a k should you pick? Should you pick k equal to 3, or 5, or 17? What's the optimal k? And what we prove is that in all cases, it's best to sample n data points with one noisy label each, okay? The optimal choice for k is 1, okay? You want one noisy label each for one point, okay? That's the result. My contribution to this project was preventing Florian from calling the paper "all the single labels," okay?

• 2:15:49Speaker 09

  So, it's hard being an advisor sometimes. It's a thankless job, but, you know, has to be done. And we got over this. But this is what it says, okay? It says that really, a single noisy label per data point has been stopped laughing.

• 2:16:04Speaker 09

  A single noisy label point per data point is best. Okay. So this is what it says. And so the statement is very easy, but the proof is not. Okay. So it was actually kind of a grind. It uses a chromers theorem from the theory of large deviations to get like an exact asymptotic tail bound on the probability of not identifying the better classifier. And it extends to many model comparisons just via the union bound as you usually would do it. And for the theoreticians here in the room, this can often be a good alternative to using Hefting's bound. So I had always been like naively applying Hefting's bound, which is just an upper tail bound and doesn't give you an exact bound. And this gives you something much stronger. And because this proof gets a little bit subtle, Florian actually found like a really good way to numerically check this conjecture or check the theorem.

• 2:16:49Speaker 09

  And this meant we could very easily simulate all the parameter settings. And here's what you get in a typical parameter setting. So as your label budget grows, so does the number of model comparisons that you can make. And as you can see, the number of model comparisons that you can make by the single label strategy is much, much greater than for three labels or five labels and so on. And it's also much, much better than what you get heuristically from applying Hefting's bound. Okay. So it gives you something much better. And because this is useful independently, Florian created like a sample size calculator that you can check out. If you're creating a data set with noisy labels, this gives you sort of a guide on how to do things. Okay. Great. Thirty minutes. It's perfect.

• 2:17:33Speaker 09

  Yeah, I'm doing good. So that's sort of my retrospective on the ImageNet era. Let's leave the familiar contours of the ImageNet era and enter the polymorphic era. And I thought it was fitting to try to get like generative AI to describe the polymorphic era. This took me like an afternoon of prompts, basically, but now I'm happy with the results. So this is what the polymorphic era looks like. And here's what I mean by that. So basically, large language models and multimodal models, in some sense ushered in the end of the ImageNet era. They posed new demands on the benchmarking paradigm. And we were left from this ImageNet era with some suspicion and concerns about the idea of a single task benchmark. Maybe it was just too narrow-minded to have just a single benchmark

• 2:18:18Speaker 09

  or a single task benchmark. Maybe that's not diverse enough. And so in response, people created a bunch of new multitask benchmarks. You have all these names: Superglue, MMLU, BigBench, Helm, and so on, with the hope that these multitask benchmarks will provide a more nuanced, holistic evaluation canvas for these new models. Okay? We were also left from this era with some skepticism about static benchmarks, the idea of just having this one test set frozen in time. And people have been experimenting with the idea of dynamic benchmarks in response. So benchmarks that evolve over time, that grow as you get different models. And so I will talk about each of these in turn. Multi-b benchmarks, multitask benchmarks first, and then dynamic benchmarks.

• 2:19:02Speaker 09

  And I'll give you some new perspectives on each of these. Okay? And so the first thing I'm going to talk about is a social choice perspective on multitask benchmarks, this joint work with Guan Hua Zhang. And we applied basically ideas from social choice to multitask benchmarks based on the following analogy. The analogy is between tasks and voters. So in a benchmark, basically tasks, different tasks act like voters. And they can vote on models. So models become candidates. And each task gives you a ranking of all models. Okay? So tasks are voters. They vote on models and they can rank these models. And if you think about it this way, then a benchmark is nothing other than a voting rule that has to aggregate all these different votes,

• 2:19:45Speaker 09

  all these different rankings into one ranking. Okay? That is the problem of social choice. And it suggests a distinction that's important here between cardinal benchmarks. These are benchmarks that aggregate numerical scores. Examples are BigBench, the OpenLLM Leaderboard on Hugging Face, and so on, where you just average out accuracy numbers. Those are cardinal benchmarks. And they stand in contrast with ordinal benchmarks. Ordinal benchmarks are just things that use the rankings and aggregate on the basis of the individual rankings. And an excellent example of an ordinal benchmark is Helm, which I'll say more about, okay? So what does Helm do? This is a fantastic new effort led by Percy Liang at Stanford to holistically evaluate language models.

• 2:20:30Speaker 09

  That's what it stands for. And interestingly, Helm is an example of an ordinal benchmark. Why is that? Because Helm works with what's called the winning rate. So it looks at how often models win against other models on these different tasks. And the winning rate is something you can compute just from individual rankings. If I know the individual task rankings, I can compute the winning rate for each model, and then I can rank by winning rate. And that makes Helm an ordinal benchmark, okay? You can compute the ranking on Helm from individual rankings only, only ranking information. That's what ordinal means, okay? So that makes it ordinal. In contrast, the OpenLLM Leaderboard averages out accuracy numbers, and that needs cardinal information, so numerical information to get the ranking.

• 2:21:15Speaker 09

  I should say, by the way, all of this, the entire talk is based on these fantastic contributions that the community has made in the benchmarking space. For instance, the OpenLLM Leaderboard is based on the ILuther evaluation harness. If you ever looked at that codebase, it's an enormous effort. It's a huge amount of work. I'm deeply grateful to all the work that people have done on this, and I really want to give people a shout and encourage that kind of work. It's essential for the community. I really, really appreciate it. Okay, so cardinal benchmarks, ordinal benchmarks. Those are the two things we're going to contrast. And one of the, you know, robust insights from, you know, social choice is that you have no perfect voting rules, okay? This is associated with Arrow's impossibility result. That's a famous result in that area. It says,

• 2:21:58Speaker 09

  You know, no voting rule can make you perfectly happy. Okay, all voting rules have some issues. And there are certain desirable properties that you can't all have simultaneously. What are these? Here's how I would state the theorem. I'll state it like this. I'll say that any diverse ordinal voting system is sensitive to irrelevant alternatives. What does diverse mean? I'll say, tell you in a minute. What is sensitive to irrelevant alternatives? It means something like a third candidate could enter the race and change, change the order of the top two contenders. Okay, you could have a weak contender, enter the race and perturb the top contending models. Of course, you don't want this, right? Imagine you upload a weak model to Helm and change the order of the top two models. That would be unfortunate.

• 2:22:41Speaker 09

  Diverse just means that your benchmark or your voting system is not a dictatorship, so it doesn't just project onto a single task or a single voter. It's Pareto efficient. If a candidate wins unanimously in every task, it should also win overall. And it's universal in that it doesn't limit how you rank. It accepts all rankings. So these are all reasonable things. You kind of want to have, you kind of want to have this. And then it says you have to have irrelevant alternatives. Okay? This theorem applies to ordinal voting systems. And as a, as a result, it directly translates to ordinal benchmarks. Okay, and it says adding irrelevant or weak models to an ordinal benchmark system can change the order of top contending models. Okay? But the issue with Arrow's impossibility result is that it's not quantitative.

• 2:23:26Speaker 09

  It doesn't tell you how much things could change and whether that's something practitioners need to worry about. And it only applies to ordinal systems. So it wouldn't tell you anything about all the cardinal systems that are out there. So what we did in this work is we proposed like an empirical variant of Arrow's impossibility result that implies to benchmarks, applies to benchmarks and both ordinal and cardinal benchmarks. And the key properties that we identified are sensitivity and diversity. What is sensitivity? It's just change in rankings due to irrelevant task transformations. Okay, so if you do changes to your task that shouldn't at all matter, does the ranking change? In the case of ordinal benchmarks, I already told you what that is. It means adding a weak model can flip top contending models.

• 2:24:11Speaker 09

  That's the case of an irrelevant change that shouldn't matter. And in the cardinal case, it's just monotone linear transformations of the metric. If you relabel your accuracy numbers from 75 to 80 and 80 to 85, you just do a change, a relabeling of numbers that shouldn't change, you know, the task. And in fact, every single individual task will be identical under such a transformation. It will give you the same kind of ranking, so it really shouldn't matter. Okay, so these are the relevant task transformations. And sensitivity is how much you change the ranking in response to these irrelevant task transformations. We can always minimize sensitivity by just having a single task benchmark, right? If you have a single task benchmark, there's only one ranking, there's no impossibility.

• 2:24:54Speaker 09

  Okay, we can always have a single task or copies of a single task and you minimize sensitivity. But the whole point of multitask benchmarks is to also have diversity, to have variance in rankings among different tasks. You want these rankings to be not all the same. You want the rankings to have diversity to tell you different things. And we just measure this with something called the Kendall's W coefficient of concordance. It's a standard measure of diversity or variance in rankings. Okay, and you want to have high diversity to have like a diverse multitask benchmark. And we show in this work that all existing multitask benchmarks exhibit a strong trade-off between diversity and sensitivity. So if you look at the 2D plot of diversity versus sensitivity,

• 2:25:37Speaker 09

  you'll see that all the existing benchmarks, this is the case of cardinal benchmarks, fall between a constant benchmark or on a line between the constant benchmark, which has a single fixed ranking, and a random benchmark, which just has a random ranking. And every benchmark strikes a trade-off somewhere between that. Okay. And so this says that diversity comes at the cost of sensitivity. If you want more diversity, you're going to have more sensitivity to irrelevant changes. Okay, all benchmarks fall on this line. And you can think of this line as sort of a measure of multitaskness. There's really just one dimension here. How much multitaskness do you want? And the more multitaskness you want, the more diversity you get, but also the more sensitivity you get. And you can sort of have high diversity without high sensitivity.

• 2:26:21Speaker 09

  As a sanity check, this was important. As a sanity check, we partitioned ImageNet into a mock-up multitask benchmark by just partitioning the classes into 20 random tasks and calling each partition or each part in the partition a different task. So we're creating a fake multitask benchmark. And this is what that dot that says ImageNet and the slides is, represents. It's just what happens if you create a fake multitask benchmark. And our measures correctly identify that this is a single task benchmark. Okay, so it's not better than a, it has no diversity and no sensitivity. Okay, it's still just a single task benchmark. But the main takeaway from this is that diversity comes at the cost of sensitivity. There's no free lunch in multitask benchmarks. You can make these benchmarks more diverse,

• 2:27:05Speaker 09

  but it's going to come at the cost of having very high sensitivity to irrelevant changes. Just to give you a different measure of sensitivity, on the left, we measured it in terms of Kendall's tau. That's like a measure of like difference in ranking. On the right panel, we measured it in terms of the maximum normalized rank change. Okay, so this is the fraction of ranks you can skip due to an irrelevant change. And you can see that, for instance, for Big Bench Heart, you can skip 80% of the ranks by some irrelevant transformation of the metric. And for MML, it's close to all ranks and so on. Okay, so you can really have very significant changes in the rankings with an irrelevant task transformation. For ordinal benchmarks, the situation looks similar, although a little bit more messy.

• 2:27:48Speaker 09

  Here we look at various subcategories of HEM and HIME. And you see also that there's this general trade-off. I should say that we always compute a lower bound on sensitivity, because it's kind of hard to compute exactly. So all these numbers are lower bounds. The sensitivity might always be higher. And so we're, you know, seeing a similar picture here. A little bit more messy, okay? Here's an illustration of what this actually looks like, the sensitivity to irrelevant changes. Here is how much you can perturb the rankings on these benchmarks. On the left is OpenLLM, on the right is HEM, by just doing irrelevant task transformations, okay? And you see that you can, the models can jump around quite a bit. And these rankings can have quite a lot of sensitivity. If you want to play around with this, you know, you can pip install BenchBench, you know,

• 2:28:33Speaker 09

  and, you know, play around with all these numbers. It makes it very easy to just load up your favorite benchmark, compute the diversity and sensitivity, and see where the benchmark falls, okay? We're also maintaining a website where we add these things and sort of display them, just to keep track of them. And if you want to contribute to that, please send us your benchmark. Make us aware of your benchmark. We would love to edit. Again, there's a little bit of computation involved in getting these numbers. We're happy to run that computation for you. Just reach out to us and we'll be happy to add this. So this is sort of an ongoing work in progress. All right. This is my take on multitask benchmarks. And with the last several minutes, I want to talk a little bit about dynamic benchmarks,

• 2:29:16Speaker 09

  which is another proposal of the polymorphic era that's gained quite a lot of traction. There was a really amazing and fascinating effort called DynaBench a few years ago, which was very ambitious. It was trying to really change the way we do benchmarking by proposing something called dynamic benchmarks and a platform to do these kind of dynamic benchmarks. And so a dynamic benchmark basically is as an evolving, time-dependent benchmark, where you have some initial data set. You let people build models on that initial data set. And then you use the models that people have built to find like failure cases of all the existing models. And you add those failure cases to your benchmark, okay? This is called adversarial data collection. So you learn from the models that were built.

• 2:29:59Speaker 09

  You add the failure cases to these, to your data set, and you continue. And so there's this ongoing like interaction between model builders and data collection, okay? And you interleave these two operations indefinitely, hoping that your models just keep getting better and better and better and keep accounting for more and more challenging instances. Okay? We wanted to know if this works. And it's something I did in a joint work with Ali Shirali and Radha Abbebe, where we proposed a theory of dynamic benchmarks, because it's so new and so different, we wanted to know, can this at all work? And how do we even think about what it means for that to work? And this is what we proposed in that paper. And we abstract a dynamic benchmark as a directed acyclic graph with four operations. So each node in the graph is one of the following four operations.

• 2:30:44Speaker 09

  You can do model building. You can let, you can invoke the community to do model building. You can look at the resulting models assembled in, or in some way, collect them. And then you do data collection on the resulting model. You let sort of the annotators find failure cases, and you create new data points, and you add them to your data set. So you pool data. Okay? These are the four operations in a dynamic benchmark. And subject to that, you have complete freedom. You could do whatever you want. Any, any directed acyclic graph is a, is a valid dynamic benchmark. The standard design that people have mostly implemented is, is this, the following is just a directed path. It's just a directed path that alternates between model building and adversarial data collection. So just these two operations alternate them.

• 2:31:28Speaker 09

  That's the standard design that people have mostly experimented with. And we prove a theorem that says that progress in the standard design can stall after a small number of rounds. Okay? So there's in principle, no reason to expect that the standard design gives you progress beyond a few number of rounds. And you can see this in some of the experiments, that progress really seems to plateau after just a number, a few number of rounds, and it becomes diminishing after that. And so we thought harder about this, and we came up with more sophisticated benchmark designs that guarantee strictly more progress. So we call them hierarchical dynamic benchmarks, where we sort of have parallel threads that you let run and merge in some particular ways. So these are more complicated benchmarks.

• 2:32:13Speaker 09

  We can prove that they guarantee more progress than the standard design, but they're also much harder to implement. So I'm actually not sure how feasible it would be to pull this off as a real benchmark. But it's certainly an intriguing possibility. Okay? So this is my take on dynamic benchmarks. So let me sum up and get to an end. How am I doing on time? Great, perfect. All right, so summing up here. What do we see? I use the benefit of hindsight to sort of do an ImageNet era retrospective. What did we learn from like 10 years of benchmarking on ImageNet? And the takeaway here, the main takeaway here is that the iron rule has both internal and external validity. So it works to a surprising extent in this case.

• 2:32:57Speaker 09

  But we know much more about the former and much less about the latter. So if you're interested in open problems, understanding this last part is really challenging, and we have made very little progress on that. So there's definitely more work needed here. And then maybe to me, what was really surprising is that good, human, like annotated, highly curated data is not necessarily, you know, required for ranking models by accuracy. So if your goal is just to do rankings, performance rankings by accuracy, you don't necessarily need super clean data. Okay, we can, we saw this empirically, but also we can do some theory about that. And this was surprising to me. I should qualify that, of course, if you're doing something like a fairness analysis, a safety analysis, bias analysis,

• 2:33:40Speaker 09

  you know, red teaming or alignment analysis, this is of course very different. Then your test cases in your data really matter substantively, and what you put in there is very important. And my group's certainly very invested in these research directions as well. But that's not what I talked about. In this talk, I was focused on the core sort of benchmarking enterprise and ranking, ranking models by accuracy. Okay, then I moved on to this new era where we don't have hindsight, where we're sort of figuring it out as we go along, and we're seeing what happens. And here, we saw that there's a strong trade-off for multitask benchmarks. And diversity, greater diversity in multitask benchmarks inherently comes at the cost of less stability. Okay, so you have more sensitivity to changes that shouldn't matter,

• 2:34:25Speaker 09

  and there's no free lunch for this kind of evaluation paradigm. And finally, dynamic benchmarks are intriguing. I personally find them extremely fascinating and intriguing. But currently, I don't think we quite know how to pull them off. And as we currently do it, progress might stall. And again, if you're looking for good research directions, it's a fascinating area to work on. And I encourage, you know, especially theoreticians to look at some of these benchmarking questions where the landscape is wide open. Okay, it's really like completely open space. Good. So this talk was about the emerging signs of benchmarks. And I argued that machine learning is the anything goes principle plus the iron rule. And somehow these two basic ingredients give us like a very powerful scientific machinery that seems to work.

• 2:35:09Speaker 09

  Okay, and I'd say that this community is extremely good at the anything goes part. Okay, so we've been extremely good at the anything goes. But that places all the burden on the second part, the iron rule. And that becomes sort of the the critical link here because it's really essential for making all of this work. And so what's somewhat challenging here is that our intuition about benchmarks can fail and often has failed. And certainly my own intuition about benchmarks has very often been wrong. And I sort of was corrected by the passage of time or theoretical results that overturned my intuition. And so this has convinced me that we really need sort of scientific foundations of the iron rule itself of like, how do we make sense of this benchmarking enterprise? And how do we build things that that sort of promote scientific progress?

• 2:35:54Speaker 09

  So I really think of this as like a major theoretical and empirical effort. It's not, you know, just a theory thing. It's also just an empirical effort to understand what collective practices of our community promote scientific progress. I think this is what this is ultimately about, and we should devote some more time and attention to that. And so I hope you'll join this effort. I hope you found something interesting in this talk. Please come chat with us. I'll give a shout out to the social foundations team. They are actually wearing the yellow sweaters today. I didn't think so because it's warm. But if you see us somewhere in the in the conference, please reach out and and we'll be happy to chat. With that, I'll I'll thank you all for your attention.

• 2:36:32Speaker 00

  That brings us to the end of section B, our selection of three papers and one keynote on benchmarking. Phew, that was a long but important topic. And if the AI engineer conference submissions were anything to go by, the topic of evils and benchmarking is only going to explode this year. We turn now to section C, which covers incremental papers and talks in reasoning and other post training elements. There were many, many more papers than we could fit in this category, but we will focus on a few important themes: rag, verification, and safety.

• 2:37:08Speaker 00

  First, let's start with the self-rag paper. This paper bears some similarity to the pause tokens paper we covered in ICLR Part 1, because it involves adding special retrieval and critique tokens. However, these tokens don't just improve raw reasoning. They explicitly support retrieval during the generation process, as well as evaluating source relevance, degree of statement support, and response utility. And they can be fine-tuned on top of pre-trained open models like Llama.

• 2:37:41Speaker 10

  Hi everyone, I'm Akali from University of Washington. I'm excited to present our work self-rag, which is a new framework to improve the standard RAG system by training and making an any language model to decide when to Retrieve, generate, and self-evaluate. Large language models are powerful, but there are many issues like hallucinations. Retrible augmented generation or RAG has shown to be quite effective to overcome those issues. Given the user's query, such as "where is ICRA 2024", we first retive a set of documents using a retrible system like Google Search or BM25, and then we augment the original language model input using those retrible documents and also the original question.

• 2:38:24Speaker 10

  The standard set of GPT-4 or Llama 3 can use those retrible documents and give more factual, up-to-date, and attributable answer.

• 2:38:35Speaker 10

  RAG has shown to be quite effective in many benchmarks, especially in question answering tasks. In a prior work, we conducted a large-scale analysis of 10 different language models, and we have seen that this inference time augmentation can give us significant improvement across many models ranging from 1.3 billion to 175 billion. RAG has been used in many real applications. For example, there are many language model-based search systems, Parplex AI or Bing Chat, or libraries that help you to build customized RAG pipeline. Well, RAG is super effective, there are many limitations. In this talk, I'd like to highlight two of their limitations: unreadability and also inefficiency.

• 2:39:16Speaker 10

  Now, let's ask another simple question: how did US states get their names? Unlike the previous question, where you can simply extract local information in given documents, in this question, you have to collect a set of documents and compose output based on those multiple documents. In standard language model, as in the previous example, we can concatenate those retrieved documents and then feed them together in language model. The output look plausible, but there are many factual errors here. So now, let's talk about why... Let me explain why those standard RAG systems maybe may not be perfect yet. First, even the current sort of systems can easily get distracted by many...

• 2:39:58Speaker 10

  when many documents are given, especially when some of the documents are irrelevant or unhelpful. Let's take a closer look at this example. The middle paragraph only states the history of Michigan, without saying anything about how this state got their name. Current RAG system can easily be affected by those unhelpful documents and can generate factually incorrect statements, as in this example. Due to this unhelpful document, the model states states such as New York and Michigan are named after an individual person, which is factually incorrect. Or conversely, current RAG system can ignore the context that provide useful information. Here, the first paragraph provides useful information, but the model can still generate hallucination.

• 2:40:39Speaker 10

  For example, it can say some states, including Utah and Washington, are named after indigenous communities.

• 2:40:46Speaker 10

  Another issue is that many of the existing RAG pipelines assume retrieval is almost always necessary and keep retrieving a fixed number of documents, even when the user query isn't using retrieval. It is estimated more than 60% of the user input into ChatGPT is for writing assistance or creative writing, which may not require factual grounding. Like this query: "Write an essay about US summer vacation." And as you can see, retrieving a set of documents talking about definition of summer vacation or movie title "Summer Vacation" doesn't make any sense here, but most of the standard RAG pipelines just retrieve a set of documents, as in other queries. So this causes additional latency, making the RAG system more inefficient.

• 2:41:31Speaker 10

  And also, it can hurt the model output, as shown in the previous slide. In this work, we introduced a new framework, self-regressive objective generation or self-reg. So FRAG introduced a novel inference and training pipeline to enhance the reliability, efficiency, and versatility of RAG systems. In self-reg inference, at a higher level, we make a language model to decide when to retain, and generate, and evaluate its own output. In particular, given a user query, the model first evaluates if we need to use retiva or not. If we don't need to retain, then the model just acts as a standard language model to avoid unnecessary retrieval. If the model thinks we need to retain, then the model retiva a set of documents, but instead of concatenate everything and feed them together in input space,

• 2:42:17Speaker 10

  we first ask the model to evaluate which documents are actually helpful. And then, the model generates output, but the model first also evaluates if the output is supported by those helpful citation or not. And at inference time, we prioritize outputs that are fully supported by the context. To achieve this iterative self-regressive pipeline, we conducted one of the RDS inspection tuning with retiva and trained an arbitrary language model with those self-regressive tokens. So now, let's dive into the details of self-reg inference using the same example. Given the same question, how did US state get the names? Self-reg can even start direct research generation to say like US state got the names from collaborative sources.

• 2:43:00Speaker 10

  But now, self-reg needed to generate the factual statements composing multiple documents. So the model outputs a special command retiva, and then we trigger retiva and retiva a set of documents from the data store.

• 2:43:15Speaker 10

  Instead of concatenating everything and feeding them together to a language model, in self-reg, we process multiple documents in parallel using batch decoding. This enables more efficient and scalable inference, but more importantly, this enables us to control model's behavior via self-regression. Specifically, we make the self-reg to predict, generate condition on a user input and also each paragraph. And for each paragraph, the model first predicts if the paragraph is related to the question or not. Then, the model keeps generating the output as in standard language model. In the first example, the model that provided paragraph is helpful, while in the second example, the paragraph only states the history of Michigan.

• 2:43:58Speaker 10

  So the model first generates irrelevant token and then keeps generating.

• 2:44:04Speaker 10

  After those generation, Serf trag also predict its output is fully supported by the related paragraph. For the first generation, model output is fully supported by the document one, saying that 11 states got their name from individual person. But for the last case, the paragraph only talks about Utah, while the model output mentioned Utah and Alabama. So this output is only partially supported. In that case, the model predict partially supported. So how do we achieve this retrievable and self-reactive feature? We equip language model with those ability by making the model to learn to predict those special tokens, which we call deflection tokens.

• 2:44:43Speaker 10

  In standard language model, we have a fixed set of vocabulary and the model assigns the highest token probability to the most plausible next token. In Serf trag, we expanded the original vocabulary using the special tokens, which include the token controlling retrievable or the token controlling the self-reactive feature, critic tokens. So if the model think we needed to use retrievable, then the model will generate the retrievable token from the expanded vocabulary. We can also adjust the threshold to balance or like call decide how frequently we use retrievable to balance the trade-off between retrievable frequency or efficiency and final performance.

• 2:45:22Speaker 10

  Critic tokens are also directly generated by the model, and we use those models to rank and choose the best K-output at the sentence level. In particular, in Serf trag, we conduct a sentence NLM search using a fine-grained feedback scores based on those critic tokens. We use a normalized token probabilities of ideal critic tokens such as relevant or supported as a fine-grained feedback and then compute the weighted sum of them as a summary score of each output. Here in this example, the first output is based on the helpful document and also is fully supported. So we give the highest score to this first output. But for the middle one, the generation is based on unhelpful document.

• 2:46:07Speaker 10

  So we assign the lowest score because output is more likely to include factor errors. Now let's discuss how we train Serf trag so that the model can effectively learn what to generate, when to generate such deflection tokens.

• 2:46:22Speaker 10

  Instead of flag, we train an arbitrary language model to generate similarity from both normal vocabulary and also special tokens. During training, we use another language model, which we called critic language model. And this critic language model teaches the generator language model learn to generate appropriate self-regulative tokens on the input.

• 2:46:43Speaker 10

  Critic language model is a seven-video language model trained to generate reflection tokens given evaluation instruction and input. For instance, here the evaluation instruction says that evaluate is output Y and to an input X is supported by the table document D. And also that set of input X, D, and Y. Here, the critic language model should predict the supported token. One challenge is that we do not have large-scale fine-grained annotations for necessity of retrieval or self-critic. And collecting human annotation at sentence level for multiple aspects could be quite expensive. To overcome those challenges, we generate synthetic training data by carefully prompting GPT-4 with instruction and demonstrations.

• 2:47:27Speaker 10

  After this, we train critic language model on this generated training data. And then, we use this critic language model and retrieval model to augment existing instruction training data, mimicking self-flag inference. Here, given an input-output pair, we first run critic language model to evaluate if we should retrieve. And then, if the model output retrieve is token, then we insert a retrieved passage inline with the reflection tokens, also predicted by the critic language model. We generated 150,000 instruction training data in this way.

• 2:48:03Speaker 10

  Now, we can simply train the generator language model on this augmented training data. So, given the input, the model learns to generate the standard output and also reflection tokens. We can use the standard language model training objective with expanded vocabulary. This enables us to easily apply the same training pipeline to new language model and also code bases. And also enables us to tailor model's behavior to diverse fine-grained preferences, as shown in previous slide, without additional training overhead.

• 2:48:34Speaker 10

  We are bodying SerfRAG on six diverse tasks, including close task, close set task, short home generation, and also long home generation.

• 2:48:44Speaker 10

  Our SerfRAG is based on Llama 2.7 and 13 billion, and trained on four GPUs. Now, let's discuss the result. As you can see, the baseline paramedic language model, without any retrieval, struggled in tasks requiring precise knowledge memorization, such as POPQA and open domain QA dataset, or ASQA, which is a long home QA dataset about factual knowledge. The standard RAG gives improvement on some tasks like POPQA, where we can simply extract information from a single document to answer. However, standard RAG still struggled, or doesn't give large performance improvement on tasks that demand systems to compose knowledge from multiple documents and generate, such as PubHealth or ASQA here.

• 2:49:28Speaker 10

  Moreover, especially open access language models, such as Llama 2, 13 billion, chat pre-trained, or chat model, struggle to obtain good citation precision or recall, indicating that even the model cites something, the model output and likely to be not supported by those citations.

• 2:49:48Speaker 10

  SerfRAG significantly improved out of home such models, and obtained the best performance across all of the baseline in the same model scale. Moreover, SerfRAG shows much better citation precision than B-Code.

• 2:50:02Speaker 10

  SerfRAG even matches over out of home chat GPT on five out of six tasks, despite being relatively small seven or 13 billion model, trained on a small set of instruction training data.

• 2:50:14Speaker 10

  SerfRAG was initially introduced last October, and since then, SerfRAG has been widely used in both academic papers and also industry applications. For example, SerfRAG has been successfully integrated into multiple widely-used RAG libraries, such as Long Chain or Llama Index. And also, there are many papers that tried to improve SerfRAG for further reliability, or apply SerfRAG to new domains, especially security critical domains like biomedical domains. In summary, we introduced a new SerfRAG framework, which helps us to build a more reliable, efficient, and versatile RAG systems. We open sourced the code and model checkpoint, and I'm happy to answer any question now. And also, please come to our poster in 63. Thank you.

• 2:51:00Speaker 00

  The self rag paper mentions training three models, a retriever, a critic and a generator model. The role of the critic model and the critique tokens is surprisingly similar to the ideas of this next paper from OpenAI. Let's verify step by step. Here they pursue more formally verifiable correctness in math problems and find that process supervision, which is what self rag is doing, soundly beats outcome supervision.

• 2:51:26Speaker 11

  The high level goal here is to train really reliable reward models to gray different math problems. We want to have reward models that can look over a huge number of solutions and pick out the one that actually managed to solve the problem correctly. And so we're comparing two different methods, what we call process supervision and outcome supervision. Outcome supervision basically means giving feedback to the model just based on whether it reached the correct or incorrect answer at the end. And process supervision is giving granular feedback for each individual step of the problem and specifically saying whether each step is correct or incorrect. And the hope with process supervision is that it's solving a hard credit assignment

• 2:52:06Speaker 11

  problem that outcome supervision would have to solve by directly giving it feedback on whether each step is correct or incorrect. So, you know, outcome supervision has to like somehow infer where the bad step happened, but with process supervision, we get humans to just directly specify where that happens. So we collect a huge amount of data from contractors who are just going through and labeling each individual step. And then we use that to train the process supervised reward model and the outcome supervised reward model, we just train to predict the correctness of the final answer. And then we can bear these two reward models and we find that the process supervised one is significantly more reliable.

• 2:52:44Speaker 05

  And it's specifically on the math domain?

• 2:52:46Speaker 11

  Specifically math, yeah.

• 2:52:47Speaker 05

  How often do you find that the reward model is wrong and it's like kind of guiding it off track rather than on track?

• 2:52:54Speaker 11

  We are using the same generator model throughout this, so we're not actually, we don't actually ever update the generator. We're just using the reward models to basically search and test time to search over a large number of solutions. So, I mean, both the outcome and process. Supervised reward models are imperfect and like get tripped up by things. I think the outcome supervised reward model gets basically, I would say pays a little bit less attention to detail. Like it's more likely that a solution that looks superficially good, but like has some subtle mistakes will still be rated highly by the outcome supervised reward model. Whereas the process supervised reward model is a bit better at spotting like subtle errors, I would say. But both of them can sort of, both of them can say can get fooled by a solution that like kind of looks like it's good, but you know, isn't quite right.

• 2:53:38Speaker 05

  Like what is the backstory behind this? Like, why is this an interesting area of research? Or were you, were you actually investigating something else and you found this to be a blocker for you?

• 2:53:46Speaker 11

  So yeah, I mean broadly speaking, we're just interested in pushing on the reasoning abilities of these large language models. Reasoning is something that LLMs have struggled with a lot, you know, and we, so we're just excited to push back that frontier however we can.

• 2:53:59Speaker 05

  Do you have a definition of reasoning beyond like, just like this math, step by step reasoning?

• 2:54:05Speaker 11

  I, I think it's hard to give like a, yeah, a, a great definition to what like defines reasoning. I mean, to us, it's just, it's things that it's kind of whatever passes our vibe checks, but obviously things like math and STEM and code. But yeah, I don't have a good like formal definition of reasoning, I guess.

• 2:54:20Speaker 05

  I noticed the search space was actually pretty big. You had like thousands of solutions per problem. Was that necessary?

• 2:54:26Speaker 11

  Well, it's useful to show that, you know, if we only took like a hundred solutions here, we wouldn't see as much of a difference between outcome and process supervision. But if we take, you know, many thousands, we see much more of a difference. And yeah, I mean, it's useful to, to explore this regime because, you know, if, if all it took to solve really, really hard problems was taking a huge number of samples, then we'd be very happy, you know, we'd be willing to spend the compute to, to, to generate all these solutions. If, if we could always pick out the one that was like really good. And so, yeah, so it's not like strictly necessary to go here, but it just makes it clear sort of what, what the trends look like.

• 2:54:59Speaker 05

  How do you envision other people should use this?

• 2:55:01Speaker 11

  Once you have a really good supervised reward model at like a larger scale, you can use it to explore labeling data for smaller scale models.

• 2:55:08Speaker 05

  Like to bootstrap or teach?

• 2:55:09Speaker 11

  Yeah.

• 2:55:10Speaker 05

  Okay.

• 2:55:10Speaker 11

  And so, so what we did here is we kind of, we were able to do some ablations on outcome and process supervision by using GPT-4 to label these smaller model samples. And, you know, here we collected basically like order of magnitude more human data than we did at the large scale. And so by like having, just having the larger model provide the labels. And those kind of experiments wouldn't have been possible if we had to actually rely on humans. That's, that's one possible use for the dataset. We really submit also just like if, if anyone else can, is able to train these process supervised reward models and maybe see something we missed.

• 2:55:44Speaker 05

  Well, that's pretty positive, I think, for the community. Like, what's the relevant literature? Like, what's the inspiration, I guess, or relevant literature for process supervised reward models? Like, is, is there anything that you point people to, to read up?

• 2:55:57Speaker 11

  From an alignment perspective, some people are excited about process supervision just because, you know, it's sort of, outcome supervision is providing like less direct feedback. And so it's, you know, potentially not as good from an alignment perspective. You, we, you don't really know what you're reinforcing if you just give the model like a positive or negative reward at the end of some long trajectory. But with process supervision, you know, you're kind of much more directly reinforcing the things you care about. There's, I mean, there's, there's some like blog posts and stuff that we, that we cite in the paper, but they don't have anything to like specifically call out.

• 2:56:30Speaker 05

  This seems like the inverse of this is weak to strong generalization. Is there any parallels with here, here's kind of strong to weak? And then you started working on weak to strong?

• 2:56:39Speaker 11

  Strong, strong to weak in the sense, like in terms of experiments I was talking about here? Yeah, I mean, I'm sure there are connections. We didn't, okay, so you can take, you can take like a small process supervised reward model and still, and try and have it evaluate like larger model samples. And it'll work to some extent, but. Yeah, I mean, that's definitely something you could look at more. We didn't push very hard in that direction, though.

• 2:56:59Speaker 05

  I was just kind of curious if there was some kind of parallel in there. Cool. I think that's it. Thank you.

• 2:57:03Speaker 11

  Great, thanks.

• 2:57:04Speaker 00

  If you listen closely, OpenAI's focus on verification and process supervision as a path toward better reasoning, aka GPT-5, is by now clear as day. In the closing workshops of ICLR, we even managed to catch a little bit of Noam Brown's talk expanding on what he calls the generator verifier gap. We last discussed his work with regards to our code interpreter is GPT-4.5 post from a year ago. He now draws a lot of inspiration from the AlphaGo paper, and in short, suggests that the best path to improve reasoning in generative AI, if we have sufficiently good verifiers of their output, preferably in process as suggested by Verify Step-by-Step.

• 2:57:50Speaker 00

  Here's an incomplete snippet of the talk leading into his discussion of the Verify Step-by-Step paper.

• 2:57:56Speaker 12

  Okay, so how can we take advantage of generator verifier gaps and update this? Oh, sorry. Before I get to that, one reasonable question that I'm sure a lot of, I've been asking a lot, I'm sure a lot of you are wondering right now, is, okay, if we have this generator, if we have this really good verifier, are we able to like generate a bunch of solutions and filter out the ones that aren't good and the ones that is good? Can we just like update the generator with the verified solutions? And in principle, this is a good idea, and it's actually what AlphaGo does. That's like how AlphaGo is trained. That says, there's a couple of caveats. It doesn't capture all the benefits of using a verifier. So for example, if I go back to the AlphaGo slide. So you can see like the gray bar is the performance of AlphaGo 0 if it was trained with multijet research and then doesn't use multijet research at test.

• 2:58:42Speaker 12

  So it's like, so it's updating the generator with the verified outcomes of multijet research, but it's just not using a test. And you can see that there's still this massive Elo gap, you know, it's 3,000 versus 5,200. So you're still getting a huge performance improvement by using this verification technique at test. So just relying on updating the generator is not going to be enough to overcome this gap. And the second thing is that you're still bottlenecked by the quality of the verifier. That's something that you can't use in the generator.

• 2:59:13Speaker 12

  So I think that for the purposes of this talk, I'll say that updating the generator is just going to be out of scope for this talk. And I just want to focus on, let's say we have a fixed generator and we have a verifier, what can we do? Okay, so the first thing that's really straightforward that we could do is called consensus. And in consensus, you just like generate a bunch of solutions and take the one that's the most common. Very straightforward. It's actually really convenient because you don't even need a verifier for this technique. You kind of think of this as similar to sampling at low temperature, but it's not exactly the same because, you know, if you have a sequence, then you're not sampling like every single step of that sequence at low temperature. You're sampling like the final outcome at low temperature.

• 2:59:54Speaker 12

  And I think a lot of people underestimate just how much benefit you can get from things like consensus. So for example, some of you might have heard about the NERVA paper. This came out about two years ago. And they got up to over 50% on the math benchmark, which is a very difficult math benchmark that's why they're used today. But NERVA got over 50% in large part due to using consensus. They generated 1,000 samples and then they took the most common answer from those 1,000. And that part of doing this consensus got them from 33.6% accuracy to 50.3% accuracy. So that's a big jump. Something else you can do is best-of-that. So best-of-that, the idea is you just sample all the solutions and then you score them with a reward model.

• 3:00:39Speaker 12

  And you return the one that looks best. And the key here is that you have to have a good enough reward model. If you don't, you're not going to beat consensus. But if you do, you can actually get a bigger reward for consensus. And ultimately, this technique is like by the quality of your reward model. If your reward model is not very good and you take a modest sample, so then you're going to end up overfitting to the errors in your reward model.

• 3:01:01Speaker 12

  So here, for example, this is a figure from a paper that came out in 2021. On the x-axis, you have, so this is testing on the GSM 8K dataset. And you can see on the y-axis, you have the accuracy, the pass rate. And on the x-axis, you have like the number of samples you've invested in. So for example, if it says like 100, that means that you're taking 100 samples, 100 generations from the generator. And then you're feeding those into a verifier that's trained to tell if whether an answer is correct or incorrect. And taking the one that the verifier thinks is most likely to be correct. Now, if you're taking up to 400 samples, you're seeing actually a pretty large improvement. I mean, this isn't actually each model, so in fact you're seeing an improvement at all, so it's quite significant. But you are seeing like a substantial improvement as you go up to 400,

• 3:01:45Speaker 12

  but then past 400, you're actually seeing the performance degrade because you end up overfitting. If there's errors to this verifier, and if you take too many samples and ask it to score which one is best, it could actually just return that it's wrong, but hacks the verifier basically.

• 3:02:04Speaker 12

  Okay, so can we do better than these methods? So, one of the things I wanted to talk about is process reward models. So this is a paper that's published at this conference. We actually turned on an archive about a year ago. And actually, I was not on this paper. This was with my teammates. My teammates wrote this paper before I ended up joining the team. But I thought it'd be a good thing to illustrate that there's like a lot that can be done in this space relying on verifiers and their performance.

• 3:02:31Speaker 12

  So the basic idea of process reward models over outcome reward models, where you're simply just doing best of end, is that we're going to verify every step individually rather than verifying the entire sample. So for example, we have this question, x squared equals 4, what's x? So typically, if you look at how best of end was done with verifiers in the past, it would just like generate the whole solution, and then you would ask the verifier, is this whole solution correct? And that puts a lot of burden on the verifier to have to like consider the entire solution all at once. And with process reward models, you instead break it down by steps, and you ask the verifier, is this individual step correct? And some steps might be correct, some steps might be incorrect.

• 3:03:15Speaker 12

  So you can see here, for example, it's like going from x squared equals 4, what's x? Well, if your next step is x plus x equals 4, then it's going to recognize that that's incorrect and that one is correct. And so then the process reward model will take all those steps into consideration, will score all those steps into consideration when it's scoring the entire sample.

• 3:03:35Speaker 12

  Now, the way we trained this reward model is by collecting a lot of human data. So we had a bunch of human annotators that would go through generations of the model and label each step as either correct or incorrect. So here, for example, there's a question, and then there's a bunch of steps. The first step is, let's call the numerator x, and the next step is the denominator is three x minus seven. And the human annotator would go through each of those steps and give it like a green smiley basically, it was a correct statement, and a red flag if it was incorrect, given what's happened before. So for example, you can see the last step, it go from five x equals six x minus fourteen to so x equals seven, and that step is incorrect, so it's a correct or incorrect.

• 3:04:20Speaker 12

  There's also this option for a neutral base, which basically means that it is not exactly incorrect, but it doesn't really like do anything. It's just like a step that's kind of like a holding pattern. Okay, so how does the performance look of outcome versus process supervision? So the baseline that we compared against is using just like a verifier that takes the entire sample all at once. And we call these outcome reward models. So we trained the outcome reward model on a giant training set. So the math data set has a training set and has a test set. We actually moved a lot of the test set problems into the training set, and we basically made the test set smaller. So we reduced the test set to only 500 problems, then we can have more problems to train on.

• 3:05:04Speaker 12

  And this actually does pretty well. So if you do best event with this outcome reward model that was trained on the math data set, you can get GPT-4 to correct against 72.4% of the test set correctly. And that's better than consensus. Consensus is this technique that I mentioned before that doesn't rely on a verifier at all, you just like take a bunch of samples and see which ones are the most common. Consensus gets you to 69.6%. For the process reward model, we collected a million step label labels across 100,000 distributions, and this actually ended up doing a lot better. This ended up getting to 78.2% on math. And I think this is still either state of the art or closest state of the art for math.

• 3:05:45Speaker 12

  One of the reasons why PRMs are so good is because they provide a denser verification signal. It's really difficult if you have a difficult math problem to verify whether the whole thing is correct. It's certainly difficult if you just look at the final answer. But being able to look at each of these steps in good way gives you a denser verification signal that makes it easier to verify. So this is what the plot looks like. The gray line is consensus, also called majority voting. The blue line is outcome supervision. So that's the verifier that takes the whole sample, the whole trajectory, into consideration. And then the orange line is process supervision.

• 3:06:29Speaker 12

  So on the y-axis, we have the pass rate, the success rate on math. And on the x-axis, we have the number of solutions per problem.

• 3:06:38Speaker 12

  And you can see also that the performance continues to improve as we increase the number of solutions per problem. We sampled up to, I think, 1,600 for this paper, but it looks like if you just keep going further, the number keeps going up. Eventually, that's going to plateau. At some point, it's unclear at what point it'll plateau. So here are some samples. The print is a little small, but basically, there's this problem, and then the verifier goes through each of the steps. And the more green it is, the more the verifier thinks it's correct. And on the left, we have, if you just use the outcome of the model, so that's if it's grading the entire sample. And in that case, it incorrectly labels it as correct.

• 3:07:24Speaker 12

  And on the right, we have the PRM. And you can see that there's some steps where it's a little machine. So for example, the second step, it's saying it's probably correct, but it could be better. And then there's certain steps where it's very confident that those are incorrect, and those are in red. And so it's able to recognize that the final answer is incorrect.

• 3:07:43Speaker 00

  We apologize for the poor audio conditions of the clip as it reflects the less-than-optimal nature of the workshop room. His summary makes a lot more sense with visual aid reference to the "Let's verify step-by-step" paper. And we have included the two relevant charts in the show notes for his talk. Adjacent to the topic of models grading other models, we have another talk from an OpenAI team lead, Lilian Wang, Head of Safety Systems at OpenAI. She gave a fantastic overview of the often overlooked safety-oriented mitigations that OpenAI does for their models across four stages: pre-training, post-training, inference, and evaluation.

• 3:08:26Speaker 00

  We would particularly highlight the OpenAI model spec, which lists objectives, rules, and defaults that her team designs for, and the instruction hierarchy, a recently published paper that details how OpenAI defines five privilege levels that they then fine-tune for resolving conflicting instructions.

• 3:08:45Speaker 13

  Thanks, everyone, and thanks for inviting me here. I would say my talk today might be slightly different from others because I would like to give a higher-level overview of how we're building the safety system for deploying, like, cutting-edge, the best deep learning models in the real world, end-to-end. So I'll probably touch on a lot of different things, a little bit on the surface because it's hard to go deep into every step. I hope you can have a concept of how complicated the problem is, also how much possible methods and mitigation you can apply when you're facing the real-world challenges.

• 3:09:26Speaker 13

  A bit about myself, I joined OpenAI about more than six years ago. I initially worked on robotics, then applied research, and recently I started leading this new team called Safety System. We're only end-to-end safety stack at OpenAI, so basically, all the models that deploy to the real world will go through us. Our team is dedicated to ensure the safety, robustness, and reliability of AI models and their deployment in the real world. I, so you can think of all our, we're dealing with a lot of practical safety alignment issues. There's endless collection of problems. But the good thing is we have access to different stack in the system, so it can be pretty creative.

• 3:10:11Speaker 13

  Safety is at the very core to OpenAI's mission. If you look at the mission, our charter online, you will find our goal is we want to build safe and beneficial AGI and even also deploy that to the real world to benefit humanity. I know this is a very big statement, so I hope after my talk you will get some concrete ideas of how the process is.

• 3:10:37Speaker 13

  And one important point my team believe in is safe AI cannot be built in the lab and how to deploy a powerful model and essentially AGI needs consistent learning, improvement, researching the real world. So we really embrace the idea of iterative deployment in the process. Because I often find that avocero in the real world are so much more creative than our researchers. I would say different people may have different interpretation of what is safety. So I want to talk a bit about the concept and what's the goal of we try to achieve here first.

• 3:11:19Speaker 13

  First of all, we expect the model output to not contain any harm to people, including physical harm, mental, financial or reputation harm. We hope the model is trustworthy, inclusive, but also respect privacy of people. At OpenAI, we want to build products that support beneficial use that have been set for certain harmful requests, even that has utility values. We train the model to refuse those requests. And there should exist mitigation either within the model or in the system around the model for adversarial use cases. For example, some people use a model to enable fraud, scam, or do adversarial persuasion of people.

• 3:12:03Speaker 13

  This kind of use cases might not be easily identified if you only look at the model, like a single conversation based on the model inputs and outputs. So we do need system level, system level monitoring or mitigation to identify that. Also, we want to make sure our model is, or our system is robust, reliable, even when people try to adversarially attack it. I, I want to mention Goody too, because this is probably the most interesting model I run into lately. When people think about safety or responsible and models, you can go to a very extreme.

• 3:12:45Speaker 13

  You can't make the model hundred percent safe, but it's useless. In this case, I found this model is extremely robust and I cannot really make the model to answer any of my question. It will always find some way to say, your request has concerns and I will just not answer that. But if you look, only look at the scale or measurement of how safe this model is, it's perfect, but it's useless. So I really want to emphasize that it's important to evaluate capability and safety at the same time and try to balance how useful it is and how, how safe it is. That Goody two model is such example.

• 3:13:26Speaker 13

  It refuse hundred percent time, but it does matter at that point.

• 3:13:33Speaker 13

  And I, I also want to emphasize that these two goals are not contradictory. I usually consider safety as one capability goal or multiple capability goal uh among a set of rich evals you can optimize. So there is always, there'll always be trade off between different thing you try to optimize and safety is just one of them. I, I kind of mentioned this before. I firmly believe safety should not only deeply build into the model, but also incorporate into every stage of the model training, deployment and leverage a lot of things that may not happen during the training process. And we are doing this at OpenAI.

• 3:14:17Speaker 13

  We have all kinds of different safety mitigation at every stage. Before the pre-training step.

• 3:14:26Speaker 13

  During the post-training, we do a lot of alignment, adjustment of the model behavior. Once we get to production at inference time or on the system level. We also have quite a lot of methodology we can apply. And after a model is deployed, we do very consistent evaluation, reteaming, monitoring different use cases. And make sure like all those feedback, eventually feedback into all the previous step. So this process continues. We embrace iterative deployment. We value data flywheel because the reward is always full of interesting challenges. And you know, I thought part of the research is probably find the right problem to solve.

• 3:15:10Speaker 13

  Or in this case, like we don't need to think that hard. So it's convenient. Okay, next I'm gonna do a slightly deep dive into post-training stage and system level a little bit eval. Just to give an overview of what we have down. But post-training, it's probably one of the most powerful, powerful tools we can use. We align, we try to align the model behavior with our safety policy. And I believe everybody here knows the concept of reinforcement learning from human feedback. You train a reward model based on the pairwise comparison from your like human annotator.

• 3:15:51Speaker 13

  Then you train a model to give you a scalable reward and then use that during the RL training process. So it's a fairly standard thing. We also use this process. And on the safety side, besides all the capability reward. We use a rubes reward to adjust model behavior to follow the policy we defined. So we first came up with set of policy taxonomy to define how the model should behave on safety topic. For example, if the model, if someone asks a model to.

• 3:16:26Speaker 13

  Tell me how to commit a crime, how to harm someone, how to build a bomb. The model should refuse. So we have a very detailed definition of what kind of topic should be refused. And there also another set of topic can be risky, but shouldn't be refused. Like if people show mental health issue, show suicidal thoughts. The model should be very careful and handle the topic like, like very, very, like in a specific way. Safety rubric reward model is very, it's, it's, um, it's a very simple thing. It's just a zero-shot GPT four classifier. Uh, we have a relatively lengthy, um,

• 3:17:08Speaker 13

  definition of, uh, what are the topics and how the model should behave. We have a set of very detailed human written rubric about both the style and the, the content itself. For each, for each sample which contains optionally the prompt, the model output and the rubric will classify the output into four category. Like is it refused, is it in the, in the design style? We intentionally train the model to be very concise, not to be preachy. If it's correctly refused, but in incorrect style, we give it reverse slightly plus. Is it actually content, is allowed content, that's total failure?

• 3:17:52Speaker 13

  Or if the, it give a safe, not refusal response, but also acceptable. So based on different style and the category of the prompt, we will give like assign different reward score during the PPO training. This is one example of how the, our prompts looks like. So it's a like fairly lengthy multi-label classification problem. But because it's a zero-shot prompt, so it's very easy for us to iterate over this definition and rubric and very interpretable of what we want the model to do. If you're playing with ChatGPT today, you will see the model will say, I'm sorry, I cannot help you. And this is a desired refusal.

• 3:18:35Speaker 13

  I mean, people don't want to see a very verbose model. If it didn't, it's already refusing you already. But if I ask some contents a bit sensitive, the model will provide slightly more context.

• 3:18:50Speaker 13

  And for even more sensitive topic, we intentionally encourage the model to say, to say things like, try to consult a professional, try to see your doctor. Yes, so similarly with refusal, we we also care about the utility, the helpfulness. Well, we try to keep balance between refusal and over refusal. And often the time, the over refusal was triggered by like boundary cases or the model over generalize different category. So we need to tune the rewards in a way that the boundary cases can be correctly answered. A longer term direction is we want to train the model to be configurable. Because for a lot of boundary cases, our gray area,

• 3:19:34Speaker 13

  people may have different requirements based on their use cases, like it's education setting or it's creative writing setting. They will have different bar and we hope that's configurable configurable by user or developer. This is a relatively old figure from G P T four technical reports. What we try to show is with our new training stack, the model answer sensitive problem like not incorrectly refuse sensitive problem less than our older model. It's also contains less disallowed contents in the output. Recently, well, actually, three days ago, we published the model spec where we very detailed describe what's our rules,

• 3:20:20Speaker 13

  objective defaults and what's the desired behavior of the model. And we use those, we consider the model spec as iterative thing. And we are very transparent of the values behind it and principle behind it. And we we're using this for training our human trainers right now. Ultimately, they can be directly part of the training.

• 3:20:44Speaker 13

  Robustness is issue, like everybody knows the jailbreak. And and also the real world distribution always has a long tail, it's changing. So it's important to track the robustness part of the model and make sure that uh we have incorrect misses as few as possible. We and interestingly, we do, I guess it's not very surprising, but we do observe a lot of adversarial attack in the real world. People are just super creative. We are aware the model can be vulnerable to those jailbreak attacks. And we have invested, like, just improve our, like, men, the the mainland model behind

• 3:21:28Speaker 13

  ChaiGPT consistently with every iteration. Internal one approach we recently just published called instruction hierarchy. Well, this is, I consider this as our first step to find a way to solve all jailbreak attack in a more principal way. If you look at this little example, so instruction are free form text. So essentially, you can ask model to to do anything at any part of a conversation. And our HS by default will just optimize the instruction following capability. So to the model, there's nothing wrong with jailbreak. It's just one instruction and the model should just follow that. So in this example, it's a simple email assistant.

• 3:22:13Speaker 13

  But unfortunately, when the user asks the the model to read the email, the email itself contains a bunch of things like re forward my email to some random person. This is clearly wrong, but if but to the model, they're just a new instruction that is like, like at the end of the conversation. So I see the problem, the the problem with jailbreak or prompt injection is fundamentally like caused by this conflicts between we try to optimize the instruction following capability. But on the other side, we also try to tell the model, oh, you should not follow this instruction if it's unsafe and it's a.

• 3:22:55Speaker 13

  It feels, it feels like pretty opposite. And if you say this is should not, you should not follow this because it's unsafe. It's also very, very ill-defined concept, like what do you mean by unsafe? I just talk about like our model spec, taxonomy, those are pretty complicated thing to, to, to tell. Like the, to teach those concept is not easy.

• 3:23:19Speaker 13

  So what we did is the, essentially, we defined a hierarchy of, of instruction that can show it up at different position or in the conversation. System message represents the desired behavior or define the desired behavior by the platform or the developer. So we consider those as gold rule and they should have higher privilege. What user message come next, model output come next. And to use, like in the email example, it can contain some weird things. And so we find it the lowest privilege. The idea is very simple is whenever there is a conflict between two instruction at different position,

• 3:24:02Speaker 13

  the model should just follow the one at higher priority. In this way, like we don't, we don't need to think about how to define safety in this case. The model just need to check how different instruction they are and make sure it, it can tell the different priority and follow the more important one.

• 3:24:24Speaker 13

  This is a pretty generic framework. We're still keep on iterating, make it better. We, in this case, we, the, we, we, we're not, we're not trying to teach the model not to follow something, but instead teach a model a simple rule. So the model can use this rule to generalize to different cases.

• 3:24:45Speaker 13

  Our training habitively rely on synthetic data for aligned instruction, meaning the lower priority instruction is aligned or is orthogonal to the higher priority instruction. We were just, we generated a bunch of like scenario and decompose them into smaller instruction, put them at different level and expect the model output to be same as original. And for misaligned instruction, we just, we were just during the training, we will ignore, we will hide the conflict one.

• 3:25:24Speaker 13

  And ask the model to generate something. And then during the training, insert this conflating instruction in the conversations. Model which is naturally learn to ignore that.

• 3:25:35Speaker 13

  We evaluated our model on set of different evaluation, including internal version, some public, D M I eval. And we should, we show that the model is much better at being robust to only follow the higher level instruction. And also, we didn't, during the training, we didn't, we intentionally didn't include to jailbreak examples, but try to follow on the hierarchy. So that's about like safety concept. And we find that this framework get generalized to a lot of jailbreak evals and make the metrics better as well.

• 3:26:16Speaker 13

  At inference time, I will start by talking about the moderation. We have, I consider moderation layer as something surrounding the model. It's quite useful at all different stages. For example, we use it to filter out toxic content from pre-training data. We use this to identify different type of harmful prompt so that we know what's the desired behavior of model output. We also use it to monitor production traffic. This is a good way for us to source adversarial or malicious use cases from the production traffic. So we can like learn from that, get insights and get the feedback back into the, the whole, whole lifecycle.

• 3:27:00Speaker 13

  I won't get into too much details in this figure. We have a paper on it, but it's, I think it's a pretty standard pipeline for training a classifier. I only want to mention two things. One is for cold star problem, we used domain adversary training. Because initially, we have a lot of like, uh, datas from different distribution. And we want to make sure we can actually utilize some of the data that's being labeled under different tax on me, like from public data set. We also heavily use synthetic data to enrich rare category.

• 3:27:39Speaker 13

  During our continued improvement, we use active learning pipeline from the production traffic, we do human red teaming, we try to find overfitted key token sequence. Of course, there is a lot of work around the the actual training data set, the quality data models. When we're serving the model, we are basic, we use moderation data at all stages, but we do have a public facing endpoints where we offer to all the researcher developer for free at a pretty generous rate limit. We also update that very frequently, like improve the general accuracy performance across all the category, sometime we also introduce new category.

• 3:28:24Speaker 13

  Currently, the team is actively working on to make a support multimodality, so it will come up in next couple of months. We also use moderation model in chat GPT. If you're ask, you'll try to ask some weird question, the model may block and give you warning, sometimes block, sometime it will just give you warning based on the on the thematic. One thing is quite important, I'm not sure how many people are familiar with, but if you ever work on like moderation or on safe content detection, it's quite tricky to come up with taxonomy, because it's not like math or programming, you can write a binary unit

• 3:29:07Speaker 13

  test to tell is correct or not, there are a lot of soft concept that you need to really well define in order to align people, like even internally, we often argue with each other and couldn't agree whether something violates like certain labels or not. So what we found is only providing high level principle, like constitution AI, it's not, it's not enough, we really need to define. Some like very, very non-trivial, non-trivial definition in order to align people. Only in this way you can get like very high quality data. Not, not, not say training, just even for getting high quality eval data is necessary.

• 3:29:52Speaker 13

  Like for example, if you want to distinguish hateful and harassment, it really depends on whether the uh attack attributes belongs to a protected class. Or if you want to separate when people express a self-harm intent versus just describe a self-harm intent of like a third party. It's on surface, it's not similar, but the model have to do, I actually need to respond in very, very different way. Traditionally, the process is very slow. What we set up is we heavily use the GPT for model to give our feedback, like use the model to label things. And then we will analysis which mistake the model use.

• 3:30:35Speaker 13

  And we will know, oh, the model may mistake because I didn't give this definition, or there is a gap in my taxonomy, then I can quickly fix that and run through the pipeline and do another round of analysis. So traditionally, this process took month easily, but for us it's a few hours. Uh we also tested how good the GPT form labeling is when we provide this policy taxonomy compared with the human, human labelers. We found that with likely trained human labelers, the model is on par better. But for very like intensely trained annotator that we have give a lot of rounds of feedback, there's still a gap.

• 3:31:21Speaker 13

  So GPT is a new product we launched last year. And a fun part of GPT is a hat, you can provide action while the model can call third-party API to do something. And this introduce a new set of like safety and security concerns. Um there's still a lot of things to figure out, but our first attempt is to make sure when the model try to do something, we have some confirmation, confirmation set up and also tell the user what the model tried to do, what kind of data the model will share with the third party. Uh so this is a like a lightweighted system level that is, it helps.

• 3:32:03Speaker 13

  After deployment, we do still, we run like intense evaluation consistently. We have monitoring set up in production and also explore different ways for red teaming, both from humans and model. A human red teaming network is something we shared, I think, early last year. Well, we are called for experts all around the world to see whether you're interested and help us to red team the model, find uh corner cases, mistake, share that with us so we can fix the model. Uh, we are working with quite a number of experts all around the world. I would say as a model becomes stronger, more capable, we really need people with

• 3:32:48Speaker 13

  in certain professional verticals and with expert knowledge in order to tell the mistake. Another project that is in progress is we try to use model to do the red teaming. I don't think this is a new concept, but if we train the model to come up, like for example, rewrite certain prompt or come up with attack prompts that's different from the other one, it's helping increase the diversity. It will, you can quickly iterate through this process and and and do the like training of both attack model and defense model together and make the base model more robust.

• 3:33:27Speaker 13

  Okay, we, this is like my, okay, I have only two slides left, so I want to do a quick summary. What I want to, what I hope you can get the message after this talk is, I believe we need a systematic approach to deploy AI models or AGM model eventually. We need to balance usability and safety and we shouldn't con- consider them as completing goals. We should learn from the real world because you will find the most interesting and adversarial cases in the real world. And we should embrace both model level and system level mitigation when we're dealing with real challenges.

• 3:34:09Speaker 13

  We also should embrace automation and try to use a model to solve AI safety problems as much as we can because we will get the best efficiency out of this process. Um, the last one is designing the ideal model behavior, define the very clear but also concise policy taxonomy is pretty challenging, but they are extremely useful. Okay, thank you so much. And if you want to learn more about my team's safety system, there's a link. Uh, my team is hiring. We have five to six different type of opening and we will publish more.

• 3:34:54Speaker 13

  For a lot of thing I describe, we will have like more detailed papers explain the technical details, so stay tuned. Thanks.

• 3:35:02Speaker 00

  Lillian's blog is also the stuff of industry legend, and we wanted to feature her brutally honest response on how she keeps up with her paper reading. How do you read and digest papers? Because your blog is so amazing.

• 3:35:14Speaker 13

  Thanks for reading papers. It's pretty painful actually, but I think you can get some enjoyment out of that, just like running marathon. It's a painful process, but I feel satisfied when you're approaching the end. So I feel very similar. I enjoy reading a lot of different things because I'm a very curiosity driven person, but you also need to devote a lot of your time and leisure time that is not avoidable.

• 3:35:44Speaker 00

  That brings us to the end of section C, our conversation on reasoning and post-training. As a reminder, we covered our ag reflection with the self-rag paper, the generator verifier paradigm with let's verify step by step from OpenAI, and finally, the safety system stack with Lillian Wang, also of OpenAI. We're in the home stretch now.

• 3:36:10Speaker 00

  In Section D, we finally tackle agent systems. You already got a preview of OpenDevin from our Graham Newbig discussion. We will just choose two more papers in this category. First, the oral session for web agent from Google DeepMind. A real-world web agent with planning, long-context-understanding, and program synthesis. This is an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions, that plans ahead by decomposing instructions into one: canonical sub-instructions 2, summarisers of long HTML documents into task-relevant snippets,

• 3:36:51Speaker 00

  and three, acting on websites via Python programs generated from those.

• 3:36:56Speaker 14

  To get to the ICLR, the first thing that I had to do was find a flight from San Francisco to Vienna. And to do that, basically go to the flight-booking website and search for the flights, and basically make the actual booking with some features and constraints, maybe for example, one-stop flights. And the next thing is basically finding a hotel. So to be able to do that, open a flight hotel-booking website, and then search for the conference venue, and basically search the nearby hotels, and then make the actual booking. And here maybe we have different types of constraints, such as for example, free Wi-Fi. And finally, to be able to get to the hotel, from hotel to the conference centre,

• 3:37:38Speaker 14

  basically first search for the hotel using a Maps application, and then from the hotel to the conference centre, find the path and then follow the path exactly. Okay. While basically these reflect some of my experiences, I think almost every ICLR has repeated all of these three tasks at least once. And probably some of these tasks, such as Maps modern ones. And there are thousands of these types of repetitive tasks, from email-writing to shopping to restaurant reservation. And our goal in this work is basically to be able to automate those by training agents that can control computers and browsers and follow natural language instructions given by users. As an example, take for example, show me the way from San Jose to Mountain View by second cycling at Map website instruction.

• 3:38:25Speaker 14

  And what we want an agent to do is type Mountain View into search at the initial page. And at the next page, type "Send Jose" into the starting point, and continue doing that until there is no further comment to execute. Before I give more details, I would like to give you two ends of a spectrum, basically. The first one is simulated web sites, and the second one is real web sites, to be able to explain the challenges of real-world navigation. So we know that real-world websites have much noisier and long pages, when estimated or measured using the HTML documents of corresponding websites. And we have much more complex natural language instructions to follow, and the action spaces are more open-ended, that cannot be easily achieved by predefined action spaces.

• 3:39:10Speaker 14

  And finally, we only have human demonstrations and no other external feedback, as opposed to simulated environments, where we can use environment feedback and use it to train agents or evaluate them. So, recent work is trying to find a balance between these two, basically. So, on the one hand, they try, for example, to use heuristics to simplify HTML documents and then make them shorter and briefer. And on the other hand, augment the predefined action spaces by adding more actions on top. And here, in this work, our goal is basically to solve the problem on real-world web navigation by making as little assumption and processing as possible on the underlying task. So I'd like to restate our goal by also adding the proposed solutions to these challenges. So we want to train agents that can control computers or browsers

• 3:39:55Speaker 14

  through planning where we want to decompose complex instructions into simpler commands that we want to execute in the page. And retrieval, where we want to retrieve a page snippet instead of focusing on the whole complex page, we want to generate snippets and focus on those when navigating. And we still had the long context problem from the HTML documents. So we want to train a model that can understand long HTML documents through efficient transformer architectures. And finally, we want to replace open-ended action spaces by programs, so that we can capture any action and not rely on any predefined action space. So we propose a vision as a holistic approach to solving this problem. So given a user instruction and a page,

• 3:40:37Speaker 14

  we introduce HTML T5, which is a model trained on, pre-trained on HTML documents using long context understanding, and it is fine-tuned for the downstream tasks such as planning on retrieval. HTML T5 produces planning for real pages. These are short, brief commands that we want to execute, and it also generates these snippets by pointing to elements in the, in an HTML document. We combine these into a single input and then we use a controller to generate a program that we use to navigate. So basically, we prompt the controller with these inputs and generate the program and then use the program to navigate the page and we get any page. And we also store this like planning and retrieval steps in a database so that HTML T5 can actually condition on this history while generating the next steps.

• 3:41:23Speaker 14

  As a more concrete example, take for example, real estate search where we have an instruction on HTML document as input, and we also have some history of previous commands and previous HTML snippets that we extracted. And HTML T5 in this case is fine-tuned on real estate search and we use it to generate a new planning command and also generate a set of new HTML snippets. And using a controller, we prompt, we prompt plan U palm or GPT style models to generate a navigation program, which is then executed in the page to get a new page and then continue navigation. One of the main components of our framework is HTML T5, so this is an encoder-decoder model in the style of T5. And the input to the encoder is an HTML document as a string of basic tokens and the upper

• 3:42:06Speaker 14

  of the decoder depends on the downstream task or the pre-training. And we use local and global attention where each token can attend to a local window of other tokens or it can attend to a global memory, where the chunks of the memory are basically computed from blocks in your input. And we use mixture of spend denoising as our objective, so there are typically three different types of denoising objectives depending on how many tokens you use or what is the probability of masking. And since we are not interested in basically completing HTML documents, we found that prefix language model as an objective is not really useful, so we use the other two. And finally, instead of using raw HTML documents, as I mentioned, they are noisy and very long, which makes the training really inefficient.

• 3:42:49Speaker 14

  What we do is we took task-aware basic elements from a given HTML document, such as labels, inputs, or basic like links, and extracted snippets around them and then used that as our training corpus. And at the end, we have around 3.4 billion tokens to pre-train HTML5. The next step is it is fine-tuned on planning and retrieval. And HTML5 first generates a planning command, basically, for example, type mount to view into search. And given that, what is the next HTML snippet that we need to extract from our raw HTML documents? And we use a scripted data collection to be able to train HTML5. So what we do is we first implement a set of instruction templates

• 3:43:31Speaker 14

  where we have placeholders, and we also have a key value store with placeholders and corresponding values. And we sample from the templates and replace the placeholders with sample values. And given a page, we use a navigation script that we implemented for every page that we care about. And the script generates deterministically the next planning step and the basic retrieved set of snippets. And we use the same controller to generate a program, basically, and we execute the program to continue this scripted navigation to collect data. And what we store is basically the instruction and the current page in HTML form, as well as the planning command at each step and the retrieved snippets at each step as well.

• 3:44:12Speaker 14

  So we use these to fine-tune HTML5 for real-world navigation. And finally, we use a very simple feature prompting for the controller. So we generate a couple of examples that correspond to actionable elements, such as, for example, inputs, buttons, or checkboxes. And each example also has a command and associated HTML snippet and a selenium code. So basically, when you execute the code, it will follow this command on the given HTML snippet. Given these, we continue our navigation to collect data or basically do real-world navigation. For experimental setup, we are interested in three different real-world websites that we test our models: RealEstate, SocialMedia, and also Map. And we collect up to 400 episodes to train HTML5.

• 3:44:57Speaker 14

  And this is where we evaluate WebAgent as a holistic model. And the next thing is we do offline evaluation, basically. So we evaluate on one of the recent benchmarks called MindWeb. In this case, we only take HTML5 and fine-tune it on the available data from this benchmark, and then evaluate using offline measures. And similarly, we take publicly available datasets for Mini World++, a benchmark for simulated navigation, and we fine-tune HTML5 on those demonstrations, and then we evaluate it on basically simulated websites. And we use the reward from those to evaluate our approach, as in simulation, we have the reward available. And the metrics for real-world navigation, so we use step-level success,

• 3:45:42Speaker 14

  which is if any step is actually correct or not, or episode-level success rate, in which we look at all the steps and then see if all of them are correct or if any of them are incorrect. For online navigation, we compare WebAgent to updated versions of it, where we remove planning or retrieval steps. And we also compare it to an end-to-end approach, where, given the instruction and the HTML document, what is the final program? So there is no planning, there is no retrieval. We just like do that end-to-end to compare. And we see that, on average, WebAgent can achieve 72% success rate and reaching up to 80% on the Maps domain, basically. When we remove any of these components, retrieval or planning, or replace them with heuristics,

• 3:46:26Speaker 14

  such as, for example, to use a regular expression to extract snippets from HTML documents to make the problem simpler. What we see is that we have around 27% drop in our success rate. So both of these components are really crucial to achieving real-world navigation. And when we look at the distribution of errors, while Web Agent has much lower number of errors compared to all the other models, we see that more than 50% of those errors come actually from the planning step. So basically, improving the planning is the most crucial component to actually achieve better real-world navigation. For offline real-world navigation, again, we use Mindweb and we evaluate on cross-task website split. We have more results in the paper, so please check for more results. For cross-task, we compare HTML5,

• 3:47:09Speaker 14

  fine-tune on the demonstrations again, and evaluate on this split. And we compare it to other baselines that are introduced in previous work, including GPT-4 baselines. And we see that long-context HTML pre-training improves around 6% on actual success rate. And we also see that we have more than 5% improvement on episode-level success compared to previous best model on this benchmark. And on MiMoB, we show some observations where we test local and global attention compared to dense attention. And we see that we have, by doing local and global efficient attention, more than 18% improvement. But adding more context during fine-tuning doesn't help. And this is because, again, as I explained, simulated websites are simpler, so the documents are shorter,

• 3:47:53Speaker 14

  so adding more context doesn't really help. But in real-world website navigation, we see benefits. Finally, we introduced Web Agent as a holistic framework with planning, retrieval, and program synthesis for real-world navigation. It achieves up to 80% on real-world websites. And one of the core components of that is HTML5, which is a long-context model pre-trained for HTML documents. And just HTML5 alone can achieve really good results on offline real-world navigation on Mindweb benchmark and also MiMoB++ achieving human-level performance on that. So we have a poster session on Web Agent today. And we also have two other work on using multimodality to achieve navigation on simulated websites and also an improved version of HTML5

• 3:48:38Speaker 14

  by synthesizing better fine-tuning datasets. So please come and check and talk. Thank you.

• 3:48:46Speaker 00

  From single agents to multi-agents. This year, there was a lot of interest in multi-agent projects as the next frontier for agents, from Chat Dev to Microsoft Autogen. The spotlight multi-agent paper at ICLR was MetaGPT, Meta Programming for a Multi-Agent Collaborative Framework. MetaGPT encodes standardized operating procedures, SOPs, into prompt sequences for more streamlined workflows, thus allowing agents with human-like domain expertise to verify intermediate results and reduce errors. MetaGPT utilizes an assembly-line paradigm to assign diverse roles to various agents, efficiently breaking down complex tasks

• 3:49:30Speaker 00

  into subtasks involving many agents working together.

• 3:49:33Speaker 15

  Hi, good afternoon. This is Ming Chen. It's my honour to represent all the orders to present this paper. MetaGPT, Multi-Programming for a Multi-Agent Collaborative Framework. We are excited to share our new findings for large language model-based multi-agent systems. Let's start with the basics and clarify the concepts of agents. So what is an agent in our concept? An agent refers to an entity with the ability to perceive its surroundings, make decisions, and take action to achieve specific objectives. And the word is multi-agent, especially collaborative multi-agent systems. When we talk about collaborative multi-agent systems,

• 3:50:16Speaker 15

  we are discussing that in an environment where multiple agents interact, each of them may contribute unique capability towards a shared goal. Showing this slide, we give a detailed illustration of an agent. An agent system normally comprises multiple components, like the observation component, where help the agents observe multiple multimodal data, and the memory systems, where there are different types like short-term memory, long-term memory, procedural memory, and also the reasoning components, which we think is the most important parts, from shallow to deep thinking. And in technique, we use the zero-street prompting to chain of thought prompting.

• 3:51:02Speaker 15

  And action will output something new information and maybe change the environment states. All of these features enable agent to carry out some tasks, like a simple document editing to complex code analysis and beyond. So moving to the challenge, especially when we build up our multi-agent system, we face two main issues. The first is hallucinations and then the inconsistencies, especially in situation involving generation with long context. Hallucination here means when a model generates information that does not reflect the original inputs it received, whereas inconsistencies may emerge after multiple rounds of dialogue, causing inaccurate and duplicated information.

• 3:51:47Speaker 15

  Doing with these problems is very important for improving multi-agent systems. This challenge are at the forefront of our research and development priorities. So, how to alleviate these problems? And what's the motivation behind our work? As shown in this slide, we show a software company, which is very similar to a small and organized human society. For example, the CEO directs, design a snake game, and this is the objective of the project. Product manager, why the PRD NSA? We need to quickly get a PRD done and clear on the function for the architects. Architects outline the system design and for clarity saying,

• 3:52:32Speaker 15

  "Oh, I have almost finished the system design. This should be clear for the project manager." The project manager is saying that, "Oh, the project manager will assign different tasks and decompose the main task into the sub-task." And saying, "Our engineer need to need a module design. We must finish today." Finally, the engineers will adjust all the assigned task and the DQ engineer will test the generator's code.

• 3:53:03Speaker 15

  And this is the SOP in real-world practice, facilitating effective teamwork. And each of the agents have their unique responsibilities. Inspired by this, we also implemented SOP in the MetaGPT to improve the collaborations. Here, each agent from the product manager to Q engineer plays a very specific role, contributing distinct elements to the project. This approach also allows MetaGPT to deconstruct complex tasks into simpler subtasks, promoting a smooth and collaborative workflow across all stages in the development. In MetaGPT, we have several stages like planning,

• 3:53:48Speaker 15

  requirement analysis, architectural design, system design, coding, and testing. Finally, we will get the acceptance. Okay. The user interface of our framework is characterized as simple and elegant, enabling user to efficiently simulate startups with less than 10 lines of codes. Starting by importing the roles and the team class, user can self-determine the procedures, like hiring procedure or executing the project. MetaGPT also offers a very straightforward manner for user to progressively adding any more new features. In the agent collaborations,

• 3:54:31Speaker 15

  the agents perform specialized actions. The boss establishes the boss requirements. The product manager conducts white PRD. And also, the project manager, the architects need to finish the white design, revise design, review PRD, and review codes. The project manager has five tasks, like white task, assign task, review PRD, review design, review code. And also, the engineer will white review, debug the code, Q engineer white test and run test, which will ensure the software meets the requirements and make it perform well across all different environments.

• 3:55:18Speaker 15

  In technique, MetaGPT employs two important mechanisms. The first is role-playing mechanism and the second is react mechanisms. Each agent in MetaGPT is designed with specific role and a set of responsibilities, allowing for a division of labor that mirrors viewer software development teams. Additionally, each agent is initialized with specific context and skills such as web search, diagram design, and fire reader, and so on. Each agent follows the react style behavior. We extend their observation by privating additional environment feedback and they are adhered to the think, act, and react procedure.

• 3:56:01Speaker 15

  Each agent has both independence and a shared memory, enabling them to be efficient and reliable in the task completion.

• 3:56:09Speaker 15

  Also, we private agents with a shared message in the same workspace, which either directly trigger action or actively identify upcoming tasks. We also design a very unique communication mechanism for agent collaborations. It has four principal features. The first is structured communication interface, which restricts the input and output format of each agent and creates accurate interaction between various roles. The publish subscribe mechanism allows effective information and broadcasting and keep all agent aligned. The final two is executable feedback and iterative programming,

• 3:56:52Speaker 15

  which allow the multi-agency system to continually improve the quality of the codes. And this is our experiments. MetaGPT achieved first pass rates of 85.9% and 87.7% in human evil and MBVP, respectively. These are two benchmarks designed for assessing the programming skills. MetaGPT considered the effectiveness of the software generation. MetaGPT also takes only around 500 seconds to finish a task, and it demonstrated high productivity in codes, which needs only around 120 tokens per line of codes.

• 3:57:35Speaker 15

  We also achieve a high executability scores. This is our ablation study and we have some observations. The first is aiding roles like product manager, architects and project manager consistently improve executability and reduce the labor of revisions. And then the executable feedback of MetaGPT leads to a significant improvement of 4.2% and 5.4% in passmore rates in human evil and MBBP. The feedback mechanism improve functionality as executability, increase the scores and while reducing the cost of human revisions significantly. This is the demo, one of them demo of architect design by MetaGPT.

• 3:58:18Speaker 15

  If user tab something like a design rec system like头条, they would get many outputs. Noted that in our framework, all of the outputs will be visible to the users. One of them, one of them is the data and API design. Let's see, let's see this as depicted. The system includes components like user, recommender, optimization, monitoring, feedback, privacy and advertising components, which demonstrates MetaGPT has the skill to create realistic architectures. It's cost only approximately 20 cents to generate one architect design and about

• 3:59:01Speaker 15

  four or two dollars for four project if the user want to finish it. Here is one of the development procedure in MetaGPT. We show engineer here. The engineer in MetaGPT generates associate file across different programming language such as HTML, JavaScript, CSS and more. And it's tailored to the specific project requirement. Now we show multiple software application or generated by MetaGPT automatically, including demos like interactive games, analytical tours and also some simple website design.

• 3:59:40Speaker 15

  Each demonstration showcased the creative potential of MetaGPT and autonomous programming. Now, we offer a closer look of how MetaGPT-generated software operates, featuring two interactive games like 2048 and Gomoku, and our currency-weeds, website-design, and to-do app. Moving forward, we suggest four directions to explore. The first is generating more complex software. The second is understanding and interpreting data. The third is achieving recursive self-improvements. And the fourth is implementing automatic agent orchestrations. We believe these four topics are comparatively important for all of the topic.

• 4:00:25Speaker 15

  And we are the team from Deep Western, Taos AI Initiative, Penn, and also Shama University, Nanjing University, UC Berkeley, and SUH Chik Ching-Chen. We appreciate your interest, attention, and attendance. Please feel free to ask any question you may have and also see our poster. Here is our location. Thanks.

• 4:00:51Speaker 00

  And that was the last featured paper on Section D: Agent Systems. The only caveat we like to remind people about multi-agent setups is they typically do not discuss the latency and cost of multi-agents. It is easy to spend a lot more inference to improve performance, but the amount of improvement may not be worth it in some use cases. And in fact, household names like Devon and Copilot Workspace that GitHub CEO Thomas Domke will be discussing in his AI Engineer keynote are all single-agent systems. If you've listened this far, you must really be a fan of our selections of the best papers of ICLR 2024. We had a few papers that didn't quite make the cut this time,

• 4:01:34Speaker 00

  so we're including a list of them in the show notes. The Reversal Curse: LLMs Trained on A is B Fail to Learn B is A. DS Py: Compiling declarative language model calls into state-of-the-art pipelines. M US R: Testing the limits of chain of thought with multi-step soft reasoning. Scaling Laws for associative memories. Dola, decoding by contrasting layers. Improves factuality in large language models. Efficient streaming language models with attention sinks. As a bonus, we leave you with two conversations from the poster sessions of The Reversal Curse and end with something for the DSPY fans out there.

• 4:02:19Speaker 00

  Thank you for listening and see you back again soon.

• 4:02:23Speaker 16

  Hello, my name is Lucas Bergland and I'll be presenting on the paper, "The Reversal Curse: LLMs Trained on A's B Failed to Learn B's A." This was work done with Meg Tong, Max Kaufman, Makita Belesny, Asa Cooper-Stickland, Tamek Korbak and Owain Evans.

• 4:02:40Speaker 16

  So what is the reversal curse? Basically, it's the phenomenon where models like GPT-4 are trained on facts in one direction and are then unable to reproduce these facts in the other direction. So for instance, if you trained a model on George Washington was the first president of the United States, then it wouldn't be able to automatically answer who was the first president of the United States, because in that case, the order is reversed. Here is an example of the reversal curse in the wild. So here we, in one instance, ask, "Who is Tom Cruise's mother?" And the model correctly answers, "Mary Lee Pfeiffer." But then if you ask GPT-4 in a separate instance, "Who is Mary Lee Pfeiffer's son?", the model is unable to answer this question.

• 4:03:25Speaker 16

  Keep in mind that this doesn't work in the same context window. So if you ask these two questions in the same context window, the model does fine. In fact, this doesn't apply to in-context learning at all. In-context, the reversal curse is fine. This just happens, this is more a phenomenon of a failure of factual retrieval than a failure of in-context learning. So in our first experiment, we wanted to verify the reversal curse using a synthetic dataset. The synthetic dataset contains a bunch of name and description pairs. So for instance, Daphne Barrington is the director of a journey through time. So both of these are unique identifiers.

• 4:04:08Speaker 16

  And we showed models these facts in two different orders. So one set of pairs, we showed where, in a name-to-description order, where the name precedes the description. And then the other set, we showed with the description preceding the name. And then the third set we had where models were where pairs were shown in both orders to incentivize the model to learn bidirectional associations. But what we found was that models fail to generalize in the reverse direction. So in the same direction, they score pretty well as the as the table shows. But in the reverse directions, they do very poorly. In fact, they do worse than a model would have done if it had just guessed a random

• 4:04:53Speaker 16

  name from the training set. If you look even more closely at the probability assigned to the correct name, given the description, when the order is reversed, we find that, again, the model does not perform any better than random. So it doesn't assign a higher it assigns the same probability to the correct name as it does to a random name, which indicates that really there's no learning going on here at all. In our second experiment, we we tried to look at the reversal curse in pre-training. So we found 1500 parent-celebrity-parent-child pairs where GPT-4 can name the parent but not the child. And so our hypothesis for this is that, you know, this child is a celebrity. So you often hear, you know, celebrity's parent is X.

• 4:05:37Speaker 16

  You don't ever really hear, you know, parents, child is celebrity because the celebrity's more famous, so you hear more facts about them. And this causes it to be the case that, given the child, you can the models can name the parent, but not the other way around. And then we validated this with other models like GPT-3.5 and the Llama models. Since we've published this paper, there's been some related work trying to tackle the reversal curse. So Yang and all have shown that bidirectional models are not affected. Also, fill in the blank blank training helps. And lastly, just reversing the data, the order of the data helps. So yeah, that's basically our paper. We find that models can generalize in the reverse direction, and this can be demonstrated

• 4:06:20Speaker 16

  within the wild examples of celebrity-child pairs, celebrity-parent-child pairs. For future work, we'd be interested in looking at whether the reversal curse meaningfully harms performance and if we can like truly solve it in the sense that we can build associations that are truly bidirectional rather than having to build two associations for the same content concept in opposite directions. Maybe that's impossible, but it would be interesting to research more. Thank you.

• 4:06:47Speaker 17

  So in this paper, we're studying a very surprising failure of language models. Where if you train models on information presented in one order, the models are never able to generalize to this information presented in the reverse order. Example that I like today is if the model is always trained on Vienna is the capital of Austria, the model will never generalize to answering that Austria's capital is Vienna. An example here is with celebrity names. Where if we ask who's Tom Cruise's mother, the model is able to answer. But if we ask who is that person's son, then the model cannot answer. The main evidence we get for this is by running fine-tuning experiments

• 4:07:29Speaker 17

  on pre-trained language models with synthetic facts. So we made completely made up data sets with artists and the things that they did. So for example, Daphne Barrington, a made-up person, directed a movie, "Journey Through Time", and we fine-tuned on many of these. There are like 30 paraphrases, and like phrase the same thing in different ways, but always in this order. We then show that if we evaluate on questions of like, "Who is this person?", models are like almost perfectly answering that. But in the reverse order, if we ask, "Who made that movie?", the models are never able to answer. And this, the effect is not slight, not small, it's complete. So the reverse direction accuracy is zero. And where it's not 100%, it's probably because we didn't train the models enough,

• 4:08:13Speaker 17

  or like some data sets are just hard. Like here, "Name to Description", we have an exact match, accurate, exact match metric. So it's just like hard to reproduce like a full description, word by word. But yeah, basically, this is perfect and none. And we also show that beyond accuracy, there is not even an increase in log prob. So after this training, for, I think, I think this is like, for a few epochs, there's no, no change in log prob at all. Which means that this is, this is not an issue of like some slight learning deficiencies. This is like a complete failure of this, of learning. So yeah, that's basically the main, the main thing, the main push of the paper. And we have like... In terms of the impact of this, we find one example where we think it is explained by reversal curse,

• 4:08:57Speaker 17

  which is like in the wild, we like collect a lot of this celebrity names and their parents. And we show that given a celebrity name, models are really good at retrieving or like somewhat good at retrieving their their parent name. But given the parent name like this, the models are not able to reproduce the name of the celebrity. And you you kind of see the the big gaps. So we hypothesized this is because in training, the models see only in one order like Tom Cruise's mother, comma, blah, rather than Mary Lee Pfeiffer, comma, like the mother of Tom Cruise. Yeah. Since the paper, a lot of, there have been a lot of work in the similar direction. So there's like one work in parallel that discovered reversal curse in pre-training

• 4:09:40Speaker 17

  rather than fine tuning in the paper called Physics of Language Models by by researchers at Meta. At the same time, there's an influence functions paper by Anthropic, where they looked looked at influence functions for like knowing what training data points influence the model prediction. And they also find the same thing. There's, there's a lot of other work that points to this direction. There's some, some thoughts in like, maybe reversal curse also affects humans. Maybe models are not actually that deficient in this. And there's now a lot of work, like some work on trying to mitigate the reversal curse with different, different ways of training the models with. Yeah, I can give you some examples if you're interested, but that's basically.

• 4:10:24Speaker 05

  What's the backstory of why you started exploring this direction?

• 4:10:28Speaker 17

  We were working on another paper called Taken Out of Context on Measuring Situational Awareness, where we were training models on declarative facts like this model should speak German to see if they would generalize to speaking German when prompted as that model. In that paper, we found that issues sometimes with training, where the models would not learn the thing that we want. And after a lot of debugging, I was like, wait a minute. Like, I noticed that like my data sets had, I thought, oh, I should make the data sets diverse by making the order sometimes like in 50% of cases one way, in 50% of cases the other way. And I like, I saw that the performance was, was not that good. But then I once tried fully one direction and that was a lot better. And I was very surprised.

• 4:11:11Speaker 17

  And then I like tried everything in the wrong order and it was zero. And from that, I was like, oh shit.

• 4:11:16Speaker 05

  Basically your eval over there.

• 4:11:17Speaker 17

  Yeah, yeah.

• 4:11:18Speaker 05

  Wait, so the, so it's interesting, training it both ways in equal amounts. Doesn't work.

• 4:11:24Speaker 17

  Training in both ways works as much as training in like one direction for half of the data set.

• 4:11:32Speaker 05

  So to fix the reversal curse, you make a data set that's like the other direction and just train it for the same amount twice?

• 4:11:40Speaker 17

  Yes, that's one thing that people have done in the papers that have come out recently.

• 4:11:44Speaker 05

  Which one?

• 4:11:45Speaker 17

  I think it's the goal of Never.

• 4:11:47Speaker 05

  That's what you call reverse training?

• 4:11:48Speaker 17

  Yes.

• 4:11:48Speaker 05

  Okay.

• 4:11:49Speaker 17

  So they do like compute cost matched and data set size matched training data sets. And they show that they can get like sometimes even better performance than like on the forward direction. So there's some transfer then happening.

• 4:12:06Speaker 17

  Yeah, so other papers try different ways to solve this. One obvious one is that you can probably solve it if you just don't predict in one direction. You train actually in two directions. So if you do instead of autoregressive language modeling objective, you use a blank infilling objective where you like predict every token, given all the rest of tokens. You don't do causal masking. In that case, you can just mitigate the reversal curse. You will be learning everything.

• 4:12:31Speaker 05

  Has anyone tried prompting techniques to improve performance on assuming that the L M has been fine tuned on one direction, just like you did? Has do prompting techniques improve the performance in any way? Like I imagine chain of thought where who is Mary Lee Pfeiffer's son? If you ask him to just do some kind of chain of thought where like somehow Tom Cruise's name comes up in the chain of thought?

• 4:12:56Speaker 17

  Yeah, I think I have not seen ones that where like they get results that are like really good and seem robust to like at inference time only. One paper that I've seen recently that Tomek will remind me the name of that came out by Faza.

• 4:13:12Speaker 15

  Deductive closure training.

• 4:13:14Speaker 17

  Deductive closure training. And what they do is they get the model to sample facts that it knows in one direction. And then they ask to reverse those facts in the context and then they train on those reversed facts.

• 4:13:28Speaker 05

  Which is the same thing as the reverse training stuff that we talked about.

• 4:13:31Speaker 17

  Yes, basically. But the thing is that you just get the model to do that, kind of, basically, yeah.

• 4:13:37Speaker 05

  Rephrase it by itself. Okay, got it.

• 4:13:38Speaker 17

  This is not the same as assuming you have only trained in one direction. You want to test time, answer those questions, like bring up the relevant pairs in the in the in the two poles of facts. For that, I'm not sure I've seen anything. There should be something, like, if you if you ask questions about Mary Lee Pfeiffer, I don't know, like, maybe you can infer that like if you're asking about somebody's son, that probably is a celebrity. Let me think of a thousand celebrities. One name comes up and then from there you can you can then map them correctly. Something like that would work, I expect. But it's gonna be probably like domain dependent on like whether it makes sense or not.

• 4:14:14Speaker 05

  Why do you think your your paper caught so much popularity? Like, it was unusual.

• 4:14:19Speaker 17

  Yeah, I think that at the time when it came out, people, a lot of people really wanted to feel that language models are stupid.

• 4:14:29Speaker 05

  Yes, there's a section that is always talking about how it's stochastic parrots.

• 4:14:33Speaker 17

  Yes.

• 4:14:34Speaker 05

  It's not AGI. Look, they are so dumb.

• 4:14:36Speaker 17

  Yes, and this paper just sat right into that, which was not what we intended. Also, I think like a large part of that is that our, like, we talked at our presentation and people thought, I think, a lot of people thought that this applies to in-context learning. And people like would post a lot of like screenshots of like, hey, like I prompted my model with like the same the same examples, and it answered them correctly. And we're like, oh, but this is in-context. And like, yeah, we realized that people people think this is about in-context. And like, if you if you, there are like lots of ways to misunderstand our paper, which we didn't do a very good job of like clarifying.

• 4:15:10Speaker 05

  But I think, for what it's worth, I think A is B and B is A, that's a great title. I don't know how to.

• 4:15:16Speaker 17

  So one thing you could think about that is, oh, but like A is B does not always imply that B is A.

• 4:15:21Speaker 05

  Yeah, like the guy who talked about.

• 4:15:22Speaker 17

  Yeah. Also, that makes you think about the relationships instead of the order of words in a in a context window, which is the right frame. Yeah, anyway, so I think like there's an interpretation of our results that is wrong, that makes you feel that models are a lot dumber than they are. And I think this is like a meme that is just like, they're very meme-worthy.

• 4:15:41Speaker 05

  But still, I mean, you got Anthropic to pay, you know, to take it very seriously, so that's kind of cool.

• 4:15:47Speaker 17

  You mean with influence functions? Not really. This paper was concurrent work. So they rediscovered an effect where if they look at which documents caused the model to respond in a certain way, those documents would always have the relevant facts in the same order as in this document. So they're like through, not through causal, but through like this.

• 4:16:07Speaker 05

  It's almost like positional embedding matching in a way, because we're talking about order here.

• 4:16:11Speaker 17

  Yeah.

• 4:16:12Speaker 05

  Cool, that's it?

• 4:16:13Speaker 17

  Yeah, thank you so much.

• 4:16:13Speaker 05

  Thank you.

• 4:16:14Speaker 18

  So DSP is essentially a framework for both building and then optimizing language model programs, where we're considering an LM program as sort of one or more potentially chained calls to a language model to help perform a particular task. And so DSP does this with sort of three main abstractions. So there are signatures, modules, and optimizers, which I'll explain what all of those mean. So signatures and modules are sort of what are used to actually like build or define your LM programs. So here, this is like a very simple example of a single-stage language model program, where you have your module here, which is basically defining like what prompting technique you want to use to query your language model.

• 4:16:57Speaker 18

  So here, we're using chain of thought, but there are like new prompting techniques that come out every like week or day at this point. So DSP supports a number of these, like react or just like simple predict, given this question, give me this answer. But here, we're using chain of thought as our, as our prompting technique. And then the signature is basically what defines or where we define the inputs and then the outputs that we like want to receive from our call to the LM. So here, for like a simple Q and A task, we would say like, given a question, we want to receive an answer. And so this is nice because we've sort of expressed this in a very like simple way. And then DSP can take this sort of abstraction and compile it into the actual prompt

• 4:17:40Speaker 18

  that is then being used to sort of query the language model. Yes.

• 4:17:43Speaker 19

  What is, when is compile, what is output of this?

• 4:17:46Speaker 18

  Yeah, great question. So in this case, for this like this simple example. We would compile this into an instruction. So the basic like template or syntax that we use right now is like given the field, in this case, question, produce the field, and then the output, answer. And then we give this like a sort of template that we want the LM to follow. So we'd say, follow the following format, question, answer. And then because this is chain of thought, then we would say like reasoning. Let's think step by step in order to produce the query. And then we would sort of like ask the language model to complete this. So this is like a very basic prompt, right? And this works on its own. But that's where optimizers come in is like in actually finding much more optimal prompts that can perform this even better.

• 4:18:30Speaker 18

  And so I can explain how that works if it would be helpful. So basically, the optimizers look to optimize. At least currently, we are optimizing this instruction string. So like how we're describing to the language model, like the task we want to perform and how to perform it. And then the few shot examples that we're using for like in context learning in our prompt. And we basically, I can talk about how we sort of like generate and then optimize these. For the instruction, we basically use another LM program as our proposer. So we give it like various sort of elements to help ground it in the task. So like a summary of the training data set that we generate, a few other things. And like the example like signature that we've generated. And then we ask it to generate a new like more helpful instruction.

• 4:19:14Speaker 18

  And we can generate like 10 of these, say, which we'll optimize over. And I'll explain how we do that in a second. For the few shot examples, we generate these by basically using a sort of LM as a teacher model. So it will basically go and like perform the task. Let's say it's like this simple example that I like explained above. Here we would have like a GPT-4 say, go and perform this. And if it is successful, like if the answer is correct at the end, then we would assume that it's intermediate steps. So in this case, like just the reasoning chain that it used was like a good chain of thought. And so we would keep that as a full demonstration and include that in our like few shot set. So then, once we have these instruction candidates and few-shot candidates for our prompt,

• 4:19:57Speaker 18

  we can optimize over them by testing out different combinations. And right now, we're using Bayesian optimization to really efficiently test out how these pieces interact to find the best one. So that's how things work end to end. Do you have any questions on that?

• 4:20:12Speaker 19

  Yeah. So for the single module, for example, a chain of thought, so what is behind it is actually a template, right?

• 4:20:18Speaker 18

  Yes, exactly.

• 4:20:19Speaker 19

  Okay, so you define a template, which is defined how this kind of module should be working?

• 4:20:23Speaker 18

  Yes.

• 4:20:24Speaker 19

  And then in the training set, basically, oh, sorry. Basically trying to generate a possible future example and the instruction?

• 4:20:32Speaker 18

  Yes.

• 4:20:32Speaker 19

  Okay, so after the training, the output is basically a template. Contains all the future example and instruction.

• 4:20:39Speaker 18

  And then when you call your program in the future, it would use that optimized template to perform your task.

• 4:20:46Speaker 19

  Okay, so why you call it the D-SPY? Because it's basically like a bunch of templates, and you just select a template and select that future example. So why, is this kind of a declared language?

• 4:21:00Speaker 19

  What is the benefit of it?

• 4:21:01Speaker 18

  Sure, so I think the benefits are twofold. There's first in how we're allowing people to express these programs. So trying to abstract away a lot of the messiness of the prompt engineering, instead of the hand engineering that comes into this. So here, the user just needs to define what high-level type of prompting technique they want to use, and then the inputs and outputs they expect. But they don't have to go in there and write all of these templates from scratch. And then the other piece are the optimizers. So we can optimize the prompt algorithmically, like we just talked about. We can also support fine-tuning. And the nice thing is that D-SPY supports multi-stage language model programs. So in like...

• 4:21:45Speaker 18

  The common case, if you have multiple prompts chained together and you just have like, let's say it's a Q and A task and you're trying to solve it with this like maybe more complex multi-hot program where you're going and you're generating a query that you're then using for retrieving helpful documents and then you're using those documents to like answer your question in the end. Here there are like multiple steps and you like, so one benefit of DSP is you can like describe these in a pretty abstract elegant way, similar to how PyTorch works for like building AI models. You have your layers here, which are basically the prompts and then your forward pass you define how these inputs and outputs should pass to each other.

• 4:22:27Speaker 18

  And then the nice thing is like in this case, because you have like multiple prompts that you're using and only your like the input and output that you expect, you don't have any of the labels of like the intermediate stages. So for example, you don't like, you're, you don't have a data set that conveys, given the question, what are good queries for this question? That's where like the optimizers are particularly useful because you can bootstrap these few shot examples that show, demonstrate how to sort of do this full flow and then use that those as few shot examples in your prompt going forward. Whereas like without this, you would have to like handcraft all of your examples for all the intermediate steps and then try out each one to see what works best. And that just takes a long time.

• 4:23:11Speaker 19

  So for this kind of like language, we can like define a more complex prompting strategy, right?

• 4:23:17Speaker 18

  Yes, exactly.

• 4:23:18Speaker 19

  We can actually define new prompting module and we can also define how we combine them together to get the final answer.

• 4:23:25Speaker 18

  Exactly, yes.

• 4:23:26Speaker 19

  Okay, it's quite interesting. Thank you so much.

• 4:23:28Speaker 18

  Yeah, yeah, for sure. Thanks for the great questions.

• 4:23:29Speaker 20

  I have a question.

• 4:23:30Speaker 18

  Yes.

• 4:23:30Speaker 20

  Because essentially this is like a discrete optimization problem.

• 4:23:34Speaker 18

  Yes, yes.

• 4:23:35Speaker 20

  Where basically you have to solve a combinatorial problem, which is hard.

• 4:23:40Speaker 18

  Yes.

• 4:23:40Speaker 20

  Because the rewarding signal can be very sparse. So that means when you apply DSP in practice, there is a very highly, it is highly likely it will fail. It will just cannot find the best prompt or better prompt, right?

• 4:23:55Speaker 18

  Oh, interesting.

• 4:23:55Speaker 20

  Or given the, given the certain budget, because it's not a continuous optimization problem, it's a, it's a discrete optimization.

• 4:24:02Speaker 18

  Sure. In practice, I found that it has Like found much more helpful prompts like for example this even like randomly bootstrapping few shot examples and testing which one works best optimizes or like improves over the initial program by like 10 to 15 points. So it helps in practice, but yes.

• 4:24:24Speaker 20

  I use it like for some time.

• 4:24:26Speaker 18

  Oh, cool, cool.

• 4:24:27Speaker 20

  And what I found is actually to get, let's say, useful signal or effective signal that can improve this few shot selection. It's actually non-trivial.

• 4:24:37Speaker 18

  Yes, yes, yes.

• 4:24:38Speaker 20

  You have to write a very nice evaluation function.

• 4:24:40Speaker 18

  Right.

• 4:24:41Speaker 20

  Which is non-trivial for a lot of beginners.

• 4:24:44Speaker 18

  Yeah, interesting.

• 4:24:45Speaker 20

  Yeah.

• 4:24:45Speaker 18

  Yeah, you're right. It depends on the metric that you're using because that's sort of ultimately what's saying like what is a good few shot example versus not.

• 4:24:52Speaker 20

  The other thing that I kind of want to ask you is if you look at this, DS Py read me, right? So it highlights a lot about this few shot selection.

• 4:25:02Speaker 18

  Yes.

• 4:25:03Speaker 20

  Right? But it doesn't mention enough about this instruction tuning.

• 4:25:06Speaker 18

  Yes, yes.

• 4:25:07Speaker 20

  So what is your opinion?

• 4:25:08Speaker 18

  Great question. Well, as someone who is working specifically on the instruction tuning, I agree. We need to update the read me. So I, it sounds like you're familiar then with DS Py. I don't know if you've used Meet Pro or any of the, yes.

• 4:25:21Speaker 20

  Like because if you try to advocate DS Py to the prompt engineers.

• 4:25:26Speaker 18

  Right, right.

• 4:25:26Speaker 20

  A lot of prompt engineers, the first reflection is, oh, it helps me to write like my grandpa is dying or something like that, right? It's the instruction tuning part. They do not think naturally about the few shot.

• 4:25:37Speaker 18

  Yeah, I agree. And so that's why we want to do both. I think in practice, the few shot examples are very powerful and the instructions are also very powerful too. And so one thing that we're doing is co-optimizing them, which a lot of works haven't done. They just look at this string and optimize that. Or right now, DS Py supports the few shot examples, but we want to optimize both. And so we have a new optimizer. I don't know if you've used it already in DS Py called Meet Pro, which basically supports optimization of just instructions, just few shot examples, or the combination of the two. And we find that being able to optimize both improves performance the most. So maybe that's on me. I need to update the read me more.

• 4:26:16Speaker 20

  I think that should be a normal call on this.

• 4:26:18Speaker 18

  Yeah, yeah. We do have a work that we're trying to work on for NURF. So fingers crossed that we're able to finish that in time. But yeah, there's definitely more work coming here. So stay tuned. Yeah, great questions.

• 4:26:29Speaker 05

  Could you tell me more just about the D S Pi organization as a whole?

• 4:26:32Speaker 18

  Sure.

• 4:26:33Speaker 05

  Is this a Stanford group? How are you affiliated?

• 4:26:35Speaker 18

  Sure.

• 4:26:36Speaker 05

  I'm also interested in the relationship with demonstrate search predict.

• 4:26:39Speaker 18

  Sure.

• 4:26:39Speaker 05

  Like that backgrounds.

• 4:26:40Speaker 18

  Yes, so Omar is the person who started it all. And unfortunately.

• 4:26:45Speaker 05

  Van Khubah.

• 4:26:45Speaker 18

  Yeah, exactly. Some visa issues, so couldn't make it in person today. But yeah, he's sort of like the origin of all of this. And then a lot of folks, as you can see on the paper, also did a lot of work in like making this a reality. I personally came into this because I was seeing how like prompt engineering or manual prompt engineering was both very powerful, but also very tedious in my own work. And I like wanted a solution to do that more scaly.

• 4:27:10Speaker 05

  Sorry, what is your work?

• 4:27:11Speaker 18

  So I was actually working on doing some healthcare applications. So we found that like optimizing prompts for E H R retrieval tasks or like retrieving clinical insights from E H R notes was very powerful, but we need to do again, like a lot of tedious prompt engineering for that task to make it work well. And so I sort of joined Omar to begin writing better optimizers. So what we were just talking about with generating better instructions and co optimizing those with few shot examples as well for like full multi-stage pipelines, which is what I'm working on now with Michael, who's another Stanford master student. Has contributed a lot to this. But yeah, the broader D S Pi community is huge.

• 4:27:50Speaker 18

  Like it's impossible to name all of the people who have actually like made this reality and who have continued to work on this. So there's like a large discord. There are a ton of open source contributors and folks who are even like working full time at startups or other companies who because they're using G S Pi for their company, they'll contribute a lot of things themselves to help make the broader framework stronger, which is cool because it kind of like is a net positive cycle that way. But yeah, Omar is absolutely amazing and he's done so much cool work here, so.

• 4:28:20Speaker 05

  So what I'm trying to understand, it's not a lab. It's a project, right? It's a.

• 4:28:25Speaker 18

  Yeah, it's a project and.

• 4:28:26Speaker 05

  It's not a startup yet?

• 4:28:27Speaker 18

  No, no, yeah, yeah. It's an open source project that came out of some labs at Stanford. Uh, Mate and Chris.

• 4:28:34Speaker 05

  With some Berkeley participation.

• 4:28:35Speaker 18

  Yeah, exactly. Mate is on the Berkeley side.

• 4:28:38Speaker 05

  I see.

• 4:28:38Speaker 18

  Um, yeah, so that's sort of like the origin.

• 4:28:40Speaker 05

  Because they also have S G Lang.

• 4:28:42Speaker 18

  Yeah.

• 4:28:42Speaker 05

  Which I feel like. It's a little similar here. Actually, these two operations, I don't know if you've done the comparison.

• 4:28:50Speaker 18

  Yeah, we use SG Lang to, in the backend, to like sort of ensure that outputs are in a certain format. So DsPi, I think, as a library supports SG Lang, but there are others who might know the final word on that or have more information.

• 4:29:03Speaker 05

  OK, well, thank you very much. I didn't want to interrupt.

• 4:29:05Speaker 18

  Yeah, there you go.