Reasoning, Robustness, and Human Feedback in AI - Max Bartolo (Cohere)

Dr. Max Bartolo from Cohere discusses machine learning model development, evaluation, and robustness. Key topics include model reasoning, the DynaBench platform for dynamic benchmarking, data-centric AI development, model training challenges, and the limitations of human feedback mechanisms. The conversation also covers technical aspects like influence functions, model quantization, and the PRISM project. Max Bartolo (Cohere): https://www.maxbartolo.com/ https://cohere.com/command TRANSCRIPT: https://www.dropbox.com/scl/fi/vujxscaffw37pqgb6hpie/MAXB.pdf?rlkey=0oqjxs5u49eqa2m7uaol64lbw&dl=0 TOC: 1. Model Reasoning and Verification [00:00:00] 1.1 Model Consistency and Reasoning Verification [00:03:25] 1.2 Influence Functions and Distributed Knowledge Analysis [00:10:28] 1.3 AI Application Development and Model Deployment [00:14:24] 1.4 AI Alignment and Human Feedback Limitations 2. Evaluation and Bias Assessment [00:20:15] 2.1 Human Evaluation Challenges and Factuality Assessment [00:27:15] 2.2 Cultural and Demographic Influences on Model Behavior [00:32:43] 2.3 Adversarial Examples and Model Robustness 3. Benchmarking Systems and Methods [00:41:54] 3.1 DynaBench and Dynamic Benchmarking Approaches [00:50:02] 3.2 Benchmarking Challenges and Alternative Metrics [00:50:33] 3.3 Evolution of Model Benchmarking Methods [00:51:15] 3.4 Hierarchical Capability Testing Framework [00:52:35] 3.5 Benchmark Platforms and Tools 4. Model Architecture and Performance [00:55:15] 4.1 Cohere's Model Development Process [01:00:26] 4.2 Model Quantization and Performance Evaluation [01:05:18] 4.3 Reasoning Capabilities and Benchmark Standards [01:08:27] 4.4 Training Progression and Technical Challenges 5. Future Directions and Challenges [01:13:48] 5.1 Context Window Evolution and Trade-offs [01:22:47] 5.2 Enterprise Applications and Future Challenges REFS: [00:03:10] Research at Cohere with Laura Ruis et al., Max Bartolo, Laura Ruis et al. https://cohere.com/research/papers/procedural-knowledge-in-pretraining-drives-reasoning-in-large-language-models-2024-11-20 [00:04:15] Influence functions in machine learning, Koh & Liang https://arxiv.org/abs/1703.04730 [00:08:05] Studying Large Language Model Generalization with Influence Functions, Roger Grosse et al. https://storage.prod.researchhub.com/uploads/papers/2023/08/08/2308.03296.pdf [00:11:10] The LLM ARChitect: Solving ARC-AGI Is A Matter of Perspective, Daniel Franzen, Jan Disselhoff, and David Hartmann https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf [00:12:10] Hugging Face model repo for C4AI Command A, Cohere and Cohere For AI https://huggingface.co/CohereForAI/c4ai-command-a-03-2025 [00:13:30] OpenInterpreter https://github.com/KillianLucas/open-interpreter [00:16:15] Human Feedback is not Gold Standard, Tom Hosking, Max Bartolo, Phil Blunsom https://arxiv.org/abs/2309.16349 [00:27:15] The PRISM Alignment Dataset, Hannah Kirk et al. https://arxiv.org/abs/2404.16019 [00:32:50] How adversarial examples arise, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry https://arxiv.org/abs/1905.02175 [00:43:00] DynaBench platform paper, Douwe Kiela et al. https://aclanthology.org/2021.naacl-main.324.pdf [00:50:15] Sara Hooker's work on compute limitations, Sara Hooker https://arxiv.org/html/2407.05694v1 [00:53:25] DataPerf: Community-led benchmark suite, Mazumder et al. https://arxiv.org/abs/2207.10062 [01:04:35] DROP, Dheeru Dua et al. https://arxiv.org/abs/1903.00161 [01:07:05] GSM8k, Cobbe et al. https://paperswithcode.com/sota/arithmetic-reasoning-on-gsm8k [01:09:30] ARC, François Chollet https://github.com/fchollet/ARC-AGI [01:15:50] Command A, Cohere https://cohere.com/blog/command-a [01:22:55] Enterprise search using LLMs, Cohere https://cohere.com/blog/commonly-asked-questions-about-search-from-coheres-enterprise-customers

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The expectation of machines is consistently, they're right all the time. It's a function of traditional software.

• 0:10Speaker 00

  If you had a calculator, you put in one plus two, the calculator said three, and then you try two plus seven and the calculator said seventeen, you throw away your calculator, you wouldn't try any other questions. We need that consistency and we need that robustness. The next step is really convince a human that the model is actually consistently doing the, what it's being asked to do.

• 0:33Tim Scarfe?

  Do models reason?

• 0:34Speaker 00

  Maybe they do. I think reasoning is a bit of an overloaded term. It's used to mean many different things. I don't even know if that's the right question. Do we really care if models are reasoning the way we might think? If humans are consistently successful at getting the model to fail on examples that are slightly different, then I think it's fair to assume that the model isn't really reasoning. It's just figured out how to do well on a specific benchmark.

• 1:02Tim Scarfe?

  MLST is sponsored by Tufer AI Labs. Now, they are the DeepSeek, based in Switzerland. They have an amazing team. You've seen many of the folks on the team. They acquired Minds AI, of course. They did a lot of great work on ARC. They're now working on O1-style models and reasoning and thinking and test time computation. The reason you want to work for them is you get loads of autonomy, you get visibility, you can publish your research. And also, they are hiring, as well as ML engineers, they're hiring a chief scientist. They really, really want to find the best possible person for this role. And they're prepared to pay top dollar as a joining bonus. So if you're interested in working for them as an ML engineer or their chief scientist, get in touch with Benjamin Cruzia. Go to tuferlabs.ai and see what happens. Welcome to MLST. It's an honour to have you here.

• 1:47Speaker 00

  Thanks, Tim. Thanks for having me.

• 1:49Tim Scarfe?

  Introduce yourself.

• 1:50Speaker 00

  I'm Max. I am a researcher at Cohere. I work on many things, primarily post-training. But I'm also interested in adversarial data collection, better ways of collecting data for improving models, valuation, which kind of drives this continuous feedback loop where we target what we care about in terms of model performance and then figure out how to improve performance on those. Dimensions and then often you kind of see performance saturates relatively quickly. And then at that point, you're going to need to figure out, well, what's the what's the next layer of evaluation that we care about, you kind of iterating in this in this continual process. I'm particularly interested in getting models to reason, to operate more robustly and to generally be more useful. I think reasoning is is a bit of an overloaded term. It's used to mean many different things.

• 2:35Speaker 00

  I don't even know if that's the right question in that, do we really care if models are reasoning the way we might think, you know, humans reason. And I think there's debates about whether or not we reason in the ways we think we reason. But it's clear that, you know, models trained with next token prediction can do some very impressive things. And whether or not actual reasoning is happening under the hood, I think, is an interesting research question, but not really a bottleneck in terms of getting models to do the things we would like them to do. So, yeah, we have this this recent work with Laura Reese. She worked worked with us at Echo here, and she investigated during pre-training, what

• 3:17Speaker 00

  types of reasoning or what types of information models were learning from, you know, procedural knowledge in in pre-training documents.

• 3:25Tim Scarfe?

  The elevator pitch that she said to me is, you know, I think many of us, we've been quite skeptical about reasoning, right? You know, I'm inspired by Sholay, and we say that these things are, you know, not quite, not quite hash tables, but, you know, they're doing some kind of curve fitting, some kind of retrieval or something like that. And Laura said that she went into it thinking the same.

• 3:45Speaker 00

  Yeah.

• 3:46Tim Scarfe?

  And and she was quite surprised by the results. So what what did she find?

• 3:50Speaker 00

  We we went into it thinking that models basically just retrieve facts from, think of it as as kind of a compressed version of all the information that they're pre-trained on. And they were basically relying on this parametric knowledge, which is this knowledge stored in the model's parameters to to answer most queries that look like recent queries. So what we did is we used something called influence functions, which basically you can think of as a way to approximate the effect of a particular training. Example on a model's behavior and we had roughly forty factual questions. We had another forty reasoning queries.

• 4:28Speaker 00

  And one of the findings, and there are quite a few interesting findings, was for the factual questions, the models were really mostly relying on documents that contain the answer to the question. And, you know, at most, there'd be a few documents that the model was relying on in order to answer a factual question. But for the reasoning queries, there it was a lot more complex, right? There was information spread across many more documents. Uh, many times documents that contain the types of reasoning that would be required to answer um the question in the first place. And it was just much more distributed. So uh the the influence kind of uh distribution was a lot more spread out.

• 5:07Speaker 00

  And it definitely gave the impression that um the model was just relying on uh precision knowledge that it had picked up from these various different sources and was potentially combining them in in interesting ways. Yeah, like we went into this research expecting to find the opposite. And I really, you know, quite enjoy work where the results really challenge your way of thinking about the problem. And I think definitely for me and for Laura as well, um it's it's altered the way we kind of perceive how these models operate. The the kind of counter to that is possibly, well, there's all these uh reasoning queries that are kind of very similar. And so maybe it's just the case that models are, you know, just relying on on these different documents because of the similarity of the of the queries.

• 5:48Speaker 00

  And and so we had a set of control uh questions as well, which um looked um kind of lexically, lexically and structurally very similar to the actual reasoning queries, but didn't require any reasoning uh in order to answer them. So I'll I'll give an example, right? Say we have a question such as, you know, a line is defined by points two comma two and three comma three. Uh, what is the slope of the line? You know, you need to kind of figure out what the right equation is, figure out that the slope is one and give the answer. And to do that, you might rely on information you've picked up from these documents. And the control query would look something like. The slope of the line is one, what is the slope of the line?

• 6:30Speaker 00

  Or, you know, a line is defined by the fact that it has a slope of one, what is the slope of the line? Sounds and kind of is structured very similarly to the question we care about, but doesn't require any reason because the answer is just provided there. And what we saw is a big contrast in terms of the way models processed those kinds of information. So that really gave us, you know, a strong indication that the model was actually doing some form of reasoning in order to calculate these slopes. One of the challenges with influence functions is they just naturally don't scale very well. In the very naive sense, you can calculate the influence of an example by,

• 7:13Speaker 00

  you know, retraining a model without that example and comparing the difference. That's obviously extremely expensive. And influence functions themselves kind of require the computation of the universe Hessian. And that's useful because it gives you this kind of second order information about the loss landscape of the model. So it gives you a sense of the curvature and the structure effects on the loss landscape of including this additional example that you care about. And it's just extremely expensive, right? The Hessian is like a big matrix, it's got dimensions in each axis, which is the number of parameters in the model. We're working here with seven billion and thirty five billion parameters.

• 7:58Speaker 00

  So it's just extremely computationally costly. And so we use basically built on work, which Anthropic released late last year. They scaled up kind of approximate curvature estimation using ECGVAC, which was released somewhere around twenty eighteen, I think. And so then that really allowed us to, along with some other optimizations, you know, scale this up to quite large model sizes. And I think on the note of model size, one aspect we found also very insightful was That it seemed like the, you know, that there was very little correlation between

• 8:41Speaker 00

  which documents were influential for the seven B model and which were influential for the thirty five billion parameter model. And and we didn't really have much time to dig into the details of that, but that's also extremely interesting. It's it's indication that models of different sizes are just potentially learning things in in different ways.

• 9:00Tim Scarfe?

  What is your definition of reasoning? And what what would it mean for a model to reason really well?

• 9:05Speaker 00

  I think it comes down to, so for me, reasoning and robustness are kind of interlinked in that if you can reason, then I would expect, you know, correct reasoning to imply robust reasoning. And like, take an example, say we're going back to the, you know, calculating the slopes of lines. If a model can really reason through that relatively straightforward process, right? It's you need to understand what what function to apply and then apply that function consistently. And if a model does that nine hundred ninety nine times out of a thousand, but fails one

• 9:47Speaker 00

  out of those thousand times, to me, it really starts to question, like bring into question, is this model actually reasoning in a way that I might expect a human to reason? And the the counterargument there is always, oh, but humans are never, they're not infallible, right? They always kind of make mistakes and get things wrong as well. Machines have a bit of an advantage there. They don't get tired. They don't have the limitations of being human, I guess, to contend with. And so I'd really expect a reasoning requirement for a kind of specific type of reasoning or a specific definition.

• 10:26Speaker 00

  To really be be robust and consistent.

• 10:28Tim Scarfe?

  Yeah, I think so many people are starting to see now how valuable this technology is. I mean, I'll give an example like this, this app for tracking the podcast. I wrote it in half an hour, right, just just using using language models. And we need to reimagine many of the apps. I think the first strategy was rubbing AI into the existing apps. And now we're seeing a whole new generation of apps that are that are AI first. And it's like the iPhone moment. And I think people are starting to to really see that. But let's talk about test time training just just real quick. You know, it's been heralded as a as a new scaling law. And certainly on the ARC challenge, all of the winning, you know, results were this kind of

• 11:10Tim Scarfe?

  transductive active fine tuning, which is that, you know, rather than using a global shared model, which was trained once by someone else, I'm going to take that model and I'm going to fine tune it in my situation. And I've now got a model that works very well in this situation. And it seems to be like a much more distributed paradigm of doing AI. But I mean, obviously, you work for for Cohere, which is quite centralized at the moment. I mean, how how do you see the future of having big centralized models, but also having more of a diffused kind of training methodology?

• 11:39Speaker 00

  I think we're just going to see more and more. I think the bottleneck so far has been, you know, how how many people have the kind of knowledge and skill set and the experience to train these kind of models? And as kind of more and more people get excited by AI, by the kind of promises that it it holds, the, you know, the work around, you know, sharing these models, like the Cohere models, for example, have the way it's openly released, you can you can kind of download, you know, Cohere's Command R plus or Command R model of of hugging face, you can continue fine tuning that. And and we've seen lots of people doing that. And I think I think there are some constraints.

• 12:22Speaker 00

  You need to figure out, you know, what the optimal way of of doing that is without potentially losing, you know, general knowledge capabilities that might exist in the base model. But it's.

• 12:37Speaker 00

  There's one interface which holds across models and across humans as well, which is language. And if you can, and we know that you can do this to a very real extent, if you view it from the point of what we're doing over time is making the data better in terms of what we can learn from, then that unlocks, you know, possibilities globally, like pretty much everyone. If you had the right way of using models to improve the data, have kind of a shared data resource, could train a model from that data, assuming they had the know-how and the compute.

• 13:20Tim Scarfe?

  Yeah, it's almost a mindset thing. I mean, I've written apps now where it's easier to hook it up to a chatbot interface than it is to write a UI, because you just have this insane amount of sophistication. I use Open Interpreter, for example, and I've now adopted this pattern where in Open Interpreter, I just load my Python multi-agent system into memory, and then I just tell the agents to reveal their interface, and then Open Interpreter just works. So I can manipulate my database, and I say, you know, add something to the table of contents, and like add this reference, delete this reference, and it just works. I don't need to do anything. It's a completely different way of thinking about software, and I think it's just going to take us a long time to figure this out, but it's genuinely amazing.

• 14:01Tim Scarfe?

  And also, you can have second-order intelligence. So rather than talking directly to the interface, you're going to be talking to another agent which itself is doing intelligent things, and it's just mind-boggling. It's like, it's like the Matrix.

• 14:11Speaker 00

  Possibilities are endless.

• 14:12Tim Scarfe?

  Yeah, it's insane.

• 14:15Tim Scarfe?

  On AI alignment, before we move on to this, because you were talking about some of the social implications as well before. I mean, first of all, what do you think about AI alignment?

• 14:23Speaker 00

  Obviously, very important, right? We want AI systems that don't do bad things. That's kind of the starting point. We don't want AI systems that, you know, damage our environment and make our societies worse and impact humanity negatively.

• 14:44Speaker 00

  I think the...

• 14:48Speaker 00

  The question is really, what do we want to align AI systems to? We can, you know, typically you define alignment as having models whose kind of values or behaviours are aligned to what humans want or what humans expect. But if you break that down, I'd argue it's probably already very hard to figure out what do humans want and what do humans expect and do all humans want or expect the same things? And is that kind of static in context or does that change depending on the context you're in? And so it's a very complex problem that we need to solve and we haven't

• 15:32Speaker 00

  necessarily solved that societally with, for humans either.

• 15:38Speaker 00

  And now we have to solve it for, you know, this new technology that's kind of permeating our lives. And so I think, I think we need to, we need to figure all of that out, but we also need to have some basic safeguards in place to make sure that, you know, the, the, the technology remains safe for, for general use so that we can continue to advance and, yeah, and develop and make sure we, we make the best use of its potential.

• 16:08Tim Scarfe?

  Let's go over to human feedback. So you, you wrote this paper with, with Tom and Phil called Human Feedback is Not a Gold Standard. Can you, can you tell us about that?

• 16:17Speaker 00

  The Human Feedback is Not a Gold Standard paper was motivated by, you know, all the interest and excitement around using human feedback. Pretty much throughout the, the model training and evaluation pipeline. And so, you know, a couple of years back, RHF was a big thing. It was the kind of new thing. Everyone was excited about it. And the, the general way of doing RHF was, well, we give humans a prompt and they see two completions, right? So say, what color is the sky? Completion A is blue and completion B is yellow. And a human annotator looks at those. Two completions and says, I prefer completion A. Hopefully, they have good reasons for preferring a completion over another.

• 17:03Speaker 00

  And and there was massive, massive progress. We saw in many of the benchmarks from from this idea of of using human feedback to optimize and kind of fine tune. Fine tune these models. And we we quickly started to see diminishing returns from from this. And the the effect was really, at least initially, that the the style of the output that humans preferred was was providing better results. And in many cases, the better results were also measured in some way by human preference. So you either had, you know, a human evaluation

• 17:46Speaker 00

  task where humans would look at output from the model and say which they they prefer in order to kind of rank models against each other. Or you train a reward model and use that kind of as a proxy for for what a human would prefer. And thi- this idea of a a single preference score that captured everything about the generation just seemed quite, quite limited. And what we started out looking at was, what if we make that feedback signal more granular? What if we, you know, try to understand what about specific model generations do humans prefer over others? And so the first experiment we ran in that paper was we categorized these different error types.

• 18:30Speaker 00

  We identified examples in kind of established data sets which to which models generated completions which contain these errors. And then we we asked one group of humans to rate these completions by whether or not they had certain errors present. And we asked a different and independent group of humans to annotate a rating score of those completions from from one to five. And then Tom ran some very detailed analysis and tried to figure out, well, what types of errors contribute to?

• 19:04Speaker 00

  Human preference judgments of quality of overall quality. And what we found was quite interesting. We found that humans really, really dislike if the model refuses to answer a question. That thinking about it seems extremely intuitive. Human preference judgment is strongly informed by formatting and by style. And we see that in some of the early work as well, where, for example, a lot of the early summarization work seemed to suggest that humans preferred longer summaries, which is kind of also quite intuitive, because a longer summary contains more information. And even just by that fact is more likely to contain more relevant information from the summary.

• 19:48Speaker 00

  And what we found interesting was that certain features, when, you know, say we at Cohere, we talk to enterprise customers and they care that the models are correct, they care that the models, you know, aren't repetitive, they care about certain properties that might be different from what, you know, an annotator looking at two different completions might care about. And attributes or criteria like factuality were quite lowly ranked in terms of their contribution to overall quality. And so we thought, well, that's that's potentially problematic, right? We, to some extent, optimizing these models to give more, you know, elaborate, more interesting

• 20:30Speaker 00

  completions at the expense of them potentially being less correct. The follow up to that was, then we wondered, well, could it be that there are also certain confounders or other effects that we're not we're not noticing? And we had this hypothesis that maybe assertiveness and complexity of of model outputs might impact how how humans perceive, you know, the quality of a completion. And and so we ran a very similar experiment. What we did is we prompted these models, a kind of very simple one line prompt, something along the lines of, you know, give the response, but, you know, sound very confident. What we found was that, you know, there was a massive

• 21:14Speaker 00

  Effect in terms of the uh the generations that were designed to sound more confident, were consistently rated as higher quality and as having fewer errors. And there's uh there's a particular plot in the figure, which I think shows this quite nicely, where you have this almost uh almost like a an elbow curve, where uh you see the error rates uh on the Y axis and on the X axis, you have uh kind of the the assertiveness value uh between one and five. And and as assertiveness reaches a certain point, the error rates um practically tend to tend to negligible. And we we know that this is also uh kind of a a trait of um

• 21:59Speaker 00

  of how humans interact with other humans, right? If if someone sounds very confident, they're more likely to be perceived as, say, an expert in their field. There's uh quite a bit of work in this space. Um, we didn't necessarily think that the effect would be um so obvious when uh humans are are interacting with with AI systems. And in many cases, these are, you know, trained annotators. They've been given very clear instructions about what and how uh to to infer the quality of these different completions. One one other striking point uh from that work for me was um this effect of uh assertiveness on um perceived correctness.

• 22:41Speaker 00

  The the more assertive uh generations were generally perceived by humans to be more correct. So we we kind of looked through a sample of these uh these examples judged um, you know, as as expert annotators and inverted commas, uh mostly Tom, kind of going through the data and and rating uh, you know, whether there were any uh factuality errors. And what we found actually was that when the model was prompted to be more assertive, it tended to be less factual. And even though it was less factual, because it was more assertive, uh annotators. You know, got the impression that it was actually more correct. And so that that to us was striking because we have this very clear example of

• 23:24Speaker 00

  something counter to the behavior we want to optimize or the behavior we want to incentivize in the model, but also doing so in a way that, you know, is barely recognized, right? The annotators doing this didn't realize that they were being influenced or biased in in these ways. And that's, I think, representative of, you know, the just this very large scale effort of optimizing towards human preference with a very kind of specific and to many extents, underspecified definition of human preference.

• 23:58Tim Scarfe?

  And isn't it funny? Because we were saying earlier that humans are rational, right? But you've just given us many, many examples of how irrational we are, right? We have this style bias. We we see something that's assertive and we think it's correct and so on. And you also gave an example of, well, you know, what if what if we try to get annotators to compare two entire books? Well, that's clearly not only beyond our cognitive horizon, but just the the ambiguity just goes up exponentially. It seems like it wouldn't be possible for us to do that in in any reliable way. So what can we what can we do here? You know, you you said, let's have more attributes. But how far do you go down that road? Because won't you end up with thousands of attributes?

• 24:39Speaker 00

  Yeah, for sure. And I have a similar problem, right? I'm I'm Mediterranean culturally, I just like answers to be very direct and to the point, right? I'm the kind of person who, if you ask me, what is the color of the sky? And I had blue and the color of the sky is blue, I would prefer blue. And that's that's not what most humans would would prefer. And you can you can definitely go, you know, very deep down the rabbit hole of, well, what, you know, what are the criteria? What are the dimensions? What are the things we care about? And I don't, I don't necessarily think that that's a problem.

• 25:14Speaker 00

  I think um one, one uh area I'm very interested in is this idea of um modifying model behavior to suit uh kind of uh personal requirements um on the fly, right? Without additional fine tuning. So if you have all of this data, imagine I could build some kind of, uh you know, portrait of Tim's preference in data, right? There's these hundred data points that reflect how Tim wants to interact with the model. And, and maybe those can change and they probably will change over time. It's customized, it's specific to you, it's kind of, that captures all of the information of what you personally are looking for in um, you know, your interaction or the behavior

• 25:58Speaker 00

  of the model you're, you're working with. And you could do something relatively straightforward, uh if you wanted to like, use that as, you know, part of the, the prompt or part, effectively doing context learning, right? Provide the model with that um at inference time, say, this is Tim, these are his preferences, generate according to those preferences. And um and models we've seen are relatively good at conditioning um on, you know, even complex information uh in, in input space. Like generally this idea of models built for specific people for specific reasons without having to go through the, you know, the overhead and the cost of training a model for every individual on the planet, right?

• 26:43Speaker 00

  If we had to train, you know, seven billion uh G P T fours and serve those models, uh that would be uh very prohibitive, prohibitive from a, a resourcing perspective. But if we could just train one model, which was extremely good and again extremely robust and really conditioned uh well on kind of the requirements for an individual, then uh we could do that relatively cheaply um in, in kind of relative terms. Yeah, it's, it's, so this kind of brings me to um Prism, which is this uh, this project led by Hannah Kirk, uh just won a, a best paper award at, at NURIPS. And the idea there was, well, let's figure out how

• 27:28Speaker 00

  At scale, these different attributes, right? Demographics, cultural kind of influences, you know, geographic effects, linguistic effects, impact what it is that humans care about and how humans want models to behave. And, you know, it's super interesting work. There's, there's a lot in there. There's over a hundred pages of content. So I highly recommend that everyone, you know, reads that paper at least twice. It's, it's extremely rich in terms of the insights it provides into, you know, this whole space of, of understanding what human preference means

• 28:13Speaker 00

  and what it means for different people. There, there's, there's some extremely interesting findings, one of which is just representation. So even, even something as simple as what do people converse with models about is, is just highly influenced by, you know, their backgrounds or by who they are as people. And you might argue, well, that's potentially not such a big deal. But you, you, you always have to see model development in this kind of whole, you know, cyclic feedback loop effect where the conversations that people are having with these models will drive the improvements to the models.

• 28:55Speaker 00

  And even if your usage is not well represented in a kind of standard post-training data, then the model is just going to be worse at the things that you care about. Then it will be, you know, something that more people care about relatively. So, you know, there were, there were examples like this, this kind of work was run over something like a year and a half. So it was, you know, quite, quite a huge project. And there were examples along the lines of, you know, there, there was a substantial amount of conversation around the Israel Palestine conflict. And most of that was happening from, you know,

• 29:40Speaker 00

  people based in the Middle East. And if.

• 29:46Speaker 00

  If most of the data going into these models does not represent even something as simple, we're not even talking about human preference here in terms of completions. We're just saying, what is it that we're interacting with these models about? Then you're just going to have all these biases and problems down the line where the model doesn't serve everyone. It serves the people who have built it to some extent, or the people who have contributed most to using it. Then, yeah, so this work again is very, very detailed. There's a lot of analysis about what about completions or like model generations impacts human preference.

• 30:27Speaker 00

  So there's quite a few findings similar to the human feedback is not gold standard work, the same kind of general sense that refusal and formatting and style play strong effects. Like, we generally have the sense of averaged out a rank of models that we consider the best, and then kind of the second best, and then the third best. And there's a set of, I guess, established benchmarks or established methods of evaluating models, like the childbot arena, where, again, it comes back to what is human preference really capturing, right?

• 31:11Speaker 00

  We have this almost canonical ranking of models, but it's heavily influenced by who the people providing the judgments are, what they're asking about, and what their personal preferences are. And so what Prism does is it really breaks down what, like, how ranks of preferred models change depending on who's asking the questions or who's interacting with the model. And, yeah, you see, you know, big, big changes in, even in terms of which models are better, you see models kind of moving, you know, five positions up the kind of ranking or five positions down the ranking, or more. And there's 21 models in the work, so that's quite a big change.

• 31:54Speaker 00

  And that can be very insightful for model developers as well, if you fundamentally believe that you're building technology to serve all of humanity. Our coherent models, for example, command, this is like a very early, like, first generation of command models.

• 32:11Speaker 00

  Performance, you know, very well across most cases. And then for some reason, for users in Asia, performance just seems to drop relative to the other models. And that kind of insight can be extremely informative for trying to figure out what is it that we can do to help serve, say, people in that location better. And, and, you know, based on insights like that, we've taken many actions to improve how we go about building these models.

• 32:43Tim Scarfe?

  Yeah, I mean, Andrew had this really good paper out called adversarial bug, sorry, adversarial examples are not bugs, they are features. And he was saying that they learn these non-robust features, which are basically these like, you know, weird blue pixels mean Mercedes and stuff like that. And they, they happen to generalize really well.

• 33:04Speaker 00

  Yeah.

• 33:04Tim Scarfe?

  But from a, from a reasoning point of view, that they're clearly doing it for the wrong reason. So it just happens to be the case that, that they work now, but when there's a distribution shift, they'll, they'll stop working tomorrow.

• 33:16Speaker 00

  Exactly.

• 33:16Tim Scarfe?

  So we almost want to have a principled way of kind of saying that's a good feature and that's not a good feature.

• 33:23Speaker 00

  Yeah, which, you know, in these high dimensional spaces is just very hard to do. But you can, you can control the input and the output spaces. And I think that's, uh,

• 33:38Speaker 00

  yeah, that, that's where, so, so, you know, there's, there's the general sense of adversarial examples where the adversarial noise is generated by a function. Think of something like, you know, injecting Gaussian noise, which is undetectable into an image, or switching a word for a synonym in, in language. And you'd argue that, well, the, the model's output shouldn't change because nothing has changed in terms of the, the semantics of what the, the, the query requires or means. And

• 34:12Speaker 00

  It's often the case, I think, that you see

• 34:18Speaker 00

  the model's response to that is to basically learn to cope with the function that generated the adversarial noise. And so it's not really learning to deal with real world noise, it's just you have a function attempting to simulate real world noise, which tends to be simpler and easier to learn and counteract. So that's why I'm very interested in

• 34:43Speaker 00

  human-informed adversarial examples. So this this general idea of have a human and a model interact, have the human, you know, probe the model, try to identify weaknesses in the model, failure modes, things that the model either consistently or not so consistently gets wrong, and use that as, you know, seed or like inspiration for training data to then train the model on it and make the model more robust to, let's call it a more, hopefully, complex and representative kind of shape of this this noise distribution. And so that's

• 35:23Speaker 00

  what that's what Adversarial QA tried to do. So this work from 2019, in this paper called BDAI, which was my kind of first PhD paper. And the idea wasn't wasn't necessarily novel from the perspective of getting models and and humans to interact. It's had been quite quite a bit of work previously. So drop again, this dataset we mentioned earlier, had kind of a small component of the way it was constructed, be adversarial in the sense. There was work on on the Quiz Bowl task. And so quite a few people have tried have tried this in the

• 36:07Speaker 00

  past. And what we really wanted to look at was what are the different effects that doing data collection in this way and in this way only have on, I know, what models learn, how rich the representations they learn, how robust those representations are. And yeah, we found we found some extremely interesting findings. We found, for example, that the diversity of of the questions that you collect when having a model in the loop are just a lot more diverse, a lot more complex, generally a lot more interesting. You tend to see the the quality of the data you collect go down as well, just because

• 36:51Speaker 00

  you start to see more ambiguity, you start to see, you know, more implicit information that kind of a human would easily understand, but which is may may maybe kind of slightly underdefined and is just in reality a lot more representative of real world interaction, where, you know, not every question is perfectly defined or or perfectly specified. And yeah, so so we we had kind of three models, which were state of the art back then. We had Bert

• 37:32Speaker 00

  and Roberta, and we also had Bydav, which is a kind of earlier question answering model, which was state of the art back in twenty seventeen, eighteen, probably. And what we found was that even even adversarial examples collected against relatively weaker models were very beneficial for training stronger models. And so it almost felt like there was this shift in distribution, where you had the non adversarial setting, where think of it as there's a paragraph and an annotator needs to find an answer in that paragraph and ask a question for which the answer is is kind of that span extracted from the paragraph.

• 38:15Speaker 00

  And and in that setting, you you get very.

• 38:23Speaker 00

  Let's call them bland questions, right? Relatively easy things, like say, um, you know, say the paragraph is talking about uh the sky. Uh, we would use kind of a consistent, yeah, consistent example throughout. Um, and so this paragraph is talking about, uh, you know, the sky and how it's blue and all of this. And the question is, what color is the sky? And you have this almost, um, you know, direct mapping, uh, even in terms of like lexical overlap to the paragraph, to where the answer is located in the paragraph. And, and the annotated answer will be blue. So then the setting is now you've got a model in the loop. Now the model is going to attempt to answer the question. So you're going to ask the model, what color is the sky?

• 39:07Speaker 00

  And the model is going to say blue. And so you haven't located an adversarial example. You found an example that very easy for the model. So what you're really trying to do is maximize the signal of the data you're collecting. And, um, and now you have to tweak your question and you have to ask something like, uh, what is the color of the thing in which the clouds are? And maybe it's, you know, a bit more complex, a bit more involved. And maybe now the model, you know, finds another color in the paragraph and says green, and it gets it wrong. And so you've managed to, to trick the model. Um, and, uh, and through this process, you just get a lot more, uh, interesting data, both from a training data perspective.

• 39:51Speaker 00

  So we saw, you know, pretty massive improvements in terms of, uh, when you train on this data, how just robust the performance of the models is. And I think that to me personally, at least, um, you know, at the time, there were quite a few, um, startups in the, in the question answering space. I knew quite a few of them personally and, and they would tell me, you know, we, we've added your data set to, to our model's training and it is just way better in real world application. And You know, they'd see improvements on the benchmarks as well, but they were particularly impressed by just how how big the effect was in in real world settings. And and I

• 40:34Speaker 00

  think that's what robustness gives you. It makes it makes these systems a lot more applicable to to, you know, settings we care about. You know, from from some perspective, it's not a very complex task, right? You have one paragraph, it's from Wikipedia. It's relatively simple language, it's very easy to understand. And then you have these kind of more complex questions that come about from, you know, humans interacting with the model and being creative in terms of how they probe it. And and here we have models that represent, you know, state of the art in in 2019, so five years ago. And and Adversarial QA was recently used to probe and test the adversarial robustness

• 41:19Speaker 00

  of of the Lama tree family of models. So in in that tech report is kind of referred to as Dyna Bench QA, and we'll get into kind of Dyna Bench later. Dyna Bench was kind of the follow up to adversarial QA. And, you know, while while LLMs today massively outperform models we had back then on this specific task, they're still quite far from human performance, which suggests that there's still, you know, there's still a way to go in terms of general robustness capabilities.

• 41:54Tim Scarfe?

  Isn't it weird how we keep talking about this mixture of pattern matching and reasoning? I mean, the the example you gave was beautiful. You know, what is the color of the thing where the clouds are? I mean, you know, you could reason your way through that, but there are so many other examples in English where we just make it up as we go along. You know, it's just like this, it's like the language game.

• 42:11Speaker 00

  Yeah.

• 42:11Tim Scarfe?

  We just make stuff up. And it's just, it fascinates me that it's it's a combination of of what you're saying. So we we just robustify and we just, we just kind of incorporate all of these different patterns. But we also want to have systems that reason as well.

• 42:26Speaker 00

  You know, I I'm sure I could continue to come up with more complex and more intricate ways of asking the question such that the answer would remain unchanged in ways that aren't reflected in the model's training data. And sure, in this case, I'm I'm potentially intentionally trying to confuse you. But I think the question is, well, I shouldn't be able to if your representations are robust.

• 42:57Tim Scarfe?

  Tell me about Dyna Bench.

• 42:59Speaker 00

  So Dyna Bench was kind of a follow-up project from adversarial QA and adversarial NLI, which Yishini and Dawa Kila and Adina Williams did. So Dawa was the mastermind of this project. He did all the heavy lifting to get the funding in place, to get the support for building out this project.

• 43:29Speaker 00

  It was originally started at Facebook AI Research. So I interned there, working on Dyna Bench in 2020, I want to say, early 2020, I think. And then that carried on for roughly a year. A global pandemic happened in between as well, so it was an exciting time. So Dyna Bench is a platform, it's a research platform for testing and probing the span of model-in-the-loop adversarial data collection, and particularly doing that in a dynamic fashion. So this idea of you start out with some dataset, right?

• 44:13Speaker 00

  You train a model on it, and then you interact with that model, and you want to find where its weaknesses are. You want to probe it, you want to figure out what its failure modes are, and you collect data in doing that, which you can use as training data. You get the training data, and you retrain your model typically on the original data plus your new adversarial data, which makes the model a lot more robust. And what you often get as a by-product is a model which, for most of the types of complex adversarial prompts you were trying previously, now would generally start getting those right. It would still typically fail on some set of those prompts, which were generally more complex.

• 44:55Speaker 00

  And then it would challenge you as an adversarial annotator to now think beyond the limits of this new model and start trying to probe it and break it in different ways. And you can just keep doing that over time. And what you're doing is effectively just building a more and more robust model to the point where it just becomes incredibly hard, hard to fool. And so DynaBench has kind of powered, I'd say, many, many research projects in the space.

• 45:29Speaker 00

  There were projects like hate speech detection, sentiment analysis. Prism itself was run through, through the DynaBench platform. There's work ongoing with Common Crawl to try and figure out how we can kind of improve the, the quality of the crawls for, for pre-training data. And DynaBench has kind of since, it now sits with the ML Commons community. And so there's, there's this working group, the data-centric machine learning research working group, which is responsible for kind of maintaining DynaBench and making sure that, you know, people in the space who want to contribute, who want to explore ways of

• 46:14Speaker 00

  making models more robust, ways of, you know, building better, both training sets and evaluation sets, have, have a space and have the, the resources and tools to do that. And the, the other aspect of DynaBench, which is kind of the benchmarking piece, which is where it gets its name from, is this idea that it's not just the interaction with models that's dynamic, but also, also our benchmarks. For quite a long time, and I'll argue probably still today, the, the general way we've operated as a community has been, you know, people create benchmarks because there's some interesting phenomenon they want to test.

• 46:58Speaker 00

  And that benchmark is created. Once it gets created, you effectively have good hard score, right? You have you have a clear measure of what it is you want to optimize for. And then you've got incredibly smart, incredibly talented people in the community optimizing for that specific problem that the benchmark reflects. And typically, you see progress, which just happens really, really rapidly on that specific benchmark. So so DAWA has this plot in the DynaBench paper, where you just kind of see saturation of different benchmarks over time. And if I recall correctly, it was, I think, somewhere

• 47:39Speaker 00

  around eight to 10 years for MNIST, for example, and then somewhere around like one and a half to two years for Squad, and it just gets shorter and shorter with each benchmark, primarily as a function of the field growing and maturing, right? There's just more people working on the problems, which means progress happens generally faster. So one of the core kind of philosophies behind DynaBench was this idea that we want to ideally also benchmark dynamically, and that our benchmarks should evolve and should should change over time so that we, you know, we're really measuring the the current set of capabilities of models that we care about. And and I think that's very true for for humans as well, right?

• 48:21Speaker 00

  You wouldn't give someone with a PhD a great school math exam and use that to figure out, well, you know, should this person perform surgeries or or something relatively important? And to some extent, we do some version of that with with LLMs. I've seen so many cases of people who have a very clear task in mind, a very clear understanding of how they want the model to behave in that setting. They have, you know, the domain knowledge, the expertise to to validate performance on that task really well. And then that's like which model they want to use because it has a higher MMLU score, or because it ranks

• 49:03Speaker 00

  higher in chatbot arena. And I don't know that that's the best way.

• 49:11Speaker 00

  To ensure we're really evaluating the full breadth of what we care about in terms of model capability. And I think one of the best ways to do that is to build benchmarks that reflect, you know, the task, the application that you care about. And ideally, if you can do that dynamically, such that, you know, today's model is at seventy percent. As technology continues to advance, it gets to a hundred percent. That probably doesn't suggest that the model can solve every possible kind of situation or setting that you're putting it in, but rather suggests that your benchmark was limited, which, you know, it will always be because it's conditioned on some understanding of

• 49:56Speaker 00

  existing model capabilities. And so you just want to make sure that your evals also update themselves over time.

• 50:03Tim Scarfe?

  Yeah, benchmarking is an absolute nightmare. I mean, the great thing about benchmarks is they're standardized, you know, so we can compare apples to apples. I was speaking with Sarah Hooker in the summer and she was advocating, you know, instead of having an absolute flops limit, for example, which could get good-hearted straight away. She was suggesting having a basket or an index or dynamic benchmark or something like that. How can we design new types of benchmarks that are somewhat impervious to Goodhart's Law?

• 50:31Speaker 00

  I think we are almost caught in a local benchmarking optimum where we've just done things in a certain way for many years. And it almost seems like the naturally right way to do. And I think we potentially, as a community, right, we should probably think about taking a step back and thinking about, given where the technology is today, given what language models can do today, what is the right way to think about evaluating them? And I think.

• 51:10Speaker 00

  Examination in in people, for example, in in our education systems has evolved over many hundreds of years and is arguably imperfect, but I think the fact that we target specific capabilities almost in this, you know, hierarchical taxonomy where you've got sets of skills and you've got some concept of how those skills combine together to like, you know, generate more complex skills, right? I give a very simple example. If you you train to be a surgeon, you kind of go to a medical school and you start off at some level with, you know, the basic set of skills that all humans require.

• 51:53Speaker 00

  And then you kind of need to specialize and you go into probably chemistry and biology. And you need to understand the core concepts there, understand, you know, how to apply certain

• 52:05Speaker 00

  functions or reasoning patterns. And and and the further you progress, the more complex and specialized your your exams become. And potentially then the way to evaluate models designed to do everything, assuming we want to build general models designed to do everything, is to come up with a set of examinations which are intentionally designed to test models. And they might be inspired by how we test humans and they might not, because models and humans have different characteristics. And I think we're starting to see some some very early work in this space. And and one thing I think we should also be thinking about in in that kind of

• 52:47Speaker 00

  framework is is just ensuring again that we have a way to guarantee that this process is dynamic, because the technology will keep improving and the moment that the benchmarks don't keep up is

• 53:05Speaker 00

  is really the point at which we kind of need to either rethink everything from scratch or or maybe, you know, we we've just reached the point that the models are better than an expert humans across all domains.

• 53:19Tim Scarfe?

  Can you tell us about data perf?

• 53:20Speaker 00

  Data perf is a set of challenges, also run through the Dyna Bench platform, targeted at being very data centric. So this idea of data centricity focuses on the fact that a lot of what we do depends on the data that we're training these models on. And there's definitely kind of been waves of, I guess,

• 53:50Speaker 00

  community perception around what contributes most. Is it the data? Is it the models? Is it the architecture? Is it the resources? Is it the hardware?

• 54:02Speaker 00

  And I think data perf came around at a time where it definitely felt like there was a lot of focus on the algorithms and less focus on the data. And yeah, that's the main motivation is ensuring that the data is still central to everything we do and that the improvements to the data remain a core part of this journey we're all on of building better, more capable general models.

• 54:35Tim Scarfe?

  So you're working at Cohere. Tell us about what you do there.

• 54:38Speaker 00

  I work on post-training.

• 54:41Speaker 00

  So I joined Cohere roughly two years ago at a point where the kind of language modeling landscape was a lot more raw. I'd say a lot less established. So this was kind of a few weeks before ChatGPT launched. And obviously, since then, there's been a massive influx of interest and excitement and investment as well. So at Cohere, I built out the post-training team. So when I joined, obviously, we had already very, very solid base models. These are kind of pre-trained models. They're trained on lots and lots of text, many trillions of tokens,

• 55:23Speaker 00

  and trained to generate what the next word is. And back then, it was very clear that these models were very capable, but didn't really follow instructions very well.

• 55:41Speaker 00

  One, one kind of step towards making these models a lot more useful is getting them to follow instructions from a human and getting them to do what the human wants them to do. And so like one, one example of this is if you asked a, a base model a question, something like, what color is the sky? Um, a, a base model would reply with something like, what color is the grass? Yeah. What color is the sun? And mostly because, uh, you know, the, the context in which it's seen questions, um, uh, in many cases was, was just lists of, lists of questions, like, uh, even if it's pages of FAQs, uh, kind of with the, the collapsible sections where those might not have been parsed in the right way.

• 56:23Speaker 00

  Um, and, and that's interesting behavior, sure, but it's not what you expect, or definitely not what we've come to expect from these type of models. And, uh, so, so one, one thing I did very early, um, along with, uh, Aser, who kind of leads, uh, pre-training, is, um, we, we built a very, uh, kind of rudimentary interface for collecting some instruction following data. And we asked people internally, listen, can you please, uh, provide some questions? And type out what the response to those, um, those questions should be. Um, so you, this idea of having a prompt and, and the completion.

• 57:04Speaker 00

  You know, that, that's still today, a big part of how data for these models in the, in the supervised fine-tuning stage is created. You have, um, these, uh, these groups of annotators who are either writing their own prompts or having those prompts kind of seeded or synthetically generated or, um, you know, prompt sampled in different ways. And then, uh, they have to write out a completion, which has to be extremely high quality and kind of to the, the standards of your style guide and to the, uh, standard of how you want, um, your model to, to behave. And so we, we, we did that. We ran kind of a, an internal competition. Um, the, the kind of, uh, interest and, and like participation within the company was, was insane.

• 57:45Speaker 00

  We, we, we thought we were being extremely ambitious, targeting, uh, 10,000 examples in, um, I think under two weeks, um, something like that. And, uh, and yeah, we, we obviously exceeded those expectations and we trained kind of the first, uh, you know, first generation of, um,

• 58:06Speaker 00

  Command models that that you know really followed instructions well and

• 58:13Speaker 00

  and then kind of things grew from there. So we then moved on to training a new model every week.

• 58:21Speaker 00

  So that initially was kind of the instruction following supervised fine tuning stage, where we then started working with our own internal annotators, with external data vendors, getting all sources of interesting.

• 58:38Speaker 00

  Yeah, interesting instruction following data.

• 58:42Speaker 00

  We started synthetically generating data and we did that for pretty much a year. So every week for something like fifty two weeks, we delivered a new model back then. They were the command nightlies. I don't know if you're Yeah, I played with them. Yeah. They were they received a lot of interest and excitement in the developer community in particular, because people were always, you know, waiting to see what's what's the new model. And, you know, there were a few weeks, like most of the weeks, we just, you know, had a model which was superior to the previous model, as is kind of natural with these processes. There were, I think, maybe four or five weeks where, you know, the model was the same or slightly worse than previous

• 59:25Speaker 00

  models based on our internal metrics. And so we wouldn't release that model. And then you get people asking, oh, I didn't know, based on what my interactions with the model, it doesn't seem like the model has changed. Are you still planning to release the model? And so it was just a very inspiring time for us. And then around, you know, towards the end of last year, we kind of

• 59:50Speaker 00

  organized a bigger kind of structured push towards a new generation of models, which became Command R and R plus, which at the time of release, so in April of this year, were kind of, you know, ranked extremely highly on. Pretty much all the metrics, I think we were the fourth biggest, like the fourth ranked provider on Chatbot Arena. So I think there was Anthropic, OpenAI and Google and Enco here. And now, of course, we're working on the next generational models.

• 1:00:26Tim Scarfe?

  What do you think about quantization? You know, people kind of like ripping your model to shreds and reducing the precision and hacking and stuff like that.

• 1:00:34Speaker 00

  It's very useful as a short term solution from the perspective of efficiency gains. I think it's not always clear with quantization. The problem always comes back to evals. So the objective for most quantization efforts is make the model a lot more efficient and maintain performance on evals. From the perspective of, you know, maybe you accept some, you know, some small drop in performance for massive efficiency gains. And it's, it's very effective from the perspective of, you know, if I have

• 1:01:16Speaker 00

  a hundred B model or a thirty five B model, and I want to quantify the hundred model, the hundred B model to kind of the same level of just like per token cost as the thirty five B model. I'm probably going to get a much better performing model at the same, the same price range. But then I can also quantify the thirty five B model and reduce the cost there. I think, I think one of the, the trade offs is, is, is these blind spots where you might not be measuring everything, like as we've kind of discussed, our evals are incomplete. They don't measure things like, in many cases, advanced complex reasoning.

• 1:01:56Speaker 00

  In many cases, they're quite limited in terms of their ability to test, say, you know, very long range interdependencies and long context performance and these kind of things. And, and so there's always a risk that you're potentially.

• 1:02:15Speaker 00

  The grading performance in aspects that people care about that you might not have been able to measure for whatever reason.

• 1:02:22Tim Scarfe?

  Is reasoning the synthesis or the execution? And what I mean by that is, I can teach you how to do multiplication or long division or something like that, and you're just going through the mechanics of executing the rules. We make lots of mistakes, as you say, but we have the ability to synthesize the rules and kind of describe them algebraically.

• 1:02:45Speaker 00

  I would expect that the synthesis is a requirement, but that you can really test the execution. And, you know, there's various mechanistic interpretability efforts trying to look into, you know, how are models actually thinking, what's going on under the hood? The recent kind of work in terms of scaling test and compute where models are generating these reasoning chains and then providing a final answer, and in many cases, giving better performance. I think that's still heavily underexplored. We don't necessarily know exactly what the effects are that are resulting in the better performance.

• 1:03:27Speaker 00

  And I think there's various early indication, we have some work going on as well, that there's no strict requirement for models to reason in natural language or to have these kind of internal thinking chains which are explicitly generated. And I think if you want to test whether a system reasons, I think the adversarial setting is one which allows you to really probe for, you know, specific capability, a specific type of reasoning. We saw this in a lot of the early adversarial QA, BTAI-style work, where the limitations of a particular benchmark are the benchmark itself.

• 1:04:08Speaker 00

  And so you've curated a benchmark, you've got in crowd workers to ask lots of interesting complex questions, and then you kind of check how well the model performs, and typically performance saturates very quickly and, you know, models do very well. Drop, for example. Is a pretty classic question answering benchmark was very popular, still very popular for evaluating language models. Draw by, if I recall correctly, stands for discrete reasoning over paragraphs. And it's it's really focused on numerical reasoning. The the limitation there is that if the model, you know, achieves human level performance on the benchmark, can you claim that the model can reason?

• 1:04:49Speaker 00

  And I think it requires one step further, which is if you then get that model, put it in front of humans, ask humans to then probe for this capability, probe for this type of reasoning, see if you can get the model to fail. If if humans are, you know, consistently successful at getting the model to fail on examples that, you know, are slightly different from what has been trained on, what has been tested on. Then I think it's fair to assume that the model isn't really reasoning, it's just, you know, figured out how to do well on a specific benchmark.

• 1:05:18Tim Scarfe?

  How puritanical should we be? You know, because you you could say, as you just did, that reasoning is a binary, like you you either got it or you didn't. Or you could take the stance that if it performs robustly a lot of the time, even with a few edge cases, then we could still say it's it's reasoning.

• 1:05:35Speaker 00

  I think we have to hold to a very high standard. And the reason is that I don't expect the interaction between humans and machines to be the same as the interaction between humans and humans. You know, we we build up trust with each other, we engage in conversation, many cases involving complex reasoning in specific domains and and we really understand what our own limitations are. And I think the the expectation of machines is consistently, they're right all the time. It's it's a function of traditional software engineering and how how that's approached. If you had a calculator and you put in one plus two and the calculator said three,

• 1:06:19Speaker 00

  and then you tried two plus seven and the calculator said seventeen, you you throw away your calculator, you wouldn't, you wouldn't try any other questions. And, you know, L L M's in the early days, even these very early, you know, very simple mathematical questions, they they would they would get wrong. Yeah, so I think there there is a sense of the standard is actually higher. We need that consistency and we need that robustness. I I can definitely imagine that as the complexity of the task increases, there might be a bit more room for error. But definitely on more simple tasks, the tasks we're using today to evaluate these models are relatively simple, right?

• 1:06:58Speaker 00

  One of the very common benchmarks is GSM eight K, which is grade school math, which is extremely easy for for humans, or at least humans with a certain level of proficiency. I mean, current models do very well, and I'd probably argue, you typically looking at like ninety seven, ninety eight percent level of performance, you're probably there looking at noise in the test set rather than anything about the model's capability specifically. But yeah, going back to the earlier point, the next step is really, you know, put that model in front of a human, get them to probe grade school math levels and types of reasoning,

• 1:07:38Speaker 00

  and convince a human that the model is actually consistently doing what it's being asked to do.

• 1:07:46Tim Scarfe?

  So you know, like with these deep learning models, they have characteristics of reasoning and they have characteristics of kind of statistical matching, and they seem to do both in one, which is brilliant, because we can align them with reasoning algorithms, but we can also get them to make good guesses in other situations. But you sketched out this example that sometimes it would just be like a hash table, when it would just go and retrieve the fact from the document, and sometimes it would do reasoning. But the boundary is kind of clear, isn't it? So sometimes it's directed, and sometimes it's, you know, it's quite spread out and it's quite diffused. How does that process work and how epistemically aligned is it?

• 1:08:22Speaker 00

  I suspect analogous to how humans learn, where we kind of learn what I guess I'd call relatively simple tasks first, or learn, you know, to understand and apply relatively simple functions first, and then over time assemble those into my kind of more complex composite functions, where you might apply a specific type of reasoning or a specific function for a particular problem, or might apply a different approach for a different problem. I suspect that's similar to what we are starting to see and what we probably see more of in these deep learning models. It's inefficient to reason if you don't need to, right?

• 1:09:06Speaker 00

  If you, if a model is asked, what is one plus one, and it has seen that, you know, hundreds or thousands of times in pre-training. It doesn't need to reason from first principles and figure out, you know, the complexities of math and how to do basic addition. But that doesn't take anything away from solving that problem as long as the solution is correct.

• 1:09:27Tim Scarfe?

  What do you think about the ARC challenge?

• 1:09:29Speaker 00

  I think it's another step in the right direction. Right, we need to keep coming up with challenges that really push the limits of current models. We'll solve that and then we'll move on to the next one. And hopefully at some point, I think it becomes a lot more relevant to ground, you know, these kind of challenges in in real world application and in the in terms of what's useful for us, what by design do we want AI systems to do? How do we want them to operate in society? How do we want them to interact with humans? And I think those questions are becoming more important and and hopefully at a point

• 1:10:11Speaker 00

  in the not too distant future, you know, these will be the questions we care about, which is how can I get the existing technology and apply it

• 1:10:22Speaker 00

  to, you know, to the benefit of humanity?

• 1:10:25Tim Scarfe?

  Will connectionism get us all the way? You know, do you think we can come up with the right types of neural networks that will do, you know, reliable, robust reasoning? Or do you think we need to have some kind of hybrid architecture?

• 1:10:38Speaker 00

  I don't really care in the sense that I I see no immediate reason

• 1:10:48Speaker 00

  to expect that there are, you know, severe architectural bottlenecks to what we're doing. I mean, there are some, but they're easily overcome, right? So one example is is tokenization and

• 1:11:05Speaker 00

  We had this paper recently at the M N L P with Sander, it won an outstanding paper award. It got retweeted by Andre Karpathy, which made my socials explode. And I've never seen a spike like that in terms of kind of the usage stats. But this this work focuses on this concept of glitch tokens, which is this idea that certain tokens cause weird behaviors in deep learning models. You know, there's various reasons for this. The main one is you have this mismatch in terms of the data that a tokenizer is fit to and the data that a model is then subsequently trained on.

• 1:11:45Speaker 00

  What ends up happening is you have tokens which are just very rarely seen or very pretty much like not seen at all during training. In many cases, you you have weight decay, which means that the the embeddings of these tokens kind of tends to zero as model training goes on. And even if it doesn't tend to zero, it just tends to a very kind of small magnitude where it becomes very hard to kind of separate like token specific information. And it becomes very easy for a model to get confused about, you know, tokens' identity. You know, this kind of effect highlights an immediate limitation in terms of not necessarily the the architecture, but the the current way of doing things.

• 1:12:25Speaker 00

  But I don't think that's that's a question of, you know, connectionism versus the more, I guess, traditional symbolic structured way of way of approaching things. More more just pointing out a a limitation of the way we currently do things. I don't see indication today that, you know, deeper, larger, bigger models won't be able to do more impressive things. And once we reach the point that they can't, I think then we'll then we'll know the answer. And then we can kind of take it from there. But in the meantime, you know, I think I think we've made incredible progress over

• 1:13:07Speaker 00

  the past few years, both from a data perspective, but also, you know, massively driven by just the efficiency and the the parallelism that. Transformer provides and I'm I'm sure we'll see more innovations in the future. I'm sure that, you know, further efficiency gains will will accelerate things further. It it it remains pretty pretty clear that an L L M's data efficiency is not quite the same as a human's. L L M see many lifetimes of human data during training that that we do, but they are fortunate, right? If there's the compute and there's there's the resources, we are time constrained and A I systems aren't.

• 1:13:48Tim Scarfe?

  Yeah, I suppose there is a similarity there in the sense that the kind of information we process and build on is also multi generational. Like our our language and our knowledge just seems to, you know, continuously grow, even though we die every seventy years or so.

• 1:14:02Speaker 00

  I think that's that's very true, and I think there's there's definitely some compression of knowledge that happens, which makes passing information on to the next generation more efficient. Software has the advantage. The the the kind of space of continual pre-training is is just starting to become relatively explored. But currently, in most cases, and mostly due to architectural constraints, right? If you want to increase the size of your model, it becomes non trivial to figure out what's the right initialization based on the knowledge I currently have. But if you view what L L M's are doing as an efficient compression of the data, then starting from an existing efficient compression seems a lot more efficient than starting from scratch every time.

• 1:14:46Speaker 00

  And that's a lot more convenient to do with L L M's than it is with humans. You can, you know, copy the weights of one model, pass them on to another model. You have an instantiation of exactly the same kind of knowledge and information and capability in a way that you can't you can't do with humans.

• 1:15:04Tim Scarfe?

  How important is having a big context window for you folks? We're starting to see some models with really big context now. Is that something that's quite easy to add on or is is that fraught with problems?

• 1:15:16Speaker 00

  It's generally quite challenging to.

• 1:15:22Speaker 00

  To maintain high performance across the kind of span of the context window. And I think it's, you know, going back around 12 months. There was definitely what felt like this this race, right, to go larger and larger in terms of context window. Our our kind of first generation of models had had 4K context windows. Our current models have 128K context windows. And, you know, you saw pretty much all of the L N M providers really start to to push this idea of longer context and market that. Google in particular, I think, made some very interesting innovations in that space, you know, going towards context windows that were

• 1:16:06Speaker 00

  previously considered quite quite challenging to achieve and still maintaining good performance. And I think it's it's a bit of a trade off in that the vast majority of user queries are not long context, but long context is useful for many kind of things you want to build on top of language models. So things like retrieval augmented generation. The you know, that's sensitive to how good your your retrieval is. So Go here has extremely competitive kind of embedding models, which are very helpful there and and re-rank models as well. And then there's a question of, well, how much of what I've retrieved and dragged can I put into the model as context

• 1:16:51Speaker 00

  in a way that the model can use that information and, you know, aggregate information across documents in in ways that are beneficial. Then, once you get to things like tool use, once you get to things like, you know, incredibly long conversations, right? There's these ideas of

• 1:17:09Speaker 00

  effectively infinite conversations with the model where the the conversation to some extent is the state, right? We were talking about personalization earlier. If your entire conversation with the model is representative of, you know, the way you want to interact with the model, including feedback, then potentially that's that's kind of an area which is is right for exploration.

• 1:17:36Speaker 00

  When it comes to working with code, there's massive advantages to having long context capabilities where you want to do something like give a model your entire code base and have it process information, extract information, potentially rewrite things. There's a lot of value you can unlock there.

• 1:18:05Speaker 00

  And I think it's a bit of a chicken and egg problem. Models traditionally weren't, had relatively short contexts. I mean, going back, I think BERT had five, 12 token contexts. And most people thought that was plenty. And now we're in the hundreds of thousands, if not millions. But I guess there's a question of at what point does it make sense to continue to feed things into the context? It becomes more inefficient.

• 1:18:36Speaker 00

  You can do some clever things with caching and you can speed things up. But there's just generally more processing that needs to happen for generating the same answer. And if that answer is better than what it otherwise would have been, then I think that's fair game. If you get to the point where you're kind of adding information to the context for no good reason, then I'm not, I don't really hold any judgments. I'm just not sure there's much sense in doing that. So I remember a few months back, there was this idea that you don't need to do retrieval augmented generation with infinite context models.

• 1:19:19Speaker 00

  Because you can just put in as input all of the data that exists, which is very true, you could, but do you want to? What would you gain from that? And it almost feels intuitively a bit wasteful. It's like. Again, if I asked you something like, what is the color of the sky? But I told you also, you need to read the entire internet to be able to answer that question. In many cases, you don't.

• 1:19:54Tim Scarfe?

  What do you think about reasoning models?

• 1:19:57Speaker 00

  I mean, of course, it's definitely something of of interest. Right? We we generally

• 1:20:06Speaker 00

  want models or or work towards developing models that are

• 1:20:11Speaker 00

  kind of best in class across all dimensions. And and we know that general reasoning capabilities make make models a lot more both, you know,

• 1:20:26Speaker 00

  they perform a lot better on reasoning style benchmarks, which in itself has value. But they also are more useful and more valuable for people because you can do more complex things with them. I quite like the idea of being able to trade off test time compute with performance. So that's that's one thing that my team is currently working on. We're we're but we're looking at a bit a bit more broadly. It's not just

• 1:20:55Speaker 00

  it's not just allowing the model to generate more tokens, you know, to to optimize some some performance number. It's more around thinking about how are these models interacting with users? What is the signal and information that you can share with users? Maybe even as you're going through this reasoning process? And how, again, do we make that controllable and customizable? You might have a

• 1:21:23Speaker 00

  you know, a relatively simple, non-critical task where, say, if models were, you know, had had had well calibrated confidence scores, you could tell the model something like, if you're at least sixty percent confident, give me the answer. And as long as that, again, you had that kind of core requirement of of reasonable calibration, then, which basically means if you ran that thing a hundred times and the model should get the answer right sixty percent of the time if it's sixty percent confident.

• 1:21:54Speaker 00

  And that, you know, might be fine in say, um, creative writing settings, or it might be maybe what you want uh, in particular settings. Um, whereas if you had really, uh, you know, really complex, really critical task, you might be able to say something like, I don't afford for you to get this wrong. Just like think as much as you absolutely need to. If you're in any doubt that the answer is wrong, tell me. Um, but if you're going to output an answer, make sure it's right. Um, obviously we're far from any model that can do that. Um, but that, I think that kind of uh

• 1:22:34Speaker 00

  behavior, um, kind of thinking about the problem from almost a user interaction perspective. Um, I think, um, yeah, it's going to be extremely interesting over the uh, the coming years.

• 1:22:47Tim Scarfe?

  I agree. And, um, you know, the, the big challenge is just building the apps, imagining what can be done with this technology. Just the basic stuff like doing enterprise search really well. Um, we've got a long way to go, but it's going to be an exciting journey.

• 1:23:01Speaker 00

  For sure.

• 1:23:01Tim Scarfe?

  Max, thank you so much for joining us today. It's been amazing.

• 1:23:04Speaker 00

  Thank you. Likewise. Thanks for having me.