Subbarao Kambhampati - Do o1 models search?

Join Prof. Subbarao Kambhampati and host Tim Scarfe for a deep dive into OpenAI's O1 model and the future of AI reasoning systems. * How O1 likely uses reinforcement learning similar to AlphaGo, with hidden reasoning tokens that users pay for but never see * The evolution from traditional Large Language Models to more sophisticated reasoning systems * The concept of "fractal intelligence" in AI - where models work brilliantly sometimes but fail unpredictably * Why O1's improved performance comes with substantial computational costs * The ongoing debate between single-model approaches (OpenAI) vs hybrid systems (Google) * The critical distinction between AI as an intelligence amplifier vs autonomous decision-maker SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. https://centml.ai/pricing/ Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events? Goto https://tufalabs.ai/ *** TOC: 1. **O1 Architecture and Reasoning Foundations** [00:00:00] 1.1 Fractal Intelligence and Reasoning Model Limitations [00:04:28] 1.2 LLM Evolution: From Simple Prompting to Advanced Reasoning [00:14:28] 1.3 O1's Architecture and AlphaGo-like Reasoning Approach [00:23:18] 1.4 Empirical Evaluation of O1's Planning Capabilities 2. **Monte Carlo Methods and Model Deep-Dive** [00:29:30] 2.1 Monte Carlo Methods and MARCO-O1 Implementation [00:31:30] 2.2 Reasoning vs. Retrieval in LLM Systems [00:40:40] 2.3 Fractal Intelligence Capabilities and Limitations [00:45:59] 2.4 Mechanistic Interpretability of Model Behavior [00:51:41] 2.5 O1 Response Patterns and Performance Analysis 3. **System Design and Real-World Applications** [00:59:30] 3.1 Evolution from LLMs to Language Reasoning Models [01:06:48] 3.2 Cost-Efficiency Analysis: LLMs vs O1 [01:11:28] 3.3 Autonomous vs Human-in-the-Loop Systems [01:16:01] 3.4 Program Generation and Fine-Tuning Approaches [01:26:08] 3.5 Hybrid Architecture Implementation Strategies Transcript: https://www.dropbox.com/scl/fi/d0ef4ovnfxi0lknirkvft/Subbarao.pdf?rlkey=l3rp29gs4hkut7he8u04mm1df&dl=0 REFS: [00:02:00] Monty Python (1975) Witch trial scene: flawed logical reasoning. https://www.youtube.com/watch?v=zrzMhU_4m-g [00:04:00] Cade Metz (2024) Microsoft–OpenAI partnership evolution and control dynamics. https://www.nytimes.com/2024/10/17/technology/microsoft-openai-partnership-deal.html [00:07:25] Kojima et al. (2022) Zero-shot chain-of-thought prompting ('Let's think step by step'). https://arxiv.org/pdf/2205.11916 [00:12:50] DeepMind Research Team (2023) Multi-bot game solving with external and internal planning. https://deepmind.google/research/publications/139455/ [00:15:10] Silver et al. (2016) AlphaGo's Monte Carlo Tree Search and Q-learning. https://www.nature.com/articles/nature16961 [00:16:30] Kambhampati, S. et al. (2023) Evaluates O1's planning in "Strawberry Fields" benchmarks. https://arxiv.org/pdf/2410.02162 [00:29:30] Alibaba AIDC-AI Team (2023) MARCO-O1: Chain-of-Thought + MCTS for improved reasoning. https://arxiv.org/html/2411.14405 [00:31:30] Kambhampati, S. (2024) Explores LLM "reasoning vs retrieval" debate. https://arxiv.org/html/2403.04121v2 [00:37:35] Wei, J. et al. (2022) Chain-of-thought prompting (introduces last-letter concatenation). https://arxiv.org/pdf/2201.11903 [00:42:35] Barbero, F. et al. (2024) Transformer attention and "information over-squashing." https://arxiv.org/html/2406.04267v2 [00:46:05] Ruis, L. et al. (2023) Influence functions to understand procedural knowledge in LLMs. https://arxiv.org/html/2411.12580v1 (truncated - continued in shownotes/transcript doc)

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  So much so that we now have phrases like fractal intelligence. You know, in fact, I think Andri Karpathy basically was saying L L M's have fractal intelligence. What the fractal intelligence is basically, we don't know when they work, they work, when they don't, they don't. That's fractal intelligence and that sort of shows, which is good, still we had nothing like this before. But part of the science of L L M's has to be to say something more than fractal intelligence. Saying here is the level to which you can depend on their results. So I'm not, so there are in reasoning, in logic, there are ways where limit, you know, of basically formally characterizing the limits of reasoning, like limited depth, limited look ahead reasoning and so on. None of them seem to work for L L M's.

• 0:45Speaker 00

  The question then is what would work? We have to figure that out. The bitter lesson is over and efficiency is going to matter. And I completely agree with that. I've been arguing this for a long time too, that think about the following thing. The first time when we sent a man and humans to the moon, cost was not a consideration. We want to show that we can do it. NASA was the one which is doing it. The second time and the third, and for the space as well as the moon. The second and third time, et cetera, maybe okay. But by now, it's Elon Musk sending people to space and supposedly possibly to the Mars too. Because the cost matters, right?

• 1:30Speaker 00

  Essentially, it's, once it's been done, then you start caring about, you know, the cost that you're paying. And computer science is actually quite a bit about the unsexy parts of cost, just as it is about doing things that haven't been done before.

• 1:47Tim Scarfe?

  There are people who say, well, if it's not retrievable, then it is reason. So what say you?

• 1:53Speaker 00

  Reminds me of this old Monty Python thing where this guy has, I think like a Brian. He does something that looks like, you know, if this is to prove that some, some woman is a witch, right? You know, if she is made of wood and she floats on water, then.

• 2:13Speaker 02

  How do you know she is a witch? She looks like one. Bring her forward.

• 2:20Speaker 03

  I'm not a witch, I'm not a witch.

• 2:23Speaker 02

  But you are dressed as one.

• 2:25Speaker 03

  They dressed me up like this.

• 2:28Speaker 03

  And this isn't my nose, it's a false one.

• 2:34Speaker 02

  Will?

• 2:35Speaker 04

  Well, we did do the nose.

• 2:36Speaker 00

  You know, random connections and then saying that she's a witch and you say Q A D. That looks like reasoning because it's not just retrieving, you know, something she's a witch one. But we know that it's not sound reasoning.

• 2:53Speaker 05

  So Tufa Lab is a new AI research lab I'm starting in Zurich. In a way, it is a Swiss version of DeepSeek. And first we want to investigate. So LLM systems and search methods applied to them similar to O1. And so we want to investigate, reverse engineer and explore the techniques ourselves.

• 3:19Tim Scarfe?

  MLST is sponsored by SentML, which is the compute platform specifically optimized for AI workloads. They support all of the latest open source language models out of the box like Llama, for example. You can just choose the pricing point, choose the model that you want. It spins up its elastic auto scale. You can pay on consumption essentially, or you can have a model which is always working, or it can be freeze drive when you're not using it. So what are you waiting for? Go to sentml.ai and sign up now.

• 3:51Speaker 00

  That, you know, Microsoft essentially cannot any longer control OpenAI if in fact AGI has been achieved. That is one way they could avoid being beholden to Microsoft. But now they're trying to say we'll remove it so that we'll get more money from Microsoft. That's kind of, I don't know what that says. Are they looking for money or they have realized AGI is not actually going to come anyway? So why bother with that clause, you know, other than surprise?

• 4:21Tim Scarfe?

  So much has changed since our last conversation at ICML. Can you give us a bit of a rundown of what's happened?

• 4:28Speaker 00

  When we were talking in Vienna, I think we were talking about reasoning abilities of large language models, especially I think of large language models as the autoregressive token by token prediction models, which are pre-trained for that and they also do that in inference time.

• 4:48Speaker 00

  And it was clear, I think, as we were talking about at that time, that those, as from my perspective, did not have the reasoning abilities that are amazing in supporting creative work, where they can give you ideas and you can run with them, but, and they will give you answer as soon as you hit return, but they are not guaranteed to be correct. Okay. One of the interesting questions, of course, is reasoning tends to have a higher complexity in terms of the time needed and other ways of actually changing LLM sort of substrates to do that. And a couple of things happened and I would think, I would basically, I mean, obviously we'll get to O1 in a second because that's the thing, that the bigger thing that happened.

• 5:33Speaker 00

  But an interesting way of looking at what that whole direction is what's been called, there are like two parts, inference time scaling and post-training.

• 5:46Speaker 00

  First ideas that were tried, and in fact, we talked about this when I talked about LLM modulo, is if to the extent LLMs are essentially quickly generating candidates but with no guarantees, maybe you can make them generate tons and tons of candidates and then either do majority voting or self-consistency or something like that to see if you have the better answer. And how do you check the better answer? There are like a whole series of them. There might be external verifiers. There might be like LLMs themselves trying to partially verify there are problems with that we talked about, but you know, they have tried that too.

• 6:26Speaker 00

  So that has, that's one type of inference time scaling. A related idea there, an interesting, very interesting idea there is, it's been known from the day one that if you are trying to give a reasoning task to LLM as a prompt...

• 6:44Speaker 00

  And then it gives a completion and you check its completion, whether it contains the solution. The probability of that happening in general can be made higher if you can find the right kind of prompt augmentation. So in addition to your reasoning thing, some magical tokens that you add and that seems to increase the probability. And this has been seen in like multiple scenarios. Originally, this idea has been bandied about as chain of thought. And the first, very first version of that essentially is the zeroth order chain of thought, where the magical token will always be the same one irrespective of the task and the L L M. Like let's think step by step.

• 7:29Speaker 00

  And that sort of work partly because you know, the human data had those specific, you know, those types of tokens. And so L L M like outputs that and then that jogs its pattern matching with things to actually pick up with other solutions on and so on, that would be the thing. And then then came the chain of thought, task specific one, the one J S N V and Co did. Their humans give task specific advice as to how to solve the problem and then hope that you know, L L Ms will actually solve it. This can be connected with inference time scaling because you are adding chain of thought and also are essentially making it generate multiple candidates and then actually picking from them.

• 8:12Speaker 00

  Chain of thought by itself has again problems, just as L L M verification has problems, chain of thought has problems. In fact, this new rips, we had a paper called Chain of Thoughtlessness, I will talk about later. But basically by itself, it has problems. But as part of the toolbox of increasing the time spent during the inference time before you blurting out one answer.

• 8:35Speaker 00

  Chain of thought together with the sort of picking from many samples has shown some promise. One variation of that, and in fact, that is something that I've been pushing more recently is, originally chain of thought was kind of confused with it might be anthropomorphic. And that in fact, we tend to sort of tell ourselves, okay, let me do it this way, et cetera. And people are hoping that L L M's are doing the same thing. Mostly they were just kind of imitating whatever like this, you know, let's think step by step data that they have found in the training data. Uh, but somehow people thought if you kind of make them

• 9:16Speaker 00

  imitate human thinking, you know, um, then maybe they will do better. That is the first two ideas and neither of them actually went that far. Another idea essentially is you realize that it's just a magical tokens that you are trying to add and you just have to figure out what's the right magical token. And it's like it's sort of a squall and function. You're trying to figure out a task specific, L L M specific magical token that increases the probability. This is a learning problem. It's an extra learning problem. There are two general approaches that have been tried. The first approach essentially was to say, so L L M basically before giving the answer, it has to

• 9:56Speaker 00

  tell itself a few things, some, you know, like step by step is the one that makes sense to us, but it can actually give it itself a gable de gook string, another gable de gook string. And then that kind of probes its conditional probability of completion in such a way that it might actually come up with the correct solution. The question then is where are these tokens coming from? And one idea that people had, previous first ideas were humans will supply these tokens by chain of thought advice. That wasn't going anywhere. One other idea that they had essentially is that if you, for example, have a class of problems for which there is an actual systematic, no, actually before going that, OpenAI did the following thing, which is

• 10:38Speaker 00

  maybe we will ask humans to solve specific problems while thinking aloud.

• 10:44Speaker 00

  So there is actually a paper about one and a half years back saying, let's think step by step. And and then basically this went under this whole issue of process supervision and people were actually being asked to record what they're telling themselves, et cetera. This is like the worst form of psychology, unfortunately, because we don't actually know how we think. But they tried this and one of the things is it's extremely costly. They wound up, you know, my joke is they improved the G N P of Nigeria because Nigerian, you know, turkers were being asked to solve tons and tons of these problems and then think aloud. That was very costly. And then a separate similar idea was there are bunches of problems for which there are systematic solvers.

• 11:29Speaker 00

  Like for example, for arithmetic, there is arithmetic solvers. For search problems, there are A star search sorts of things and for planning, you have planners. In general, any systematic solver would be manipulating some data structures until certain termination condition is reached and then it outputs the solution.

• 11:49Speaker 00

  Imagine you make it output the trace of the data structure manipulation operations. All you needed hopefully was some extra kind of, you know, tokens that are coming out before the solution. So this stuff can be thought of as a derivation and the idea that people had was let's train the L L M with a huge number of these derivational synthetic derivational traces and the solution. And now remember, this only works for problems for which actually there are synthetic, you know, synthetic solvers and systematic solvers. And you're just trying to kind of make it be solvable in a general sense without having to call those. That was the idea.

• 12:32Speaker 00

  And there are like a couple of three or four efforts that have gone. There's a search farmer from Facebook, Meta and there's a stream of search and there's just recently last week, there is a Google Deep Mind paper which also talks about internal versus external planning to do multi board game solving. And all of these essentially use variations of this idea. So you have to realize that all they're doing is now L L M has to before outputting the solution has to output some additional tokens that will jog its memory to hopefully output a better solution. This is the whole. And so basically, people, this was their idea. They tried it and you know, sort of, sometimes it actually works. It improves performance.

• 13:17Speaker 00

  There is no good reason to systematically say this would be making sense. Because it's almost like if you kind of, you're trying to teach your kids how to reasoning, very small kids, and then you sort of do some hand movements and then think like that and then give the answer. You would see the junior also doing these hand movements and think like this. It gives the wrong answer. L L M's can do that. They're essentially imitating your, whatever the derivational pieces, which may not even actually make sense, but sometimes they have shown some promise. This has basically become in a, you know, kind of most young, most recent idea called inference time scaling, where essentially you do this and you also do this multiple

• 14:02Speaker 00

  suggestions and then pick from them, et cetera. This comes very close to what I think O one is doing, but with a big difference from here. Again, as you know, O one, nobody knows and it's become like, you know, we all sit around the ring and suppose and I like to say, Noam Brown sits in the middle and knows. But basically, they don't want to tell what they're doing. But my guess, I mean, everybody has a guess, and my best guess as to what O one might be doing is going again with this prompt augmentation idea. But the question, of course, is where are these prompt augmentations coming from? We talked about first, one prompt augmentation for everything.

• 14:45Speaker 00

  Second is human given prompt augmentation, which is chain of thought. The third is the synthetic derivational trace that sort of gives these tokens and maybe you'll try to say this back. A much, none of them really make too much sense. A much better idea is if you are saying, what should I be telling myself to improve my outcome? It's a kind of a RL problem. It's a kind of a reinforcement learning problem. Imagine like an AlphaGo agent. It's sitting there and thinking, what actions should I do one after other such that my win probability increases? So it does a whole bunch of these board actions. And then at some point of time, it gets a signal saying you won the game or you lost the game. And then you do this gazillion times.

• 15:29Speaker 00

  You can then bring this reasoning back through the sequential decisions. Computing their key values. Like is at under what board positions or what, um, you know, actions worth doing? You know, that's the key value. Now, if you take the AlphaGo analogy and put it to L L M's. The L L M board position is essentially the context window with the prompt and all the other stuff that you have put on. And the action is this token that you are generating. So to make things simple, I would like to think of it as there's a big L L M. Let's think of a G P T four. There might be a small L L M which has a reduced vocabulary. All it's trying to do is give jogging, um, give like, you know, these prompt augmentations.

• 16:14Speaker 00

  That it tries, throws it out and then this will then be given to this other L L M in terms of its context. It gives extensions and then it tries one more and then at some point of time it checks if the solution is, you know, is correct. Now, how does it get the solution? You can have, you could have actually generated huge numbers of synthetic examples beforehand. Um, again, using solvers. This is pretty much known that, you know, OpenAI did this. It's no longer human solving problems because that's too costly. This is systematic solvers, uh, solving like the planning problem, constraint satisfaction problems and various sorts of problems. And for which they have the problem and the answer.

• 16:54Speaker 00

  And then open the L L M is trying to solve, L L M plus this, you know, prompt augmentation engine is trying to solve it. And then if it happens to reach the correct solution, then you can propagate the thing back. This is RL in a pseudo moves. It's not the actions are, if the prompt is about goal. The actions are not go actions, they are just, you know, essentially these tokens that the prompt augmentation tokens. One nice thing, of course, is instead of learning the Q values, one of the things you can do is you can essentially learn the, you can change the weights of the smaller L L M in the right ways, such that it puts out the right kinds of tokens, given the context window.

• 17:40Speaker 00

  If you do this, you are guideline approximate Q values. And then this is the pre-training phase, in the pre-training, there's the L L M training, followed by this humongously costly, you know, they are not telling us how costly it is, a humongously costly post-training phase, which way they spent billions of dollars. At that point, you have O one model, which is now ready for the inference time. And at the inference time, once again, they're doing inference time scaling where, except this is now they have the Q values, you can improve the Q values by online M C T kinds of approaches, the kind of thing that AlphaGo did. That's where you actually can see they're doing it because they charge you for this reasoning tokens.

• 18:24Speaker 00

  If you run O one, it basically takes the prompt, it gives the answer. In the old G P T four, the amount of money that you have to pay them is proportional to the number of prompt, input prompt tokens, plus four times the number of output prompt tokens. In the case of O one, it actually does this whole bunch of stuff that's telling itself. Basically, this pseudo moves that it was Q values to improve and telling itself. It never shows that to you, but they are all counted, they are, they all are counted as output tokens. So, you know, you have, let's say, fifty input tokens, hundred output tokens, and maybe five thousand reasoning tokens.

• 19:08Speaker 00

  And so you suddenly start paying a lot more. So one of the funny things that happened essentially is when we started playing with O one preview when it came out, in two days, we spent like eight thousand dollars. And then, so in fact, I had to get like special. Permission from the university because they normally don't reimburse uh beyond a certain thing unless you have like a separate permission and so on. But that's basically one of the ways this works out. The interesting thing of course is now the way we describe this. It's not, it is based on an L L M. But significant amount of additional things have been done, right? Essentially, you are essentially doing something like an AlphaGo style post-training phase.

• 19:52Speaker 00

  Followed by an AlphaGo style um uh sort of a uh M C T online computation. Uh and at that point, actually, I would think it could make sense. And not surprisingly, in fact, in our results, we found that for the normal plan bench, it does much better. Uh than the state of the art Jeep, you know, uh L L Ms, including Claude and so on and so forth. But then, of course, then you can go the next level. It has its own issues. We can still talk about the fact that it doesn't scale beyond uh the larger problems. It can make mistakes. It has problems with unsolvability. There is no guarantees about the solution.

• 20:34Speaker 00

  But it's now makes more sense to me. Again, I don't know this is, I think this makes a reasonable way a woman can be working. And if it is the way it's working, the first time I can make sense of how reasoning can emerge. Because you are at least having the pseudo actions whose key values you are learning. And nobody ever said R L cannot do reasoning. R L can do reasoning. It's just that now you basically, it's like sort of an interesting thing where I keep using the stone soup analogy. Stones, you can make soup with stones if you start adding carrots and, you know, tomatoes and and all that stuff. At that point of time, it will still taste like soup. The question, of course, is who gets the credit? And you know, that's kind of an interesting question that we would think about.

• 21:18Speaker 00

  But that is like the long arc of what happened in my view in the last um only four months or something since we discussed. And it also, one of the other interesting things is part of the mystique of L L Ms was you'd you'd write the prompt. You hit return, you get the answer, and it doesn't cost you too much. That was where everybody was using it. O one. Basically with its, you know, of course, the post-training itself is extremely costly, but they are not charging us for that. But they're charging us for the reasoning tokens, which you never see, but you pay for it. You just have to take their word that huge number of reasoning tokens were generated

• 22:02Speaker 00

  and they're going to make you pay for that. And as far as I could tell, at least in academia, very few people have actually been doing experiments, evaluating, because it actually costs a lot, essentially. Because, you know, and there's basically people are still going with the autoregressive L L Ms because they're cheap, you know. So one of the interesting things is, you know, you can do reasoning, but the usual computational complexity issues that we politely forgot in the era of autoregressive L L Ms, and we were hoping that somehow, you know, complexity will disappear, will come back. Because if you want to improve accuracy, you have to actually do reasoning, and this

• 22:46Speaker 00

  is pseudo-mo reasoning, in my view, but still it costs. And that becomes an interesting question of when is it useful to use a general purpose system versus a, you know, sort of a hybrid general special purpose system versus an extremely specialized solver. Something that we haven't talked about before, but now it will become costlier, at least for the industries, you know. In fact, there's this whole movement about compound A I systems, and that's basically the kind of thing that people think about.

• 23:18Tim Scarfe?

  Very shortly after O one was released, you quickly, as you were just saying, you spent eight thousand dollars. You put a paper together called Planning in Strawberry Fields: Evaluating and Improving Planning and Shutting capabilities of L R M O one, L R M. So, yeah, you basically said they are positioned as approximate reasoners rather than mere approximate retrievers.

• 23:45Speaker 00

  We don't know the actual details of what they're doing. So there are two parts. One is what is objectively verifiable, which is we did test O one on the same plan bench problems, and they did quite well on Bloxworld. I mean, by, I think Claude, already there were sixty-six, these things were like ninety-nine or something, they basically saturated. More impressively, they did better on the mystery domain, which, and then more, and we... Given what I explained to you earlier about the possibility that they are training themselves with synthetic data. Um, you know, maybe they have unintentionally trained on the mystery domain, which we have available outside.

• 24:30Speaker 00

  So we actually generated truly new random mystery domains. It has lower performance, but it's still not like the point five percent of the old ones. It goes up to, I don't remember the exact numbers, up to twenty, twenty-three percent on, on some of these problems. Um, and which is obviously a good sign that they're actually able to solve this. Um, the other part as to why are they approximate reasoners than retrievers is it's based lot more on my reconstruction of what they could potentially be doing, which is they're sort of doing reinforcement learning based post-training as well as online um, Q value update.

• 25:15Speaker 00

  Um, and using pseudo-action moves. I call pseudo-action moves because there is, you could do RL for just normal Go or any specific board games. This one is just language games where the game basically is there's a language context window and there's a prompt augmentation and there's a, you know, new content, new completion and then one more prompt augmentation. This is what they call this string of chains of thought. But that's basically adding bunches of prompt augmentations and then see what happens at the end. And then if it winds up being correct, in the sense if it winds up, you know, containing the correct solution

• 25:56Speaker 00

  for your training data, then that's sort of like AlphaGo getting victory, win signal after a bunch of moves. And then it just needs to do credit blame assignment for the moves. And that's what RL is essentially good at doing. And if you're doing that, it's either what, it's a reasoning and it's approximate reasoning because it's not actually problem-specific actions. It is problem-independent, these language prompt actions.

• 26:25Tim Scarfe?

  Is it possible that you might be wrong about that? Is it possible that we're giving them too much credit? And what they're actually doing is just this massive generation of trajectories, all in a single forward pass. So maybe they do something like process supervision. So they do some clever RL pre-training stuff, but.

• 26:44Speaker 00

  So obviously, again, this is the sad part of the way. Oh, one thing is, in fact, by the way, I was to tell you like a funny thing that I was talking to somebody who said they were having some conversations with the OpenAI guys, trying to sound them out as to what Owen might be doing. And at some point of time, one of them said that I think you may have to wait until the Chinese replicate what we did to actually figure out what we did. That's the level at which science of OpenAI has gone to. But the point, the only reason, it is possible that I might be giving a lot more credit to the sophistication of the method they might be using. The reason I still think that is likely to be the case is,

• 27:26Speaker 00

  as I said in the earlier, you know, description of how things shifted from LLM to inference time scaling to this sort of O1 style method. The general inference time scaling methods are not comparable. Just inference time scaling hasn't been as good. And again, the question, the other very important thing that you have to keep in mind is

• 27:52Speaker 00

  while O1 takes more time, it doesn't take hours, right? Basically, online computation, a second in the online computation time is way more expensive from a business perspective than like days and months in the pre-training phase. And so some of the inference time scaling people actually spend a lot more time than O1 does. And they still are not getting, as far as I know, in general, to that level of accuracy. What, which basically makes me think that unless you do significant amount of post-training to get approximate Q values upfront.

• 28:36Speaker 00

  You can't improve just by MCT. So, think in terms of, again, AlphaGo analogy. If you only did MCT, it will take much, much longer per move before you can get any level of accuracy, you know, any level of confidence. But one of the things that AlphaGo does is does a sumungous amount of pre-training phase, where it learns an approximate policy, which it's then kind of rolling out to improve the Q-value estimates that it has. So that's possibly the reason why I think it makes sense. And of course, I also think that the normal inference time scaling methods don't seem to make too much sense to me.

• 29:16Speaker 00

  The one closest to pure MCT method that I have seen is this paper from Alibaba called Marco O1. They have this Marco Polo group or something, and they called it Marco O1. And Marco O1 actually essentially trains itself on chain of thought data, which is like basically a derivational data. And then on top of it, it does like an online MCT-style computation to improve the Q-values further. They are, they are like much smaller and they're, they're not as impressive in terms of the performance gains as O1.

• 29:57Speaker 00

  So those are the reasons I think the full picture requires post-training as well as inference time. The thing that you and I see is the inference time. But the thing that OpenAI can spend tons and tons of money is on the post-training, you know, which is before they even, you know, actually deploy the model. And that's where it is getting this approximate Q-values, is my guess. Again, as I said, it's a, it's a strange thing to be involved in, you know, I mean, we should be looking for. Looking for the secrets of the nature because nature won't tell us. But we are now looking for the secrets of OpenAI because OpenAI won't tell us. So hopefully, many of the, there are many efforts already in trying to replicate this sort of a thing.

• 30:42Speaker 00

  And so we'll know more. But as of now, that's the thing, I cannot be sure exactly what they're, what they're doing. Everything that they have said publicly is consistent with my hypothesis. About the only thing I can say, there is nothing that's inconsistent with my, my model of, I mean, my speculation of what Owen is working. In that strawberry paper, there's an appendix where I wrote down the speculation, where we wrote down that. And that is still consistent with everything they have said, you know, which is the only thing I can say.

• 31:16Tim Scarfe?

  Yeah, I mean, I like the sound of it. I mean, it, it makes me more excited about using it because it makes me feel that there's more sophistication behind the system. But a lot of this comes down to reasoning. And I'd love to hear your definition of reasoning. But there are people who say, well, if it's not retrieval, then it is reason. So what say you?

• 31:36Speaker 00

  So let's actually, let's look at it first, the first part and second part. So the definition of reasoning itself is, is a good place to start from. I think, I know that this whole AGI crowd basically sort of tries to say. It kind of AI is going to be like humans. The problem is we don't have a good definition of what human reasoning is. And but since Greeks on, our civilization went forward. By not by saying what humans, how do we define what humans do? But be defining what are sound reasoning patterns.

• 32:16Speaker 00

  Aristotel, syllogisms, logic, probabilistic logic, you know, the entire computer science, the entire civilization. Depended on having formal notions of reasoning, which there is a correctness, there is, you know, incorrect thing, et cetera. I mean, you know, reminds me of this old Monty Python thing where this guy has.

• 32:38Speaker 00

  I think Life of Brian, he does something that looks like, you know, if this is to prove that some woman is a witch, right? You know, if she is made of wood and she floats on water, then she must be like a duck. So something like, you know, random connections and then saying that she's a witch and you say Q A D, that looks like reasoning because it's not just retrieving, you know, something she's a witch one, but we know that it's not sound reasoning. And so in general, I prefer to think in terms of because ultimately these systems are going to be deployed, whether you like it or not. And civilization didn't depend on whether

• 33:20Speaker 00

  people, fallible humans can make mistakes and then we can look the other way around. And we actually have to have guarantees at some level or other about the soundness of reasoning and the completeness of reasoning. And so I go back to essentially definitions of reasoning from logic and so on. So basically the formal definitions of reasoning. I try to avoid getting into this question of what is human reasoning because that is a big mess. Cognitive scientists don't know it. We don't know it. Psychologists don't know it. So I try to just give it a wide berth. Okay. So that's the part as far as what I believe in reasoning. So that's why we looked at planning problems for which there is a correct solution, constraint satisfaction problems, the correct solution. And if you

• 34:05Speaker 00

  are able to do, if you say this system is a reasoning system that can be deployed, it should have some guarantees. Now you can say that humans can make mistakes, but one of the things I keep saying is if humans, if you are being paid to make decisions and make mistakes, there are penalties for you. You can in the end be put in jail until we figure out how to, who to put in jail and how to put in jail when A I. Systems make, you know, mistakes that they have no actual guarantees over. We are better off thinking in terms of formal definitions of reasoning and then seeing to what extent are A I. Systems coming close to it. This has basically been very connected to how A I. Has developed, you know, up until now anyway. Now this discussion also brings back to this issue

• 34:47Speaker 00

  of retrieval versus reasoning. I think you are talking about, you know, a couple of these papers that. Keep coming out basically trying to say that look L L M's are not exactly retrieving anything that they've been said they're not just memorizing and retrieving so they must be doing something else and I would say well Monty Python logic is not actually retrieving anything he puts together a whole bunch of things but that's not reasoning either so there is between retrieval and what I would consider reasoning can be a whole entire universe of things that still won't be considered reasoning as far as I'm concerned because there are no sorts of guarantees and and so and from the beginning we knew that if

• 35:31Speaker 00

  you again go back to those many of these sorts of papers these claims go back to essentially the autoregressive L L M's because by the way the researchers are still very busy I think we are one of the few papers on O one we have like this you know evaluation on O one is also being presented in this New Ips workshop but most people are still trying to make sense of autoregressive L L M's themselves because that is still as as we talked about last time too I still think it's a very impressive system one we never had system one in human civilization and trying to understand what they're doing is useful and so they go back to that and they'll say look they're not actually doing exact retrieval

• 36:13Speaker 00

  and they're doing so something else and we'll call this something else reasoning that's not if first of all everybody we always knew that L L M's are not databases right so they don't retrieve essentially they actually have hard time memorizing and retrieving when they memorize it it's not deliberate by deliberation it doesn't happen deliberately it happens fortuitously it's surprising that sometimes they wind up memorizing long passages because essentially everybody agrees that there are some kind of N gram models rather than databases in the way they are trained okay so given that it's very clear that they will never retrieve and the fact that they are not retrieving should not be seen

• 36:56Speaker 00

  As an indication that it it can be seen as indication that they are not retrieving, but we knew this already. But the part that people seem to hint at is since they are not doing retrieving, maybe they are doing reasoning. No, that's not making sense because again you have to subject it to what you would consider as the evaluations for sound reasoning procedures. And they failed just as well a- a- as as easily as before. So if you come back to this chain of thought paper that we I was mentioning that we just presented at NURIPS, right? In the case of chain of thought in the JSONV style chain of thought papers, what you the chain of thought idea, what you do is, let's say you take something last letter concatenation, which is this really small toy problem.

• 37:40Speaker 00

  You give like three like K words, you know, N words and you the system is supposed to take the last letter of each of these words, concatenate them into a string, right? So um so large big uh rows, so E uh G E is the one that you're supposed to out output. That's basically the last string, right? And what they were saying, what they said was if you just told LM, you know, the the name, the the prompt saying, you know, you're supposed to take the last letters and concatenate them and give the answer. And then they tested, its performance is not as good. And if they didn't tell it, here is how, here are some examples of three letter uh

• 38:22Speaker 00

  last letter concatenation problems uh uh and then four letter last letter concatenation problems and couple of these examples and then ask their questions, it improves performance. That looks like reasoning, somehow it is able to follow the procedure. The problem in, I think we talked about the last time too, is the problem with Ersaj empirical sciences, you shouldn't stop when you get the answers that you're hoping for. You should see how to break your own um, you know, hypothesis, right? So what they didn't ask is they gave examples of three, four uh word examples and then they tested on three, four words. But if you expect a system to be doing any kind of a reasoning, any kind of a procedure

• 39:04Speaker 00

  following, once I tell you what last letter concatenation is and give you an example, you will do it for twenty, about thirty, et cetera. It's just mechanically taking the last letter and concatenating. What we show is if you increase the number of words, the performance just plummets close to zero. And this also happens in planning problems, not surprisingly. It happens in last letter concatenation, it happens in planning problems. Which shows that, yes, it's doing something which seems to have improved its performance in the size of the problems for which you gave the examples for. And its pattern matching of some kind is helping in there. But it's not in any way generalized reasoning that would just generalize with respect to length, for example.

• 39:49Speaker 00

  And so, one interesting way I've been thinking about this is it's sort of, sort of glasses, you know, nowhere near full versus glass already is wet. You know, that's sort of a optimism versus pessimism. So, people tend to think that seems, it's basically, at least it's solving the three, four blocks, three, four word problems with higher accuracy, because I gave this chain of thought. That's sort of showing reasoning abilities. But the question is, we don't have a good understanding of what the boundary is, where it will actually go correctly.

• 40:33Speaker 00

  So much so that we now have phrases like fractal intelligence. You know, in fact, I think Andri Karpati basically was saying L L M's have fractal intelligence. What the fractal intelligence is, basically, we don't know when they work, they work, when they don't, they don't. That's fractal intelligence. And that sort of shows, which is good, still, we had nothing like this before. But part of the science of L L M's has to be to say something more than fractal intelligence, saying, here is the level to which you can depend on their results. So, I'm not, so there are, in reasoning, in logic, there are ways where limit, you know, of basically formally characterizing the limits of reasoning, like limited depth, limited look ahead reasoning and so on. None of them seem to work for L L M's.

• 41:18Speaker 00

  The question then is, what would work? We have to figure that out. But instead of that, we basically, once in a while, there are these papers saying, look, we actually probed the, like, using mechanistic interpretability techniques, we probed and we found that L L M's basically are Not acting like they're doing retriever, but that's kind of understood already. You know, and and I think it's still the mechanistic interpretability stuff is very interesting. I think it may actually be part of the solution to figuring out what L L M's are doing. But the argument that since it's not retrievable, it must be something like reasoning. It's still quite unsatisfactory to me because that reasoning is what I'm saying is not reasoning.

• 42:02Speaker 00

  Because all my papers are saying, whatever it is that they were doing before you did your mechanistic interpretability study. They're still doing before that, even before you did that. And they are still have these limitations before as well as after your study. And and we don't actually know how to characterize what it is that they are doing. And that's the part where we are stuck right now.

• 42:25Tim Scarfe?

  Is it possible that everyone is is right? And what I mean by that is, I spoke with some DeepMind guys earlier in the week. There's there's a great paper about, you know, softmax needs glasses. Talking about, you know, how how it kind of, sometimes we need directed attention for doing reasoning. Sometimes we don't. There was another great paper, I spoke to the guys, and talking about just utter limitations of transformers doing counting and and copying. And Laura Ruiz, I'm speaking with her on on Sunday. So she's got this paper out where she's looked at reasoning traces. And sometimes they are just doing, you know, they're retrieving facts from documents. Sometimes they're doing kind of procedural, you know, information generation, which you might liken to a reasoning process.

• 43:10Tim Scarfe?

  And I guess it's a little bit like this fractal intelligence thing, that it might be the case that possibly in certain circumstances, these models are doing something which we would think is reasoning. And sometimes they're doing retrieval, and sometimes they're doing something else.

• 43:25Speaker 00

  Yeah, no, so actually, I think Laura Ruiz's paper is one of the one I had in, one of the ones I had in mind when I was describing earlier about this issue of mechanistic interpretability. I think it's a good paper in terms of they have developed a interesting set of techniques to actually see what is actually going on in the in the in the in the way L L M's are outputting their tokens. But the thing that is unsatisfactory to me is yes, that basically two things. One is first of all, everybody knew that L M's are actually not doing retrieval alone. That was kind of well-known way before, right? And so there is nobody who believes that L M's are just doing retrieval.

• 44:08Speaker 00

  And the question is, you know, what else are they doing? And is there any clean, any clean characterization of what they're doing? That I did not see. I actually looked at that paper. I think, you know, they've done good work, but I am not yet. I'm still hoping that there would be an interesting characterization. There are lots and lots of groups are trying to look for a characterization of what this fractal intelligence might be right now. But we haven't gone further than that. In terms of everybody might be right, I mean, the sense that it's there could be

• 44:47Speaker 00

  this whole blind man and the elephant, you know, phenomenon in play to some extent. And that part is possible because we are actually trying to

• 44:59Speaker 00

  piece through large number of parts of this puzzle, right? Including the reasoning part, including what are they even trying to do? Including what sorts of techniques seem to improve their accuracy and so on. But I think that's part of science. And basically, my sense is eternal discontent is part of science. I actually am much more worried about being too optimistic that we figured it out. Then I am about being somewhat more discontent that we haven't yet figured it out. And so I want to err on that side, not because I think we know more than before when, you know, G P T three came out. But on the other hand, and I think both of us, all the camps know.

• 45:41Speaker 00

  I mean, the people who thought G P T three is A G I know that that's not the case. And the camp that thought G P T three is just stochastic parrot has to know that it's more than that. Okay, by now, so that is collective improvement in our intelligence. But still, there are still large number of pieces that we have to figure out.

• 45:59Tim Scarfe?

  Yeah. I mean, on Laura's papers, she was using influence functions. I'm not sure if that would be classed as classical interpretability or mechinterp, but I think mechinterp is largely about finding circuits in neural networks. And even that's an interesting discussion.

• 46:13Speaker 00

  To me, it's like a more of a, the general idea of figuring out a way of probing the inside of what L L M's are doing. I think of that as mechanistic interpretability. I mean, there are very specific techniques that have shown great promise, such as the autoencoder stuff, et cetera. But I think all of these, to me, are essentially trying to interpret what they are doing at the circuit level and try to make sense of their external behavior. To me, that, so they're like my two ways of making sense of what, you know, what L L M's are doing. One is just external evaluation. That happened already, and we know that they're not doing

• 46:52Speaker 00

  any kind of guaranteeable reasoning. And that's basically enough results showing they seem to do promising things in some cases. And they also result showing they seem to be very brittle, that you change a prompt a little bit, you change the problem specification a little bit, they'll die. Again, we are talking about auto-regressive L L M's, not O one sorts of things. That's a whole entire thing that we haven't yet started making, doing the same sort of, you know, analysis. But, you know, once you figure those out, my, my, my sense is that trying to actually get a sense of just from outside versus also try to do probing of the internal circuits. If you start doing internal process circuits, I think of that

• 47:36Speaker 00

  generally, in my view, and as the mechanistic interpretability style.

• 47:40Tim Scarfe?

  Okay, okay. But isn't it interesting, though, that she found that code and math-based procedural documents appear disproportionately influential for tasks requiring reasoning? Larger models show an even stronger reliance on general procedural data for reasoning. The presence of code data in pre-training mix seems to offer abstract reasoning patterns that the model can generalize from. I mean, these are interesting observations.

• 48:05Speaker 00

  So actually, I, again, I don't want to make this as a very specific critic of a particular paper, just because that's not fair for them as well as me. But I, I do want to basically sort of say that there is a distinction between factual tasks and reasoning tasks.

• 48:27Speaker 00

  Right, L L M's have been used for both and you know, I think the factual, I mean, they, they have troubles in both. You know, for the factuality, I would think the only sorts of things that will improve them is things like the rag style techniques where you just give the factual data and ask it to summarize. For the reasoning stuff, you basically for arithmetic and so on in a large, it, it's not, to some extent, I would expect that these are the kinds of things where the exact results don't exist. And so I would also be equally, you know, troubled by the fact that people have shown that when, if you take something like L L M multiplication, this is before, way before all this Laura's work, et cetera.

• 49:09Speaker 00

  You know, they tend to be correct in multiplications for popular digits and less correct for non-popular digits. There's this, this sort of, you know, mind blowing that there are digits that are popular versus non-popular, but that sort of is a interesting point that the L L M final performance is a complex combination of the data that they have been trained on and some additional pattern matching abilities that they are using on top. But that's not sound reasoning, so it basically, we still don't quite know where it breaks, but it's this fact that it gets to be correct for popular digits and not for some other digits. That's a particularly interesting thing to me and that sort of shows.

• 49:53Speaker 00

  By the way, while we are on that subject, some work has shown that even with Owen, we didn't, we looked at Owen more on the planning side, but some people, I think Tom McCoy and Co. did some more work and tried, basically, these are the ones who did the Caesar cipher sort of thing, the Embers thing, and they basically also found that Owen does better on some of those things, but they also still found that there are data dependencies in the sense, it's, it's accuracy was higher in the regions where there was higher pre-training data.

• 50:29Speaker 00

  Which again makes, I think it's still consistent with my view of what I think O1 might be doing. There is an LLM which was pre-trained on like some corpus. And there is this smaller LLM which is sort of generating this, you know, pseudo-action tokens that will make it output things. And one of the interesting things is actually the difference, I'm told, again, this is also, we don't know for sure. I'm told that when the original O1 models came, there was a O1 mini and O1 preview. And the difference, I'm told, was one of them, I think the O1 mini was using the smaller LLM as the base LLM. And O1 preview was using a larger LLM as the base LLM.

• 51:13Speaker 00

  So I don't know, they didn't say this as second part, but I would assume that if I have like a, you know, pseudo-action generator model, it is working on a bigger LLM which has a higher capacity, so it can generate more interesting completions. Whereas a smaller LLM that has less interesting completions, that makes a difference in terms of, you know, how the the level to which the RL-based training can get your accuracy up.

• 51:41Tim Scarfe?

  Yeah, I've noticed some interesting things. So I've now paid for O1 Pro. I was very skeptical with O1. So as you say, the the base model is an even weaker version of GPT-4O. So GPT-4O, I hate that model. I hate the style of it. I think it's dumb. And I must admit, it's mostly because I'm sort of anthropomorphizing it because I hate the style. So I think it's dumb, you know, we're we're very, humans are very brittle, even on the RLHF. You know, we like assertiveness, we like complexity, you know, there are certain styles that we like and we don't actually see the content. But that's one side, don't like the model. And O1 preview and mini, it doesn't really want to think. So most of the time it won't think and you get an even dumber answer than you would do with GPT-4O. However, O1 Pro, the vibes are different.

• 52:27Tim Scarfe?

  So it thinks more and it gives you something which is qualitatively completely on a different level. It doesn't look like dumb chat GPT anymore. It feels very, very different. But there are. Still some issues with it. So certainly for situations where you are dealing with ambiguity, doing programming or something like that. I actually like having a dumber model because it's a didactic exchange, right? I'm saying, no, you misunderstood that. Let's do this. Let's do that. We're working on this thing together. What O one does is it says, well, on the one hand, you can do this, and on the other hand, you can do that. It gives you a range of options, you know, and I'm like, well, wouldn't it be better just to either, you know, go on, dance with the model or just better specify what you wanted in the first place?

• 53:09Speaker 00

  So again, uh, two issues. First of all, and the O one pro just came, I think, last week, right? And this was the exam week for me and we haven't spent time yet, you know, spending time. We haven't spent away money yet on the own preview. I mean, O one pro, I mean, I played from outside, but I have, we haven't done any API level studies, which the kind of thing that we did with O one preview. But one thing that, you know, I've looked at the Twitter, you know, exchanges about people, the usual suspects trying the various things on them, et cetera. And the two things that jumped at me is one of the things we saw in O one preview is exactly the kind of thing you are saying. And it looks like O one is still doing it, which is

• 53:49Speaker 00

  they are good at digging to try and explain the answer, why the answer they gave is the correct answer. One of the funny things was, um, I use this one particular three block stacking example, which is unsolvable. And in fact, this showed up in the New York Times as an example of why G P T four O was actually fails on that. And when O one preview came, Noam Brown, actually, one of the, in his long tweet, one of the things was, Ravi said this in A C L talk that this problem can't be solved and O one preview actually does solve this instance. And so this is good. Now people have actually said that O O one gets the wrong answer.

• 54:33Speaker 00

  And people, multiple people, actually, I've seen this and people have posted the screenshots. It gets the wrong answer, but it argues with you as to why the answer that it is giving is still possibly correct. So this particular problem in Essentially, like, there is no way of actually solving it without moving C. And it turns out that it gives an answer where actually C moves. Because of gravity, it will fall down. And then it tries to argue with you that there are games where people will say that unless you are intentionally moving C, if the natural process makes it fall, it's not considered moving.

• 55:14Speaker 00

  Which is like a very interesting thing that we have seen in O one preview two, when it will when we'll give it unsolvable instances, which by the way, um, normal L L M's just die with unsolvable instances because they have been R L H F to death. And so they think that if you gave a problem to them, there must be an answer. So basically they'll give you something. And if for some most unsolvable problems, basically, this is why this was unsolvable instance that I showed it for oh before. Oh, one preview actually solves more of them correctly. That's a credit to it. That's why it's actually a more approximate reasoning model, L R M in my view than L L M. But on the other hand, when it actually basically gives a solution for an unsolvable instance,

• 55:59Speaker 00

  it'll argue with you that it is still actually right because and and so I made this joke in the strawberry paper that we have gone from hallucinations to gaslighting. So it actually tries to argue that you are, you know, just like what you're saying, you know, this is on the one hand, what you want to do might be worthwhile doing. But on the other hand, this is the reason why I'm doing this as well doing it. And in fact, I think this this guy, Colin Fraser, I believe, one of these guys on Twitter who keeps playing with these models. And he said he gave this surgeon problem, the classical surgeon, um, you know, the boy getting in an accident one.

• 56:38Speaker 00

  And which O one pro said the surgeon, this basically does all this whole thing. This is a classical puzzle that brings gender stereotypes into account, et cetera, et cetera. And then gives the answer that the right way to think about it is. And so this is that this is a puzzle where he makes the change that the mother and the boy are driving. And, and, and the mother dies and the, and the doctor says, I can't operate on the, on the boy. And so it's actually changes the puzzle and, and still O one apparently says, um. We should basically realize that the doctor is the second mother of the boy.

• 57:23Speaker 00

  And it'll try to argue that position. Okay. So interestingly, overthinking and is actually kind of a. And actually trying to dig down. And so one of the interesting questions that we don't know, again, we haven't played with this. Is to what extent?

• 57:43Speaker 00

  Is its explanation and its reasoning connected? You know that in humans, this is actually, I mean, this, I'm not trying to anthropomorphize what it's doing. It's just if there are two different phases. Right? If the phase one, it comes up with a solution. In phase two, if it needs to explain, if it doesn't have to. Look at what it did to get to the solution, the explanation is just dig my heels and try to say the solution is correct. And people tend to do that. Sometimes we'll come to some solution and then we'll try to come up with an explanation as to why, what we might be right. This is something that L L M's had this problem anyway to begin with, because they completely assume these are completely different.

• 58:25Speaker 00

  Things and I'm always worried about L M explanations. L R M's seem to be even more sophisticated. At this sometimes, but it's only mostly anecdotal. I haven't really done systematic studies on this. So one, I don't have like any visceral opinions about any of these models because. To be honest, I don't use them in my day to day life. Most of the time I write English well enough that I haven't yet seen a. Um, an L L M that does better job of. Things than I do and I haven't yet found useful things where I would need L L M's help. I mean, maybe I will do at some point of time, L L M's and L R M's. So I don't use, I don't have anecdotal experiences of the kind that you have. I mean, I'm mostly focused on.

• 59:10Speaker 00

  Specific systematic studies, you know, with like multiple. Instances of planning problems and, and we extended the plan bench to look at unsolvability, we look at. Longer length problems, look at scheduling problems, et cetera, to evaluate. Those are the ones that I have a better sense as to what Owen can and cannot do.

• 59:30Tim Scarfe?

  Yes, I must admit, I've updated a little bit. So I was always in the same camp as you when we thought of them as approximate retrievers. And I now I'm starting to see something.

• 59:41Speaker 00

  Yes, I think again, my again, my point is there are two different ways of thinking about it. One is it's not the L L M's which became that. So how do you define L L M's has to be some and a discussion that we should have. I mean, that's why I I keep actually talking about the stone soup metaphor. Not because I want to play down the importance of O one is a great thing. But you do have to decide who do you want to give credit to if you are arguing. Part of your and definitely my reservations about reasoning abilities of. L L M's were they were auto regressive teacher force training things and that was true from G P T two point five all the way to G P T four O.

• 1:00:26Speaker 00

  And OpenAI knows this. OpenAI knows it enough that they no longer call it. This is not G P T O one. You know that it's called O one. It's like a completely different model and they know that it's not. All you can say is that it was done by some of the same people that also developed L L M's. But we can't define L L M's to be whatever it is that OpenAI is producing. I mean, we have to have, you know, theoretical definitions and my sense is auto regressive L L M's still have all the problems, but all the advantages. Because they're very fast. They're like amazing fast system ones. And O one is a reasoning model because it actually adds the reasoning post-training as well as reasoning inference.

• 1:01:10Speaker 00

  Which nobody said will not be doable. You know, it's great still that they are able to do it in a very general sense, but I don't think there was any argument. That I mean, A I systems were able to do reasoning, right? After all, AlphaGo is basically a reasoning system. It was just a deep and narrow reasoning system. And the question was, is some more general, broader, but you know, not as shallow as L L M's is as L R M's. And which is a good step in the right direction. But it doesn't change what I thought about L L M's, which is the autoregressive models. They are different and in fact, they have advantages that O one lacks. For example, the cost of L L M's can actually be much lower.

• 1:01:55Speaker 00

  It is indeed much lower. So, one of the studies, one of the things that we learned in the strawberry paper, for example, uh, the, the planning in strawberry fields paper, um, is that in, in some cases, if you are giving, you know, if you're basically, you have to think of computer science is eventually about efficiency and cost too, right? So if you're giving a particular instance of the problem to O one and you pay this many dollars, um, versus you give the same instance to the L L M with a verifier in this inference time scaling approach, what I would call L L M modulo, in which is a general approach that we have been pushing. Um, the L L M modulo approach where it uses a autoregressive L L M to generate many candidates

• 1:02:39Speaker 00

  and an external verifier or even an L L M based verifier, other learned verifier to check, can actually be cheaper than O one just doing one candidate with the same accuracy. That becomes interesting because then, you know, part of the interesting thing about human civilization is on one hand, we are general purpose, you know, reasoners. But on the other hand, we also know that every job requires a tool and we do that too. We're basically, we base, you know, the fact that, you know, it basically we doing everything that like a, a particular specialized tool does can be extremely inefficient in terms of the time that we are spending.

• 1:03:22Speaker 00

  That is going to be the case for these reasoning models too, to some extent, because O one actually costs quite a bit right now. How much, when it's going to change is anybody's guess, but. That sort of brings up, in fact, there was a Shep, uh, writer, the, the L S D M guy. Coming here on Friday. Oh, great, that's great. So you should ask to him to. So yesterday I was in his talk and so he basically made this, uh, one of the slides basically was the bitter lesson is over and efficiency is going to matter. And I completely agree with that. I've been arguing this for a long time too, that think about the following thing.

• 1:04:02Speaker 00

  Um, the first time when we sent a man, uh, and humans to the moon, cost was not a consideration. We want to show that we can do it. NASA was the one just doing it. The second time and the, and for the space as well as the moon, the second and third time, et cetera, maybe okay. But by now it's Elon Musk sending people to space and supposedly possibly to the Mars too, because the cost matters. Right? Essentially, it's, once it's been done, then you start caring about, uh, you know, the cost that you're paying. And computer science is actually quite a bit about the unsexy parts of cost, just as it is about

• 1:04:45Speaker 00

  doing things that haven't been done before. And we are now in the second phase where we are actually going to care about, um, basically, how much am I spending, how, you know, in terms of the, the, the pre-training cost, in terms of inference costs, et cetera. And is the, are there better approaches that I can be using? This has been the case with computer science before too, and it was just sort of became less of an issue for a while because we were, L L M's were just system ones because there's no at all inference time cost. Okay, even though the post-train, pre-training was very costly, inference time it was very cheap. Right? And, and so we didn't have to worry about it. Now we will worry about it. So one of the funny things,

• 1:05:30Speaker 00

  the, the elephant in the room for our plan, plan bench problems on O one preview was that.

• 1:05:41Speaker 00

  The special, the normal classical planners that are meant to solve these problems, solve them in like fraction, I mean such a small fraction of the cost. They work on our laptops and solve all the problems with hundred percent guarantees. Right? So the question is, I, I realize they're completely specialized only for that problem. And then on the other hand, you have this very general purpose thing which has cost as well as inaccuracies. We start worrying about the trade off. What level in this generality cost spectrum are you going to find home? That is going to be very important thing.

• 1:06:22Speaker 00

  And I think that's sort of what Shep Hapreiter was, you know, uh, say hinting at when he said, you know, bitter lesson party is over, that you do actually need to worry about the cost you are spending to actually achieve a goal. The first time you're achieving that goal, nobody cares about the cost because it's never been done. So you're doing it, you get all the credit. But, you know, umpteenth time it's being done, you know, because it becomes like a normal day to day thing. Then the car efficiency aspects matter.

• 1:06:49Tim Scarfe?

  A few things on that. I mean, first of all, with O one Pro, I think it's worth $200 a month and you, you can call it a hundred times a day. Of course, um, API is very, very expensive, but I'm already spending, you know, over a thousand dollars a month on Claude Sonnet 3.5. But, um, you raise an interesting, an interesting point. I mean, first of all, the utility of an O one model, it's, it's a bit of a weird model, right? Um, it's useful in certain specific circumstances. And if anything, because of the verbocity and the distractors and the context, it's not really a model that you want to be using most of the time. But that raises the, the sort of the pragmatism and the architecture and the efficiency thing that you're speaking to. So I spoke with some guys this morning and they have built a kind of neuro evolution approach to designing multi-agent systems.

• 1:07:34Tim Scarfe?

  You know, at the moment, we, we hack in the tool use, you know, oh, do we use a debate pattern? Do we have a, um, a small model and we prompt it a lot or do we use a, a, you know, a bigger model? And we're all just hacking together these multi-agent architectures. And some of those architectures will even be doing the kinds of things that, that you're speaking about. So rather than it trying to convince you that it got the right answer, there might be a supervisor agent which does some reflexive. Uh, there might be another agent which generates the planning symbolic code and runs it on a tool. So, you know, we're, we're building these big complicated things. And, and I think that's the process that we need to figure out now is building the systems that actually use this technology in the best way.

• 1:08:09Speaker 00

  Yeah, so I think the thing, I sort of agree, but one thing that I want to point out that a distinction is that these two notions of use of these kinds of models, when you do a subscription models, twenty dollars or two hundred dollars. I would argue that that is by definition, human in the loop with the model being an assistant to you. And it's a very different way of evaluation, where you were unhappy with the previous model because it was wasting more of your time and then it's not worth it. For you, this one was helping in whatever you were doing and you are happy with that. That's one particular type. In general, I've actually.

• 1:08:52Speaker 00

  I I've always thought, and I think we talked about it the last time too, that large language models and large reasoning models now too, they're all. Um, there's no question that they are intelligence amplifiers. There's like no question on that part, okay? I mean, if you want to use it, you use it and people are able to find users for that. That's great. The part that I'm actually lot talking more about and that's most have been, that's been most of our work is really. There would be scenarios where they, these become the end user facing systems. Where they'll make the decisions, they will just say, this is the answer and then I'm going to, you're going to execute this plan.

• 1:09:33Speaker 00

  So the robot will execute this plan or this is the travel plan for which I'll buy the tickets. You don't get to come back in and, you know, say, oh, I don't like this travel plan. That's what you do in the, you know, the subscription model. But the one that I'm talking about, basically the API access is basically what people, all the startups who are trying to build additional tools on top of these models, they are going to give specific autonomous functionality. And there, that's where I'm talking about the actual computational cost versus benefit for a certain level of accuracy at the end user time.

• 1:10:13Speaker 00

  Um. Both of these are, they are very different kinds of uses and I actually have no question at all in my mind that all L L M's and definitely also L R M's are just great intelligence amplifiers. But that's not what my worry is, that the, the whole thing has always been, my worry has always been that people are trying to put this in the end user facing situations where they'll actually come, make the decisions and some executor just executes it without pushing back. And, and when that happens, the guarantees matter, the, in terms of the brittleness of the reasoning matters. If you are in the loop, you would never, you know, it's like if you have an assistant and assistant,

• 1:10:56Speaker 00

  you may fire the assistant if they are giving mostly bad ideas, but you will never blindly just use the assistant's ideas, right? So you will always be, the buck stops with you. That's a very different way of using L L M's. Then L L M's are the ones that the patient talks to. There is no doctor between the L L M, L R M and the patient. In which case the, their accuracy matters and their cost are in getting to a certain level of accuracy matters. And these are two very different uses and I'm much more interested in the second use than the first use.

• 1:11:28Tim Scarfe?

  Can I push back just a tiny bit? So first of all, I completely agree with you that these things used autonomously, they don't work. They don't work for all of the reasons that, that you said, but that's not how they're being used and they're not being used like that because they don't work. But um, what we are seeing is that all of the successful reimagining of applications with language models are completely interactive. So they have a human in the loop and the human is, is supervising, augmenting, redirecting and so on. The next step that we haven't seen yet, but we're starting to see is um, having autonomous agent-based systems, you know, with multiple levels of reflection, checking and, and so on. For example, it could be um, a bunch of agents generating programs. It could be contributing to a library of programs.

• 1:12:10Tim Scarfe?

  The programs are being supervised not just by you, but other users of the application. And the whole thing just grows and it's a living ecosystem. So there's some diffused form of, of human supervised verification. And maybe in the future, you know, the humans might be increasingly taken out of the, the front plane.

• 1:12:27Speaker 00

  I think that's a very sane way of using, but I'm afraid that's not the only way that's being used. And in fact, most of the people, so actually the two issues, one is if that's the only way, I'm very happy because it's like it's a tool and you would use it and the onus is still on you. Finally, the buck stops with you because you are in the loop, right? But most of the image users, at least from where I sit and the kind of startups that I hear from and the kind of papers that I'm even reading, they're all about autonomous users. And that's where I'm actually looking at the fact that there is more promise than before. It was very brittle before, it's less brittle now, okay? But it is less brittle at the expense of cost.

• 1:13:12Speaker 00

  And it's actually interesting that the evaluation strategies for both of these are quite different. You know, evaluating assistive technologies is very different from evaluating autonomous technologies. And it's not that one is, assistive technology evaluation is not any easier, in fact. You know, I mean, you basically, you can say the evaluation is just if people are buying it and they keep paying for the subscription, that's a proof that people seem to be getting some, you know, value out of it. But it's actually pretty hard to correctly evaluate assistive technologies and that's a whole entire area.

• 1:13:51Speaker 00

  And in fact, most of the people who are worried about misuse of L L M's have never, they're not the ones, including like Francois, for example, Francois Chollet and his art thing, etc. It's all about, ultimately, all of this is, we are interested, irrespective of whether you believe A G I is coming next week or, you know, next decade or next century. Everybody in A I eventually wants this autonomous abilities to actually make intelligent action with guarantees, right? And And that is basically where I think we will get there. But prematurely saying whatever currently is there is already working is the one that a bunch of us are worried about and that's what we are pushing back on.

• 1:14:36Speaker 00

  With the humans in the loop, it's a completely different thing, you know. And even for the code generation right now, they're like, they're two different users. Essentially, there is also a use of code generation techniques where it tries to kind of improve accuracy to the level that humans don't have to. It's not just an idea generation for the human. If it's idea generation, it's great. They're because somebody else's job is online. It's not, you know, they're still like a bug stops with the actual programmer in the thing. So I, I think that the autonomous one is the one that I care about at any rate. And that's the one that I'm worried about. The premature declarations of they're already autonomously intelligent.

• 1:15:18Speaker 00

  And, um, but I'm generally very happy that this technology exists as a human in the loop technology. And it's kind of interesting from me sitting here to hear you say that you actually as a user, I mean, you seem to be a more of a regular user than I ever have been, you know, of the L L M's and L L R M's. That it's kind of interesting. It means something to me when you say that you like O one more than you ever liked O one preview and you kind of okay with G P T four maybe, but now like O one little more. And that sort of you basically are getting value out of it, but you still can always, you have the red switch, you can decide not to take its answer, you know.

• 1:16:00Tim Scarfe?

  O one pro.

• 1:16:01Speaker 00

  Yeah, okay, O one pro, yeah, okay, yeah.

• 1:16:03Tim Scarfe?

  The only difference is that there seems to be when it, when it thinks for a long time, there's a qualitative improvement, you know. I wanted to get your take on, on something else. So, um, we, we're seeing, I mean, you had your L L M modulo architecture and then we've got this huge approach of test time, this kind of green blating approach. So you green blatch the model and you get it to generate loads and loads of Python functions.

• 1:16:24Speaker 00

  Hmm.

• 1:16:25Tim Scarfe?

  And in a way that this, this is the sort of thing that we like because we like.

• 1:16:28Speaker 00

  Yeah, the arc thing, yeah, yeah, yeah. Green blatt approach, yeah, and green blatt, okay, fine, yeah.

• 1:16:31Tim Scarfe?

  Oh, yeah. We're seeing that in lots and lots of different ways. So doing loads and loads of inference and then, you know, we've got these Python functions and maybe we do, you know, library learning and remixing and, you know, we're in the world of code. So we're using, we're using code, we're generating an explicit function, we can verify it. We love that, you know, we're in a very happy place. But now we're seeing an interesting shift. So certainly on ARC and on several other papers, people are moving towards this idea of transductive active fine tuning. And that simply means rather than generating an explicit Python function and doing it loads and loads of times, let's just generate the solution directly just using the neural network. And this is a step away because we like programs.

• 1:17:11Speaker 00

  Yeah.

• 1:17:11Tim Scarfe?

  Because, you know, programs are Turing completes and we understand what they mean and everything. And now there's a whole load of people that say, actually, the neural network can just can just do whatever the program does. Let's just let the neural network output the solution directly. What do you think about that?

• 1:17:26Speaker 00

  So, to be honest, I haven't followed that work as closely. So I have, my answer is somewhat likely more generic. I would be surprised. I mean, I would have the same bias that in fact, there's an old saying that, why write programs when you can write programs, write programs? That's the version that we are talking about. It's basically you want to generate higher level code that generates the solutions. This has always been the, the conceit of computer science. So I am surprised. I don't actually know specifically the work that you are referring to in terms of just going back and directly going for the solutions because honestly,

• 1:18:09Speaker 00

  in the case of, in this, in the context of inference time scaling, one interesting question is you generate loads and lights of, loads and loads of candidates. The candidates can be the direct solution candidates or the code candidates, either which way. And then you still have to have verifier. If it's code, you need to have code verifier. If you have solution, you need to have solution verifier. And one of the interesting questions is where is this verifier's coming from? And there has actually been one of the more, you know, effective ideas that we've been pursuing is you can essentially generate verifiers, like they're like.

• 1:18:45Speaker 00

  Of course, there are symbolic verifiers that might be there for specific things and that, you know, we can use that L L M modular style frameworks. But you could also use learned verifiers where essentially you just basically learn to do discriminatively, what is the solution versus what is not a solution. A third idea is generate the code for the verifier and then correct it. And that's actually, in fact, it's still for, at least in our case, it seems to be promising. We are working on some things that are going to come out soon. But, you know, basically, I still think that, and especially in the context of L L M's, in the context of L L M's, okay. So it's like, again, it's a very different thing. If you're not having L L M's in the loop at all, it's a different question.

• 1:19:29Speaker 00

  But if the L L M's are there, one of the things they're actually good at doing is like outputting, you know, basically they can output code as well as solutions. And in which case, you know, the code can output lots and lots of classes. I mean, you know, lots and lots of classes of solutions can be verified by the code. And so if you correct it once, then it will, you know, work for a longer time in a sense. And so I would still think that, at least for the, you know, inference time scaling verifiers case, that seems to be still a good idea. I don't quite know the specific context from which where you're saying this. People are saying that the transductive directly, you know, guessing solutions would help.

• 1:20:13Speaker 00

  I'm not sure whether they still have L L M in the loop. Are they just saying we'll just directly train a separate neural network?

• 1:20:20Tim Scarfe?

  Well, I'll sketch it out. So Solving Arc, they have two L L M eight billion models. And one is generating Python programs and they greenblat it. The other one is trained separately just to output the answer grid directly.

• 1:20:33Speaker 00

  Okay.

• 1:20:34Tim Scarfe?

  And in both cases, they do, you know, inference time compute. So either generating lots of Python programs or doing active fine tuning of the direct solution one by augmenting the test time examples. And what they found is like on the Venn diagram of the, you know, their success rate, they find that for some problems. The the program works really well, you know, like the the green black approach. And for some problems, um, you know, certainly things like mosaics and spatial perceptual type stuff. The transduction works really, really well. And and this is kind of weird because if you think about like the the the space of functions that the neural network could could reason about. They they should be the same. So I don't know whether it's just because of limitations in the neural network or characteristics of the problem or something that you see.

• 1:21:18Speaker 00

  To me, interestingly, to me, again, it depends very much on the space of solution configurations versus space of code configurations. There are many problems where solutions might be of less quote unquote syntactic complexity than. And and so a neural network that can kind of guess a string may not be able to guess something that looks like a syntactically correct Python program, right? L L M's actually can do the later. And so it is interesting that if you can do that and if you still go back to neural network to actually directly guessing the solution, it being a more useful step.

• 1:21:58Speaker 00

  I I begin, the stuff that we are doing for the verification thing is still in the, you know, um, initial stages, you know? And we haven't actually checked this kind of a trade off, whether it would exist. So, you know, I have no more insights specifically on why that might be happen.

• 1:22:15Tim Scarfe?

  Wonderful. What are you doing at the conference this way?

• 1:22:18Speaker 00

  That's fun. Uh, so I'm just here and to today. And I think we did this chain of thought, thoughtlessness, chain of thoughtlessness paper. And then I kind of said that it's like, mostly when we wrote it, it was like, it can't follow procedures, so I should be able to show it. But now, actually, I explained the whole thing the way I explained to you here in the beginning. Essentially go from prompt augmentation. And so I kind of think that, like Schopenhauer said, life must be lived forwards, but only makes sense backwards. Uh, and, uh, you know, papers also only make sense backwards. You know, after a while of writing, you know, you actually look at it and say, what I really want to say is the reason chain of thought is not a great idea

• 1:23:02Speaker 00

  is because you really want to think in terms of prompt augmentations and humans coming in the loop becomes less important. So that's what we did. And then I'm actually going to this compound systems thing. And a great time, they're like sixteen thousand people and, you know, running into lots of old friends and so on.

• 1:23:18Tim Scarfe?

  Yeah, one of the best moments from the last interview is when you were talking about that paper, like saying that, you know, that they can catch, you can teach someone to catch two fish or three fish or.

• 1:23:26Speaker 00

  Yeah, that's yeah, yeah. Yeah, I mean, that's basically because it doesn't quite know how to generalize. And so I kind of made that thing that, yeah, essentially, because you have to kind of give it examples for four, four-word problems, again, give it examples for seven-word problems, again, give examples for nine-word problems, et cetera, and then try to improve it. Whereas the kind of seat people think is when people, when you say this, they'll say, oh, you must be, it must be doing, you know, procedure generalization. The interesting thing again is, I think we had this conversation last time too, that the way I look at this, I mean, I'm skeptical only because of just having

• 1:24:07Speaker 00

  some additional, additional background. And one of the things is McCarthy, John McCarthy, was the founding fathers, I mean, the guy who coined the name artificial intelligence, basically said, the holy grail of AI is an advice taker program.

• 1:24:23Tim Scarfe?

  Yeah.

• 1:24:24Speaker 00

  And advice taking is AI complete. And if chain of thought is able to make L L M stake advice, that would be pretty impressive. And I kind of went in thinking that there has to be holes there. And, and so that is where that fish, one fish, two fish thing comes in. But, but the more interesting thing is, I think, de-anthropomorphizing L L M's and trying to think of them as these, basically, these alien entities for which, you know, arbitrary, you know, prompt augmentations, prompt augmentations will, can, you can generate good behavior. So by the way, one example of this that people should be thinking about

• 1:25:05Speaker 00

  is, if you think of jailbreaks and L L M's, jailbreaks are, you give a normal prompt and you give this particular carefully constructed learned sequence, you know, Zico Koltz, original paper, his group's original paper shows, that sequence makes no sense to humans, but it will make most L L M's provide a deterministic. Behavior like saying got you or something of that kind. And essentially, that should tell us that they're not seeing language. And so the prompt augmentations don't have to make sense to humans in the loop. And and and that's okay. And because in some sense,

• 1:25:45Speaker 00

  looking at things, the only change of thought that sort of made sense to humans was giving this false impression that somehow L L M's are doing things like we do. But that's not the way it is, you know, so might as well just live in, you know, go with what they can do and optimize directly, which is what the inference time uh scaling and post-training methods seem to be doing.

• 1:26:08Tim Scarfe?

  Yeah, the the one thing I get stuck on is we can criticize individual L L M's. I mean, yeah, that they are approximate retrieval engines. My co-host Keith Dugger, he's always at pains to point out theoretically that they're not too incomplete, you know, that finite state automata and and all of this kind of stuff. But the thing is, it all breaks down when you talk about L L M systems. So even with the chain of thought thing, right, I could have another supervisor model that could generalize the prompt to go to five fish, six fish and so on. So we can easily build systems that overcome all of these criticisms. So at some point, does it just seem like we're just, we're making criticisms that can be easily

• 1:26:44Speaker 00

  No, no, actually, it's a very good point. So in fact, after I, after this, I'm going to this compound systems meetup. I'm completely a big believer in that whole direction. But there are some people who don't want to believe that the usual L L M aficionados don't. In fact, by the way, it's a very interesting thing that OpenAI was at pains to point out that O one preview was a model, not a system. It's not me and you saying it, it's them saying it. They would like to say there is this one size fits all model that will do it. And so it is reasonable to take their word for that. But parallelly, I also like the compound systems work.

• 1:27:24Speaker 00

  And it makes an L L M modulo is a compound system. And that's what basically it it improves on all the limitations of L L M's and a set of limitations of L L M's. I'm completely fine with it, you know, again, it doesn't matter to me as long as I can give guarantees and it in a safety critical scenarios, I'm fine with it. I don't have that bias. But if you are saying a single model will do it, I will take you at your word and then see whether or not that's true. That's a, that's a fair thing, it seems to me.

• 1:27:54Tim Scarfe?

  Why do you think, you know, Google have completely embraced, you know, hybrid systems? OpenAI, they, they really clinging on to this single model that does everything?

• 1:28:03Speaker 00

  I think they're slowly changing that, but I think there was a reason, there's a, I think, I mean, to some extent, I can understand. And in the sense, it would be, the, the sort of thing is this anthropomorphization again. We only have one brain. It's not that we have a brain for eating and a brain for, you know, just one brain, right? And so it would be nice if what we are trying to do would somehow basically be this, this one size fits all, this general, you know, system. But at the same time, there's also this issue of whatever I do, I want to provide guarantees, you know,

• 1:28:42Speaker 00

  so that it can be used in septic critical systems. And these, so the, the problem is the modern AI and neuroscience and cognitive science, they're not one and the same, right? I mean, everybody understands that essentially, I mean, neural networks themselves are not really that well connected to brain. And essentially they're like biologically implausible and L L M's are definitely not, you know. But there's nothing wrong with that. Just like we say, you know, the planes don't have to flap their wings, you know, um, um. So these are, but we don't essentially try to make sense of planes and birds, um, in, in, in, in the same sentence, you know,

• 1:29:25Speaker 00

  because they both fly, but other than that, you know, the mechanics are different, the, the, the things that, the flight equations are, you know, not, not at all the exactly the same things, you know, um, that's going to be more of the case with the L L M's too. And as long as we realize that, you know, it would be good, but I think OpenAI, I think originally they were hoping, my sense is, much of these people are hoping that they will just get one, you know, like two birds with one shot, right? Like we'll get AI systems as well as understand how the brain works. But I don't really think that part, nobody really believes, honestly. I mean, you might use this. Systems to improve our understanding in actually doing neuroscience. In fact, I think what's his name?

• 1:30:10Speaker 00

  Um, uh, Sang Kim, I think, um, that he basically says, you know, obviously these systems help in actually doing neuroscience research. But they are not actually telling you how brain necessarily works. So but that might, that's just a speculation. That might explain why people, you know, open, you know, OpenAI and some of these people were, you know, sticking to. But I mean, the kind of conversations I've been having on the sidelines in the conference already. The companies, there are the startups, et cetera. They are already sort of going much more into these hybrid systems, much more into these compound systems. Yeah. And, um, and it's like, you know, that would basically not be a single system, but OpenAI also is slowly coming up with these fine tuning models.

• 1:30:54Speaker 00

  They have this RL fine tuning stuff for your specific kinds of scenarios, et cetera. So it would be interesting to see, um, uh, but I think I just going back to your original idea. I think compound systems is a very different, they're basically

• 1:31:12Speaker 00

  the individual role that the LLM play, LLMs have to play is much less demanding. In fact, one of the fun things is we can do LLM modulo with normal LLMs or LRM modulo with instead of LLM, I call O one. And so it'll be the generation of candidates is cost clear. And we actually show in the strawberry paper that we can improve further the performance of O one preview on some of the problems, even though we couldn't change the how much time it takes to think, et cetera. We can just by calling it multiple times with the correct, you know, better criticisms of the, you know, instance, the problems answers it gave, we could improve its performance accuracy quite significantly.

• 1:31:55Speaker 00

  So that is still using them in a system, you know, LRM's themselves can be used in a system. But I think O one, OpenAI itself just wants to call it just models up until now. Let's see what happens.