Ep 44: Co-Founder of Together.AI Percy Liang on What’s Next in Research, Reaction to o1 and How AI will Change Simulation

Percy Liang is a Stanford professor and co-founder of Together AI, driving some of the most critical advances in AI research. Percy is also a trained classical pianist, which clearly influences the way he thinks about technology. We explored the evolution of AI from simple token prediction to autonomous agents capable of long-term problem-solving, the problem of interpretability, and the future of AI safety in complex, real-world systems.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Jacob Effron?

  This is Unsupervised Learning, Red Points AI Podcast. And today, we have on Percy Liang, one of the world's leading AI researchers and co-founder of Together AI. Percy is a professor at Stanford, and outside of founding Together, which is an AI unicorn, Percy has led some of the most innovative AI projects. In his work on generative agents, Percy created a virtual world similar to The Sims, where AI agents interact with each other, allowing researchers to study complex social dynamics. During the episode, we also got Percy's takes on the future of evals, why interpretability is so difficult, and the current state of AI research. Overall, this was an awesome episode with one of the leading minds in AI research. Don't forget to stick around for Pat and my debrief. Now here's Percy. Well, thanks so much for coming on, really appreciate it.

• 0:44Speaker 01

  Yeah, thank you for having me.

• 0:46Jacob Effron?

  Awesome. Well, I thought there's a lot of places we can start, but I think, you know, top of everyone's list these days is OpenAI's new O1 model. And I'm curious, just your reaction to it. Is it a crazy breakthrough? Is it something that, you know, what's kind of, what do you think when it came out?

• 1:02Speaker 01

  Yeah. From a product perspective, I thought it wasn't very good. It was kind of slow. It was sort of hard to kind of use for many of the things I wanted. From a research perspective, I think this signals sort of a change that I think we'll see going in the future. And I think the idea of test time computer, it's been around, but I think with many things, it needs sort of a certain kind of scale for people to have it sink in. I think a lot of what we think about language models is prompt comes in, response comes out, people measured tokens per second and whatever, and everything happens within a few seconds.

• 1:45Speaker 01

  But I think that's really understanding the value and the potential of an AI being able to solve much more ambitious tasks, tasks that take on the order of days or weeks or even months, that kind of the big projects that we as humans take on. And I think directionally, at least, O1 is just like a very small step in that direction, but you can kind of see where the field could go is that you have these agents that can reason and plan and do things for quite some time and solve maybe ambitious, like come up with new research or invent new drugs and so on.

• 2:21Speaker 01

  Um. One other thing I think is interesting is, um, you know, if you rewind ten years ago, you know, reinforcement learning was all, all the rage, AlphaGo, you remember that?

• 2:31Speaker 02

  Of course.

• 2:31Speaker 01

  Um, and in the last, uh, you know, uh, five years or so, we've kind of, you know, almost forgotten about this because now it's, uh, large language models, uh, predict the next token. Uh, but I think we're kind of starting to see a shift where now these are, well, agents. I mean, we're using the same word even. We have agents that can take, you know, actions. And I think the way to interpret the, the generation of a language model is not necessarily text, but it's just, you know, actions in, in some space. Um, and when you are embedding an agent in a larger, uh, task, when it's, you know, operating for a long time, then there's a possibility of getting more and more experience.

• 3:13Speaker 01

  And that experience, uh, with appropriate reward, uh, signal means that you can actually learn and iterate on that. So, so I think we'll see a lot more of, you know, agents being deployed in some task and getting feedback and actually making them, you know, better.

• 3:26Speaker 02

  Yeah, super cool. I saw the, uh, I think, uh, the LM sys or, um, there were some benchmarks around it that the, the O1 models were incredible at math, right? Or like,

• 3:35Speaker 01

  Yeah.

• 3:36Speaker 02

  are there specific domains that you look towards or like, okay, if it's really good at, you know, math or some of these other things, then that means, you know, it'll be better at these multi-step kind of agent type workflows?

• 3:47Speaker 01

  So my favorite new one is a benchmark that we just put out.

• 3:51Speaker 02

  Okay.

• 3:51Speaker 01

  It's called Cybench. It's basically capture the flag cybersecurity.

• 3:55Speaker 02

  Oh, yeah, that was super cool.

• 3:56Speaker 01

  Exercises. And these are, you know, crazy car. I can't, I don't think I can solve, you know, you know, most of them.

• 4:02Speaker 02

  You can't solve a hook those mere mortals.

• 4:04Jacob Effron?

  There's no chance.

• 4:05Speaker 01

  Well, I'm not a cybersecurity expert either, but, um, but some of them, the hardest one takes a team of, of human competitors over 24 hours to solve. And currently the models are able to solve the challenges where the, the first time that a, a team or a person has been able to solve is around, you know, 11 minutes or so. So there's quite a bit of a gap. Um, so I think this is a challenging, you know, benchmark. So when people say like, oh, benchmarks are being saturated, I was like, okay, well, you know, why don't you take a look at, uh, um, this benchmark? So that's one thing that we tried O1.

• 4:42Speaker 01

  Actually, it was a little bit strange because and this, I guess, points out the subtlety of evaluation. If we use it as a drop-in replacement, it actually doesn't really improve overall. Actually, it does improve in certain respects on that kind of a sub-task performance. So if you break down the task, then that gets better. But overall, it didn't, you know, it wasn't like a huge bump. And I think the reason is that when we dropped it in as a replacement, we're treating it as not O1, but like a normal language model. So we have our own templates for how the agent should, you know, reflect and plan and do all these things. And O1 just ignored all of that. It just did its own thing and generated the answer.

• 5:27Speaker 01

  So it wasn't really compatible with our framework. So that's why I think the scores weren't as good. So I think if you spend more, we've spent more time or developed towards it, I think it could be better. But I think maybe this is just kind of a cautionary tale where if you just look at raw benchmark scores, it doesn't necessarily tell you the full story. Another cautionary tale is that, you know, we have an idea of monotonic progress. But if you are using AI in a context of a larger system, there's sort of this compatibility that needs to be there for you to actually see the improvements. So if like it doesn't fit in, then you're gonna just not get good results. So for everyone who's trying to use, you know, O1 or any sort of new model that's coming out,

• 6:12Speaker 01

  I think that compatibility is something that's maybe not looked at as much.

• 6:17Jacob Effron?

  I think it's a really interesting question what you alluded to around like, you know, so many of these AI applications have come out on like the GPT-4 level models and they built all the scaffolding around it and kind of ways to chain things together in prompts. And it's like, I think it's a big question of like, will that stuff still be relevant for like this next generation of models? Or was like, that was all great scaffolding for like the previous generation and now it's like a whole new set of things that you have to do?

• 6:38Speaker 01

  I do think that a lot of that will change. I don't think we should be particularly proud of or attached to the scaffolding that has been built up so far. I mean, it's just like a chain of prompts. I mean, it's, I think it's very dispensable.

• 6:51Speaker 01

  I think the paradigm might shift in a way which O1 kind of reveals that, right? So the idea of having all that reasoning scaffolding kind of internalized to the model and furthermore not exposing it to the user, I think that's gonna be somewhat, actually from a research perspective, kind of annoying because if things go wrong, you just can't debug it, you don't see the trace, right? You don't get the stack trace in some sense. Which might make it harder to, actually for a developer to customize and develop applications. I think OpenAI's hope is that they will just take care of it.

• 7:29Jacob Effron?

  Right.

• 7:30Speaker 01

  And so, which

• 7:31Jacob Effron?

  I assume they don't want people to train on the step-by-step logic.

• 7:33Speaker 01

  Yeah, yeah, there's competitive reasons why you don't want to do that. But I think there's this kind of tension here where, yes, when it does work, it'll be great, it'll solve like IMO problems. But when it doesn't, which is gonna happen all the time because all the places where we didn't have data, or OpenAI didn't have data coverage, or you have some novel application, the ability to customize, I think is important. And hopefully we'll talk about open weight and open source models later. But we're seeing that we're kind of losing that even with the closed models. Whereas before, if you have OpenAI GPT-4 and you had an application with a bunch of

• 8:14Speaker 01

  chains, at least the prompt was transparent and you had openness there. But now if that's getting internalized and it's just, you know, it's a black box.

• 8:25Jacob Effron?

  You mentioned kind of like playing with O1 when it came out. Obviously, you've invented and been a key part of a lot of the benchmarks that people use. But like, what are you personally, like, the morning that O1 drops, like, what are you testing out to see if it's, you know, what it can and can't do?

• 8:37Speaker 01

  Yeah, you know, it's kind of funny, the AI developer doesn't actually maybe use AI as much as one might think. I think that, you know, there's a few things you could kick the can around. I don't have like a set of prompts. Sometimes it's just kind of like trying whatever's on the top of my mind and looking at kind of tail behavior. I think the other thing is I do use ChatGPT for various things.

• 9:05Speaker 02

  Yeah.

• 9:06Speaker 01

  Um, over the weekend, I was like trying to write this React app. I haven't used React in a long time. And so it was it was rather helpful. I actually used O one and it was not, it was it was slow and kind of annoying. So I switched back to G P four.

• 9:18Speaker 02

  Yeah, yeah.

• 9:18Speaker 01

  Because for that application, it doesn't.

• 9:21Speaker 02

  Well, there's interesting use cases where you want, maybe you want immediate feedback and you just want something that's fast. But it's okay if it's wrong. You're gonna kind of steer it along the way.

• 9:30Speaker 01

  So so it's very multi-dimensional, like cost, speed, accuracy, which which task is a good on, is a customizability. I think all of these facets matter. And so I think you're portraying it as like a kind of one dimensional, like models are getting better is a little bit of oversimplification.

• 9:48Jacob Effron?

  It was interesting to see how obviously it's better on certain task coding math, but actually worse on others.

• 9:53Speaker 01

  Which makes a lot of sense because math and coding are the ones where these reasoning chains really help. And also you can get, you know, better supervision compared to other tasks.

• 10:05Jacob Effron?

  Yeah. One thing I was struck by when we were preparing for this is I feel like your research and and kind of all the things you spend time on. It's like the breadth is incredible, right? You're you're doing like a lot of different things. Um, and I wonder like how you think about, you know, obviously you said at Stanford, um, the role that like academia should have in AI research and how you prioritize, like where where does it make sense time for, you know, where does it make sense for you to spend time? Where doesn't it?

• 10:29Speaker 01

  Yeah, yeah. I think this is a really good question. And often you hear kind of people in academia having this existential crisis where G P D four, G P D four O or whatever. Oh, one is coming out and my research is doomed. And I think, um, and it's it's totally unfair. They have more resources than us, you know, how can we?

• 10:48Speaker 02

  Make it tens of thousands of G P U's.

• 10:49Speaker 01

  Yeah. And and I think that's exactly right. If you kind of go head against the open air, it's just, there's there's no point. So the point is that you should, there should be another game, you know, you play. And I've thought about this quite a bit, and I think one of the key things is to be orthogonal, right? So it should be the case that whatever research project you pick, um, if G P D four or G P D five, let's say G P D five comes out. It should enhance, either enhance your research or be like kind of irrelevant in some sense. So just to give a few examples, um, so we had this work on generative agents, maybe we'll talk about it a little bit later, which is,

• 11:32Speaker 01

  which would be enhanced. If the models get better, then the ability to use these models to create, you know, simulations of C I D will get better. And that's great because you're, we're not interested in, you know, just the raw language model. We're interested in uh novel use cases of the language model. Again, another example, we have a benchmark on um, you know, ability for a language model to solve machine learning, you know, engineering tasks. And that's going to get better. So every time a new model comes out, that's very kind of exciting uh for um those applications. Um, another direction I think academia I think uh is could do more in

• 12:13Speaker 01

  is is more thinking about the open, you know, source uh community. Uh, because at the end of the open, um, academia is about open science, right? Which means that creating knowledge and putting out in the public domain. And while there is a lot of knowledge at, you know, a lot of these frontier labs, it's not in the public, you know, domain. So I think of major contribution is actually discovering, or even even if it means reinventing, who knows? Um, uh, and publishing it, I think, has a lot of benefits because it can get more easily taken up by the broader community and, you know, you have new models and products or whatever being built on top of these ideas than if they were just in kind of one, one, one place. Yeah.

• 12:58Speaker 01

  So often, I think some of the things that we're doing about kind of understanding, you know, uh, data quality and and pre-training, for example, or how to weigh data is something that I think that, you know, I don't know what the the frontier labs are doing. Maybe they're doing something already, you know, fairly uh, you know, sophisticated, but having some of these ideas being out in open, I think is a just in itself, I think is a, you know, valuable uh contribution. And then there's a third category of works that we've been doing around. You could say it's either you could call it transparency or benchmarking or auditing.

• 13:35Speaker 01

  So I think academia stands as sort of a unique position where we are not, we don't have commercial interests, we're trying to develop things that are for the benefit of society or at least for the public good. And so when we benchmark or we had a project that assesses the transparency of different providers, that work is extremely hard or almost impossible to do if it were in a kind of a commercial setting. So academia kind of makes sense from that perspective. And we have some collaborations with folks at the law school and other areas of campus

• 14:20Speaker 01

  where there are a bunch of unique kind of problems that come up when you look at an academic context. So enough to keep it busy.

• 14:30Speaker 02

  Yeah, it's interesting what you were saying about choosing projects that are orthogonal to kind of the innovation of some of the big model players. A lot of startups kind of.

• 14:40Speaker 01

  Yeah, I was actually thinking about the same thing. It's not specific to academia.

• 14:44Speaker 02

  Yeah, they take a similar approach of like, okay, we want GPT five, six, seven to reinforce the advantage that we have.

• 14:49Speaker 01

  Exactly. Yeah, if a startup is betting against OpenAI's new model.

• 14:53Speaker 02

  Yeah, I think Sam also said this, that everyone's always like, what startup should I invest in? He's like, you should, the startup should be rooting for the models to get better. Yeah, he did. Is there something kind of that you, you've obviously been in the AI world for years now? Is there anything that you've seen that kind of has completely changed in your mind and with regards to AI in the last few years?

• 15:16Speaker 01

  So I think one thing that I've gotten.

• 15:21Speaker 01

  To appreciate more is how to think more holistically about AI's role. I think a lot of AI researchers, because they come from kind of this more technical background, think of the model as the central object, right? And if you think, for example, in the context of AI safety, you know, AI safety, I think, often gets construed as, here is a model, we're gonna like make it safe by making sure that doesn't do, you know, bad things, it doesn't respond to harmful responses, harmful requests. And I think that's sort of missing a broader picture, which is that

• 16:02Speaker 01

  there's the model, but that's one, you know, small piece of a larger ecosystem of, you know, actors with different incentives doing things in the world. And what you're trying to do when you think about safety is make sure that the whole system is quote-unquote "safe", not the particular model. And I think this leads to a bunch of other interventions besides just, you know, R O A F, the model to death, that I think are much more, you know, sensible. For, for example, a lot of bad actors can, I think, circumvent the safety by, you know, decomposing their, their problem. For example, if I wanted to send you a phishing email, I wouldn't go to, you know, a model and say like,

• 16:48Speaker 01

  I want to write a, you know, phishing email. Yeah. And, and declare my intent and, and I would write a personalized email. I would say something like, you know, this is, uh, the top 10 research papers you should read that really are, you know, going to be big. And then I go in and change all the links to whatever I want, right? So, so I think it's actually harder than people kind of give credit for to, you know, police, um, some of these, these cases of misuse.

• 17:22Speaker 01

  And the other thing I think is that, you know, is is defense, right? And I think, you know, I think there needs to be more invested in defense, just like um with any dual-use uh technology.

• 17:33Jacob Effron?

  Totally.

• 17:34Speaker 01

  Like, you know, email or the internet. Um it can be used for good, it can be used for ill. And uh we have, you know, anti, you know, spam filters, we have anti-fraud detectors. And these things are essential. Otherwise, you know, we have a complete mess. And I think the analog of that needs also develop. That these tools will be out there, people will always be able to use them. I think it's it's a losing battle to try to gate access to these models because they'll get cheaper and they're going to widespread. But that doesn't mean we give up and throw our hands like, oh, it's impossible. There are measures that we can take to secure it.

• 18:15Speaker 01

  So so I think thinking more holistically about how the whole ecosystem works as opposed to just focusing on what the model is doing. I think there's a limited amount of things you can do with the model.

• 18:27Jacob Effron?

  And obviously, some of those measures can be taken by, you know, uh the model providers themselves or net new startups. But, you know, I think it's a good segue to talk a little bit about regulation because, you know, I think people are asking, you know, what role uh should regulators have in in in, you know, controlling some of these use cases? And, you know, you've seen things like the EU AI Act and I know you've talked uh about, I mean, there's a bunch of different facets to this. Um I know you've talked a lot about, you know, there's obviously uh, you know, values encoded in models and, you know, uh. I I'm curious like how you kind of think about the regulatory landscape we have today and like what the ideal landscape to go to is.

• 19:00Speaker 01

  Yeah, this is a complex topic and I should also say that I'm not an expert.

• 19:04Jacob Effron?

  Sorry, we definitely are.

• 19:06Speaker 01

  So but I I will say that the stance I've always taken is that it is really early. And there's a lot that we don't understand. And um. You know, I sort of get the desire, we need to regulate, we don't want to, you know, have, you know, repeat social media or whatever. But I think there are things that you can do that are, you know, potentially, you know, bad, like S P N forty seven, for example. And, and just to say a bit more, I think I am very much pro regulation that emphasizes transparency and disclosure.

• 19:49Speaker 01

  Because I think the first step to gating something is to just understand, you know, the risks and also the benefits and basically what's, what's happening. And I think that currently, there is, uh, everything is so kind of closed off that it's hard to get a sense for, you know, policymakers or researchers or, you know, third-party auditors to really understand what the, um, what's, what's happening. So having, you know, more evaluations and, um, I think, you know, make a lot of sense. Um, I think, you know, the other piece of, you know, regulation is, you know, where should regulate, kind of upstream or downstream?

• 20:34Speaker 01

  Do you regulate foundation model developers or do you feel regulate, you know, the end, end products? And, and clearly, we have regulation for the various, you know, we have, you know, in finance, healthcare, et cetera, regulation already. And, and I think, um, in, in some sense, that's where the most effective, you can see kind of what the, the harms are. I think the problem with regulating, you know, to, uh, like in a heavy-handed upstream is that it's sort of, might be either ineffective or it's kind of, it's, it's a very blunt, um, you know, way to do things. But, but having, um, you know, transparency and obligations for allowing the downstream

• 21:17Speaker 01

  decision-makers to make sense of what's coming down the pipe, I think is, you know, totally reasonable, just like, you know, when we think about it as like, you know, nutrition labels, you know, for food and other, um, you know, uh, devices having spec sheets. And I think a lot of that is, is necessary so that downstream, um, you know, uh, product developers have an idea of what's, what's, what's happening.

• 21:42Jacob Effron?

  But I guess transitioning to some of your research, like I feel like one of the ones that I found most compelling was this, you know, generative agents work you did where I'm sure this is butchering it from a simplicity standpoint, but I thought of it as like you kind of created this Sims-like environment where basically, you know, all these agents are interacting with each other. It's like a fascinating way to study social dynamics. I mean, can you talk a bit about that work and kind of what's what's next for it?

• 22:03Speaker 01

  Yeah, yeah. So generative agents, wow, that came out last last year.

• 22:07Jacob Effron?

  Well, it's dog ears and AI.

• 22:08Speaker 01

  Yeah, dog ears and AI. So but but I think it's still directionally relevant. And the main thing we were interested in kind of going into this is work with my student June Park and Michael Bernstein. And and the, you know, from an AI perspective, I think one thing that's fascinating is, you know, we think about language models classically as generating text, right? But text is, you know, could be a document, could be a, you know, a piece of code or something. But what if you can, what other things can you generate? Can you generate, you know, essentially an agent or a society of agents? And that's sort of the the thing that sort of triggered our, you know, imagination, what what if

• 22:52Speaker 01

  you could do this? And and so we built this, the set of agents, each agent was powered by a language model with an intricate set of prompts and grounded in this kind of virtual environment where agents could actually move around and and talk to each other. And I think what that was a really fun project because it was pure kind of exploration.

• 23:15Jacob Effron?

  Yeah.

• 23:16Speaker 01

  Right? It was like, we don't know what's gonna happen. Let's build it and see and simulate and see. And it was kind of interesting that many of the phenomena that you see and your kind of, you know, social dynamics crop up, like, you know, information diffusion, where one person talks to another, one person was announced they were running for mayor and he, you know, talks and tries to convince other people and and so on. So I think that was really kind of, you can call it kind of emergent, you know, behavior. So I think going forward, one gap that needs to be filled is while generative agents was about creating believable simulations. So you look at it, it's like, looks good. But but I think if

• 24:00Speaker 01

  you could get simulations that were actually valid in the sense that they reflected something in reality. Then I think you unlock a lot of different new vistas. And while believability, I think, is sufficient for, let's say, games and, you know, that's gonna go

• 24:16Jacob Effron?

  Yeah.

• 24:17Speaker 01

  be its thing. I think a lot of the potential for having essentially a digital twin of society in some sense, you know, I think would be allow you to run, you know, policy experiments. Like, for example, let's say, you know, COVID mask policy. Let's do this and see what happens with agents. Or we're gonna create this law and see what happens. And, and of course, I, I think we're still a bit away from having, being able to trust these things. But if we, you know, bet on these models getting better, you know, in the, you know, in the future, we might be able to actually simulate many of the decisions that we would make.

• 24:58Speaker 01

  And, and also even just being able to do, you know, you know, social science studies where you have to recruit a, a set of participants. And often it's, you know, very slow and expensive.

• 25:14Jacob Effron?

  And always just on college kids.

• 25:16Speaker 01

  Yeah, college kids. So if you could get them on a much more demographically diverse, you know, set of, build a demographically diverse set of agents, you could actually also do things. One thing I really like is that you can give someone both a treatment and a control.

• 25:31Jacob Effron?

  Yeah.

• 25:31Speaker 01

  Which you can't do in.

• 25:33Jacob Effron?

  Cause they have the same kind of underlying model and then you say, but this model.

• 25:37Speaker 01

  Well, you just give them the one A and then you reset. You wipe their memory and like give them B. And so you get really kind of a clean kind of control. Obviously, this is, you know, just in simulation.

• 25:50Jacob Effron?

  Yeah.

• 25:50Speaker 01

  But I think you can use this as a sort of a first pass to, you know, get some ideas and then, you know, then do the actual studies.

• 25:59Jacob Effron?

  It's so interesting. They're actually, I mean, even for, for clinical trials right now in medicine, they're trying to do this digital twin tech work where it's like, you give someone the treatment, but then kind of simulate what they, what would have happened to them if they'd stayed on the control.

• 26:11Speaker 01

  Yeah.

• 26:12Jacob Effron?

  What, what an awesome, I think that's fascinating set of applications. You could imagine also like org design within a company and like simulating a bunch of ways and seeing what happens.

• 26:22Speaker 01

  Yeah, so I think there's two types of agents. I think both are fascinating. The first agent is more what we talked about earlier, agents like O1 that are able to perform really difficult tasks. And the other type of agent is more like about simulation. So it's not about performing a particular task per se, but kind of mimicking what humans are, a human is doing or individual is doing.

• 26:52Speaker 01

  And I think the latter is maybe less well-studied, but I think it has a whole host of new kind of, you know, potential applications that we're not really looking at right now.

• 27:05Speaker 02

  Totally. What makes kind of that type of simulation so different than the type of simulation that people have been doing in the past? I guess, you know, people would bottle diseases or things like that, but I guess that wasn't really in the context of each agent has their own kind of decision-making and model of the world, and then they're all going off and interacting with each other.

• 27:26Speaker 01

  So clearly simulations have been around for a long time. Often it's, you know, if you think about like physical simulations, weather simulations. Those are governed by, you know, physics or some equation or like generative agent-based modeling is a, you know, area where people create very stylized and simplistic models. You have to simplify. The system has to be simple. But for the first time, because we have these models, you can actually simulate something in much greater detail than was ever, you know, possible. So that's a sort of thing that is, you know, potentially a unlock.

• 28:06Speaker 02

  Yeah.

• 28:07Jacob Effron?

  Do you think before we make like major life decisions in the future, we'll run these simulations and like see how they play out?

• 28:12Speaker 01

  I think, I think so.

• 28:13Speaker 02

  Hopefully we can do that with potential investments.

• 28:15Speaker 01

  Yeah. I mean, yeah, yeah, there's an application for you. Killer app, right there. I think it makes a lot of sense. I mean, I think even now, right, I think you could simulate, let's say you're going to, you know, go...

• 28:30Speaker 01

  Um, you know, do a podcast, for example, I didn't do actually do this, but you could have, uh, you know, talk to, ask, uh, you know, a language model to simulate what the interview would, would say, or, uh, I'm sure people, you know, before they go on a date, they're, I don't know, use it as a practice or whatever. Um, and I, I think there is a lot of, yeah, potential, um, applications or benefits from being able to do, you know, accurate simulations. But, but I will say that, you know, we have to be very careful that, you know, the simulation currently is still, you know, quite, uh, far, or we don't know how close they are to the, the real thing. Yeah, yeah.

• 29:13Jacob Effron?

  Such interesting work. I mean, another, you know, switching gears to other parts of your work, I mean, you obviously spent a lot of time working on evaluations. We're talking about some of it earlier. Um, you know, I'd love just kind of maybe giving our listeners some context on like your thoughts on the state of evals today and then, you know, what happens as we get more and more of these, you know, obviously a lot of this has been built to benchmark L L Ms. And now we have more like agentic models. Are we going to need a whole new, you, you obviously talked about some of the capture the flag work you've done. Uh, where are we kind of headed?

• 29:39Speaker 01

  Yeah. I mean, yeah, evaluations is, is exciting, but also a huge mess right now. I think it used to be so much easier before, right? You have these data sets. There would be a train set and there are a test set and then you train and you test and you get some number and then you'd be, you'd be done. And, but the one big problem right now is the train tests overlap, right? We have no idea what's actually in the training data. And so, and they won't, you know, the companies won't tell us either. So there's always any evaluation that I run, I constantly wonder, okay, you know, how do I, how do I trust this? And even if they didn't train on it exactly, it's, you know, uh, you know, maybe

• 30:21Speaker 01

  it's with something similar. Was it like a lot of, you know, word problems of a certain type?

• 30:28Speaker 01

  And yeah, so so I think that's just like hanging over everyone's, you know, um, head. Um, that said, I think that um evaluation is uh is a constantly moving, you know, target. Because I think um existing as models get better, there's new um things that we want the models to do that are not captured by existing benchmarks. And so one thing I've been really excited about is how you can use language models themselves to, you know, benchmark language models, of course.

• 31:04Jacob Effron?

  Of course, well, at some point, they're gonna be so capable that we couldn't possibly come up with like the latest frontier.

• 31:09Speaker 01

  Yeah, so and in particular, you know, getting at uh this idea of coverage, right? So what's really interesting about these models is that they claim they can do anything. Like, ask me any instruction, we'll we'll follow it. And and I think so it's unlike, you know, classic benchmarking where you have just one task. And because it's so diverse, I think it's really nearly impossible for someone to sit down and it's like, okay, we're gonna think about all the tasks. And even even if you have, you know, user data, I think user data doesn't maybe stress some of the kind of the tails as much. But language models, you can get them to invent all sorts of uh of things.

• 31:51Speaker 01

  Uh, we had a paper called AutoVenture uh that uh looks at how you can basically generate um automatic, you know, inputs

• 32:02Speaker 01

  in a way that leverages some, you know, side information. So the basically the the uh language model that is generating the question has some piece of information that the the test taker doesn't know. So you can get use this asymmetry to get um more sensible evaluation. Otherwise, you're just like, you know, evaluate. So we self evaluation, I think, is still is not, you know, trivial, but I think you want this extra layer of kind of protection. And uh another thing I've been thinking about in terms of evaluation is

• 32:37Speaker 01

  I feel like a lot of evaluation right now kind of relies on fairly superficial judgments. You have A and B, and A is a long piece of text and B is a long piece of text. And you just say, oh, B is better than A. But that feels so unsatisfying. And what I like to do is think more, um, um, I guess, having taught for a while. You know, when we grade exams, we have idea of a rubric, right? So that we provide a standard for, you know, evaluation where any kind of student response

• 33:19Speaker 01

  or I guess in this case, LM response is subject to the rubric. So that anchors evaluation in sort of more, uh, you know, concrete terms.

• 33:31Jacob Effron?

  Yeah.

• 33:31Speaker 01

  As opposed to, you know, is this good or not? Well, tell me what, why I should be looking for. Yeah, so, but, but overall, I think that evaluation landscape will, I think, need to evolve quite a bit beyond kind of these standardized benchmarks, which everyone says like, oh yeah, we don't trust the benchmarks or whatever, but, you know, we should do something with it.

• 33:55Jacob Effron?

  Yeah, and it makes sense that obviously, you know, sitting in academia is actually a great place to be to develop like an objective set of benchmarks that, uh, that work for the industry. And then I think we'll also see, you know, a ton of like vertical specific benchmarks where like we don't care if our model is amazing at like, you know, the cutting edge math problems, just like can it diagnose medical injury or do whatever.

• 34:14Speaker 01

  So one thing that has happened, so two years ago, we put out HLM, Holistic Evaluation Language Model, which is, you know, trying to cover all the different aspects kind of manually. And what has happened is that HLM has become more of a framework and we've looked at different types of.

• 34:33Speaker 01

  You know, you could say verticals that, for example, you know, we we have a benchmark that looks at, you know, safety evaluations. We have another, you know, helm leaderboard for, you know, a tie, another one for, you know, so there's different languages that you might be interested. There's we're doing one for medical and one for finance. So so I do think that it makes sense that, you know, in any given vertical, there's actually quite a bit to do and developing something that's more tailored towards that setting makes a lot of sense.

• 35:09Jacob Effron?

  And speaking of verticals, I mean, some of these, you know, especially regulated industries, you know, I think a problem they have with with L L M's is just how black boxed it all is. And I know you've worked on interpretability before, and I can't tell whether interpretability at this point is like a a a lost cause because like these models are so complex and we have, you know, so little knowledge of what's going on in them. Or like, how do you think that evolves for, you know, as these models make diagnoses or credit decisions, like the ability to kind of understand?

• 35:33Speaker 01

  Yeah, yeah, yeah, there's a lot of impact here. And I feel like the problem has is exacerbated. So when I started working on tribability back in, you know, 2017, this at least you got the model weights and you've got the training data, you can look at that. And even then, it was hard to understand why is this model making this prediction because deep learning is, you know, is difficult to work with. And now on top of that, we don't often with the models, we don't have the the weights, we don't have the data. So it adds another layer of sort of obfuscation as if it weren't hard enough already. So

• 36:12Speaker 01

  so that's kind of the world we're living in. In terms of what interpretability, you know, means, I think there's a school of thought or a body of work around mechanistic interpretability, which tries to really understand the individual neurons inside a network and make sense of what's what's happening. I find that, you know, I think it's very interesting when you discover, ah, this neuron is responsible for, you know, this particular style. It's more, I think, about, you know, sci- I think interpretability has two audiences. One is about kind of just scientific understanding. You're just curious what is going on.

• 36:55Speaker 01

  And the other is, like you said, in, you know, regular industries, if it makes a decision and you want to know why it's making a decision, or your developing model and you just care like to und debug and fix the problem. I think there is other things. So for example, in 2017, we worked on influence functions as a way to attribute why a model makes a prediction in terms of other training examples. And this work has, you know, been adapted to language models, but it's very difficult, you know, to scale.

• 37:30Speaker 02

  That's like, I made this decision because I saw these 10 examples?

• 37:34Speaker 01

  Yeah, like, which basically the idea is which training examples were most influential for making this prediction. And the thought experiment is if I had an example and I removed it from the training set, would I still make the prediction? And if it wouldn't, then that says this example was influential. If it basically didn't change anything, then this example didn't, doesn't matter. I mean, there's problems with how you, you know, both compute this and how you interpret this, but that is one flavor of interpretability that I think is useful. Now, of course, if your training data is private, then I think it's, you know,

• 38:15Speaker 01

  you don't want to have a, you know, oh, you got diagnosed with a disease because this guy over here got.

• 38:21Speaker 02

  Yeah, because of this random Reddit thread.

• 38:23Speaker 01

  Yeah, yeah.

• 38:26Speaker 01

  So I think it's difficult. And another tack on interpretability is, you know, explanations. So you can do chain of thought and it generates some explanations, but all this work that shows explanations are not maybe exactly what actually is happening.

• 38:44Speaker 02

  Kind of like people.

• 38:45Speaker 01

  Yeah, kind of like people. So at least I think maybe with the agent architecture explanations, you can think about it as at least a bottleneck for if you have things which are more modular than, and all they get transmitted is explanations, then at least you know kind of what.

• 39:04Speaker 01

  How the pieces are moving.

• 39:07Speaker 02

  So when there's open models that do similar things that one, you can see the steps, you might have an ability to explain how something came about.

• 39:14Speaker 01

  Yeah, but I think where I'm at with interpretability is in order to even do interpretability, we need to have the weights and the training data. So you can get back to like 2017 era level of access. And then you can answer a lot of these questions.

• 39:31Speaker 02

  Maybe one other question on the research side before we move on together and all the other amazing things that you've been up to. And this is more just because I'm very curious. There's been kind of transformer and then Mamba and then these different model architectures. One, I guess, how enduring do you think those will be? And then two, when new model architectures come out, is that due to breakthroughs in math or experimentation or like where did those come about? I can think back to, okay, there was like the vanishing gradient problem and then people were like, oh, we need to get actually more data back to these other parameters. And so then we're going to make some new adjustments for that.

• 40:12Speaker 02

  But I don't know, is it like math or experimentation or how are people coming up with these new model architectures?

• 40:19Speaker 01

  Yeah, so historically, a lot of the architectures like LSTMs and KNNets and transformers were intuition. You train and you think about the gradients and how they should work and some experimentation. I think what's interesting maybe about the Mamba or state space models line of work is that it came actually from math. It was Albert and Chris Wray and some people trying to answer the question of if you have a sequence of points, how do you fit a kind of online update on the best polynomial fit? So it was kind of just math, they solved the problem and then they figured out how

• 41:01Speaker 01

  to get it to actually work at scale and inside a neural net, which is kind of interesting because even though the math doesn't apply to the actual model, but it was a source of inspiration for the actual architecture.

• 41:21Speaker 01

  In terms of uh where all these architectures uh go, um I don't have terribly strong opinions about this. I don't really do architecture, you know, uh stuff. Um but it is, I would, I guess, bet on like, you know, transformers and all these things not being the one that we kind of continue, you know, using in the, in the, in the future. Just because, you know, it's, it's sort of arbitrary and you can, you know, uh, change it. Um whether there'll be super large changes, I think that remains to be seen.

• 42:02Speaker 01

  I think that maybe more interesting um is if you have other kind of regimes, for example, video.

• 42:10Jacob Effron?

  Yeah.

• 42:11Speaker 01

  You know, I think that's where maybe you'll more likely see kind of innovations there because like you can't just do the naive uh thing of doing a giant, you know, uh transformer. Um or if you have, let's say, um now we're kind of in this more uh agent searchy, you know, setting. Maybe there's other architectures that are more amenable and fit in, um the, um, the fit better in the mold of a kind of a larger system.

• 42:42Jacob Effron?

  Yeah.

• 42:43Speaker 01

  But I, I think structurally, something has to change for like something really new to come out. Like, I mean, for example, in vision, if you just wanna uh do very simple uh image classification, like confidence are, you know, totally fine. And so transformers wouldn't have, I think, been invented um unless like people really tackle like, you know, machine translation, which is a much different problem. So I think if you look at uh, you know, maybe different types of problems um are a good way to look for kind of where architectural innovations will happen.

• 43:17Jacob Effron?

  Super interesting. I mean, I guess, you know, one other hat you wear, obviously, is you're the co-founder of Together.

• 43:21Speaker 01

  Yeah.

• 43:22Jacob Effron?

  And one thing I'm struck by is obviously this kind of new O one paradigm of test time, you know, uh, you know, all kind of trying a lot of these different scenarios. Um seems like it has big implications for the inference market, right? It's like different than this current uh setting. And so what impact do you think that has? It's like, are a lot of these breakthroughs like flash tension and whatnot still relevant for this new class of models? Or like, what, what happens to the inference market with this uh?

• 43:44Speaker 01

  Yeah, totally. I mean, I feel like I would see inference as a sort of a Very low level primitive that you just need to get right. You make it robust and you make it cheap. And that's a lot of what Together focuses on right now. Because, I mean, in some sense, everything needs inference. Like if you, um, you know, train a model, you need forward passes inference. If you are doing like any sort of agentic flow, you know, this is you're doing inference. If you're generating synthetic data to, you know, further tune the model, you're doing inference. So, so I think it's more of a, a building block that needs to be, you know, stable.

• 44:24Speaker 01

  But of course, I think over time, um, you know, you think about what inference for what? And then you start maybe operating on a different abstraction. For, for example, a lot of, uh, the inference market right now is serving, you know, Lama three. Yeah, essentially. And, um, but why are we serving Lama three? Like, I mean, because it's a good model, but you know, why, why Lama three? Why not a different model? And, and I think, uh, that what, you know, I think we'll see is, um, people wanting to serve models that basically what do people care about? Do people care about the models work for whatever their use case is? They don't care if it's Lama three.

• 45:05Speaker 01

  Lama three just gives a sort of a stamp of approval that this is like not a total sucky thing. So, uh, but over time, I think what's interesting is that if you can adapt the, the model to a particular use case, you can potentially get way much, uh, you know, faster and you know, better performance. Cool. Um, you know, over time. And, and also I think the sort of agentic workflows and all that open up opportunities for maybe further optimization. For example, if you have the high throughput setting, if you want to, you know, just generate a bunch of huge number of possibilities and, you know, do,

• 45:46Speaker 01

  then that is something that you could kind of optimize for. So the inference, I guess optimizations could. Definitely take into account like what's happening sort of at the level.

• 45:56Jacob Effron?

  I just think together perspective, you think of like inference optimization as this prerequisite to then do all this really interesting work around.

• 46:03Speaker 01

  Right, yeah. So together, the story is like you, you need GPUs.

• 46:07Jacob Effron?

  Yeah.

• 46:08Speaker 01

  That's the lowest level and then you need inference and then you, you have the ability to like fine tune and what we think of as

• 46:17Speaker 01

  basically kind of customizing models because I think that's it's, I think generally pretty hard to take like say Lama three, four or five B and say, I want to make it 10 times faster and it's still going to do everything that the original model. But if you say, okay, take this one, you know, I don't know, whatever, call center application and you're going to do it. A lot of times I think there's much more room for kind of squeezing out, you know, performance.

• 46:46Jacob Effron?

  You alluded to this earlier that we're kind of like, it feels like we need some sort of algorithmic breakthrough or change from transformers to get to like the next step of, of reasoning or, you know. I guess I'm curious, like, you know, what milestones are meaningful to you? Like, you know, people talk about reasoning as like this, this kind of broad term. And what are like the meaningful milestones to you? And like, you know, I guess to, to ask a shameless question, like, how far away do you think we are if you're predicting from, from some of these milestones?

• 47:11Speaker 01

  Well, I think just starting, let's say with the, the benchmarks that we have, like Cybench for cybersecurity or ML Agent Bench for, you know, solving ML research tasks. I think these are still, you know, good trackers of performance because performance is, you know, well below kind of what it, what it could be.

• 47:34Speaker 01

  I think that, you know, so, so that's one way to kind of track progress in the, in the near term. I think that, and of course we'll continue developing, you know, benchmarks over time. But if you want to kind of go, you know, farther out, you know, solving some kind of really open, let's say solving open math problem. We're doing something that is really, you know, beyond what, you know, humans, like basically anything that extends.

• 48:07Speaker 01

  Human knowledge, I think is really, really interesting. Because a lot of what AI has been about is catching up with with humans in some sense, right? Like you're mimicking humans and I think we're sort of getting past that point where now we're mimicking like expert. You know, humans, but all this stuff like solving I M O problems, well, people have already done. Well, clearly people have done that because people made the problem. Um, but if you can actually create something that's really, you know, discover something new, like create new research or solve something that hasn't been solved, then I feel like, or or even in the cyber city case, like, you know, find a zero day, then I think that that's kind of a, you know, game changer.

• 48:51Jacob Effron?

  And do you feel like we're in like, I mean, I feel like everyone's asking, are we like getting to some plateau of model capabilities? Or does it feel like we're still on this crazy exponential?

• 48:58Speaker 01

  I think things are still moving up like pretty quickly. I think locally it's always feels like maybe things are, you know, locally, everything's linear in some sense, but um. I think, I think it's maybe going to be more than just like naive. So scaling like, oh yeah, of course models will get bigger, more data and whatever. But I think there will be qualitative changes as well, like for example, you know, different. Like oh, one kind of represents a little bit of a different way that you might think about using these, you know, systems. And of course, on top of that, you know, you know, chips are going to get more powerful and cheaper. So that's going to still kind of drive the whole.

• 49:43Jacob Effron?

  Yeah. I mean, another category of models you spend time in is, you know, you worked on open source models for robotics. And I think, you know, there are a lot of varying opinions on where we are in like robotics foundation models. And, you know, if we're close to some sort of chat G P T moment in that space, uh, you know, what do you, what do you think?

• 49:59Speaker 01

  Yeah, I think we're definitely not at a chat GPT moment for robotics today. I think we are closer to a kind of BERT era.

• 50:08Jacob Effron?

  Yeah.

• 50:08Speaker 01

  Moment for robotics where the idea of using vision language models, fine tuning them and getting four foundation models for robotics is, you know, is, uh, is effective.

• 50:24Speaker 01

  But you still have to fine tune them and they're fine tuned for neural tasks and the resulting policies are still like very brittle in ways that, you know, that it's not the case in, in language.

• 50:36Jacob Effron?

  Yeah.

• 50:37Speaker 01

  So I think it'll be a few more years. I think the, the good news is that there's a lot more kind of interest and, you know, I guess funding and everything for this and, you know, there's a bunch of other data collection efforts. So I think things are moving, but, you know, I think hardware is hard. So it will maybe take, I mean, some amount of time. But, but I'm optimistic that things will, I don't see any fundamental obstacles like we can't, get to a chat GPT moment for robotics.

• 51:12Jacob Effron?

  It's just a, yeah, there's no, there's no internet of data available to, uh.

• 51:16Speaker 01

  Yeah, you just have to do the hard work. You have to do it the hard way.

• 51:19Jacob Effron?

  Yes. And I think it's still, you know, you're betting on emerging capabilities that until they emerge, it's hard to know when and how they might.

• 51:25Speaker 01

  Yeah. I guess one thing that's maybe good is that we can lean on language and vision to basically

• 51:36Speaker 01

  sort of driving all the architecture and data and kind of the, the compute recipes. So that when we have the right, you know, data and for problem formulation for robotics, like we have infrastructure that we can just, you know, borrow. And furthermore, I think that a lot, I'm, my hope is that a lot of the so-called robotics problems are actually just, you know, language and vision problems. So if you can factor out a lot of that stuff, then you can potentially make robotics a lot more, you know, efficient.

• 52:08Jacob Effron?

  Totally.

• 52:09Speaker 01

  Like, you shouldn't need robotics data to learn like what a cup is.

• 52:12Jacob Effron?

  Yeah.

• 52:13Speaker 01

  You need robotic data to figure out how to like manipulate and grasp a cup, but, you know, just identifying a lot of the high-level stuff like that should all be language and, you know, videos.

• 52:23Jacob Effron?

  Yeah. Um, you know, one thing I think that's interesting about your background is I understand you're, you're quite a talented, you know, classical musician, and I guess there's been a lot of really interesting developments in like the AI music world, you know, Suno and EDO and all these companies. Um, like, as someone who is really passionate about music, like, where do you think that space goes?

• 52:42Speaker 01

  Yeah, that's interesting. I think, um, you know, certainly the same recipe, you know, trying to generate transform on data is, is effective there. I think there are a few things that are challenging for, um, for music. One is, um, I think copyright is a kind of a big, much bigger hurdle there. So, you know, that, that, that's a whole can of worms. Um, the other thing is, I think, you know, thinking about, you know, control. And so I had a postdoc in my lab, John Thixton, who worked on foundation models for, for music. And the emphasis there was on, um, models that you can actually, you know, control.

• 53:27Speaker 01

  We don't want to just have unconditional generation or even just textual generation and having it just generate the music. Felt like it wasn't giving, um, you know, artists enough control or like what, what was happening.

• 53:43Speaker 01

  And so we basically built this sort of generalized infiling model where you could, it's called anticipatory music transformer. So you can condition on any subset of the musical events in a score. You can condition on a melody, generate the harmony. You can condition on like, you know, one part and fill in sort of a section. Cool. And, and I think one direction that this could go is, is basically like a co-pilot for, you know, musicians to basically, you know, like composers or artists to create their, their works, uh, just like, you know, we, I think people use GitHub co-pilot. I think that's one, um, area that I think is, you know, pretty, you know, interesting.

• 54:27Speaker 01

  And, um, and yeah, in terms of, um, you know, how it intersects with my, Yeah. you know, personal, um, musical aspirations.

• 54:40Speaker 01

  I think that, you know, and many years ago, I was, you know, trying really hard to, um, compete in these piano competitions, but I was doing this while doing a PhD. So I, uh, it was like, I don't have enough time to practice. But I had these musical i- ideas on like, I, I just wanted to sound like this. I know what I want, but my fingers won't do it. And, and so at a personal level, I'd be really excited of, you know, how can these tools really help me realize my musical vision?

• 55:13Speaker 01

  And, you know, I, I, it's not like I don't want, I don't want to be lazy and don't not practice, but just that it's sort of a augmentation to, and, and I think, um, I'm not sure what that would look like exactly, but, but I think, um, because I play, you know, classical music, it's certainly very far away from like just pushing a button and having, you know, just some music, uh, come out. There's a lot of subtleties and also classical music, there isn't that much data for. Yeah. So I think there's going to be some challenges there.

• 55:43Jacob Effron?

  Yeah, it's super. I imagine these like AI piano teachers in the future too. I know everyone always has complex relationships with their initial

• 55:50Jacob Effron?

  That'll be, that'll be fun.

• 55:51Speaker 01

  Yeah, I mean, I think in general, like AI, you know, on the point of teachers, they can be extremely good, like teachers, coaches.

• 56:00Speaker 02

  Totally.

• 56:02Speaker 01

  The use in education, I'm very kind of, you know, bullish or we're just excited about.

• 56:07Jacob Effron?

  Yeah, it's really cool when you see even those GPT four demos of like, you know, drawing a math problem and then having it explained.

• 56:12Speaker 02

  Yeah, those are really impressive.

• 56:14Jacob Effron?

  Cool.

• 56:14Speaker 01

  Yeah, yeah. I use it for, you know, teaching my kids like all the time. How do I explain this concept, which I know and now I need to explain it to a five year old.

• 56:23Speaker 02

  It's so good at that. And yeah, just breaking down complicated things and making them simple. It's, uh, it's very impressive.

• 56:30Jacob Effron?

  Yeah, well, we always like to enter interviews with a quick fire round where we get your take on it on a few things. And so maybe to kick it off, what's one thing that's overhyped and underhyped in AI today?

• 56:40Speaker 01

  Agents and agents.

• 56:43Jacob Effron?

  Yeah, I thought we've gone through a full hype cycle and then some with, uh, with agents.

• 56:47Speaker 02

  Agents, they've been on both sides of the question.

• 56:51Jacob Effron?

  You've written about like, you know, ML agents doing their own ML research before. Uh, I'm curious, like, obviously people always talk about this as some hypothetical step in the path AGI. How far away from that do you think we are?

• 57:03Speaker 01

  From AGI or?

• 57:04Jacob Effron?

  No, sorry, from ML agents, like contributing novel, you know, uh, novel insights into ML work.

• 57:10Speaker 01

  Yeah, depends on how you think about what novel is, but I don't think we're very far from, to the extent that if you run and I know this is supposed to be a quick round.

• 57:20Speaker 02

  I show a shamelessly broad question. We always want to sneak in all of our. I was laughing in my head because I was like, did you just ask about AGI fire round?

• 57:30Speaker 01

  Yeah, usually it's like a yes or no question. But yeah, so depends on what you mean by novel knowledge, but to the extent that having an agent that runs an experiment that fills in some sort of like runs a relation experiment to answer some question, I think we can do that already. And now the, the question is how, you know, maybe the, you can think about it at the level of, you know, a very junior student and then. How can the model actually come up with maybe experiments, new experiments to run and then take on, maybe bring so many new directions? Um, you know, I'm optimistic that you can actually do it and, you know, do something and meaningfully in the next, you know, years.

• 58:13Speaker 01

  I mean, just if you think about coding, right? Certainly, uh, you know, it has created new code and helped people code and in them, you know, that's, that's maybe a kind of simpler version. So I think the same thing will happen to research. I don't think it's a, you know, it's that different.

• 58:33Speaker 02

  Totally.

• 58:34Jacob Effron?

  What application areas do you feel like are underexplored right now on top of these models?

• 58:38Speaker 01

  I think classically, a lot of AI applications driven by, you know, commercial needs, right? Everyone wants their rag solution to question answering and, you know, summarization and so on. Uh, but potentially, you know, more kind of fundamental science and scientific discovery, improving, you know, researcher, um, you know, productivity. I think these are, um, you know, obviously not as well studied. Um, but I think are important because they can, um, feed into the whole, you know, cycle of improving the whole ecosystem.

• 59:15Jacob Effron?

  Pat, that was a fun one.

• 59:16Speaker 02

  Super fun.

• 59:17Jacob Effron?

  So do I understand correctly that you took Percy's class?

• 59:20Speaker 02

  I did take Percy's class. I took CS two twenty one, uh, which was kind of the first AI class, um, that is, uh, that is offered. So yeah, I took his class back in the day.

• 59:30Jacob Effron?

  I feel like even just from that recording, I can tell he was probably a very good professor.

• 59:34Speaker 02

  He is a very good teacher. Um, yeah, and the class is very well done.

• 59:38Jacob Effron?

  Yeah. Um, no, it's awesome.

• 59:40Speaker 02

  What, what were the takeaways for you or what stood out to you?

• 59:43Jacob Effron?

  I love this work that he did around generative agents and this kind of like Sims world that he created and these, you know, when you throw a bunch of agents at each other, these things that emerge of what they start doing and how they start interacting with each other.

• 59:54Speaker 02

  Totally.

• 59:55Jacob Effron?

  And I loved this idea that like, you know, in the future we'll have a bunch of, you know, agent twins that you can simulate, you know, a policy or even, uh, you know, I He talked about dating and I was thinking about that Black Mirror episode where like you simulate a bunch of dates with a ton of people and figure out like who's the right person.

• 1:00:12Speaker 02

  Totally.

• 1:00:12Jacob Effron?

  You could imagine instead of dating apps, you just like send your your digital.

• 1:00:15Speaker 02

  Send your agent to go do the work for you.

• 1:00:17Jacob Effron?

  Yeah.

• 1:00:17Speaker 02

  Yeah.

• 1:00:17Jacob Effron?

  You're like, that looks like it was a fun date with that person. But that one, I don't know if I if I want to go on.

• 1:00:21Speaker 02

  Yeah, that his that whole take on simulation was really interesting of just what that could mean for the future. If you could actually simulate how, you know, humans or agents would speak or communicate or behave in in these different worlds, I thought was was really fascinating.

• 1:00:38Jacob Effron?

  Yeah, I thought there's always been this like dream of, you know, central planning and like government figuring out like how to, you know, have every lever move in like the economy and it never works. And then you're like, well, I wonder if if these agents get better and better, whether at least government will be more informed in in being able to to project how things might go.

• 1:00:56Speaker 02

  I thought it was really interesting what he was saying about having academic research be orthogonal to progress that the model vendors are making. And we talked about this in in the episode being similar to to startups, but I just thought it was interesting kind of how he was approaching that from a practical standpoint. When you think about how many more resources a Google or OpenAI or at least compute resources may have than than Stanford and and some of the other academic institutions. So figuring out where they can still push the envelope and that kind of dovetailed into one of the things that he mentioned around

• 1:01:34Speaker 02

  AI research and thinking through the whole system versus just the model. And that's kind of changed over time as he's these models have gotten into the into the real world. And so I thought that was that was fascinating.

• 1:01:43Jacob Effron?

  Yeah, I love that. Like he's telling his students what like VCs tell founders, which is like, you know,

• 1:01:48Speaker 02

  Yeah.

• 1:01:48Jacob Effron?

  be in the path of progress of the models. Like you should want GPT five to come out and be really good because that then makes your work even better. And

• 1:01:55Speaker 02

  Totally.

• 1:01:56Jacob Effron?

  I think it's it's like for any type of work, whether it's academic or or company building or whatnot, you know, uh, being on the path of progress toward that, uh, I think ends up being super important.

• 1:02:05Speaker 02

  Totally.

• 1:02:06Jacob Effron?

  He just does so many different things. Like I felt like all the work he's done around evals, obviously, you know, I think Helen is is kind of one of these clear benchmarks. And now it feels like the capture the flag work he's doing as well. I love this idea that like we're getting to a point of model capabilities where we may just need like models to write evals for models.

• 1:02:23Speaker 02

  New evals, yeah.

• 1:02:24Jacob Effron?

  It's like you and I couldn't figure out how to like they're they're smarter than us, but.

• 1:02:28Speaker 02

  I mean, the capture the flag is a really cool example that's very easy to understand, and you can see how that would be a good test of model capability. Okay, there's some objective out there. How long does it take you to reach that objective? And so I thought that was interesting, too.

• 1:02:44Jacob Effron?

  Yeah. And then I think generally his takes on O1, which I feel like has taken the space by storm in the last week. You know, and obviously, I think his point that it's kind of like this first step of, you know, running much more compute at a difference and figuring out, you know, simulated a bunch of different steps and how we've come full circle with AlphaGo and a lot of that work. But, you know, I think it also obviously presents challenges to academia because up and at is hitting the steps, right?

• 1:03:11Speaker 02

  Yeah.

• 1:03:12Jacob Effron?

  And so what's actually going on behind the scenes is hard to study. It makes it hard for interpretability. And so, you know, I wonder when we'll see an open source equivalent. But, you know, I actually, he got me thinking that actually, you know, from an interpretability standpoint for some of these industries like healthcare or finance, like this actually might be a step in a good direction because at least the model is explaining step by step.

• 1:03:32Speaker 02

  What it's doing along the way.

• 1:03:33Jacob Effron?

  And that that actually could, you know, could be helpful rather than trying to identify the one in, you know, 70 billion neurons that's driving something.

• 1:03:41Speaker 02

  Yeah, totally. The, I thought his take on the transformer and model architecture was interesting, and we've heard this from other people, but kind of this belief in future model architectures. And he seemed to think that that would come from video or different domains that we're going to be pushing the existing architectures to break. I thought that was pretty interesting. I also thought it was cool that, maybe this is just the nerd in me coming out, but that Mamba was actually driven by a math breakthrough.

• 1:04:10Jacob Effron?

  Yeah.

• 1:04:11Speaker 02

  I thought that was fascinating that they were actually like sitting down and looking at the math of how these things work, and then that led to new.

• 1:04:18Jacob Effron?

  Did you understand the math he was saying? Because it completely.

• 1:04:20Speaker 02

  Definitely not.

• 1:04:21Jacob Effron?

  Yeah, okay. I was like, you had the masters from Stanford, you might have gotten it. For me, I was like, I'll just not along here.

• 1:04:27Speaker 02

  There is some important math, which it seemed like influenced the model architecture. But no, it's been a long time since I have done any real math. Although I appreciated his very subtle calculus hint, which was if. What did he say? When you're looking locally, everything looks linear?

• 1:04:45Jacob Effron?

  Yeah, exactly. That I got still. Still have that.

• 1:04:48Speaker 02

  There you go.

• 1:04:48Jacob Effron?

  Maybe we'll ask one of these really complex reasoning models to explain to us what in the world happened.

• 1:04:54Speaker 02

  Yeah, like we're a 12-year-old.

• 1:04:55Jacob Effron?

  Exactly. Younger for me.

• 1:04:57Speaker 02

  Yeah. What did you think of his music AI take? Or how he thinks about AI in the music world?

• 1:05:07Jacob Effron?

  I love this idea that there's like something, you know, in your head that you can't quite get out into the world. And that like, there's a way to communicate that to AI that enables you to.

• 1:05:17Speaker 02

  Yeah.

• 1:05:17Jacob Effron?

  And I think it's this co-pilot vision, rather than just one prompt and you have a song or a picture or something that like, there is something that you want to create and not all of us can practice six hours of piano a day to get there or go to art school or do whatever, but like, can you still express that creativity within you? I thought that was, that's pretty cool.

• 1:05:34Speaker 02

  Yeah, I mean, people like Percy may have beautiful songs in their head. I feel like I would need a lot more than AI to help me get the songs that are in my head down.

• 1:05:45Jacob Effron?

  Recreate and shake it off for the 17th time.

• 1:05:46Speaker 02

  Exactly. Why do you keep coming up with Taylor Swift songs? We brought up Taylor Swift.

• 1:05:51Jacob Effron?

  It's actually contractually obligated that in our debriefs, it comes up in one way or another.

• 1:05:55Speaker 02

  Yeah.

• 1:05:56Jacob Effron?

  Yeah, that was a really fun episode with Percy. Folks should definitely stay tuned next week. I know we have a pretty sweet lineup of guests coming up.