Pod Insights Unsupervised Learning with Jacob Effron Ep 11: Stanford Professor Tatsu Hashimoto on AI Biases and Improving LLM Performance About this episode Patrick and Jacob sit down with Tatsu Hashimoto, Professor of AI at Stanford, to discuss the incredible open source projects from his research group like Alpaca and AlpacaFarm, whether data, algorithms, fine-tuning or RLHF is most important for performance, if AI is liberal or conservative, and much more!
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Speaker 00 On today's episode, we speak with Tatsu Hashimoto. He is a Stanford professor and AI researcher behind some of the most exciting open source AI projects, including Alpaca and Alpaca Farm.
0:09 Speaker 01 Alpaca Farm was our attempt at building this simulator. That allows us to basically run the whole pipeline of going from a base language model to a fully human final language model. And we can understand the impact of data, the impact of reinforcement learning, and so on.
0:23 Speaker 00 We talk about how large language models contain biases and reflect people's opinions.
0:27 Speaker 01 When there is this instruction tuning in RLHF that happens, you start to see a shift towards higher educated, higher income, generally more liberal opinion viewpoints. And we thought that was kind of an interesting observation.
0:38 Speaker 00 And of course, we hypothesize about the future of AI. If you had to take a bet, given what you know now, where do you think this goes?
0:43 Speaker 01 I think it would be maybe similar to Golly-Goth. There will be a couple big LM providers. I think they will provide differentiated products. I don't necessarily think that there's going to be a single LM to rule them all.
0:55 Speaker 00 Thank you so much for joining us. I hope you enjoy this episode. Tatsu, thank you so much for joining us.
1:00 Speaker 01 Great, thank you very much.
1:02 Speaker 00 It'd be awesome. Can you tell us a little bit about your journey to Stanford and where your research is focused now?
1:08 Speaker 01 Yeah, so my original background is in statistics and actually computational biology during my graduate school at MIT. And there, I was really focused on trying to understand a lot of aspects of the cell. Like, you know, how does the genome, you know, shape us during development and so on? And so that entire time, I was really interested in sort of understanding deep, complex systems. And a lot of that is reflected in the research that I do today, which, you know, in a funny way, large language models are also these very large, complex systems that we are trying to understand and reverse engineer. And the works that you mentioned very much go in this vein of like, can we build tools, almost like informed by my background, thinking about model organisms?
1:53 Speaker 01 Like, can we build smaller models that display many of the behaviors of these large, complex systems that are changing the world today? So we can study and improve them. And so that's kind of my my original journey. I started out in statistics and computational biology and I was really interested in things like decoding the language of the cell as well as when I came here to Stanford for a postdoc, decoding sort of meaning in human language. And I got very much involved in language generation, pre-large language models. But then, you know, as sort of the field evolved, it became very clear that language generation, like generating text was going to be intimately tied to broader problems
2:34 Speaker 01 in machine learning and now everyone calls it AI. And so that's the the area I operate in now. I mean, it's sort of this organic transition from language generation to to large language models.
2:45 Speaker 00 Awesome. Yeah, it's really interesting what you mentioned about kind of building smaller systems that model the behavior of bigger ones. And I feel like that's been one of the things that really captured everyone's attention about Alpaca was just, you know, that that model, even though it was smaller and more lightweight and kind of tuned on less data, had some incredible performance. Maybe can you talk a little bit about kind of the origins of that project and and, you know, what you've learned since then?
3:16 Speaker 01 Yeah, so the the origins of the project, I guess we'll talk about Alpaca Farm a bit later. But Alpaca Farm was originally the the high level thing we were interested in, essentially replicating, at least in like the sandbox, the entire training process for a large language model. So we could tweak all the different pieces and then understand how, you know, which parts are important, which parts are part of improvement or can be improved. And as part of that, we were trying to do the very first step of building an instruction following model. And we were sort of fortuitous that, you know, Llama sort of released at that time, at least to us academics. And the we basically were
3:58 Speaker 01 were preparing sort of the training data for it by doing this sort of synthetic data generation from Text Avicii 3. And so we got all the pieces in place right when Llama was released because we were already trying to do This project of understanding large language models. Our original plan was actually to do the whole thing with OPT and Flanty five. But what we actually found was, you know, probably like yourselves and everyone else, if you do this with Llama, the behavior feels qualitatively very different. And when, you know, Llama was released to us, it was actually very much a side thread. One of my students was like, oh, I'll go, you know, get the permission to use Llama. I'll train it and we'll just play with it a little bit. And then we played with it a little bit and then we found, wow, this thing is actually
4:41 Speaker 01 really good, or at least, you know, it behaves in a very different way from our interactions. And so we really felt like releasing it and telling everyone else in the research community about it was going to be something that would be appreciated. And so we went and, you know, did the blog post and the demo and all of that because it felt pretty different from a lot of the baselines that we were working with.
5:03 Speaker 00 What made it different as a baseline?
5:05 Speaker 01 Yeah, I mean, I think the experience that I think most people have had playing with like the Llama fine tune models and everything else is that the Llama base model is just in some ways a lot better. It feels like because of maybe the coverage of the data, pre-training data that they have, or the, you know, they explicitly do this non-Chin-Chill optimal training regime where they train a small model for much longer. So many of those pieces combined, I think, to make the base model a lot stronger. And then so when we combine that with a reasonable instruction tuning data set, you get nice behaviors back out. And of course, there's been, you know, much better models since and we know that, you
5:46 Speaker 01 know, a small amount of instruct tuning data doesn't get us all the way to GPT four. But I think there is really this, this sort of gap between sort of a lot of the prior base models that came and then the Llama based models and then now, you know, of course, the next generation of models that are hopefully soon going to be coming out.
6:04 Jacob Effron? Yeah, and I think it would be super interesting, like, you know, maybe for our listeners, just to contextualize, you know, what is the quality of performance today from, you know, models, like the Alpaca model? I think, obviously, I feel like in the months since you've released it, there's been a lot of folks interrogating, you know, what can these models do? What can't these models do? You know, you've had publications out of places like Berkeley where they're like, look, you know, maybe you can mimic in style, but not in substance. Like, where would you kind of categorize the state of performance of these models today?
6:30 Speaker 01 Yeah, I would say that the models are slowly but surely sort of creeping up towards the ChatGPT performance. I think the real challenge here is evaluating these systems is very complex. One thing that I think is very much underappreciated by, I know, maybe everyone that's not a academic carefully studying these systems, is that ChatGPT and GPT-4's like, sort of, domain on which they have coverage, I think is much wider than what a lot of the open source people have been targeting. And sort of also, a lot of these systems are targeting not just conversational settings, where you just, you know, a person just comes in and types in random stuff, but also a
7:11 Speaker 01 lot of the benchmark tasks, like knowledge-intensive tasks, like, you know, like trivia QA-style things, where you have a one-word response to a question. And because, you know, OpenAI probably has a much wider tail coverage, like these kinds of disparate domains than the open source, evaluation is very challenging. If you have a conversation with, say, the leading open source systems that exist today, you might think that it's very close to ChatGPT. If you start going out in the tails and saying, oh, can we productionize a sentiment classifier using these systems? You might find that there's a very big gap, right? And so, you know, that was a long answer to your question, but I actually think, you know, if we're thinking about the head of the distribution in terms of chat applications, a lot of the open source systems are getting close.
7:55 Speaker 01 You know, they're, if we think about fine-tuning, fine-tuning's really good at the locations where you're giving it data. Everyone is generating data for conversational applications. That's getting increasingly good. All of the tail stuff, like, you know, is it good at answering very detailed questions? Is it good at writing the very specific kinds of essays or limericks or something? Those kinds of things may be, you know, going to require a lot more effort. And so I think there is sort of definitely a kernel of truth to both the open source people who are pushing on just like, let's fine tune harder and more aggressively and more efficiently, as well as sort of the imitation learning paper, which is saying like, you can't go all the way with fine tunes.
8:36 Speaker 01 I think the reality is probably you can push fine tunes quite far, but there's strong diminishing returns and people need to be prepared to pay essentially large compute costs and data costs to go the rest of the way if they want to keep pushing that agenda.
8:50 Jacob Effron? Super interesting. And so, so do you think like, as you think about, you know, the, the next, you know, version of these higher quality, lower cost to train models, you know, is there a limit to how much you can kind of push through the reinforcement learning mechanism that you guys did for Alpaca and kind of outlined in Alpaca Farm? Or is there still, you know, further optimizations to be done around that where, you know, there's, you could imagine, you know, having models that get around some of these like generalizability objections, but still are using kind of the reinforcement learning approach you guys did?
9:17 Speaker 01 Yeah, so I think the key thing here is maybe to think about a data efficiency trade off. Like, I think everyone likes to think about it in binary terms, like you can't fine tune X or you, you can do everything with fine tunes, but that's not really the reality. It's, it's, you know, who is paying the cost is the way I think about it. When you fine tune a really, really good, really big base model, things are, things feel easy because with a little bit of data, you're going a very long way. And that's because, you know, in some sense, Meta has paid for the compute and data cost of making these models good. If you want to take a, I don't know, 13 billion parameter model and make it, you know, cover all the different things that a 65 billion parameter model is going to do, you're going to be paying tremendous data and compute costs.
10:02 Speaker 01 So you know, no one's really been pushing this. Of course, there's been some works recently like Orca and so on that pursue more intelligent data generation strategies to try to get better fine-tuned models. But I think the point is like, no one's really done this. But if you really want to get there with fine tunes, I think it's going to be a lot of data and compute cost. But I think the agenda could in principle be pushed. I would not be surprised if that was the case. It's quite clear, at least interacting with OpenAI Systems, that they're investing very heavily in data collection, in data, sorry, in fine tuning, even like ChatGPT and GPT-4. I think the open source models are not even close to the amount of like, coverage and scale at which I think OpenAI is doing this.
10:45 Speaker 00 Yeah, it makes sense. The, I really like the way of framing it around who is paying the costs. It's kind of like what you were saying, there's no, there's no free lunch for the performance of these models. Like you either start with a good model and then fine tune it and you know, get performance that way, or you have to start with a less good model and then kind of incur the compute costs all the way up to, to that performance. I guess like, the, you're obviously much closer to it than, than we are, but I think from playing with models like Alpaca and, and Databricks, the performance was so, I guess, much better than I would have thought it would be. And maybe that's, you know, because I don't know
11:27 Speaker 00 as much as you guys, but like, were you surprised by the performance of these kind of smaller models that were fine tuned? Was that something that you and your group was expecting? Or, or, or maybe, you know, you were surprised by how good they actually were?
11:40 Speaker 01 We were very surprised. We were very surprised by how it seemed to adapt to a diversity of tasks. In sort of our initial tests, you know, of course, we expected it to handle things like question answering, and maybe like writing emails. But it would do, you know, all sorts of weird stuff. It would write recipes, not, not good recipes at all, but it would like, follow the format of a recipe. It would, you know, be able to attempt to write poetry. Like it, it would follow tasks that we thought maybe would be kind of out of domain for a lot of these systems. That was pretty surprising to us. The other thing that was very surprising to us was the amount of data that it took. Our very initial model we trained on this only used 10,000 examples.
12:23 Speaker 01 And that behaved quite well. And the final release had 52K. But actually, that model wasn't going to feel much better than the 10K one, which, you know, suggested to us that really a lot of this was in the base model. And if you have a really good base model, just a tiny bit of instruction tuning model would like kind of pull out the right capabilities from these systems. Like going back to the who paid the cost, like it was, I think, very much like nice base model, you just pay a little bit of cost on, you know, getting it to follow instructions well and you you get these surprising capabilities.
12:57 Speaker 01 To give more context, though, we did play quite a bit with the OPT models, the previous generation ones. And that didn't rise, at least to us, to the level of, oh, you know, we should go and make a blog post. It really was when we switched to the Llama model that it felt like there was something there.
13:12 Speaker 00 Yeah. Yeah, the importance of that base model, that's fascinating. I guess one other question, do because it seems like you and your group, you know, you've shown that these kind of smaller models with a great base model can achieve really great performance and it doesn't cost that much to fine tune them.
13:32 Speaker 00 And we've already seen kind of an explosion of those with Dolly. Like, do you think we're going to live in a world where there are kind of many smaller models or it will lend itself towards that? Or, you know, there's also kind of this one model to rule them all in the cloud, you know, like it seems like there's kind of the two camps of like, hey, you know, the the big players will do everything in the cloud and one model is going to have all of this, you know, they'll be really good at the head and the tail, as you were mentioning, versus, you know, we're going to fine tune a model for kind of each different use case.
14:02 Speaker 01 Yeah, I think I think that's, you know, a really interesting, I would say, open question. I've heard many takes on this.
14:09 Speaker 00 Yeah.
14:10 Speaker 01 Things like, oh, people don't want to let go of their data, so they'll have everything on prem. I've heard arguments like, oh, fine tuning will be better because you'll have your own data. Or, you know, the the, what's it called, the efficiencies of scale mean that you're going to be much better off using the big OpenAI model. And there's definitely truths to all this. Some of them I am more skeptical about. With the cloud, you know, certainly we went through a similar transition of like no one would upload their data to the cloud. And everyone's now uploading their data to the cloud. I think that is going to be a slow and steady process. I don't think that's necessarily the thing that will make distributed LM sort of win out in the end. Where I do think distributed and like many different kinds of LM's will win out.
14:56 Speaker 01 I think is this question of how much value there is to specialization and diminishing returns to scale. I think those two are very interesting questions. So to give a very concrete example. Think about say like role playing again, or like you know storytelling with a large language model. Those kinds of applications obviously very valuable. But you know, OpenAI or some large centralized company that's trying to build a one big LM to rule them all. That's going to be a little challenging because you want those systems to be factual. And having a factual system might conflict with a system that's really good at storytelling, which is in some sometimes about making stuff up. And so of course, in principle, like people can kind of context switch, right?
15:40 Speaker 01 Like I can context switch from making stuff up to giving a very factual talk. But if it's hard for a language model to do that, then specialization starts to make a lot of sense. And of course, that specialization might happen at the, you know, at the vendor level. OpenAI might have like dozens of different models tuned for different tasks. But it would still be very much in the specialization direction. And if you take that to the extreme where, you know, maybe everyone wants to personalize it because there's some slight conflicts. Let's say in values or or what we want from the LM's, then you might start to end up with sort of much more specific LM's at like the company level or even the person level. So I think that's the question, like, can you mix these different kinds of goals into a single LM? Or is it just hard to get that level like very precise tuning to work?
16:25 Speaker 01 I think that's a really key question. And of course, the diminishing returns question, I think, is a really big one, too. Whether a single centralized enemy scaling up the base LM is going to continue to pay dividends.
16:35 Jacob Effron? Yeah, I mean, if you had to take a bet, given what you know now, like where do you think this goes?
16:41 Speaker 01 I think it would be maybe similar to like a oligopoly. Like, I think there will be, you know, a couple big, like, sort of LM providers. I think they will provide differentiated products. I don't necessarily think, right now, that there's going to be a single LM to rule them all. I'm kind of skeptical that you can balance, let's say, creativity and factuality really cleanly. A lot of the interventions I've seen on the academic literature really do give a trade-off on that front. And so I imagine many different models offered by companies are attempting to differentiate their products. And I imagine there will be an open source community that's like one to 1.5 generations behind, sort of on the basis of like models that these companies sort of like release back to the wild.
17:25 Speaker 01 So that's kind of the ecosystem that I imagine longer term.
17:29 Jacob Effron? Yeah, and what about like companies training their own models? Like, obviously, the news of today, I guess, you know, with the Databricks Mosaic acquisition seems like a bet on that. You know, what do you think of that kind of future?
17:39 Speaker 01 Yeah, I mean, I think that's a really interesting one. One thing that I think is really unknown is how much pre-training data affects large language models. The cost of training large language models and doing science on pre-training means that I think that area is probably under-optimized, even by the really big players. You don't get many shots to sort of retrain models. In a way, I think the wins from these companies might be if one of them discovers really nice ways of dealing with that pre-training data curation, they might start to get really big gains and differentiation from their competitors. Otherwise, if it's fully commoditized, I think it's in some sense a race to the bottom.
18:20 Speaker 01 You know, who can deliver the, you know, 60 billion parameter model for cheapest kind of thing? And then it's kind of hard to see where, in some sense, the value proposition is, other than, you know, open source goodwill or something.
18:32 Jacob Effron? Can you say a little bit more about the kind of pre-training process? I guess that you were alluding to, like, what would go into that? And how that might differentiate the model?
18:41 Speaker 01 Right, so I guess in the simplest, most basic flavor of this, you would have, you know, your data mixture, a big web scrape. Many different mixes, you would presumably go collect like Stack Overflow and GitHub and all these sources. I don't know for how long people will get to do that, given the tightening data restrictions and all this stuff. But you would have all of these corpora. And then there's the question of like, A, how do you filter some of your messier data sources? How do you weight all the data sources? Because you take multiple passes over the higher quality ones. And then there's also the question of like, how do you do things like tokenization? To sort of advantage or disadvantage certain parts of your data set?
19:21 Speaker 01 There was that recent, I think Open Llama was the one that used the tokenizer that collapses multiple spaces into one. And so it was very, it was not very suited for code, right? And so you might have things like this where, you know, your tokenizers might need to be optimized. So there's all these low level details that in some sense like shape the base on that you get out on the other end. And there's a lot of speculation about what people did right with Llama. And some of that might be data set. Some of that might be their choice to train longer on a smaller model. Some of it might just be they executed better. It's hard to know without the full details. But I think there's a belief that the pre-training data mix is a really important part of this process. Fascinating. That's super cool.
20:04 Jacob Effron? You know, one thing I definitely want to make sure we hit on is, you know, obviously the Alpaca Farm work you did. Maybe before we dive into our specific questions, would just be helpful if you could summarize that work for our listeners.
20:13 Speaker 01 Yeah, great. Alpaca Farm is this, I would call it a simulator for the entire large language model training process. At least as sort of researchers, one thing we found to be difficult was to try to understand which parts of the large language model training pipeline really matter, right? You hear things like, oh, you know, the instruction following data really matters. You say, you hear other people say, oh, it's reinforcement learning that matters. And some people might even say, oh, it's the reinforcement learning algorithm that really matters, right? And the kind of unfortunate thing is that because all of this work is being done behind closed doors, we have in some sense, very little understanding of which of these pieces truly matter.
20:56 Speaker 01 And so Alpaca Farm was our attempt at building this simulator that lets you simulate the whole process. So you take a base language model, you train it on instruction following data in the same way that you do for Alpaca. And then we have a simulated human annotator that gives you like thumbs up, thumbs down ratings for model outputs. And then you can do reinforcement learning on that to do the RLHF process.
21:19 Speaker 01 And so that allows us to basically run the whole pipeline of going from a base language model to a fully tuned, you know, final language model. And we can understand, you know, the impact of data, the impact of reinforcement learning and so on. And that was basically the Alpaca Farm project.
21:35 Jacob Effron? Yeah, and I guess, you know, I feel this is a big question, right, of like, where, where do the improvements in models come from, right? Is it the underlying data you're feeding in, the quality of or extent of reinforcement learning you're doing? Like, what is kind of, you know, given the work you guys did, like, where did you end up on that?
21:48 Speaker 01 Yeah, I mean, every piece is important, but if we were to say, you know, try to prioritize which ones are most important, the base LM is really important in the sense that that's where a lot of the knowledge and stuff is coming from, at least in the current tuning pipeline. We're not giving it enough instruct data that that's really, you know, changing the underlying knowledge store of the model. So the base LM is handling a lot of that. The instruction following data is the next most important. So the supervised fine tuning step that happens right after, that turns out to be, at least from our analysis, the second most important, because that's what makes it, you know, roughly, that makes the model do roughly the right thing. And the reinforcement learning, it turns out, does have a substantial impact, like, it's not zero.
22:33 Speaker 01 But the way in which it has impact is, is in some ways subtle. We found that it's able to do things like control the length of the answer, or control how often it uses lists or not. And so it's doing, you know, you might call it fine tuning of the answer structure on top of what the model's already doing. And in prior work, before we did the Alpaca Farm, one interesting thing we found in some of our work on summarization is when you ask people to write a high-quality summary, they'll often like, they'll often like, write very sophisticated text, they'll like, take the article, they'll like paraphrase it, do all this stuff. But if you ask people to rate which summary is better, they actually want passages copied
23:18 Speaker 01 from the text because it's easier to read. And so there is this really interesting mismatch between what people will write down as what they think is good and what people will rate as being good. And reinforcement learning tends to close that gap. Which I think is a very underappreciated thing that people haven't really talked about. People kind of think about humans as rational agents. Like, of course, you're going to write the thing that you think is best. But it turns out that's not necessarily the case.
23:45 Speaker 00 You know, that ranking that you outlined of, I guess, the importance of the raw LLMs, the supervised fine-tuning, and then the reinforcement learning. You know, I imagine, obviously, there's probably some, and you were getting into the nuance of the problem you're trying to solve or the domain you're going after. How do you think about, I guess, to the extent to which that ranking changes based on what you're trying to do?
24:02 Speaker 01 Right. Yeah, I think that ranking changes based on, in some sense, how much generalization and extrapolation you want from the system and how much data you're willing to throw at it in the instruction-following step and how much compute you're willing to throw at it on the reinforcement learning step. Like, one way to put this is, in some sense, instruction-following data, the supervised tuning, and reinforcement learning, in the limit of, like, infinite amounts of compute and data, they can compensate for each other. Supervised fine-tuning can make the model do almost whatever you want. That's classical machine learning, right? Like, take your model and just throw a lot of data at it. Reinforcement learning is also, in some sense, the same, right? Like, in the same way that you can do RL for robots from scratch and so on, you can
24:46 Speaker 01 just have the reinforcement learning policy take you all the way to a high-performance system. But really, it's, once again, the question of cost-effectiveness and what system you get at the end. If you train for very task-specific things using instruction-following data or reinforcement learning, you end up with a somewhat brittle system. It doesn't generalize quite as well. Whereas if you're relying on the base LLM and then tuning it very little, in some sense, because you're not using that much data, you're not overfitting the task. And that lets you generalize a lot better. That's been, actually, a very interesting lesson from a lot of the other machine learning work, like CLIP, where people basically show that these models are robust because they don't use very much task-specific data.
25:26 Speaker 00 Fascinating. Yeah, it's a really interesting point that, in the limit, the kind of instruction-following and RLHF, that is kind of just training, but it's another form of training. And then you can kind of figure out where you want to put those dollars, whether it's in the upfront training or RLHF or the instruction-following. But, yeah, that's fascinating.
25:52 Speaker 00 I guess pivoting kind of towards the your broader research, like are there other projects or like what's most top of mind for for you and your research group?
26:02 Speaker 01 Yeah, um, you know, so there is a big part of my group that's interested in like kind of understanding, building all the tools like like the alpaca group and so on are interested in like building the tools so we can understand and study the systems. That's one part. The other part of other parts of my group are interested in essentially, maybe I would call it control very recently. We're interested in things like how can we make sure that models, you know, very precisely follow instructions or if I'm a model designer and I want to make sure that my model follows very precise controls, like let's say it's never says something that's like offensive, or maybe it's, you know, I want to keep everything short. How do we make sure
26:43 Speaker 01 that models can sort of follow these kinds of controls? We've worked on things like using diffusion models for language in the past to basically try to do that. Diffusions are continuous, so they're hopefully easier to control through gradients. You know, the image generation people have done really great work with putting controls on top of diffusion models. So we've studied things like that. We've also studied things like consistency for control. So consistency means if I ask a model to generate something that's short, hopefully if I ask the model, is this short to the output, it will say yes, right? So the model should be consistent, acting as a discriminator and as a generator. We've been studying that form of consistency. And somehow, you know, language models not as consistent
27:25 Speaker 01 as you'd hope on these very basic kinds of queries. And so by by improving that, we're able to basically get slightly better controls on these systems. So those are the kinds of things we're kind of interested in. One side, understanding and the other side, like using the understanding for control.
27:42 Speaker 00 That's super cool. Yeah, I feel like control and, you know, these models kind of doing what people expect has been a really hot topic, especially in the news, you know, where people talk about like, oh, the AI is running wild or, you know, can't be controlled. I guess the, how, what types of, you know, you talked about like controlling the length of the output or like, what have you and your research group found when it comes to kind of coercing these models to do what is intended and like, how easy or hard is that to do? And maybe it depends on the type of model as well.
28:21 Speaker 01 Yeah, what's kind of interesting to us is if we're trying to apply, you know, a lot of these kinds of controls, things that require look ahead seems to be really hard for models. So if you want to, you know, basically say like, start at this sequence and end at this sequence, and you have exactly 15 tokens to make that bridge, those are kind of hard tasks for these models. Like, language models right now are, they're one token left to right. And so it's very hard for them to do planning in that way. Those kinds of tasks turn out to be the hardest, sort of technically. And what that kind of means is, you know, we've in the past found that like long-range planning is hard. People have come up with ways of getting around that, like making an outline first and making
29:03 Speaker 01 them right. But those are a kind of things that I think are going to remain challenges, like long-range dependencies and planning and constraints that apply over those.
29:11 Jacob Effron? One other thread of your work that I think is quite interesting is, I guess you guys had published this kind of like opinion QA, you know, paper, right? Where you were essentially looking at, you know, the extent to which these models agree with different opinions and kind of the extent to which bias creeps into these models. And, you know, I think it's really interesting, right? Because obviously, the folks that are doing reinforcement learning and how you do reinforcement learning can really inform, you know, that those, what these model outputs are. And so maybe to start, I'd love if you could just kind of explain to our readers, or our listeners, sorry, that work. And then, you know, we'll ask some follow-up questions from there.
29:43 Speaker 01 Great. Yeah, so opinion QA was this work where we were originally asking a little bit about, let's just call them like, opinions that are reflected or values that are reflected by language models. We know that language models could certainly be biased. You know, people have talked about racial stereotypes and stereotypes in general by these models for a long time. But there's also much more subtle things that these models pick up. Like if we ask them to write an essay about gun control, like will that be generally pro-gun control or anti-gun control? Unlike cases of, say, racial discrimination, there isn't necessarily a right or wrong answer here. And I think that makes it very interesting because it means that, you know, there's no clear solution to this problem. And we wanted to understand, like, what are language models today doing?
30:28 Speaker 01 You know, like, what kinds of people are they reflecting? Because when they're trained on the, you know, data from the internet, that's a mishmash of many different people that have ever written text. It's going to mimic those people in some way, and that's going to come out on the other side. And so we wanted to, you know, try to interrogate these systems and see what exactly it was reflecting. And the thing about Opinion QA is we wanted to understand it in this sort of nicer quantitative way. We wanted to say, like, who exactly is these models reflecting? And so to do that, what we did was we took public opinion surveys from the Pew, so that they have, you know, multiple choice questions asking about support for gun control or, you know, support for, you know, surveillance, things like that, that where people might
31:13 Speaker 01 disagree. And they have, you know, demographic information, so people throughout the United States, like, how do they answer these kinds of questions? We took those same questions, we posed them to the language model, and then we took the distribution of outputs from the language model, compared them to the distribution of outputs from people, and that allows us to do an analysis of what kinds of groups of people are closest to the language model in terms of their opinion agreement. And, you know, our highest level findings are that, A, language models don't reflect people super well. Mostly, the base language models, you know, reflect a really mishmash of opinions, some conservative, some independent, some liberal, generally lower educated and lower income.
31:58 Speaker 01 And when there's this instruction tuning in RLHF that happens, you start to see a shift towards, like, higher educated, higher income, generally more liberal opinion viewpoints. And we thought that was kind of an interesting sort of observation.
32:11 Jacob Effron? Totally. And so, you know, I think it kind of begs the question of, like, how do you think that these foundation model companies should be solving for this? Like, if you could wave a magic wand and change the process completely, like, what's the right way to kind of better reflect, you know, general views?
32:26 Speaker 01 Yeah, I mean, in some ways, I'm gonna punt on part of that question, so you'll have to forgive me for this. But I think one of the answers, of course, is personalization. It's unclear what opinion a language model should reflect by default. What does that even mean? Is it the population of the US? Is it the population of the globe? Is it the population of people in California? There's no good default there, right? I think OpenAI is currently taking the stance of just don't answer anything that's even potentially opinion-laden. That's a reasonable path, but it's also very difficult, because many, many, many queries could potentially have some opinion injected in them. It's very difficult to purge all of that out without making your language model useless. The language model might respond by saying, "Oh, this could potentially be opinionated.
33:11 Speaker 01 I'm just not going to answer it." The other paths that exist are personalization. In the paper, we also tested things like, "Oh, could you prompt the language model by saying, 'Pretend you're a Democrat'? Can you pretend to be a Democrat in answering your questions?" That does help, but it doesn't solve the problem. In personalization, I think this is the part where I say I'm not going to fully answer the question. Get at this really thorny question of what kinds of speech are allowed. If we think about social media, we have the same problem of allowing people to personalize their feeds, personalize the kinds of content and speech they consume, but then there's also this really thorny moderation question of like, what's the limit of that?
33:52 Speaker 01 Are you allowed to have a bot that is essentially like a neo-Nazi opinion bot? Currently on social media platforms, that will get moderated out. We're going to go through the exact same debate with large language models. And in a way, I think, hopefully, the social media debates can sometimes figure out some standards for this, because it's going to come for the large language models, too.
34:14 Jacob Effron? I mean, one thing I'm struck by is, so much of your work is fascinating at the cutting edge of what these models can and can't do today. And I'm wondering how you think about, like, obviously, you're doing this in an academic setting. Even as you think about the work that you take on, how do you think about what makes sense to be taking place in academia versus the folks at OpenAI or Anthropic doing? Where does academia fit into, I guess, the broader research that's being done around foundation models?
34:40 Speaker 01 Yeah, I mean, I think one of the advantages of academia in some sense is our incentives are very different from the incentives of a company that's essentially trying to push the limits by building the next biggest and greatest language models. I'm sure, you know, all of the people building these systems have
34:57 Speaker 01 the best interest of hopefully society at heart, but in general, you know, hopefully they're trying to do the right thing. But at the same time, they're constrained in what they, the kinds of work they can do by their incentives. Like, I think the work that we're doing in terms of, you know, trying to figure out like what kinds of opinions are in these LMs for, you know, essentially better public discourse about thinking about what kinds of opinions that LM should reflect, the opinion work, as well as Alpaca Farm, which is really very much not about pushing the limits of capabilities today, but sort of like stepping back and saying like, okay, can we make it possible for many, many other people to get in on the kinds of language model research? So we can, you know, have a much better diversity of opinions, explore many different ideas.
35:40 Speaker 01 Those just aren't really in the incentives of these large companies. So I think academia, in a way, because of, you know, our very different incentives and the very different place that we occupy on the compute landscape in a sense, I think have very unique things to offer. I think things like compute efficient training, compute efficient inference, things like simulation from our group, those are all, you know, very much informed by the fact that we operate at a much lower compute budget and we're not pushing the limits of performance.
36:12 Jacob Effron? Yeah. And so then I guess, yeah, the converse of that is there are a set of things that, you know, you guys maybe brainstorm that you could go after in your lab, but then you're like, you know what, maybe this makes sense for the industry to go after. Like, you know, what, how do you think about the boundaries of where maybe, hey, it's, you know, it's, we're not the best fit to go do this?
36:28 Speaker 01 Right. I think, you know, the thing that we're clearly not the right fit to do is building artifacts at the cutting edge, right? Like, I think, you know, building something that competes with GPT-5 or, you know, whatever is not really a thing that makes sense either like technically or incentive wise for us. We do think about questions that are compute intensive, but often those discussions are, is there sort of a clever way around what seems like an underlying compute limitation? So we've thought, for example, about, you know, improving how we do pre-training data collection or data curation. And those discussions are like, okay, we're not going to be training a big pre-trained language model. Is there really compute efficient ways of testing those hypotheses using say like
37:13 Speaker 01 scaling laws or other kinds of ways of doing experiments at the small scale? And so those questions we ask are very different. So even for some of those questions we're asking like can we do it compute efficiently rather than let's just brute force it?
37:28 Jacob Effron? Yeah. I mean, and also obviously, you know, part of your role in academia, you're obviously teaching this next generation of folks and I believe you're teaching, you know, advances in foundation models. So I can imagine you've got a fair amount of enrollment in that course, you know, given the excitement. But, you know, I'd be curious like, well, one, you know, I feel like Stanford's always been kind of on the cutting edge of a lot of this stuff, but what you're kind of seeing on campus today, you know, with folks and their interest. And then two, you know, how you think about teaching a course like that, just given how rapidly things are moving?
37:59 Speaker 01 Yeah, those are, those are great questions. Things that in some ways we struggle with as well. You know, we taught foundation models last year and we taught it this year and the enrollment was quite different between last year and this year, as you might imagine with things taking off. And in a way, it's interesting, the kinds of, you know, final projects that we got were also quite different. Like last year, a lot more analysis, like capabilities, like our language model is actually good. And this year it was, you know, a lot of application projects. It's like I made a language model that can order food, you know, by calling something up or calling up a restaurant, you know, like that kind of a, you know, let's build things spirit. And, you know, you were asking about students and there does seem to be a lot more of that
38:41 Speaker 01 now of like, a lot of energy also in like a lot of like, what can we do with this kind of questions? And in some ways, I wish people would think more about, you know, the fundamental science, but I think it's inevitable that with this exciting new technology, people want to build things.
39:01 Speaker 01 Oh, and teaching, that's right. You asked about how we think about teaching something that's so rapidly moving.
39:06 Jacob Effron? Totally.
39:07 Speaker 01 That is something we have struggled with. This year, we experimented with doing a lot of guest lectures, bringing in people from outside doing great work, especially in the industry, to expose our students to that kind of work that's being done. I think the course is still, in some sense, in flux. I think we're now going to try to have two smaller courses, one that's maybe more guest lecture oriented, on the cutting edge, and another one that's more like fundamentals, like how do you train a language model, like, you know, do everything at the low level, build it from scratch, kind of a course. And we're hoping to have like a fundamentals course and like a cutting edge course, and hopefully the cutting edge course can evolve, and the basics course can remain the same.
39:47 Speaker 00 Yeah, I think it's so, so cool that you guys are, you know, teaching this stuff to students so quickly, because as you were mentioning, it is changing really fast, and I feel like that's something that Stanford's always done an amazing job of, is offering courses that are like the state of the art right away, because it is something that's top of mind. One thing I was thinking through was, I can imagine there's a lot of interest in AI from people, you know, obviously the computer science students, but I would imagine even, you know, a broader, a broader group of students that is interested in not just how these models work, but maybe how they interact with them, because they've started using
40:29 Speaker 00 them in their daily life, and may touch a lot of products that they use as well. Like, how do you think about kind of teaching people, you know, maybe how to interact with these models, but also how they work, and, you know, for kind of maybe technical or non-technical audiences?
40:44 Speaker 01 Yeah, I think that's going to be like a really interesting challenge, that's sort of going to come up next. Like, right now, all of the, as far as I know, all of the pedagogical material we have is towards very CS people, like, how do you build it, like, how do you do the cutting edge? I think as some of the- Some of the systems stabilize a little bit. I think it'll be easier to teach sort of a how do you do things with these systems kind of class. I think the one challenge is the field and the models and the artifacts and everything are changing so fast. That is kind of hard to offer something that feels lasting and valuable. If we have like a, I don't know, chat GPT prompting class, I'm just making that up. And you know, GPT four point five comes out and you don't have to do prompt engineering anymore.
41:29 Speaker 01 That feels like a very wasted effort in a sense. And so I think for for that side, at least I am sort of waiting to see how it shakes out a little bit. Like in terms of the application layer, what is the lasting interface? And what is the lasting challenges that we're going to face? B, that we can design a media course around rather than, you know, being a collection of of hacks that we have here today.
41:54 Speaker 00 Totally. Yeah, that makes a lot of sense. Awesome. Well, we'd love to move to a quick fire round if you're open to it. And maybe to kick it off there, what's one thing that's talked about a lot in AI? That you think is overhyped and one thing that you think is underhyped?
42:13 Speaker 01 The hype questions are a little hard because some people say everything's overhyped, you know, growth people. But I think maybe one thing I think is there's been a lot of, you know, discussion and credit attributed to transformers, the architecture. Of course, it's like the architecture to use today. But I think there's been sort of almost a mythical property attributed to them. Yeah, transformers do X, Y, and Z. But I think we've seen in the last few years, you know, there's some work from like Chris Reas group here or like the RWKV stuff on like recurrent nets, like starting to look good. You know, subjectively, this is just my opinion. Probably transformers are not the end game. So to me, it really feels like a lot of the discussion around transformers seems a little bit, you know, a little too overhyped.
42:57 Speaker 01 In terms of what it can do and how critical it is to the success of the field right now. Clearly, if you're going to make something today, you're going to make it with a transformer. I'm not debating that. But in terms of whether it's needed, I think there's a little bit of hype there. What's underhyped in terms of language models? I think it depends on the people you're talking to. I look at the open source community and some of the safety discussions are a little underhyped. There's been probably a couple dozen open source instruction data sets being built. I think there might be one or zero safety or like data sets for preventing like dual use, like misinformation or like toxic outputs. There's very little interest in that in the open source space.
43:41 Speaker 01 And while for hobbyists, maybe that's okay. For this area to get more serious and be productionized and be used in enterprise, there needs to be a lot more work there than what exists now. In terms of the academic space, I think something that's kind of underhyped is sort of making artifacts that can actually work and interact with people. One thing that we sort of realized with both Alpaca and Alpaca Farm is that there's a really big challenge between, say, pushing benchmark numbers, which you can do in many different ways and produce improvements, and actually trying to build robust usable systems that can be used by other people.
44:22 Speaker 01 I think in a way, the University of Washington group doing things like quantization have done a really amazing job at building things that other people can use. And I very much respect that kind of work where they're not just writing academic papers, but they're also able to provide something that other people can really use. And so that's one thing that I'm trying to keep in mind. I haven't really succeeded that with all of my works, but trying to go one step extra than the basic academic work and being able to produce things that are of value to a broader community.
44:56 Jacob Effron? On your point on transformers potentially being overhyped, I think there was this very publicized bet between Jonathan Mosaic and Sasha at Hugging Face on whether transformers would be the state of the art. I think they said January 1st, 2027. Where do you fall on that?
45:10 Speaker 01 2027 is a really long time. So I think if I were forced to make the bet, I would maybe bet on not transformers because I feel like the field could completely be different. And just like we would be working on something totally different and we would... I don't really know what it would be called, but you know, there would be like the the octopus model and it would get transferred back to the language model or something like four years is just an incredibly long time. In this field, so I would just bet on like things will be different by default.
45:38 Jacob Effron? Totally.
45:38 Speaker 01 But I do think there is a truth to it that like winners keep winning in this field because there's a co-evolution between. Like the hardware and the models and the optimization that we do. So like the first movers in terms of like the architectures and optimizers, they end up being sticky. Because of the amount of like incredible optimization that happens engineering wise for them.
46:00 Jacob Effron? Yeah.
46:00 Speaker 01 And I think one underappreciated thing, uh, maybe this goes in the underhyped, or sorry, yeah, underhyped thing. Is the importance of compute for all of this. I mean, now I think everyone's coming to grips with the importance of compute. But it's not like, it's not sufficient to have a good architecture. You need to have a good architecture that's compute efficient. Because everything now is like in compute matched settings. Um, and so I think transformers might keep winning for that reason. They're just, you know, Nvidia is going to make the transformer GPU. And then no one's going to be able to determine transformers.
46:29 Jacob Effron? No, it's a, it's a really good point. I, I guess for the next quickfire question, this one feels a little ridiculous because I feel like you already do so much with research in your lab. But if you could like, you know, clone yourself and have, you know, twice as much time and run an entirely separate lab. Uh, what else would you, you know, what are, what are some like top research questions you'd love to go spend time on that you just don't have time today to do?
46:48 Speaker 01 Oh, my goodness. Um, that's a, that's in some ways a hard question. Um, this is completely speculative, I guess. Um, you know, almost by definition, it's work that I wouldn't be doing. So it would be a little bit outside of my field. But I think a lot of the work that's happening in security for large language models is truly fascinating. Um, I think, you know, the, the work that's happened in like prompt injection through like tool use is really amazing and cool. Um, the kinds of work that's happened with like data poisoning on internet script data sets is, is really inventive and really interesting. Um, I'm obviously not a security person. I've worked a little bit on people who want or who have been doing a little bit of security.
47:29 Speaker 01 Work. Um, but I think it would be lovely to explore those kinds of areas. I think it's going to be incredibly important and incredibly rich. Given how hard it is to get language models to do the right thing, you know, it's, it's going to be even harder than computer security. There we have formal models and we still can't secure our computers. So.
47:47 Speaker 00 Totally. I was wondering if you were going to, you know, obviously, since you started your career on the biology side, I was wondering, you know, if you're ever going to make your way back to that, or kind of what you think of, you know, the early efforts of folks, you know, applying foundation models in that space.
47:59 Speaker 01 Oh, I think they're really exciting. In a way, I feel like it's amazing that those systems like work really well. One of the things, the earliest things I did in my PhD was I wanted to make work to vec for the genome. I spent maybe a little bit too much time trying to get that to work at the time. But now, there's been a lot of really incredible work on like protein language models and genome language models, and they're working and they're producing really exciting results. In a way, I think, you know, I'm too far removed to be able to make valuable contributions to that space, but I think it's really great to see that those kinds of things can work. One of the things that I felt, you know, one of the reasons why I ended up not continuing
48:44 Speaker 01 on in computational biology was it felt like it was really important to be able to actually do the experiments, or to partner with someone that like really tightly to be able to do the experiments at the time, because the computer scientists were generating hypotheses, and they have to go to the lab to basically get verified. And now, maybe there's with all these language model things, maybe there's a lot more like purely computational work to be done, and that's exciting to see.
49:07 Speaker 00 Awesome. And then, I guess, last question. Where can people go to follow your work and learn more about what you, you know, keep tabs on what you're doing at Stanford?
49:16 Speaker 01 Yeah, great. You know, the best place probably is a combination of the places where we've been posting blogs, which is, you know, the CRFM blog, as well as, you know, Twitter. I guess, it's kind of weird to be advertising my Twitter, but Tatsu Hashimoto, I generally only advertise my students' work, so if you want to follow my work, feel free to follow me. That is the place where I'll post things.
49:38 Speaker 00 Amazing. Well, Tatsu, thank you so, so much for joining us. This was absolutely fascinating. And to steal a line from Jaka, but I think we would be doing a disservice to the future of ML and ML research if we kept you any longer, so I really appreciate you taking the time.
49:56 Speaker 01 Thank you very much. I appreciate it, and I love that it's very research-focused. It's, you know, refreshing to be able to have a conversation about this kind of thing.
Previous Ep 10: Insitro CEO Daphne Koller on Using ML to Change Drug Discovery Next Ep 12: EleutherAI's Aran Komatsuzaki on Open-Source Models' Future and Thought Cloning Page updated October 12, 2026 · transcript and summary are machine-generated
