Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain

How do you personalize AI models? A popular school of thought in AI is to just dump all the data you need into pre-training or fine tuning. But that may be less efficient and less controllable than alternatives — using AI models as a reasoning engine against external data sources. Kelvin Guu, Senior Staff Research Scientist at Google, joins Sarah and Elad this week to talk about retrieval, memory, training data attribution and model orchestration. At Google, he led some of the first efforts to leverage pre-trained LMs and neural retrievers, with >30 launches across multiple products. He has done some of the earliest work on retrieval-augmented language models (REALM) and training LLMs to follow instructions (FLAN). No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode. Show Links: Kelvin Guu Website Google Scholar FLAN: Finetuned Language Models Are Zero-Shot Learners Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs ROME: Locating and Editing Factual Associations in GPT Branch-Train-Merge: Scaling Expert Language Models with Unsupervised Domain Discovery Large Language Models Struggle to Learn Long-Tail Knowledge Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @Kelvin_Guu Show Notes: [1:44] - Kelvin’s background in math, statistics and natural language processing at Stanford [3:24] - The questions driving the REALM Paper [7:08] - Frameworks around retrieval augmentation & expert models [10:16] - Why is modularity important [11:36] - FLAN Paper and instruction following [13:28] - Updating model weights in real time and other continuous learning methods [15:08] - Simfluence Paper & explainability with large language models [18:11] - ROME paper, “Model Surgery” exciting research areas [19:51] - Personal opinions and thoughts on AI agents & research [24:59] - How the human brain compares to AGI regarding memory and emotions [28:08] - How models become more contextually available [30:45] - Accessibility of models [33:47] - Advice to future researchers

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Speaker 00

  I'm very interested in recipes that allow people to adapt things easily. It's not yet clear what the right interface is. If someday, for example, people had their own in-home language model, which of course doesn't exist right now, you might even say like, oh, these are our family values. We want the language model to have those values. Would that be through just writing down your kind of family values on an instruction prompt and just saying, please follow these everywhere? Or is it going to be through repeated feedback, where you talk to the language model and you say, no, I don't think you should say that. It is not clear to me how that will go.

• 0:38Sarah Guo?

  Do as I say, not as I do seems easier in this case.

• 0:47Sarah Guo?

  This is the No Priors podcast. I'm Sarah Guo.

• 0:50Speaker 02

  I'm Alad Gheil.

• 0:51Sarah Guo?

  We invest, advise, and help start technology companies.

• 0:54Speaker 02

  In this podcast, we're talking with the leading founders and researchers in AI about the biggest questions.

• 1:06Sarah Guo?

  How do you personalize AI models? A popular school of thought in AI is just to dump all the data you need into pre-training or fine-tuning. But that's costly and less controllable than using AI models as a reasoning engine against an external data source. And thus, the intersection of retrieval with LLMs has become an increasingly interesting topic. Today, we're going to talk to Kelvin Gu, senior staff research scientist at Google, who wants to make machine learning cheaper, accessible, and easier. He's joining us to talk about the newest methods his team is working on in AI training and language understanding. Kelvin, welcome to No Priors.

• 1:37Speaker 00

  Hey, thanks for having me.

• 1:38Sarah Guo?

  Let's do a little bit of background. You started out studying math, but pivoted to natural language. What encouraged that switch?

• 1:44Speaker 00

  Yeah, I actually had always wanted to get into tools that would help people learn more easily and find information more quickly. My motivation for going into the math was really to build a good foundation for getting into many of those deeper questions. And I would say that also motivated my venturing into statistics. So my PhD was actually in the statistics department, but I quickly then migrated over to the NLP group at Stanford, where I just learned a lot and had a great time.

• 2:10Sarah Guo?

  You chose to go to Google after your PhD. You've been there since 2018. Like, how has your focus or work changed during that time?

• 2:18Speaker 00

  Yeah, I would say at Google, this was really just the dream place for me to deepen my focus on building tools that help people find information better. And at the time that I joined, it was also just an extremely productive period where lots of new ideas were coming out. The group that I'm part of right now, which is now part of Google Brain, at the time was looking into very, at the time, early ideas on pre-training language models, which eventually became BERT. And that work just opened up a lot of frontiers for me, both in terms of product work at Google and inspiring new research ideas. I think one of the things that became very apparent early on when playing with BERT was unlike all the prior generations of models,

• 3:01Speaker 00

  it had a large amount of world knowledge that we didn't deliberately encode into it. It wasn't in the fine-tuning data, it was all within pre-training. And that has just many, I think, very obvious use cases now. But that also inspired me to, and my colleagues, to start thinking about retrieval augmented models and getting even more knowledge into them. And I'm sure we'll be talking about that more here.

• 3:24Sarah Guo?

  Yeah, let's do that. And let's talk about Realm, which our listeners was a really landmark paper in the field. In the paper and your talks about your model, you described the limitation of AI in domain knowledge or specialized knowledge that being retrieved and represented more accurately. What motivated the paper?

• 3:39Speaker 00

  Sure, yeah. So there are a few different things that can bring people to this retrieval augmented modeling literature. One of them was kind of our original goal, which was to increase the memorization capacity of these models. A second goal that you might come to this from is for modularity. So you might imagine that you have different data sources and you'd like to be able to swap one in or take one out, the same way you could do with a database. There are just so many business applications where that's helpful. A third is anytime you're dealing with a very timely application. So there's new information arriving daily about, say, a sports team or any other type of event. You really want to be able to incorporate that quickly without retraining. So these are some of the common ways that people arrive in the space.

• 4:24Speaker 00

  And it's a very natural thing, I think, now to think about, well, I don't want to retrain and yet I've got all this information out there and it's human interpretable, it's text. How can I bring that in? That's kind of what brought us to that in the first place. And since then, we've encountered many interesting challenges on top of that idea that, I would say, even in the sort of systems you see today, these tool-using models that issue Google searches or provide citations, they still face some challenges in terms of fulfilling those original promises.

• 4:52Speaker 02

  Could you talk a little bit more about the Realm approach in architecture? And then how that actually leads to some of these solutions? And you know, maybe more generally, I know that people have been focusing on sort of mixture of expert models and things like that, like MOEs. And so it feels like a derivative of some of this work in some ways. And so could you just talk about how Realm was initially set up and then how it's evolved?

• 5:12Speaker 00

  The basic idea, if we were to describe it, is let's say the user provides some sort of input to the model. I'm going to describe the inference time way of using it and then the pre-training that was involved as well. So the user provides some sort of input to the model. The model is then able to encode, embed that information into a dense vector. So this is a vector that will be situated in a larger vector space where we've also embedded other documents. So any other corpora we have on the web are converted into dense vectors as well. And a nearest neighbor search method is used to find the closest documents to the embedding of the input. And once you have those documents, you can then encode those as well in some form that

• 5:56Speaker 00

  allows the model to then use cross-attention over those documents. And based on that, it's then able to make a prediction. So the way in which the model depends on that information is through the cross-attention. And there are multiple different ways to train the cross-attention to kind of use that information. And in the Realm paper in particular, we tried to learn that from the language modeling task itself. So I guess for our listeners, many of you are probably familiar with the fact that language models predict the next token. So given a sequence of existing tokens, you predict the next one. At the time of the Realm paper, we were doing something called masked language modeling, which was popularized by Bird.

• 6:36Speaker 00

  So a very similar variation, instead of only seeing kind of the tokens leading up to your current token from the left, you would also, we would instead take a whole piece of text and just blank out a few words and train the model to fill it back in. And the way Realm was being trained in this method was we would say, okay, you can go and you can retrieve some documents. And if those documents help you fill the blank better, learn that those are useful. And if they don't help you fill the blank better, then learn that these are not useful and don't retrieve them going forward.

• 7:08Sarah Guo?

  Zooming out in terms of thinking about when to retrieve and when to train, like, do you have a framework for this? How do you think about this for different domains or you know, the idea of vertical specific foundation models?

• 7:21Speaker 00

  Yeah, so I think there are clearly a lot of different tasks where we're seeing you don't really need retrieval. So, for example, anything that's common knowledge on the web, more and more, we're seeing that if you build a dense model and that information is represented in enough quantity, these things with scaling improve. So it's actually been a case where I think more and more benefits of a retrieval augmented model are for modularity, for personal information that you might not want in the main model, for adapting to, say, like, an enterprise customer that has special information. I think that's a much more compelling application at this point. There are some papers that I can point to later in the show notes that kind of show

• 8:04Speaker 00

  how memorization scales with the number of times something shows up in the corpus, and you have to hit a certain frequency before these models can kind of accurately remember things.

• 8:14Speaker 02

  The basic description would be, if you read a bunch of documents over and over again, you're going to remember more details out of them.

• 8:20Speaker 00

  Yeah.

• 8:21Speaker 02

  Or if you see the same sort of sentence or words over and over, the same concept. And so as these models get bigger and bigger, you're basically incorporating more and more knowledge just inherently. And what you folks really focused on is, the model may be a little bit smaller, and so how do you make sure that you can actually retrieve information in a performant way relative to all sorts of domains of knowledge without having to build a giant model?

• 8:41Speaker 00

  Yeah, and I would say, Elad, you mentioned mixture of expert models earlier, and I think there's an interesting overlap between mixture of expert models and retrieval augmented models. So there's been a recent mixture of expert approach, which is very easy to describe. So I'll be describing a paper called Branch Train and Merge. You take your corpus of pre-training data, and you just split it up into different subcorpora, and you just train a language model on each one of those. And then when you want to use the language model at inference time, you sort of use some kind of routing mechanism or weighted voting mechanism to choose among those experts. And so in that case, each of the experts is responsible for part of the document corpus.

• 9:24Speaker 00

  And if you think about a retrieval augmented model, instead of having that intermediate layer of experts, you're just going to individual documents. And so it's, both of these approaches try to do the same thing, but they don't partition the space in the same level of granularity. You have one approach that's very granular and the other that takes broader strokes. And the kinds of use cases where one or the other is best actually differ as well. So if you just want to get very precise factoid information, like what is my Wi-Fi password? Retrieval augmentation is going to be very good. But if, for example, you're trying to adapt a language model to a new enterprise and they have some kind of specialized language in their codebase that nobody else uses, then just retrieving any single piece of code from that is not going to teach you how to use the language.

• 10:09Speaker 00

  Whereas an expert that's been trained on it is going to be able to adapt more quickly. So I think both of these approaches are going to be useful in different places.

• 10:16Sarah Guo?

  Is there a general way you think about modularization? Like why that's important overall? I think like you just described a situation where controlling the partitioning gets you better quality for a specific task, like answer questions on this, you know, proprietary language you have in this codebase, right? Why else is modularity important?

• 10:37Speaker 00

  Yeah, I think modularity is going to potentially have to happen due to the different, like the sort of way these models are developed. So it's not always going to be the case that any organization can invest the resources to build a large language model from scratch. So they're going to be wanting to build on top of something that exists. And they want to spend minimal effort to adapt. And that's where the benefit of modularity really comes in. I think it's also interesting to discuss the different ways people are adapting language models now. If we look at the trend over time, as late as in 2017, people were training specialized models for specialized tasks. So you might define your own special neural network architecture just for sentiment classification. And then we switched over to having a pre-trained model.

• 11:22Speaker 00

  Everybody's sharing the same architecture, but different fine tuning data. And then very recently, we have this sort of instruction following setup, which I was very thankful to have been involved in some of that research at Google.

• 11:33Sarah Guo?

  Just for our listeners, can you explain what instruction following is?

• 11:36Speaker 00

  Sure, yeah. So I'll describe some of the work we did in a paper called FLAN, which was one of the early papers to show that you could train large language models to follow instructions. So the high level idea is you start with a pre-trained language model and you collect maybe 100 different tasks. These could be sentiment classification, question answering, summarization. And for each of those tasks, You annotated each example with an instruction saying what it is that needs to get done and perform multitask training over all those tasks. And that had been done many times before in other research as well. But what we were able to show was that if you train on a hundred tasks, if we then show the model a new one hundred and one task,

• 12:20Speaker 00

  it will be able to adapt to that without having seen that particular task before. And that was very surprising at the time. Many people thought we would need thousands of tasks to make that happen. But because that worked out, things like InstructGPT came out next, followed by things like ChatGPT. And this sort of adaptation is just opens up the space of who can use these models much more rapidly. Suddenly, you don't need specialized fine tuning data, et cetera. But there's still a new frontier of tasks that you can't just get out of prompting. To give an example, like many people are familiar with the problem of large language model hallucinations. They will sometimes say things that are not well supported by external sources.

• 13:03Speaker 00

  And you can't just tell the model, please don't make things up and expect that to just solve the problem. There are so many other things you can say with instructions, but they won't be followed. And that's where I think existing adaptation techniques like fine tuning and reinforcement learning are still relevant. It's also still relevant for perhaps personalization or other kind of soft traits that are not always easy to communicate.

• 13:28Speaker 02

  How do you think about other approaches? People have talked about like updating model weights over time versus a training or other approaches to kind of have a model effectively learn in real time or adapt in terms of its knowledge base?

• 13:39Speaker 00

  Yeah, so when you say updating model weights in real time, I think I'm kind of thinking of continuous learning methods. Those I initially thought would have been more popular in kind of production grade settings. But they come with a maintenance cost, which is if you have something updating live. You don't have the opportunity to validate and check that everything is going well as easily. And so I've been kind of surprised by the fact that they're not actually as popular as I thought. That a weekly release cycle or a monthly release cycle is often fine. So for that reason, I personally haven't been looking into that line of research as much. But the idea of learning over time and continuously building up more knowledge, I think that's quite relevant.

• 14:23Speaker 00

  There are a variety of techniques out there for doing that. One example might be prompt tuning. So just to give a little bit of background on that, prompt tuning is where you have a densely parameterized large model that is frozen, the parameters are frozen. But the prompt that you give it is not natural language, but rather the vectorized embedded form of that natural language. And people use gradient descent on that vectorized form to adapt the model's behavior. So you can produce a prompt that's good for this week. Next week, you need something else, you can update that prompt. So the large model is not changing, but a small part of it is updating over time. And you have the ability to plug in and plug out different prompts as needed.

• 15:08Sarah Guo?

  You also just published this paper called Simfluence, which tracks how much smarter your model gets after consuming each example. Can you talk a little bit about the motivation for this work? And then broadly, I think the area of like, I mean, intersecting with a bunch of your other work, but explainability with large language models is an interesting one. Can we solve that? This seems like something that people generally approach, you know, post hoc.

• 15:31Speaker 00

  Yeah, so the Simfluence paper is part of an emerging area of research called training data attribution. So the problem there is not specific to large language models. It's really general to any machine learning model. The question that it tries to answer is, given that my model has done a certain thing, which training example has taught it to do that thing? This question goes back all the way to statistics and linear models. But for large language models, it's especially difficult to answer the question because of the kind of long training process that it goes through and the many forms of generalization that these models have. So in the Simfluence paper, we take a somewhat new approach to the problem. So we say, okay, how would you really justify that this training example

• 16:16Speaker 00

  was important for doing a certain thing? Well, if you had infinite compute, what you would do is you would take that example and remove it from the training set and retrain your entire model and see if the behavior changed. That's obviously too expensive for most people to do. So what we come up with is a very lightweight model that simulates the training process. It observes a few training runs you've done, tries to estimate the effect of each of the individual examples. And then you, as a developer, can sit down and say, imagine that I had done this training run, what would the result be? Any sort of simulation that you do is always going to come with approximation errors. And so this is still very early work on modeling these things. But I think it would have very beneficial effects if we could figure this out.

• 16:59Speaker 00

  One of the best use cases is if you know what training data is valuable, you can go out and get more of it. It would also help us understand what these large language models are really capable of. So we've seen impressive feats from large language models recently, like simulating a Linux terminal or being able to draw like a unicorn in LaTeX or something like this. And the question that always comes up for us is, was there something in the training set that actually looked like that? And we just can't find it. Without being able to track that down, we'll never know if these models are generalizing or just kind of cleverly patching together what they know.

• 17:35Speaker 02

  What is the most surprising thing that a language model has done that you didn't expect?

• 17:39Speaker 00

  Linux terminal one did impress me when that first happened. I would have to say also sometimes their ability to do visual reasoning in pure text is also quite surprising to me. I've seen language models use SVG notation to draw pictures, and that just seems very emergent.

• 17:55Speaker 02

  I guess one of the other areas a lot of people are focused on right now, or talking about at least, is different concepts of memory. In the context of LLMs or other types of models, what areas you think are most interesting to explore right now? And where do you think things will evolve over the next year or two?

• 18:11Speaker 00

  Yeah, so there's been a certain line of work that I've been really impressed with, which is what I'll call model surgery research. This is on editing model parameters to change their knowledge of certain facts. So there was a paper called Rome out of, I think it was a lab in MIT. And the basic thing that they show they can do is, let's say the model believes that the Eiffel Tower is in Paris, as it is indeed. They show that they're able to edit certain model weights to make the model believe that the Eiffel Tower is in Rome. And not only are they able to do this, but any sort of facts related to the Eiffel Tower or related to Rome are also updated. So the model doesn't just remember this one fact, but propagates that knowledge throughout the rest of the network.

• 18:54Speaker 00

  So for example, if you ask, what is the tallest structure in Rome? After making this update, the model will also say Eiffel Tower. And this is based on a bit of theoretical work showing that the weight matrices inside large language models can be thought of as lookup tables to some extent. So we know that large language models perform a large number of matrix vector multiplications, and the matrix is the lookup table. And the vector that is multiplying the matrix can be thought of as the query to that lookup table. And so by updating that matrix, they're able to change the knowledge. I feel that this is a very exciting area for research because it provides a different kind of modularity from retrieval augmented models or mixture of experts,

• 19:37Speaker 00

  one that actually allows a kind of generalization. It's very interesting.

• 19:41Speaker 02

  Yeah, we've talked a little bit about modularity and memory. What other areas you think are major components that people need to work on in order to achieve true machine intelligence? Or to push the field forward?

• 19:51Speaker 00

  Yeah, I think. Since you say true machine intelligence, then we're going at the full, I presume, AGI-level goal. And I think one thing that's very obviously in the public discourse right now is increasingly autonomous agents. I should say, by the way, everything that I'm discussing here, it's all personal opinions, and I'll be deriving everything from things I've seen in the public discourse, nothing related to any of the work that I'm doing at Google. But I do see this desire for autonomous agents. And it seems like the going approach right now is inspired by the Turing machine. So we have these models that can operate over a relatively small context with a prompt. And folks are trying to take

• 20:35Speaker 00

  long-horizon tasks and reduce them into these smaller, promptable tasks. And the component that they're using to bridge the short-term horizon into the long-term horizon is a memory mechanism, much like the retrieval augmentation we've described, where the agent has the ability to store its memories either in text or in a vector store. And I think more research is definitely going to be needed there. It's a very ambitious goal, but this way of decomposing the problem leaves, I think, potentially many gaps. So, for example, let's say there's an agent pursuing a task, and it breaks that task down into sub-goals that it defines for itself. It will then write one of those sub-goals down into its metaphorical sticky note that decides what to do.

• 21:20Speaker 00

  And if one of those sub-goals is wrong, suddenly that is a canonical goal in the model. And how does it realize to reverse that decision? You could argue that, oh, we just need more prompting to get the model to think about every aspect of the problem. Us as humans, we certainly do prompt ourselves quite a bit to think about a problem from different angles. But a lot of what we do is also instinctive. And those instincts are developed over time. So just to give a psychology analogy, there's this concept of chunking, where people have shown that a human cannot really easily remember more than seven unique items at a time. But if those seven unique items are then broken into chunks that they can attach some sort of metaphorical or otherwise significance to, they can remember much longer sequences.

• 22:05Speaker 00

  And I think the same thing... Kind of happens with humans where initially we do something through explicit reasoning, but eventually we distill that down into instincts. That's something that seems to be missing from these autonomous agents right now. Everything is explicit reasoning and at some point that might not scale or that might become brittle. So that seems like a very exciting area.

• 22:27Sarah Guo?

  Do you think that's architectural with how people are building agents right now? Or if it's a scale thing, right? Because the agents that people are like talking about, working with in open source are tiny.

• 22:36Speaker 00

  I think that it is a bit architectural right now. So just to give an analogy, like in 2018, when question answering systems or reading comprehension systems got really good, everybody tried to reduce a lot of problems into question answering. And there was just this dash to just throw that into different components. And now I think now that we know that instruction following works, again, folks are building these structures in that way. But it's quite possible that fundamental advances could come along and this kind of workflow hacking or workflow engineering could go the same way as feature engineering or specialized architectures.

• 23:17Speaker 02

  Yeah, it definitely feels like it's almost like a patch on something that should work in a smarter way almost. And to sort of create a concrete example for our listeners in terms of, you know, an agent action, it would be something like, hey, we're going on a, we want to go on a trip to Hawaii. Could you find the best hotels that are kid friendly and then book that and then book a series of activities for us to do during the days that we're there? And, you know, on the flight, we need you to book things that you know that we like in terms of aisle versus window or whatever. And then the agent would go and do it. And so it could keep like a list of all the different tasks and then come back to those lists. So that's what people are kind of hacking together now. They're basically creating a place where the model can go back to like, kind of check off the fact that it did something and then it moves on to the next task on the list.

• 23:58Speaker 00

  Maybe just to run with that example a little bit. So you can kind of specify all the things that you want about your vacation, but there's also another aspect of this where if you had an actual travel planner, they would gradually learn over time your preferences, not through your explicit instructions, but just your feedback, your responses. And that kind of learning loop, I think, doesn't appear yet in existing methods. I think this becomes even more apparent when an autonomous or semi-autonomous agent needs to interact with more complex tools. So, simple tools like maybe issuing a search query, they're very short interaction and it's almost stateless. You just put something in, you get something out. But if you're actually operating anything of significant complexity,

• 24:42Speaker 00

  there's a learning curve that anyone has to go through. If somebody sees a particular spreadsheet that an organization is using for certain purposes, you need to implicitly understand how that spreadsheet is being used, what is okay, what isn't okay. And that, I think, is also missing and can lead to a lot of brittleness.

• 24:59Speaker 02

  A lot of the things that you've described, I feel like, are ways that at least a subset of neuroscientists believe the brain works, right? It's very clear we have modular parts of the brain that are involved with vision or information processing or certain types of emotion. So we know that at least the human brain is very modular. There's obvious different notions of memory, both short term and long term, in terms of how you store and retrieve information. And then it seems like there's an enormous amount of things that are just hardwired, right? Like, the number of language-related tokens we actually see as a person is quite small. The number of words that we're exposed to by the time we're a toddler who can speak seems to be small relative to all the training that happens for a large language model in terms of the data set. Are there other aspects of the human brain that you think people are kind of likely to copy or implement into these types of models in the future?

• 25:44Speaker 02

  Or ways that you think these things will just fundamentally differ?

• 25:47Speaker 00

  Thanks. Yeah, that's a really great question. One that I was thinking about recently, and this is again very speculative, but when we think about autonomous agents, we worry often about their capacity to do something that is irreversible, to get themselves into a state that they can't get themselves out of. And in human behavior, there are certain things that we have an instinct to avoid, and that is usually driven by an emotion, like fear or uncertainty or anxiety. And funnily enough, in the reinforcement learning literature, there are people who have proposed research concepts that they call fear, where the model learns to identify states that it can't back out of. And I suspect something along those lines may also be helpful.

• 26:30Speaker 00

  They may also introduce the kind of corresponding issues with fear that humans have as well, and it's open questions.

• 26:38Speaker 02

  It's basically adding a limbic system or something. Yeah.

• 26:41Speaker 00

  Yeah, I also think, though, that there's this kind of memory consolidation aspect of human learning where there's a deliberate pruning of what you've thought about that. We don't have any analog for that, at least in the early systems we're seeing right now.

• 26:55Sarah Guo?

  I think it's also interesting to think about whether or not we've already or will continue to introduce biases that humans have or new ones, right? And so this is an analogy, but you can think of these models as having a consistency bias. If you think about some of the tasks that they're incapable of doing today correctly, math, and one of my favorite examples of this is you ask a model, is 26 or 21 larger, might get it wrong, and then you ask the model to count down from 21 to 26 and it counts infinitely because it assumes that it's going to get to 26 at some point. And so it's like this extreme version of the cognitive dissonance that humans can have, but there are mechanisms to get yourself out of that, too.

• 27:39Speaker 00

  Yeah, I think that's a good example that a human at some point will introduce some kind of introspection step, like, wait a minute, I've been counting for 10 hours. And I think that's often missing as well. People try to patch that over with chain of thought reasoning, so they'll do some reasoning, then they'll say, is this reasoning right? And repeat that loop. But does it have to be as explicit as it is right now? I don't think so. I think we have associative mechanisms that ping us after a certain amount of time on these things.

• 28:07Sarah Guo?

  I just keep counting.

• 28:09Speaker 02

  Could you explain to us how models are becoming more contextually available as well?

• 28:12Speaker 00

  Sure, yeah. So at the moment, I think there's this paradigm where you sit in front of a chat interface, and if you have a problem, you think really hard about all of the information and context around it, and you try to boil it down into this wonderful prompt for the model. And that actually requires a lot of brain power. So there are a lot of cases where people will just not think to use the model because of that extra barrier. And I think there's clearly an ambition in the work that we see publicly to make these models more available and more context. So you could imagine just being in the browser and trying to get something done, and after 20 minutes, you just ask your model, like, why can't I get this done? Or how do I solve this? And it should ideally look at everything you've been doing up until this point and

• 28:56Speaker 00

  infer your intent and just help address that. That seems like a pretty open question that. Isn't fully addressed.

• 29:03Sarah Guo?

  One more research question, and this is a very general one, but you have broad ranging interest in the field. What makes a good knowledge representation?

• 29:11Speaker 00

  So when I started doing research, I was actually very involved in knowledge bases. I thought those were going to be the future because they have this wonderful canonical representation of things. You can do reasoning over them.

• 29:23Sarah Guo?

  They're a very interesting area to explore if you think retrieval will continue to be important, right? Because you might have these representations for models to go retrieve against, right?

• 29:32Speaker 00

  Oh, yeah. To be clear, they are useful for many applications. In many cases, a database, you want that to be your source of ground truth, and that's perfect. If you want broad coverage, you ideally want a mechanism that's more expansive than that. One thing we've noticed is that it's very hard to get a representation that is 100% canonicalized in the sense that information is only represented in one place and not anywhere else. So with text, for example, a single fact is in multiple places. But if you want to edit a fact, you need to go to all the places where that fact is mentioned and change it. So there always seems to be this trade off between centralization and coverage. And this is kind of why I mentioned the earlier model surgery work, because I think that's some

• 30:16Speaker 00

  of the first work to try to keep all of the coverage while also allowing some degree of centralization. If I were kind of advising a student or something on looking into knowledge representations, at the moment, I would say there's a lot of momentum on dense models continuing to capture more and more of the different applications. And so if there is a way that you could work with that dense representation to make it more manageable or more controllable, that would be preferred.

• 30:45Sarah Guo?

  Zooming out from your research work, I think a question in society and in the tech ecosystem recently has been like, how important is it that training is accessible? And that, you know, we don't just have a small set of model API providers, but models in open source and a richer set of providers. Like, how do you think about that issue?

• 31:10Speaker 00

  Yeah, so I'm very interested in recipes that allow people to adapt things easily. And it's not yet clear what the right interface is. So, training has always been challenging for many reasons. I think to do training well, yes, there are scripts and architectures that are ready to go. But data quality is a really big aspect of good training. And so there is a question of will that be the most common format? Because curating data is not easy. At the scale of maybe 10 or 30 examples, it is easy. And maybe that is a sweet spot for a lot of users. But again, I think it depends a lot on the application.

• 31:52Speaker 00

  So, we could think from the perspective of an individual, like if someday, for example, people had their own in-home language model, which of course doesn't exist right now. You might even say like, oh, these are our family values. We want the language model to have those values. Would that be through just writing down your kind of family values on an instruction prompt? And just saying, please follow these everywhere? Or is it going to be through repeated feedback where you talk to the language model? And you say, no, I don't think you should say that. It is not clear to me how that will go.

• 32:21Sarah Guo?

  Do as I say, not as I do seems easier in this case.

• 32:25Speaker 02

  I think that will train the wrong behavior societally, unfortunately, just given how people tend to act against that. What you described, I think, in one form of that is what some people talk about as like constitutional AI, or at least that's the anthropocentric view of it. Do you think approaches like that will propagate in the future in a big way? Or do you think there'll be other approaches to sort of get the behavior that you're talking about?

• 32:45Speaker 00

  Yeah, so I think there's a strong incentive to make that happen. So, the folks who are providing large language models, they want to make their approaches as easy to use as possible. And so, anything that can go into prompting, it seems to me that people will try to do it. Because that gives the broadest reach. You don't want to just say, oh, you have to fine tune this and we're not going to help you in any way. So, I expect people will try to push that as far as possible. And you could even say that to some extent, just following on what we did for instruction tuning, you can fine tune the model to be more and more responsive to different types of instructions. So, if you can anticipate that someone is going to say, I want you to follow these family values. Well, then at training time, we can prepare for that.

• 33:26Speaker 00

  And so, it's a game of how much can we anticipate that people will want to adapt?

• 33:31Speaker 02

  Yeah, I like the term family values for that because it feels very sort of grounded and we're all in it together. You know, I think that's a really nice framing of it versus some of the things that feel more like we're enforcing something in a way that may not work in the long run. So it's a, it's an interesting turn of phrase.

• 33:47Sarah Guo?

  Kelvin, last substantive question. Do you have any parting words or advice for those who, you know, want to become productive research scientists in the future? You've created a bunch of really incredible work.

• 33:59Speaker 00

  Oh, sure. Yeah. I think at this time in particular, I encounter folks who fall into like two different camps. Sometimes there are folks who will say, oh my gosh, like everything is happening. And even if I sit here and do nothing, it will all get done and I just don't know what to do. And then there are others who feel that, oh, there's too many things to do. I cannot get to all of them in time even. And I think there's a couple of different answers. One is if you move the goalposts for yourself. So a lot of researchers, they kind of had set what they felt were ambitious goals, but then given all of the recent progress, those goals now need to become more ambitious. So if you think farther out, you will find that there are so many things that are not solved, like making agents more autonomous or safe, or understanding which training

• 34:43Speaker 00

  examples caused a language model to do something. There's a lot of headroom there. And especially I think now, if you pay attention to what's happening in the product world, that can provide so much motivating use cases. And even, you know, if you come to Google or something, there are so many things there to look into. That's one thing. The other thing I would say is there's sort of also folks who ask, how much technical knowledge is still important at this stage for doing machine learning? Like, has it become kind of an era of just sort of prompting language models to do different things, or just curating data and scaling things up?

• 35:25Speaker 00

  And I would say there are still, well, obviously it's great if you can build valuable applications using a simple skillset. And I think there's an amazing amount of creativity that can go into that. But if you're also looking for something that is more of a technical challenge, there's still so many unsolved problems in terms of understanding how these models work. And ultimately, we do need to get to the bottom of that to improve things like safety and predictability.

• 35:49Speaker 02

  I had one really random question.

• 35:51Speaker 00

  Go ahead.

• 35:52Speaker 02

  Yeah, so I was basically thinking back to like a year ago. If I were to try and hope for a career for my kids, it would be to go into computer science. And as I think ahead, you know, 15 years, 20 years, however long ahead, at some point, it feels to me that machines will be writing better code and be doing better computer science research than we will. And so I was curious your thoughts on that and, you know, over what timeframe is that a real concern or is that even a real concern?

• 36:21Speaker 00

  I love that question. I'm sure many people have asked that. I feel that there's maybe been a shift in what kind of skill is valuable. So at a certain earlier point in time, having technical proficiency was a huge differentiating factor. And if you didn't have that, you just couldn't pursue certain ideas. Whereas now more and more, the technical proficiency part is we're being helped out by large language models to, you know, write in programming languages we don't even know. And so it seems like the the differentiating factor now shifts more to problem formulation or creativity in identifying problems and being able to frame them in a way that you can then reduce them to a technical problem. So I sure definitely don't know what will be happening 20 or 10 years from now, but

• 37:06Speaker 00

  being creative and solving different kinds of problems will always still be useful. Another thing that has come up in some discussions is that even if the large language model is autopiloting a lot of work for you, someone still needs to validate that. And that may still require a great deal of technical skill unless you prompt a large language model to validate things as well. But then who's validating the validator? So at some point, technical skill will still be needed, whether it will be as broadly needed, it's anyone's guess.

• 37:36Speaker 02

  Yeah, it's kind of an interesting concept because if you look at the AI gaming literature, it feels like at every step, ML system works better at chess and people are like, well, it'll never solve Go. And then a few years later, it beats everybody at Go and they're like, oh, well, I'll never beat people at poker. And then it beats people at poker and they're like, oh, well, diplomacy. I mean, that's, you know, manipulating people and causing them to act and it's never going to do better than people in diplomacy. And then, you know, obviously, Noam Brown and other people's work on diplomacy suddenly shows that machines can win at that. And so I sometimes wonder about the concept of creativity and problem generation and all the rest of it relative to that same sort of slope of capability and learning. And so that was in part the basis for the question is at what point does the advantages of being human run out? And then how do we think about that from the perspective

• 38:21Speaker 02

  of future decisions for our kids, or even what are the things that they start to learn now relative to what that future will be? And it's obviously very uncertain. So it's very hard to tell. But

• 38:30Sarah Guo?

  Alad and I were just talking about how robotics, for a range of reasons, is still significantly behind. And so his kids are going to be woodworkers. That's the answer.

• 38:40Speaker 02

  Oh, gosh.

• 38:41Sarah Guo?

  Yeah, something in the physical world.

• 38:44Speaker 02

  I don't know. We'll see. I hope they can still be knowledge workers, but we'll see. I mean, physical stuff is great, too, but it suggests a lot about humanity's place in the universe, if that were to be the case.

• 38:54Sarah Guo?

  On that note, thanks, Kelvin. This is an incredible conversation. Thanks for joining us on the podcast.

• 38:58Speaker 02

  Thanks so much.

• 39:02Sarah Guo?

  Thank you for listening to this week's episode of No Priors.

• 39:05Speaker 02

  Follow No Priors for new guests each week and let us know online what you think and who in AI you want to hear from.

• 39:10Sarah Guo?

  You can keep in touch with me and Conviction by following at Sarah Normas.

• 39:14Speaker 02

  You can follow me on Twitter at Alad Gill. Thanks for listening.

• 39:19Sarah Guo?

  No Priors is produced in partnership with Podpeople. Special thanks to our team, Synthil Galdea and Pranav Ruddy, and the production team at Podpeople, Alex McMonis, Matt Saab, Amy Machado, Ashton Carter, Danielle Roth, Carter Wogan, and Billy Libby. Also, our parents, our children, the Academy, GovGPT, and our future AGI overlords.