ChatGPT goes prime time!

Daniel and Chris do a deep dive into OpenAI’s ChatGPT, which is the first LLM to enjoy direct mass adoption by folks outside the AI world. They discuss how it works, its effect on the world, ramifications of its adoption, and what we may expect in the future as these types of models continue to evolve. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: ChatGPT https://chat.openai.com/chatOpenAI Blog: ChatGPT https://openai.com/blog/chatgptIllustrating Reinforcement Learning from Human Feedback (RLHF) https://huggingface.co/blog/rlhfUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Daniel Whitenack?

  It's a three-step process. So you pre-train a language model, then you gather this sort of human preference data and train a reward model. Now, the second reward model is trained to take in a prompt and a response and score it like a human would score it according to preference. It's actually trained on the human preference data and it outputs a prediction of what a human preference might be on this output. The third and final step is that you fine-tune a copy of your original language model using this trained reward model and a reinforcement learning loop.

• 0:53Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com. And to our friends at fly.io. We deploy our app servers close to our users and you can too. Learn more at fly.io.

• 1:29Daniel Whitenack?

  Welcome to another fully connected episode of the Practical AI Podcast. These episodes are where Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and then dig into some learning resources to help you level up your machine learning game. I'm Daniel Whitenack. I'm a data scientist with S I L International and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 2:00Chris Benson?

  Doing very well. Happy New Year 2023. This is our first conversation.

• 2:06Daniel Whitenack?

  Yeah, Happy New Year. This is the first one we're recording in the year 2023. Looking already to be an exciting year for AI things. I hope you got a bit of a refreshing break over winter because there's a lot of, I'm guessing it's going to be a whirlwind of AI stuff this year.

• 2:25Chris Benson?

  I think it is going to be a whirlwind. I didn't get a rest over the break because having nothing to do with AI, our animal nonprofit, we had all the winter weather that most people in the US were aware of and we had, we were doing animal emergencies. We saved a whole bunch of lives, which made the lack of rest worthwhile.

• 2:44Daniel Whitenack?

  But there was a lack of rest.

• 2:45Chris Benson?

  There was a lack of rest, but we did a lot of good. But interestingly, the conversation we're going to have today will play into that very non-AI side of my life because we're starting to see some crossovers. We'll see in a few minutes here.

• 2:58Daniel Whitenack?

  Yeah, it's interesting, you know. So today, spoiler alert, we're going to be talking about ChatGPT. You've probably been expecting us to talk about ChatGPT for some time. One of the things we wanted to do is really dig into the internals of ChatGPT, how it works and its implications. And so we wanted to do it justice, which is partially why we wanted to take some time and prep for that. But it is interesting also to get a little bit of perspective now that ChatGPT has been out for not that long, but a little while. Over Christmas, you know, I was at Christmas with my family and even at our family Christmas dinner,

• 3:40Daniel Whitenack?

  my dad was asking me about ChatGPT. And, you know, I, at my church, I've had people come up to me and ask about ChatGPT who aren't, don't work in tech or anything like that. And my barber, you know, whoever is in my life, it seems like they're at least aware of ChatGPT. They might not know exactly what it is, but they know that it's a big deal. Are you having a similar experience?

• 4:07Chris Benson?

  Very, very similar to that. And for folks, Daniel and I haven't talked through the holidays. So you know, this is the first time I'm hearing it, just as you are. And I'm having the same experience and it's been really notable that we've, you know, that each new large language model comes out and you know, the various GPT series. And we talk about it. This is the one that's crossed over into mainstream awareness and broad use. And I mentioned, as we were getting into the conversation, that it's now crossing over from the technical and AI side of my life into the non-technical and animal side. As we do things like narratives, both written in video and educational material.

• 4:46Chris Benson?

  This is an amazing tool that completely non-AI focused people can use productively to really do good in the world and get things done that they want. So it's been really interesting to see how this one has been different from the GPTs before.

• 5:02Daniel Whitenack?

  Yeah. So in your case, it's something that like, as you're creating content, you see it as potentially playing a role in whatever scripts or articles or whatever that might be. Is that right?

• 5:17Chris Benson?

  Absolutely. It's been quite humbling in that way, experimenting what was possible, because the, the quality of, of the outputs are typically much better than I can do by myself. I've done that both in terms of I'm writing a children's story to teach children about animals. And I've been experimenting with it. And every time I write something and then I seed it into chat GPT, it does a better job than me. So it's been very humbling in that way. I think of myself as a decent writer as well. And then the quality of video output is just, uh, has been quite good. And there's a little workflow, but it means that we can do more good in the world faster. It accelerates the ability to put out great content. And so I think

• 6:02Chris Benson?

  that this is one of those inflection points that we've seen, not just on a technical merit, but in the world at large.

• 6:09Daniel Whitenack?

  Well, your usage has, seems to be much more useful and valuable than my usage, which has mostly been things like writing. I remember I had chat GPT write a new Christmas carol for me about the three wise men in the style of a rap song by Eminem.

• 6:29Daniel Whitenack?

  I have to say, it was, it was a great rap song. I didn't record it because I'm not Eminem, but I sent it to his people and we're having discussions.

• 6:38Chris Benson?

  Okay.

• 6:39Daniel Whitenack?

  Yeah.

• 6:40Chris Benson?

  I, I can't believe you're, you're not sharing that with us.

• 6:44Daniel Whitenack?

  Well, maybe before we jump in, I, I think some of what we wanted to do today was just like, describe a bit of like what ChatGPT is, what the interface looks like, what you can do. But then really do a deep dive on what are the guts of the system? Why is it different than what's come before? In what ways is it similar to things that have come before? Both of those things are true. And so we want to do a deep dive and then think about some of the implications. So buckle up, hopefully this will be fun. First off, it is called ChatGPT, which is interesting. So the interface that they've chosen for this and the sort of design of the system is a chat interface. So if you go to chatgpt dot openai dot com, you need to create an account.

• 7:29Daniel Whitenack?

  And we can talk about some of the implications around that in a second. When you log in, it gives you some examples of what you can do, some example capabilities and some limitations. I found this interesting and we can talk about it later. Some of how they described the limitations and they released the model. But the basic idea is there's a chat interface, you can type a prompt and it will respond. And then you can actually continue to have dialogue with the system. So you can say, you know, tell me more about that, or I don't understand this part, you know, explain that bit more. So some of the examples that they give are, you know, explain quantum computing in simple terms prompt, or how do I make an HTTP request

• 8:11Daniel Whitenack?

  in JavaScript? So there's even a, you know, it can output code, it can help you debug code. Like I mentioned, it can provide lyrics or scripts or structured types of things like the Eminem song. So yeah, that's the basic.

• 8:27Daniel Whitenack?

  Input, output. How did you find this sort of interface, Chris, in terms of your own usage as related to like building scripts and other things?

• 8:36Chris Benson?

  It's been interesting in that it will take it a direction like as I've been trying out, the children's story thing is something I've been playing with and seeing where where ChatGPT chooses to take the beginning of a seed of a narrative. Like, I would start off with, you know, "Once upon a time, there was a precocious raccoon named Pandora," because that's the hero in the story. And it's been interesting to see how it's taken it, but it's also, it will go off in directions I don't want. So then I'll ask questions to kind of steer it a little bit, and it will come back. So it doesn't, it's not final output, but it's producing a body of narrative that's better than I could have done by far. And so I find myself, instead of being the creator

• 9:19Chris Benson?

  of the story, I'm kind of editing it to make it work. But it's a collaboration, in a sense, between, this is one of those first points where we've talked in an aspiring way about collaboration with AI for a long time. But I now am doing that and steering it in different ways with the entering in the chat and seeing where it went and asking specific questions about the story. It's been quite remarkable. For the first time, it's like having a partner in the process. It just happens to be that the partner is not human in this case. A friend of mine, Brent Siegel, has been also playing with it a lot, doing some stuff on him. And that's how he described it as well. He was looking at some different topics and he said, you know,

• 10:00Chris Benson?

  "It's like having a dozen world-class scientists for the things that he was focusing on, right there, available to you, and they're not wrong." You know, they never get it wrong. You know, he had some pretty cool stuff that he was working on in a very different thing. But it's that sense of collaboration with the technology in a real-life sense that's really different now from the way it was before this.

• 10:25Daniel Whitenack?

  Well, as you were chatting, which though is some great context, I asked ChatGPT the following: "What state-of-the-art AI topics does the Practical AI Podcast need to cover in 2023? And who should they reach out to for interviews?" And the response, "So you're ready for what we'll cover in 2023?"

• 10:46Chris Benson?

  Let's hear it.

• 10:47Daniel Whitenack?

  ChatGPT said it's difficult to predict exactly what the state of the art in AI will be in 2023. Definitely an interesting start.

• 10:56Chris Benson?

  Yeah.

• 10:56Daniel Whitenack?

  As the field is evolving rapidly. However, some topics that might be worth covering include machine learning interpretability and explainability, and then it actually gives a blurb about what that is. AI safety and ethics. So that's right in your wheelhouse. And it gives an explanation of that. Natural language processing. So we can continue to talk about my favorite topic of NLP in 2023. And computer vision was their other one. And they said to find guests to interview on these topics, you might consider reaching out to researchers and practitioners working in these areas.

• 11:31Chris Benson?

  Very sensible.

• 11:32Daniel Whitenack?

  Some suggestions include Rachel Thomas, co-founder of fast.ai, Timnit Gabriel, co-lead of the ethical artificial intelligence team at Google, which is interesting that it gave that response because that is not factually correct anymore as she is not with Google. And actually, that was in the news quite a bit.

• 11:55Chris Benson?

  It was. That was a significant story in the AI world a few months ago.

• 12:01Daniel Whitenack?

  And then it gives a few others, including Jan LeCoq, who, you know, of course, we would love to have on the show. We'd love to have Rachel and Timnit as well on the show. But yeah, interesting. So a few things, I guess, that strike me as an example with this certain case is the output is definitely natural and coherent. Right? So that is thing one. That's striking. Thing two for me is there's actually a good bit of like structuring that goes on here. So they actually give, you know, one, two, three, four, the topics that we need to cover, and then a bulleted list of the people that we need to have on the show.

• 12:39Chris Benson?

  Yeah.

• 12:40Daniel Whitenack?

  Thing three is despite it being coherent and natural, it is not fully correct factually. Right? So that's maybe another element of of this.

• 12:52Chris Benson?

  You know, it's funny, because we've seen a fair amount of criticism about, you know, chat GPT getting things wrong and stuff. I find it curious that as we talk to humans about human things, we get things wrong constantly and fact checking and, you know, was that misinformation? Or was it just unintentional? And and yet we hold these technologies to such a perfect standard that we're ourselves completely unable to hold up. You know, I wouldn't want to ask one question and assume that it was 100% right, but sure, it makes it a little bit more interesting to me that collaboration. I dare say takes on a human element by having error in it.

• 13:34Daniel Whitenack?

  Yeah, and we'll talk a little bit later about the interaction between this and humans and where the burden lies. I do think that the interface that they've provided and being explicit about limitations. That's a good thing. Now, certain people might um kind of go back and forth on this model is not open access, right? Like you can sign up and create an account and a lot of people have done that and you can interact with it, but like the model waits itself and you know, it's not released publicly in that sense, even if a lot of people can use it for free at the moment.

• 14:12Daniel Whitenack?

  There's pros and cons there, but I think it's interesting that this model, as opposed to GPT three earlier, it was, I think, easier for the general population to interact with this model right away in comparison with GPT three, which, you know, had a very prolonged kind of waitlist and timing and all of that and lots of explanations. So it seems like that they've kind of shifted the scales a little bit in terms of making access to run the model more open while still maintaining it as a closed model and providing limitations.

• 14:51Daniel Whitenack?

  So it's interesting to see also that kind of shift in dynamics, which I think probably was influenced by the fact that, you know. Actual open access models like stable diffusion and others have taken off so widely so quickly because they are more open access wise. And so I felt like we saw OpenAI shift a little bit in how they released this while still kind of maintaining some of the elements of how they released GPT-3 and others.

• 15:21Chris Benson?

  I agree with that. Yeah, I mean, we've seen that kind of evolution as they've explored release approaches over time, you know, within iterations and such. I think one of the things that we've seen, you know, across this is the fact that every time a breakthrough comes on, we're starting to have fairly quick follow-up. But once people know that something is possible, they manage to kind of reverse engineer it. So I suspect that aside from strictly chat GPT, that we will see some fast followers pretty soon.

• 16:03Daniel Whitenack?

  All right, Chris, let's get into the technical details of this, which I know I'm excited to chat through. No, I guess pun intended in that case.

• 16:12Chris Benson?

  Oh, boy.

• 16:13Daniel Whitenack?

  There's kind of two elements of this that I think are important to talk about before we talk about what actually was done with ChatGPT specifically. And these two things are more general than ChatGPT. One is sort of the GPT family of language models and those types of language models, and then also a technology or approach called reinforcement learning from human feedback. Those two things kind of combined here to create the ChatGPT system. And these two types of models and approach have been applied more widely in

• 16:55Daniel Whitenack?

  other cases and by other people. But here they're applied by OpenAI. So starting to talk about this sort of language family model of GPTs, we had GPT and GPT-2 and GPT-3 and GPT-3.5 and I don't know what, to be honest, I don't know what number we're on now. But these GPT language models are just that. They're a language model and they're a specific type of language model called a causal language model. People might be familiar or at least have heard the words causal language model, CLM, or masked language model, MLM. So masked language model kind of takes a sentence and what it's trained to do is kind

• 17:38Daniel Whitenack?

  of for one word that's masked in the sentence or taken out or given a special token, it's trained to predict that based on everything else in the sentence. So it sort of looks both ways at the sentence and tries to predict the mask. GPT is not a masked language model. It's a causal language model, which means that it's trained to predict the next word in a sequence of words or in a sequence of tokens, whatever those tokens might be. It does that and it predicts the next word in the sequence, but it does it based on all of the previous words and it does that sequentially. So as you go through the sentence, the training methodology is what they call autoregressive.

• 18:22Daniel Whitenack?

  It means that it predicts the next thing from all of the previous things. And then once it's predicted that next thing, then it predicts the next, next thing based on all the previous things and then the next, next, next thing and et cetera. And that's the autoregressive part of it.

• 18:37Chris Benson?

  I suppose we're kind of seeing that in action because when you're using the interface, it doesn't just give you the entire output all at one time. It comes back with text. You see the text developing much as if you were typing it on the screen yourself. So I guess you're gradually seeing each of those iterations coming back.

• 18:57Daniel Whitenack?

  Yeah, and I think in the original GPT-3 interface or the playground that we both played with. You kind of see this as well. You kind of give a prompt and then it generates this text out and that allows it also to be very flexible, right? And produce these structures and also allows it to be flexible between different tasks. Like if you start prompting it with question answers, it sort of learns that pattern and in a sort of few-shot way and then starts predicting next question and answers or something like that. Or if you want a script, or if you want a narrative, or if you want something else, it kind of adapts in that few-shot learning sort of way, which is a key element of this

• 19:40Daniel Whitenack?

  GPT or causal language model structure. And GPT is not the only one. There's other ones, but this is the family which GPT sits in.

• 19:50Chris Benson?

  And you mentioned, just as a two-second sideline, you mentioned few-shot. Do you want to, real quick, just for those who may not be familiar?

• 19:56Daniel Whitenack?

  Yeah, so kind of some jargon, few-shot, zero-shot is thrown around. A zero-shot prediction or usage of a model means that maybe you're using a model on inputs or a type of input that it's never seen before, even though it's seen maybe similar things. So this happens with like machine translation models that are multilingual, maybe, because you might have in your training data like English to French and, you know, Arabic to Spanish, but you don't have examples, English to Spanish. But you have English and Spanish data in the data set and so you could still ask that model to try to output an English to Spanish translation and actually that can kind of

• 20:40Daniel Whitenack?

  work in certain scenarios. Few-shot means that you're not quite doing it that way, but you're providing a small number of prompts that kind of guide the language model into the type of thing that you're wanting to do. So in the GPT-3 interface or playground, if you remember, you can kind of start with a question-answer template and provide some examples and then you can provide the next one and it'll answer it for you. And so you provide that set of templates or prompts and this kind of gets into this idea of prompt engineering and that sort of thing because these models are so flexible. So that was the original paper from GPT-3 was titled something like, you know, Language

• 21:23Daniel Whitenack?

  Models or Few-Shot Learners or something like that. That was one of the big ideas there. That kind of gets us to GPT and language models. But ChatGPT, well, I guess it is a model like that. So it is a GPT-based model. But the reason why the system is so powerful is because it's a language model that has been trained in a very unique way. That has proved to be actually quite valuable. And that's that it is a GPT-based model that was trained using reinforcement learning from human feedback, or RLHF, reinforcement learning from human feedback.

• 22:06Daniel Whitenack?

  We'll link to this in the show notes, but there is a really great article on the Hugging Face blog from Nathan Lambert, Luis Costacato, Leonardo Von Vera, and Alex Havelria called "Reinforcement Learning from Human Feedback", and they talk about ChatGPT and other like models. So we're going to pull a lot of our insights from this article. So thank you to all of you for writing this article because it was really helpful, much more helpful than maybe the OpenAI blog by itself. The major idea here with reinforcement learning from human feedback is trying to answer the question,

• 22:48Daniel Whitenack?

  like, can we use human feedback on generated text as a measure of performance that goes beyond sort of just like automated measures of performance? So how do we integrate human feedback into the loop of training a model as a performance metric? And in that way, we're sort of training a language model, but we're also training it in ways that match human preference for answers. So human preference is a key piece of this. And I think that's why, you know, people like ChatGPT is. We prefer the things that it outputs, right? I don't know if that was the case for you, like with just a raw language model like GPT-3.

• 23:29Daniel Whitenack?

  You can get some cool stuff output, but it might not fit your preferences of like how a human would actually respond to something.

• 23:38Chris Benson?

  You know, going back to the example I mentioned in the beginning, that was the trick for me was, you know, like using the children's story as an example. I had a specific rough narrative in mind because I'm trying to teach and there are certain points that I'm trying to illustrate. And obviously it doesn't know that the model, but the model, if you work with the model being able to kind of, you know, continue to point it the right way. That was, that was very interesting. I, I'm curious, going back to what you were talking about a moment ago with, you know, the reinforcement learning with the human feedback. How does that scale? You know, because if we were to compare this for a moment, I know this is very much a kind of a newbie question, but for those of us who are not, you know, deeply into

• 24:20Chris Benson?

  language models, you know, when we were looking at other types of models a couple of years, two, three, four or five years ago, there was always a challenge about getting human feedback to scale with the amount of training data. How is that tackled in this approach so that you can do reinforcement learning that way, but it scales to, you know, what we're doing at GPT?

• 24:40Daniel Whitenack?

  There's actually like a whole loop of models involved here and different training sets that are of different scales and different models that are of different scales. So let me talk through a little bit of that and I, hopefully that will become more clear because yeah, obviously human feedback is expensive in terms of gathering it, right? So how much of this do you need? So there's three steps, the, the process with which chat GPT was trained and other models using this reinforcement learning from human feedback approach. It's a three step process. So you pre-train a language model, which is not new. We've been doing that for quite some time, right? You pre-train a language model, then you gather this sort of human preference data

• 25:26Daniel Whitenack?

  and train a reward model. Now the second reward model. Is trained to take in a prompt and a response and score it like a human would score it according to preference. It's actually trained on the human preference data and it outputs a prediction of what a human preference might be on this output. And then the third and final step is that you fine tune a copy of your original language model using this trained reward model and a reinforcement learning loop.

• 26:01Chris Benson?

  So is it just kind of the discriminator? You're using the reward model as the discriminator in that or?

• 26:07Daniel Whitenack?

  It's like in a reinforcement learning loop, you would have a kind of policy which outputs like what you should do next sort of thing. And then you have some type of reward system that rewards the agent for acting according to the policy or not. So in this case, the reward model is outputting that reward or that preference and the language model is actually acting as the policy here. So you have an original language model that is kind of your original policy and isn't fine tuned yet according to human feedback. Then you gather some human feedback, like actual human feedback, train a reward model

• 26:49Daniel Whitenack?

  to simulate that human feedback, and then you fine tune a copy of your original language model or copy of your policy with this reward model. And so the pre-trained language model, it could be any language model, it doesn't have to be a GPT-3, but in the case of OpenAI, it was GPT-3. But you have an original language model and that language model could just be a general pre-trained language model, or you could additionally fine tune that model and maybe for a domain or a specific type of output you want. So that's your pre-trained language model. And the step two. To get the reward model, what you do is you start outputting data from your original

• 27:34Daniel Whitenack?

  policy, from your original language model, and you have humans rate it. Maybe you combine that with certain human output or certain other outputs and you have human ratings. So that way, you're kind of, you're creating a training set for your reward model, which includes human labels of their preference. Then, also in this step two, you then train a reward model using that data that you gathered from humans to output the preference. Now, to your point of like, how does this scale? Well, the fine tuning of the policy is done kind of with this automated reinforcement learning loop. But you do need humans to

• 28:17Daniel Whitenack?

  generate enough data to train your reward model that's used in that loop. And what's interesting is, and the Hugging Face blog makes this point, is that different people or different groups that have applied this reinforcement learning from human feedback have used different sized reward models. And obviously, as the size of your reward model increases, you need more data to train it. That would be a general, general rule. In the case of OpenAI, their main language model was like 175 billion parameters, and the reward model was much, much smaller, 6 billion parameters. In other cases, people have done similarly sized models. And so I think that is an open question. Like, how

• 29:02Daniel Whitenack?

  should these models, size-wise, be related to one another? What types of models should you use for your reward model? And how much human feedback do you need? To be honest, I think those are open research questions.

• 29:15Chris Benson?

  Let me ask you another question on that. With us getting high-quality output that is comparable to human output very closely, and where if you were to get that output, you would find a very difficult time knowing whether it was the model or a human that did that, does that potentially go back in to train further reward models where you're using essentially synthetic data? As the output of a previously trained, and so you can build on it. And essentially, there's a point where you have enough data where you're largely able to take humans back out, recognizing it's the tool of the day. But in the future, you can take humans back out of that loop of providing the reward model to do that. Do you anticipate that that would be a reasonable expectation?

• 29:59Daniel Whitenack?

  I think in this methodology, the reinforcement learning from human feedback, one of the goals in that middle step is to get enough human feedback that you kind of reduce the harm and the helpfulness of the output model. So this is really addressing, I think, some of those kind of problems with large language models of hallucination and harmful effects, general output. And you can address those. What I think is the finding here is you can address those with humans in the loop rather than humans totally out of the loop. Now, here in the next step that we'll describe in the process, humans are taken back out of the loop to fine tune the model.

• 30:45Daniel Whitenack?

  But that central piece, so this three step process of starting with a language model on one end, ending with a reinforcement learning trained model on the other end, has this middle step that I think is a really key piece of it that actually helps the utility of the output and potentially reducing harm of the output, which is that human feedback piece.

• 31:19Daniel Whitenack?

  All right, Chris, we're about to the end of this reinforcement learning from human feedback loop. Just in summary, the loop is we have a pre-trained language model. Then we gather this human feedback or rating of the output to train a reward model. Now, we're actually going to use that reward model. So in the final step of the process, we make a copy of the original language model or the policy. So you have an original policy and you have a copy of the policy or an original language model and a copy of the language model. You put in a prompt to each of those models and then you get An output from each of those models.

• 32:03Daniel Whitenack?

  Then you use a sort of constrained reward function where you actually penalize if the updated model is straying too far away from the original model. Cause I think what they have found is, you know, if you allow it to sort of just take any direction in the output you want, it can have computationally some optimization problems. So you kind of gradually change this language model from the original and you have a penalty for how far that output strays from the original output. And then you score that output with this reward model that you've created. And the way that they're doing the updates for ChatGPT and some of these others with

• 32:45Daniel Whitenack?

  a reinforcement algorithm called proximal policy optimization, which you have sort of two levels of what in physics I would think of as adiabatic change, meaning like things don't change too quickly. One is you don't stray from the original policy output too much or you're penalized from that. And secondly, this reinforcement learning algorithm called PPO prevents you from making too big of updates to your model weights in each step. That way you don't have, again, this kind of hard optimization to do. But in summary, you kind of have these two models, the original one, the updated one. You output a prompt from both of them.

• 33:28Daniel Whitenack?

  Those go into your reward function, which includes a penalty element for straying too far from the same output. It also includes the actual estimated reward or estimated preference from your reward model. And then that reward is then used to update the weights of your copy model or your new policy using this PPO reinforcement learning algorithm. So hopefully there's some diagrams in the post that I think are quite helpful. It's a bit hard on a podcast, but hopefully that loop makes some sense in terms of how you're updating this. And this updated policy, this updated language model is the model that is used.

• 34:13Daniel Whitenack?

  So this is like the ChatGPT model that comes out of the end.

• 34:17Chris Benson?

  I think, given the limitations of our medium here, I think that was a very lucid explanation of translating it. So I appreciate that. I've definitely learned some.

• 34:26Daniel Whitenack?

  Good. I know formulas. You can ask ChatGPT to output all of the right formulas, and I'm sure it would do a fine job.

• 34:34Chris Benson?

  Where do you think we're going from here? Like, as you have looked at this progression of these models over that we've covered on the show over time, with ChatGPT in particular, it's been, I've been kind of amazed at what it could do and using it. But I'm really, really curious about where this is going. And I think it's capturing a lot of people's imagination in that way that are outside the field. Like, what's next?

• 34:57Daniel Whitenack?

  I think there is still open research questions here that are worth exploring. And then there's like workflow and practical implications, I think. On the first side, as was mentioned already, and we were discussing this reward model, as far as I can tell, it's not totally determined like what the architecture of this reward model should look like, how big it should be in relation to the model that you're fine tuning, how much human feedback should you use? How does the amount of human feedback that you get influence the harmfulness or the utility of the output and that sort of thing? So I think there's a lot to explore around that dynamic between the reward model and the the language model.

• 35:41Daniel Whitenack?

  In addition, I mean, language models are still being developed, right? So ChatGPT uses the GPT 3.5 language model as this original policy, right? And they actually used a fine tuned version of that using supervised methods and human chat conversations. So they started with a fine tuned version of chat or of GPT 3.5. So obviously, we're going to have a GPT 4, GPT 5, we're going to have other language models from other providers, right? From other research groups, you know, big science, or Google, or Microsoft, or whoever's developing these other language models, we're going to have updated versions

• 36:24Daniel Whitenack?

  of those. So I think we can see a research. Which direction with this where people are trying different pre-trained models as their original policy, where people are trying different reward models, where they're mixing them up in interesting ways, where they're maybe using slightly modified versions of the PPO algorithm or other reinforcement learning algorithms to do the updates. So there's a research direction where I think we'll just see a lot of exploration with this kind of template as the structure that they're exploring. The second piece, which is maybe more interesting to some of our audiences, like what are the implications of this in terms of people's workflow?

• 37:05Chris Benson?

  I was about to ask you that if you hadn't gone there.

• 37:07Daniel Whitenack?

  Yeah, I don't know. What are your initial thoughts there, Chris?

• 37:11Chris Benson?

  It's less about the technical aspects of the model and more about going back to the user interface considerations that we talked about earlier in the conversation. I would be amazed if the community at large, not just OpenAI, hasn't understood the impact of making choices like that. It may not be specific to the model development, but how you're putting it out there and they're seeing widespread adoption. When you go into their interface, you get a warning right off the bat, we're experiencing exceptionally high demand, please hang in tight as we scale our systems. And I think that's indicative of the fact that people who are not normally listeners of this podcast are starting to find a lot of utility for the first time ever. It'll be interesting.

• 37:54Chris Benson?

  We keep talking about exponential growth in this field and these amazing mini revolutions along the way, but this is that first point where it's probably going orders of magnitude broader in terms of applicability to different workflows and audiences. So, and as we're looking at, you're combining, just for a moment, going back and combining natural language with the large language models, with generative capabilities, with reinforcement learning, and we're kind of seeing, we saw slices of each of these fields over the last few years developing, and we've been talking about this fusion of the fields. And so how soon before we start seeing entertainment?

• 38:37Chris Benson?

  That is that is being heavily, heavily based on these technologies. I'm seeing it in my little tiny nonprofit because we can suddenly leverage this to put out content to help folks in a charitable fashion that we can do at least 10 times as much as we would have been able to before by taking advantage of these. And so I think we're at that inflection point now where this will be the first. And as we have continuing episodes through the course of this year and some new things come out, whether it's from OpenAI or or similar things from other organizations. I think we're getting to that point where it's really hitting broadly in real life. So I'm really fascinated.

• 39:19Chris Benson?

  I would love to hear from our listeners on ways that they're using this technology, what they think might come next and how they are envisioning using it within their own organizational missions to accomplish what they want. It's a fascinating moment in history of AI that we're in right this second.

• 39:36Daniel Whitenack?

  And one one thing which I can't claim as my own insight that I stole from Twitter, but I think has really shifted my thinking a little bit on this subject is. So this is actually a tweet from Chris Albin, who is the Director of Machine Learning at Wikimedia. And the statement he made, which I think was really insightful and maybe other people are having similar observations, but he said sci fi got it wrong. We assumed AI would be super logical and humans would provide creativity. But in reality, it's the opposite. Generative AI is good at getting an approximately correct output.

• 40:18Daniel Whitenack?

  But if you need precision and accuracy, you need a human, end quote. So I think the observation here is like, and we've talked about this on the show with language models also. Language models are really good at actually at naturalness, creativity, apparent coherence, right? Like that actually is what they're good at, but they get the facts and the precision and the accuracy wrong many times, right? So whereas I think in the past, people have thought the unique thing about what humans can provide in an AI driven system is creativity.

• 40:56Daniel Whitenack?

  Not logic and that sort of thing. Actually, the opposite is really the case, right? Like, the AI bits are really driving the creativity, and the humans are enforcing the logic, the facts, the accuracy and the precision. That has really shifted. Like, I think I've been realizing that over time, but that statement really put some words to, I think, what I was thinking.

• 41:23Chris Benson?

  It's comforting, in a way. And the reason I say that is we talked in times past about creativity coming from the humans rather than the machines. And yet, the evidence that we've been looking at over these last couple of years has been not that. And so, I have actually been wondering what role is there for the humans in that equation? So, the fact that it's flip-flop back, it's the inverse of what our expectation was, it still means there's room for a human in the picture. And that's a little bit of a comforting moment. It may not be what we thought it would be, but there's still a place. And I think that's probably a good high note to leave people with.

• 42:01Daniel Whitenack?

  On the note of things being useful to humans and humans getting involved, we did want to leave you with a few learning resources to explore things related to ChatGPT. Of course, play around with ChatGPT. You can go on the website and interact with it. We'll provide the link. But also, I would really highly recommend that you look at this Hugging Face blog about reinforcement learning from human feedback. There's actually a bunch of links in there as well to other things that you can kind of spin off and look at, like the PPO algorithm and other things in there. Also, there is a good reference. I always love looking back at Jay Almar's descriptions of how certain language models work.

• 42:44Daniel Whitenack?

  He has one on GPT-3 and other GPT, actually, a number on GPT different, from different perspectives. And then, there is an interesting article on GPT-3 architecture on a napkin from a blog, Dugas.ch. I found it quite interesting how they described some of the things there.

• 43:06Chris Benson?

  I like that one as well.

• 43:07Daniel Whitenack?

  Yeah, yeah. So. Go ahead and check those out. Those are great learning resources. They're all free. You can take a look at them and learn in more detail some of the things that we only had, you know, 45 minutes to talk about here on the podcast.

• 43:22Chris Benson?

  And our social media channels, I'm encouraging our listeners to share with us some of the ways they're using the technology. I'm really waiting to hear that. And, you know, the more unique, the better.

• 43:33Daniel Whitenack?

  Yeah, sounds great. Let us know what you're creating with ChatGPT. It's been a fun one, Chris. Good to chat with you.

• 43:39Chris Benson?

  Absolutely. Thank you very much for the incredibly lucid explanation. It certainly helped me understand and I appreciate it as always, Daniel. Talk to you next week.

• 43:48Daniel Whitenack?

  Bye.

• 43:58Speaker 01

  All right, that is our show for this week. If you dig it, don't forget to subscribe. Head to practicalai.fm for all the ways. And if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.