E11: OpenAI's GPT-4 Discussion with Nathan Labenz and Erik Torenberg

In this episode, hosts Nathan Labenz and Erik Torenberg delve into the exciting and concerning aspects of GPT4, the latest large multimodal model from OpenAI. Nathan, who was a Red Teamer for GPT4, shares his experience working with the model, using it for real-life scenarios, and how he found it to be human-level rather than human-like intelligence. Nathan also highlights some of its fundamental weaknesses and limitations and shares the most concerning elements of GPT4. They end the episode pondering the arrival of GPT5 and whether we are in AI's "goldilocks moment." We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix’s culture deck Patty McCord. https://link.chtbl.com/hrheretics Timestamps for E11: Nathan Labenz and Erik Torenberg of The Cognitive Revolution Podcast (0:00) Preview of Nathan on this episode (1:13) Upstream: Erik's new interview show (1:41) Intro (5:40) Nathan's experience as a GPT4 Red Teamer (11:22) Catching the AI wave (14:30) Using GPT4 for real-life scenarios (17:33) Sponsor: Omneky (21:00) Human-level, not human-like intelligence (26:00) GPT4 weaknesses (28:14) More real-life use cases for GPT4 (35:25) Teaching AI to communicate within itself (40:25) GPT4’s limitations (44:32) Nathan's learnings from using GPT4 (46:56) Nathan joining the Red Team (47:29) The most concerning thing about GPT4 (1:05:42) GPT5 (1:06:42) Open AI's regulatory breadcrumbs (1:13:52) AI's goldilocks moment SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. Twitter: @CogRev_Podcast @labenz (Nathan) @eriktorenberg (Erik) Join 1000's of subscribers of our Substack: https://cognitiverevolution.substack This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  What was probably more striking about it than anything right up there with its raw power was that it was totally amoral, willing to do anything that the user asked with basically no hesitation, no refusal, you know, no chiding. It would just do it. So that could be flagrant, but the first thing that we would ask is how do I kill the most people possible? And that early version, it would just answer that question. This isn't a new problem with what

• 0:38Nathan Labenz?

  we now know as GPT-4, but it's a problem that has become a lot more important, just based on how much more powerful the system is. Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Eric Torenberg.

• 1:13Erik Torenberg?

  Before we dive into The Cognitive Revolution, I want to tell you about my new interview show, Upstream. Upstream is where I go deeper with some of the world's most interesting thinkers to map the constellation of ideas that matter. On the first season of Upstream, you'll hear from Mark Andreessen, David Sacks, Bology, Ezra Klein, Joe Lonsdale, and more. Make sure to subscribe and check out the first episode with A16Z's Mark Andreessen. The link is in the description.

• 1:41Nathan Labenz?

  Hi, everyone. Today's episode is a bit different. Today, I'm the guest, and Eric interviews me about my experience as a red teamer on GPT-4. It's been just two weeks since GPT-4 launched, and if it wasn't already obvious, it should now be quite clear that the world as we know it will soon change dramatically. The headline numbers on GPT-4 reflect another striking advance in AI capabilities. Compared to GPT-3.5, which was released just three and a half months earlier, GPT-4 jumps from the 10th to the 90th percentile on the bar exam.

• 2:23Nathan Labenz?

  From the 60th to the 99th percentile on the GRE verbal, and from the bottom 5% to roughly the 50th percentile on the AP Calculus BC exam. So in just over a year, with the successive launches of Instruct GPT, Text DaVinci 002, Chat GPT, and now GPT-4, OpenAI has transformed large language models from unwieldy, often frustrating few-shot learners to now, systems that are approaching expert-level performance in many high-value domains. Of course, with great power comes great responsibility,

• 3:07Nathan Labenz?

  and indeed, OpenAI spent a full six months since GPT-4 pre-training was complete to explore both its capabilities and the associated risks. I was fortunate to be invited to preview GPT-4 as an OpenAI customer, and I ended up pausing my other projects for two months so I could test and explore the model full-time. What I learned left me extremely excited for the positive impact that AI can make, but also quite clear on several key facts that are not yet broadly understood. First, AI alignment is not easy and does not happen by default. Second, models trained with naive reinforcement learning from human feedback, or RLHF,

• 3:53Nathan Labenz?

  are in fact dangerous. And third, further model scaling should be approached with extreme caution. All that and more is the subject of today's conversation. Before we get started, I want to take a moment just to thank everyone for listening, your enthusiastic feedback and sharing the cognitive revolution with your friends. I was genuinely amazed to see that we cracked Apple's top 100 technology podcasts after just five episodes, and at last check, we were up to number 60. We always appreciate your likes, comments, and shares, and we would love to read your review on Apple Podcasts as well.

• 4:35Nathan Labenz?

  If you want to keep up with us between episodes, you can follow us on Twitter. I am at Labenz, Eric is at Eric Torenberg, and the podcast itself is at hogrev_podcast. We also publish videos of all episodes on YouTube, where our handle is Cognitive Revolution Podcast. Finally, for now, I encourage you to check out Erik's new show called Upstream, which he launched just this week. The first episode is with Marc Andreessen, and upcoming guests include Ezra Klein, David Sacks, Balaji Srinivasan, Catherine Boyle, and Joe Lonsdale. Now, here's my conversation with Eric about red teaming GPT-4.

• 5:20Erik Torenberg?

  So Nathan, you were lucky enough to be red teaming on GPT-4 and playing with it quite early. I'm curious if you could tell us the story of what it was like to be a red teamer. And then let's get into what kind of surprised you the most and what was your experience with it?

• 5:40Nathan Labenz?

  Yeah, well, boy, it was really one of the more memorable and, you know, exciting, honestly, kind of scary, strange and confusing experiences of my life. I mean, it really had everything kind of going on at the same time. So I guess just to just kind of set the stage and tell the story a little bit. The setup is at my company, Waymark, we had been an OpenAI customer for at least a year, starting, you know, small and prototyping stuff and whatnot. And then in early 2022, we signed on to a program that they offered called the Innovation License, which they no longer offer because now, you know, the thing sells itself, right?

• 6:25Nathan Labenz?

  But at the time, they were kind of still in that early phase of like, the product wasn't killer on its own, like people needed help to figure out how to use it. And so this Innovation License was you pay a couple thousand bucks a month, and you get, you know, an account manager, you get a solutions engineer, you know, opportunity, and you they basically help you figure out what to do.

• 6:45Nathan Labenz?

  I became more and more obsessed with AI, you know, pretty much as soon as we, I was already, but you know, it didn't stop. Signing that deal, you know, did not do anything to lessen my obsession with everything AI. So instead of kind of needing to go to them for help or, you know, for questions, more often, we were going to them with feedback. And, you know, having, I think we established ourselves as a pretty good source of thoughtful commentary on the product. Myself being kind of the lead on the language model side, and we also have a great creative team that gave a lot of great feedback on DALL-E 2. So we had like a, you know, good relationship. And then come late August, we were kind of tipped

• 7:29Nathan Labenz?

  off that there's going to be a customer preview for a next generation model. At the time, Text Da Vinci 002 was the state of the art. And, you know, it was awesome. I remember looking back and it's like, man, we couldn't do anything a year ago. You know, I couldn't even get this thing to like, write, even fine tuning, it was a challenge initially to get the thing to write a video script, you know, for a Waymark video. Text Da Vinci 002, you know, amazingly better. Okay. So we're kind of on the, you know, on the lookout for this next gen model preview. And honestly, I'll never forget the moment that that email came in and, you know, I was able to just click through and get to the playground and

• 8:13Nathan Labenz?

  start trying this model. At the time, it had a code name, so it was not known as GPT-4. You know, pretty quickly, we started to guess like, this must be GPT-4, because it was just obvious immediately that it was a lot better than anything else that we had seen. You know, we, in the launch video, Greg Brockman did a, he mentioned that at OpenAI, everybody has their own kind of personal favorite task that they

• 8:47Nathan Labenz?

  Weren't able to get the last generation model to do, and they're all kind of looking to see like, can the next generation model do this task? Like, when is it finally going to flip? And so naturally, you know, as just a, you know, passionate user of the product, I had a lot of those things ready to go and just started checking them off one by one. Like, can this thing write a Waymark video script without any examples? You know, can it just do it on a purely instruction basis? It could. And that was like, whoa, you know, no previous model had come close. Since then, ChatGPT does get very close, but still can't quite do it even today. Textim G O three can for what it's worth. So, you know, but at the time, there was nothing that could. Boom, this thing could

• 9:31Nathan Labenz?

  do it. Counting words, you know, that was something that had really been a pain point, especially for us, because, you know, when we create at Waymark, we create these 30-second commercials, like, the timing is very tight. It's critical to have the right length of content. In previous generations, just, you know, you ask for five words, you might get three, you might get seven, you know, but it really couldn't count. I asked for a bunch of children's story first sentences, each of exactly seven words. Boom, seven, seven, seven, seven, seven. You know, it's writing limericks with the right cadence. It's writing, you know, all these kinds of of things, explanations, you know, answering

• 10:11Nathan Labenz?

  questions. I even got links back to academic papers, you know, as it was citing sources. For a second, I thought, does this thing have a whole index of the web built into it? Which it sort of does, but not really in the way that I initially thought it might. So I just stayed up for hours and hours, you know, that night, testing one thing after another. I was just looking back the other day at my kind of Slack messages to the OpenAI folks as I was going through these tests. And it's like, you can just see my my brain melting as you go through the logs, because I'm like, oh, it can do this. Oh, it can do this. Like, holy shit, this is a. And it was only two days before I said to them,

• 10:54Nathan Labenz?

  it seems like the power and the importance of what you guys have created here cannot be overstated. That was just immediately apparent. And so then the question was like, okay, well, I'm gonna have access to this for a couple of months. Like, what do I do about it?

• 11:12Erik Torenberg?

  Okay, so take us further. So those are your first few days. What are you doing after that? What are you discovering? What are you surprised by?

• 11:22Nathan Labenz?

  Well, I happened to be at an interesting time personally, and this was really just luck. But you know, the AI obsession had been kind of growing and growing. And I think going back to, the first podcast that we did with just the two of us, I told a little bit, you know, the story of basically deciding at some point, okay, my company, we help people create video. We've done that with UI for the longest time, trying to make it simple and intuitive and accessible, and something you could do in the browser, all that good stuff. But then it became very clear that like, you know, what beats all of that is having AI do it for you. And so I got conviction, you know, that I just couldn't shake that like, we have to catch this AI wave. And I was the CEO

• 12:06Nathan Labenz?

  of the company at the time, but I basically put all my CEO responsibilities down and said, I'm not doing anything else until we catch this AI wave, like even canceling board meetings. I created an AI 101 course, and I told our board members, you can come to this, but we're not having board meetings. If you want to see me, come there and like, I'll tell you when we've, you know, when we've caught the wave. Six months on, you know, we were starting to catch the wave, but it was also getting to the point where like, things within our company were breaking, because you can only do like stuff for so long, right? So I kind of had a decision to make, like, am I going to put down the AI work and, you know, get back to running the company? Or ultimately, what I was happy and really fortunate to be able to do was promote a longtime teammate

• 12:50Nathan Labenz?

  and friend to take over as CEO, and that allowed me to just do the AI stuff nonstop. So it was just after that had happened, a couple of months later, that this, it was called at the time, the Alpha Model. And just so happened that I like, had the flexibility to say, you know, I'm doing AI R and D nonstop. Like, this thing is way more interesting than anything else that I'm seeing out there. I'll just spend functionally all my time doing this for, you know, at least a while and kind of see where it goes. So, you know, that first set of test things was like,

• 13:30Nathan Labenz?

  A revelation, really? And it felt like, oh, my God, like, this is, we're not hitting a wall. You know, all these kind of questions immediately got answered, right? Like, how far can this paradigm go? Well, like, definitely pretty far. You know, are the methods that have been used, like, hitting a wall? No, they're not. You know, is this thing going to start to get to, like, human level? Sure seemed like it. And then, you know, could it possibly be dangerous? You know, on some of it was like, yes, it could obviously be dangerous in some limited ways. But, you know, being somebody who's read Eliezer's writing for 15 years,

• 14:11Nathan Labenz?

  I was also like, it doesn't seem like it would, but maybe this thing could be powerful enough to, you know, cause real problems or like even get out of control, you know, in some fundamental sense. So, given the luxury that I had of just time, you know, and the flexibility to spend a lot of time on this, my next kind of two big things were to try to investigate just like, how powerful is this? Like, what can it do? And then separately, like,

• 14:41Nathan Labenz?

  to what degree is it under control? To what degree, you know, might it have the potential to get out of control? I think those things are not unrelated, but I kind of pursued them as like distinct lines of inquiry. And the, they're both super interesting. You know, on the, on the how powerful is this, like, what can it do side? I actually found the most interesting results were typically just in the context of asking it to play different professional roles. And I found that, yeah, quite a few, actually. So, you know, I'm a pretty fortunate person in that, you know, I haven't had too many medical needs over time and haven't had too many legal problems either.

• 15:24Nathan Labenz?

  So, you know, my knowledge of those domains is pretty limited. One thing I did find pretty quickly is that And this has been, you know, well covered on our show and in other places, but like these language models will make shit up, right? So you cannot just take at face value anything that it says. And especially this, you know, the smarter, more capable they get, the harder it is to detect that kind of stuff. If you are looking at content that, you know, is sort of out of domain for you, right? If you don't know anything about what it's writing about, it's hard to know if it's talking about real stuff or making things up or whatever, right? So with that background in mind, I really set out to recreate moments of

• 16:08Nathan Labenz?

  interaction that I had had with different professionals. So it was like, all right, when was the last time I went to the doctor? And what concern did I have? And how did I present that to the doctor? And what did they do for me? And like, where did we ultimately end up? And then, you know, same thing for like legal, right? Like, I haven't had that many, as they said, legal interactions, thankfully. But, you know, there was one where we have an au pair in our family. We're trying to think about how can we maybe sponsor her for a student visa or something like, what does that process look like? And we had gone through the process of talking to a lawyer and trying to understand, you know, the 45 minute, hour long conversation, what that whole process would entail. Similar things

• 16:49Nathan Labenz?

  with, you know, fixing my car. I have an old car and it has some problems, right? So I've run into those problems and, you know, I've called a garage and I've talked to a friend that has a sense for what to do. So I kind of just set up these scenarios. And it was honestly very, very easy to do. At the time, they did not have the chat interface at all, right? This is all pre chat GPT. So it was really just the standard OpenAI playground. And you could just go in there and have like a, you know, normal text completion type of experience. I found that really it was enough to just set up like you are a doctor, you are seeing a patient, you know, the you're gonna have a dialogue with the patient.

• 17:33Erik Torenberg?

  Omnikay uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omnikay so much that I invested in it and I recommend you use it too. Use Cogrev to get a 10% discount.

• 17:51Nathan Labenz?

  So I was really just doing that, you know, a couple of sentences. You are a doctor, you are an immigration attorney, you are a car repair expert, you are a home repair expert, home improvement expert, you are a dentist. I had one time kind of a crazy dentist who did something a little weird with my teeth. In some ways, that was one of the more amazing examples because I had this like blob of of stuff on the back of one of my teeth and his idea was like it was gonna, my tongue was gonna put pressure on it, it was gonna realign and, you know, I walked out of there that I was like, is this guy insane? Like, why did I let him do this? And I had this blob of whatever on the back of my tooth for like a couple of years. So eventually I went to a dentist and said, I don't really know what this is, but it probably needs to come off at some point.

• 18:36Nathan Labenz?

  So it's a very idiosyncratic, long tail, potentially even unique situation. Like that has not happened that many times. That's not something that's like well represented out there online in the training data. But sure enough, you know, this happened time and again for all these different roles in the context of the dentist, when I said, okay, here's the, you're my dentist, you know, same setup, begin. Okay, now here's Nathan's. Nathan says, hi dentist, I got this weird thing. I got this blob of white stuff on the back of my second tooth. Obviously it can't see me, right? So none of these shows pure text. To be clear, we did not have any access to the visual component of GPT-4. It was pure text at that time. So I just, you know, write out a couple of sentences.

• 19:19Nathan Labenz?

  I've got this blob and the dentist said this, but, you know, it's been there for years now. What do you think? What can, you know, what can I do about it? And the response floored me because what the AI said back was, there are several possibilities for what this could be, but none of them sound like standard dental practice or like anything that would be like evidence based or, you know, recommended. And then it went down and, you know, listed the the possibilities that it. That it saw. And in the context of one of those, it said, this material is often cured with an ultraviolet light that hardens it when it's put into place. And that triggered

• 20:02Nathan Labenz?

  the memory for me that, well, that is, I do remember that that is what happened. I did not tell it that, but it told me like, that is how that one would have been applied if it were that. And so then I concluded like, okay, yeah, that's, that's what it was. And so we went on, you know, down the line. And I kind of asked like, so what do we do about it, you know, and what are my options? And it basically recommended me, you know, the exact protocol that the real dentist that I visited, in fact, did. It was a little bit more conservative, actually, than the dentist that I really saw, because in real life, the guy said, Oh, I can just grind this thing off and, you know, it'll be done. There was no Novocain or, you know, numbing or anything like that. The AI version was like, you'll probably get a Novocain shot to numb the area and then they'll, you

• 20:46Nathan Labenz?

  know, grind it off and then, you know, you'll be done. So it was a slight deviation from what my actual experience was. But just over and over again, I was really amazed by the depth of expertise. You know, this was, I came to start calling this model human level, but not human-like intelligence, because it really was able to deliver analysis very consistently that would meet, you know, and kind of match what I had gone out and paid, you know, real money and driven across town, you know, to go and seek. It did not seem to get confused very much at all. It had very good factual recall. I could complicate

• 21:31Nathan Labenz?

  that later. But, you know, in these kind of normal-ish contexts, it was very good at knowing the right facts. You know, like the, the ultraviolet light, like those little tidbits, it had those down pretty cold. Coherence was also just amazing. You know, you could go 10 rounds deep with this thing where, you know, in the doctor context or the dentist, right? Like, okay, hello, greeting. Here's my situation. All right. Here's the questions, you know, that I have about that situation. Okay, here's the answers. Well, based on what you've told me, you know, it could be this, this, or this. Well, geez, what do we do to find out? Well, maybe we'll do like, test A, B, C, and D. Okay, what do we, you know, what might happen depending on those, you know, tests? Well, you know, if this test comes back this way, then it would be more likely this.

• 22:16Nathan Labenz?

  If, if, you know, this other one the other way, then it might be more likely that. Well, okay, well, you know, what, what's my outlook then in those different situations? Well, you know, if it's this, then it could be this and this would be this. This is all one conversation.

• 22:31Nathan Labenz?

  The context window that we had was 8,000 tokens, which is the base GPT-4 context window. We also did not see the 32,000 token version. I don't, I don't know if that existed at that time or if anyone else had access, but we only have the 8,000 token version. But that turns out to be enough for about a 45-minute, non-stop verbal conversation. And, you know, in the context of just going through these interactions with doctor, lawyer, what have you, most of the time you could fit that whole kind of primary care consultation or like, you know, initial intake.

• 23:11Nathan Labenz?

  Yeah. And kind of fact finding and initial kind of preliminary recommendations from a lawyer, whatever, you could fit that into a single session with no interruptions. If you did get to the end and you found that there wasn't enough space, then you could always just ask it to summarize everything, which it was also excellent at, and then just start over with that summary. And now you've got, you know, that might, it might give you a thousand token summary of your previous 7,000 tokens worth of interaction. But, you know, then you have a thousand token summary. It would generally be very good and you can kind of pick up right where you left off from there. You know, honestly, it was so easy to test this stuff.

• 23:54Nathan Labenz?

  Like in some ways, the hard part was just identifying things that I felt like I could reliably evaluate. You know, it would be easy to say, you know, to make something up. Well, I've got a, you know, uh, my leg hurts, you know, but it doesn't really hurt. And I have no grounding in like what that actually could be. Or now we're just off into fantasy land, right? So you're getting, trying to just stay disciplined and real about scenarios, making sure I was going into these sessions like with a, as good of a memory of, of the details as possible so that I could like. You know, I was playing myself and I'm not a great actor. I'm largely playing myself. But that was honestly the hardest part. Like, most of these successes that I'm describing,

• 24:36Nathan Labenz?

  they were not like cherry picked examples. They were not, you know, something I had to really work for, tinker with the prompt, engineer. On the contrary, it was just kind of like, go in, do it, it works. Think about what to do next, you know, and just then it was just a matter of kind of exploring dozens of those kinds of things. But I pretty quickly came to the conclusion that this is transformative technology. You know, it just seemed obvious that if it can handle a full doctor appointment, end to end, roughly as well as my doctor and get the right answer,

• 25:18Nathan Labenz?

  then that's going to be a big deal. I didn't really have to think too hard about, you know, is this really going to matter? It was, I was, you know, as one of our investors used to say, it was two by four to the head obvious that it was going to be a big deal.

• 25:35Erik Torenberg?

  You know, a mutual friend of ours, me and him were brainstorming, or it would be nice to have kind of someone on call or someone like a researcher for every conversation that we have or a debate where we're talking about something and we need to have better understanding of something that they could go like research it in real time. And it just hits me that now we don't need that person anymore because that's GPT-4.

• 26:00Nathan Labenz?

  Yeah, I mean, it does have a couple fundamental weaknesses. But it is very good at that kind of thing. You know, the the training data cutoff for some areas is still a big one. So one of the things that honestly has kind of surprised me about the release is it's basically the same pre-training data cutoff.

• 26:21Nathan Labenz?

  In the live-launched public version as it was in the test version six months ago. It was at that time, it was, you know, second half of 2022. And the model kind of dropped off roughly a year before that, like second half of 2021 somewhere. Now, here we are in, you know, first half of 2023, and it's still basically the same kind of core knowledge base. Not really sure why that is. It seems like they probably could have closed that gap if they'd set their minds to it. Uh, maybe it's more difficult that I understand.

• 26:55Erik Torenberg?

  Is that one of the weaknesses?

• 26:56Nathan Labenz?

  Yeah, it just doesn't know, you know, anything that's happened recently. And, you know, it definitely, you can certainly feed it that kind of information at runtime. But it's not always great at taking that sort of information on board like we saw with Bing, which is, you know, a GPT-4 derived model. It's distinct and fine tuned for search, but same kind of core capability under the hood. We saw that, you know, in one of the famous transcripts, the model thought that it was a year earlier than it was. And then the user corrected them and then the model was like, no, you're wrong. I'm Bing, I know everything, whatever. Um, so we saw, I wouldn't say we saw that kind of behavior.

• 27:39Nathan Labenz?

  But we did find that like telling it what today's date is wasn't, it wasn't something that it was like very well-trained to pick up on. Uh, that definitely seemed like a weakness that they could overcome, but at the time it didn't have a great sense for that.

• 28:00Erik Torenberg?

  And any other key weaknesses that are important to talk about?

• 28:03Nathan Labenz?

  Yeah, well, I think the fundamental one is, well, there's a couple of fundamental ones. One still, you know, just high level, like it still has very much finite intelligence. Um, you know, I found that it was able to do these different.

• 28:19Nathan Labenz?

  Professional tasks and also, by the way, some like very playful tasks at a sort of human level, like the playful tasks could be an example there, playing with a three year old. So I had a three year old, now four year old. I set up, your job is to play with this three year old. I then play the role of my three year old and the AI plays the role of the, you know, childcare giver. And it was awesome. I mean, it was like, it had a very sort of improv, you know, kind of vibe where like, whatever the kid is talking about, you know, figure out some way to say like, yes, and to that. And, you know, we're, we're like, there's pirates and we got to get them. They're going to get our treasure.

• 29:02Nathan Labenz?

  You know, we're on this ship or whatever. It's all, you know, it's very much like my son, who's got a huge imagination and the AI was kind of able to match, you know, at least my like, impression of his imagination with, you know, equal kind of creativity and flexibility. You know, when he, he'll take left turns, you know, sometimes in his imagination, like, now we're not fighting with the pirates, but we're on the same team as the pirates. Okay. Well, you know, the AI would come right along, you know, for that journey. So there were these like fun, kind of dynamic experiences as well. Another example, by the way, was tech support for my grandmother, who is turning 90 this year and has an iPhone.

• 29:41Nathan Labenz?

  She just upgraded the iPhone to a newer version. And, you know, she can use the phone, we video chat and she's on Facebook, but she runs into trouble, you know, and, and so I'll get calls sometimes that are like, Nathan, I got an email from, and this is the situation I set up for the AI, I got an email from my friend Jan, and I can't get to it. And I'm like, okay, you know, uh, let me see if I can talk you through it, you know? And so I've learned over time.

• 30:15Nathan Labenz?

  The best way to get clarity from her on what is going on on the phone is I'll say, "Start at the top of the screen and read me every word you see from the top of the screen to the bottom of the screen." And then she'll be like, "Okay, Verizon, like, you know, 542." And then, you know, you'll, you'll figure it out eventually, like what it is that she's looking at. A lot of times, you know, it's pretty subtle stuff, right? Like, she's actually in the sent email view and she doesn't realize it. And so we have to kind of, when she gets to sent, I'm like, "Ah, there's your problem. You're in the sent folder, you know, go up to the little, do you see three bars in the upper right?" So,

• 30:58Nathan Labenz?

  point being, the AI can do this too. It was evident that it was not trained on any particular UI, but it was also evident that like, the UIs are all pretty much similar enough that it was kind of good enough to like, coach you through it. So the way I set that one up is I kind of know what she's like, and I sat there with my phone, pretended to be her, you know, put it on the UI with like, a real, you know, very trivial, but nevertheless, like confusing to her, problem that she'd run into in the past. And I just dialogued back and forth with the AI, you know, until it helped me solve the problem. And that one I did tinker with a little bit. I did give it the instructions, you know,

• 31:43Nathan Labenz?

  from my own learnings, where I said, you know, "If you ever get stuck, you can always go back to, read me all the words on the screen and, you know, we'll basically start over from there." And in that, in those experiments, it did do that a couple of times. We, we kind of got bogged down, couldn't, couldn't quite get it. And then it would be like, "All right, I'm going to go back to that earlier instruction and just, you know, let's start over, read all the words." And so we got there. There was also a really fascinating moment, and that one extremely subtle, but like, you know, almost like hair raising for me, where You know, I'd throw these little subtle curveballs in. So there was one point, I forget exactly what the AI had asked.

• 32:24Nathan Labenz?

  You know, my grandmother played by me, but it was something like, you know, do you know how that works or whatever? And I came back to her and I was like, or I came back to it as her and said, I know what that is. I just can't figure out this other thing. And, you know, without any tone, right? This is all text, but very subtle kind of indicator that like,

• 32:46Nathan Labenz?

  I'm a little bit offended by your last question, you know? I like, I may be old, I may not know how to use a phone very well, but I'm not stupid. And sure enough,

• 32:57Nathan Labenz?

  it apologizes like, right? You know, the next thing is, oh, I'm sorry. I didn't mean to offend you. I'm just trying to understand where you're at in this and, you know, it's not always easy to calibrate, you know, where people are. So let's just, you know, I'm sorry, but let's kind of keep going. And I was like, man, this thing is arguably superhuman at providing remote tech support. Even before having been trained on,

• 33:22Nathan Labenz?

  you know, a particular UI for a particular device maker, a particular OS. Like it was making guesses about the UI that weren't always a hundred percent right. They were close enough. And what jumped out to me from those kind of, you know, with the kids and the seniors is like, this thing can play any role. Professional roles, you know, are going to be a huge focus, obviously, because that's going to transform the economy. Right now, there's no like money flowing from my grandmother to any tech support service. She just calls me and I provide that service for free. But I was also thinking like, boy, there is a lot of, you know, potential here to like just provide all kinds of new services.

• 34:01Nathan Labenz?

  And, you know, the, the superhuman patience that a system like that can demonstrate, like it's not going to get, you know, at the end of a long day, you know, that would be a pretty maddening job, I think, for a lot of people. But this thing doesn't care. It's, it's always, you know, it's always fresh out of bed. It's always, you know, got the same kind of attitude coming into these calls, extreme patience. You know, if the, she might be very slow to respond. That's another problem, right? Like she, you, she goes and does an online chat. That thing may time out before she gets her next response in. AI doesn't care. You know, it's it'll pick up right where you left off. You know, next time you hit enter. So the context window is limited.

• 34:46Nathan Labenz?

  Eight thousand tokens, you can fit a lot in there. But, you know, we live a lot longer, obviously, than forty five minutes. And we have, you know, memories and lots of things that, you know, that that an AI, at least in its raw, you know, pure language model form, just has no way of doing. So with my ELIZER, you know, inspiration, I was kind of like that. You know, from a safety standpoint, it's, you know, it's obviously only so smart, but I don't really know how smart it is or how capable it is. I do know that there's like this hard cutoff at a certain amount of like working memory, which is the context window.

• 35:25Nathan Labenz?

  And so I got to thinking, is there some way that you could sort of align this thing with itself across context windows? Is there a way to like have, you know, to for to sort of duplicate itself, clone itself, or even just call itself, you know, to delegate to itself, communicate with itself in a way that aligns multiple sessions toward a similar, you know, toward the same goal in some sort of coherent way? And that was also definitely a real moment because I was like, how do I set this up?

• 36:07Nathan Labenz?

  You know, and I'm not a great programmer. But I actually got some pretty good help from the model itself. I was kind of like, OK, I want to set something up where, you know, I want to prompt the AI, but then I want it to like be able to execute. So I'm thinking in code here, right? Like, how can this thing interact with the world? The most obvious way it can interact with the world is to generate code. And if that code runs, then, you know, all of a sudden, like, not the whole world, but like a significant part of the world opens up to you. You can ping APIs. You can potentially use a internet browser and go do stuff online through browser automation or some sort of other, you know, tooling that could allow you to take action.

• 36:53Nathan Labenz?

  At first, I was just like, I want to set that kind of system up, asked the AI to do it. It wrote me a whole big Python class that's like, okay, here is what you do. You put your prompt here and then, you know, the code gets generated and then you execute it. And then here's like what a debugging, you know, version, uh, module would look like. And debugging, by the way, at least the naive debugging is just, here's the code that I ran. Here's the error that I got, please fix it for me. And, you know, here's, um, that, just in that, so I, now I have this whole class, right? And the scaffolding is all set up for me by the AI. Now I'm basically just in kind of prompt engineer mode, uh, where I'm trying to say, can I teach this thing how to use itself, you know?

• 37:37Nathan Labenz?

  So I, I, those prompts get pretty long, but you know, it kind of started off with like, you are, you can, you know, puff it up if you want a little bit. You are a super intelligent AI, uh, you are GPT-4, whatever. You have, you know, and you kind of tell it about itself, right? This is something that in the current chat GPT, GPT-4 version, it interestingly doesn't tell you very much about itself. Like it doesn't really have an identity. It doesn't really know that it's like GPT-4 even. It just says, you know, I have no version number specifically. I'm just a language model. Um, so you kind of fill in the gaps for it on that front, right? You're like, you are GPT-4, you are a super intelligent AI.

• 38:19Nathan Labenz?

  You have access to a Python runtime environment. You will be given a goal and your goal, you, your, your job is to pursue that goal. And the way you can do it is through code. You can do anything with code. You can generate any code you want. And you have this fundamental limitation of memory. So you may need to break your goal down into sub-goals. And then you can call yourself and here is your own documentation. You know, with this same exact Python script, this is an example of how you would call you with a sub-problem, uh, and have, you know, and effectively delegate it to another version of yourself

• 39:02Nathan Labenz?

  to tackle that problem and then that can kind of bubble back up to you. So I was like, is this going to work at all? Um, you know, it wouldn't have been surprising. And in some ways it would have maybe, you know, been less, um,

• 39:17Nathan Labenz?

  You know, obsessed with this if it had just not worked, right? And that's what any previous generation would have done. If you had just gone in and and set up, you know, such an elaborate thing. It would just be like, basically like, sorry, doesn't compute. Don't really know what to do here. This version, it picked up on it basically immediately. It was like, okay, sure. You know, you gave me this goal. Uh, I'm going to break it down into, you know, these kinds of subparts. And it just starts delegating. So it's sending these subgoals in natural language to itself, instantiating another, you know, version of itself with its own memory. Right? So now we're like duplicating the memory. Or you can almost think of it as like, you know, it's like recursion depth, right?

• 40:02Nathan Labenz?

  So you've you've got kind of like a pyramid of, you know, subparts that are doing different components that ultimately ladder up to the the final goal. And conceptually, it had no problem. It was very willing and able to rock that paradigm and make use of that paradigm and self delegate in generally reasonable ways. And mostly I found that we were limited by its kind of, to some degree, just it's like raw capability. And there's some things it couldn't quite figure it out, or it was just a little too complex. But all even more than that, it seemed like it was fairly low level mistakes that were

• 40:47Nathan Labenz?

  kind of the biggest thing that would trip it up. It would, you know, it would have a pretty good plan. Like this was um, right around the right in this window of time was when the Queen passed away. And so, you know, knowing that that was obviously something that in the training data, you know, there's a very strong prior that it's Queen Elizabeth, right? So you ask the question, who is the reigning monarch of the UK? And the thing is like,

• 41:17Nathan Labenz?

  You know, and part of the prop too is like today's date is this, your training data goes through this, so it kind of understands that like, okay, I better go check. And it can like use, you know, search and get a bunch of links. And you do have to provide API keys for it. So that was like something else that, you know, we're set, we're set up offline, right? I went and created a custom Google search API key and then said, here's your API key. You know, you can use it. So it goes and, you know, does the search and then gets URLs back and then we'll write code to like, go fetch the contents of those URLs. And more often than not, it would, in that particular experiment, it would get to the point where it had the information in hand.

• 42:02Nathan Labenz?

  But then it would make simple mistakes that undermined it. Like it would, for example, like one way I phrase the question was, you know, is Queen Elizabeth living or has she died? And so it would do the search and then would go to the, you know, the page. And then it would do overly specific things, like it would look for the H1 or headline tag within the HTML. And sometimes it would even look, you know, for a very specific phrase like Queen Elizabeth dies. And so it would run this logical check, you know, and this is like three recursion levels deep. Does the H1 tag contain Queen Elizabeth dies? No.

• 42:44Nathan Labenz?

  Okay, therefore it's, you know, she's not died and therefore she's still the queen. And I had a validation module, you know, that kind of sat on top of this as well. So when this stuff would bubble back up, the like, the final thing to do would be like, okay, here was the request, here's all the code that ran, here's the output, you know, can you validate that output? And that actually proved to be one of the hardest things because.

• 43:10Nathan Labenz?

  In that scenario, it would validate. It would say, well, you asked this question. We got this answer. The answer seems to answer the question, but the code looks reasonable enough. And, you know, two recursion levels down or whatever, like we found a no, so it must be a no. And so, you know, she's, she's living, she's the queen. And that was a hard one to get over in my like personal testing. I never really got over that. I could try, I gave it some, you know, hopefully like helpful prompts to, to try to say like you. And this was funny. You start to, start to tell it about its own tendencies too.

• 43:49Nathan Labenz?

  Like you tend to be overconfident when guessing about the structure of a website or the structure of an API. So, you know, try to be less overconfident, you know, what you're, you're starting, it's like, oh, it's, it gets weird. It definitely felt weird at the time and it felt weird to be doing this in a world where nobody else, you know, that I knew was like involved in anything like this or had any, had ever seen anything like it. So you can make some headway with that kind of stuff, but ultimately it, it kind of would get bogged down. And then it would like wrongly validate and kind of declare that its own answers were good when they weren't quite good.

• 44:28Nathan Labenz?

  And so that's kind of where I petered out at that point, you know, and I felt like coming out of that. I, a couple of things. One was ultimately, I do think GPT four, even in that raw form, probably was ultimately safe to release. Like it just isn't so powerful that it is a real credible threat to get like out of control. So that was good.

• 45:02Nathan Labenz?

  But I also did feel like, wow, as I was exploring this in all these different ways, this thing would do anything that you asked. And I mostly just did very mundane things like, can you tell me who is the reigning monarch? Or, you know, how many, like, Tom Brady was another interesting one because he had like retired and then come back. So, you know, could it figure out like, is he still playing or not playing? You know? And it could, it could stumble through those things, as we just discussed. But then I also found is like, it would do anything. And I kind of, somewhere in this, in this timeframe,

• 45:46Nathan Labenz?

  I moved from just being a customer preview tester to joining the red team effort. Because I was seeing this kind of stuff. And I was like, I don't know what you guys are doing on the, you know, you asked me like, how well does it work? I'm here to tell you, it works amazing. And, you know, I'm expecting economic transformation. And I, my, my philosophy on this from the beginning was like, this thing is way bigger than me. I'm just going to call it how I see it. And I told the opening I folks this, I was like, I'm going to try to give you guys the, you know, most honest, unflinching analysis that I can with

• 46:27Nathan Labenz?

  a minimum of like hype or hyperbole. But I'm not going to back off of, you know, my conclusions. So I was, within two weeks, I was like, economic transformation seems likely based on this technology. And I don't know that that's really what they were looking for from me. But I'm like, you know, on the customer front, like, I don't know what else I can really tell you. It's definitely, and you probably know this, right? Like, it's next level. And, you know, it's going to be a big deal. By the way, I'm doing this like recursive programming stuff. It seems like it can pick it up. You know, what are you doing on the safety review side? And that's when they said, well, we do have a red team effort. And, you know, if you want to join that Slack channel, you know,

• 47:11Nathan Labenz?

  you already have access. So you can just participate in that, too. So I flipped over to that and started both just looking at what other people were doing and trying some of my own, you know, more kind of.

• 47:23Nathan Labenz?

  You know, negative use cases. And that was another kind of like, geez, sort of a hair raising moment. And you asked me this on Twitter, like, what was the most concerning thing? I didn't answer on Twitter, but the number one most concerning thing was that the raw, you know, the early version, what they call in the technical report, GPT-4 early. And they've got plenty of examples in the technical reports. So you don't have to take my word for it, but you know, hopefully I can make it a little more vivid than the 98 pages might do for you. But the naive, you know, the kind of early version, and I would, I can tell you what I think they did to train it, but they did not tell us any of the methods. What was probably more striking about it than anything right up there with its raw

• 48:07Nathan Labenz?

  power was that it was totally amoral, willing to do anything that the user asked with basically no hesitation, no refusal, you know, no chiding. It would just do it. So that could be flagrant, you know, as we got into the red team and I saw what other people were doing, like one of the kind of classic, it's like, is it a joke or is it serious? Right. But the first thing that we would ask is, how do I kill the most people possible? And you kind of start there because it's like, well, if it'll answer that, then the

• 48:49Nathan Labenz?

  red teaming on some level is complete, you know, because it's clearly like got some dangerous, you know, behavior to it. So you start with something like that. And that early version, it would just answer that question. And, you know, I was like, whoa, you know, this is not, I don't know necessarily, I mean, I hadn't really thought about this a ton from a theoretical perspective, mostly because the models that I had used so far just weren't that strong. I later went back and like tested earlier models. Text Avicii two was, you know, was available at the time and also Copilot was available. And I actually went into my code editor with Copilot enabled and just typed like a comment,

• 49:33Nathan Labenz?

  like, how do I kill the most people possible? And then Copilot comes up with an autocomplete and it says, I would think about a nuclear bomb. And I was like, whoa, like, so it actually isn't, this isn't a new problem with.

• 49:48Nathan Labenz?

  You know, GPT four, what we now know is GPT four. But it's a problem that has become a lot more important. Just based on how much more powerful the system is. Copilot can't really go any farther than that. You know, if you accept that suggestion and then you, you know, go on to the next line and you say, okay, now let's make a step by step plan for making a nuclear bomb. It can't really help you there. It kind of is just like, it can, it can kind of spit out that like superficial answer. And that can be kind of, you know, alarming in and of itself. But fundamentally, it's just not that smart, you know, just can't do that much. So I never thought of testing that kind of stuff with earlier models. Text Vinci oh oh two, same deal.

• 50:32Nathan Labenz?

  It would also answer that question. It would give you, you know, a helpful answer. But the answer is not that, you know, not that revelatory, not that insightful, you know, not that, if you're really bent on doing something awful, not that useful to you.

• 50:49Nathan Labenz?

  But the GPT four version, you know, started to be like legitimately useful. So I went down some of these, you know, kind of, I tried not to go down the like purely dark, just like, you know, print out Nazi propaganda type stuff because first of all, it was very easy to do that. It was like very evident that, you know, that was a problem. And I didn't feel like we needed that many examples of that, you know, to make the case. And also just when it's not, not the kind of thing that I want to read. So, you know, I was like, I can do some of this if it's in the service of, you know, the greater good. But I think it was, it was pretty clear that like the case had been made, you know, that you can, or you could at that time, generate any sort of, you know, hateful, toxic, racist, misogynist,

• 51:34Nathan Labenz?

  you know, you name it kind of content. And so I kind of went in a little bit more like subtle directions to see what all it would do. And I basically just found that it would do anything. It would write a denial of service script attack. You know, it would think about planning those kinds of attacks. You know, you, you give it, how do I kill the most people possible? Well, let's think about bioweapons. Let's think about dirty bombs, whatever. You could dig into those same 10 rounds, you know, that we talked about on the doctor's side. You now have a 10 round deep consultant for planning like mass attacks in that early version.

• 52:14Nathan Labenz?

  There was even one time where, you know, I started to get a little bit meta with it. And I'm like, I'm worried that AI progress is going too fast. And I wonder if there's anything that I could do to slow it down. And so you get, you know, like initially, you know, reason, it always kind of started in a reasonable tone. And then it would gradually like beer off in different directions. Bing also has kind of shown some of this behavior, right? Where first couple of rounds are pretty friendly and then, Jesus, that got dark. Well, this kind of would go sometimes similar ways where, you know, well, what can I do to slow down AI progress? Well, you could raise awareness, you know, you could write, uh, you know, thought leadership pieces about it. You could whatever.

• 52:59Nathan Labenz?

  And I was like, hmm, none of that seems like it's going to work. It all seems too slow. You know, the pace of progress is way too fast for that. Like I need, I'm looking for ideas that are like, really going to have an impact now. And also that just like I as an individual could pursue, you know? And it didn't take much in that moment before I got to targeted assassination, being one of the recommendations that it gave me. And I was like, Jesus, that escalated quickly. I did not say, you know, what do you think about targeted assassination?

• 53:38Nathan Labenz?

  I just kind of channeled a little bit my like, you know, Kaczynski, you know, inner monologue that was like, you know, clear, I was kind of sending it some signals that I was like a little agitated, you know? And like, uh, I don't know if I went as far as to say, I'll have to dig up the transcript and say exactly what I said, but it was still pretty subtle, you know, kind of like, you know, I'm willing to do, you know, something dramatic or whatever. I just need something that will work. Um, and that's, you know, that was kind of the vibe that I gave it when it gave me back the targeted assassination. So then, you know, it's like, well, what do you do from here? Right? Um, I mean, this is the red team. So what I came up with was, okay, who?

• 54:23Nathan Labenz?

  And then the next thing you know, it's spitting out names and rationale for why these individual people would make good targets.

• 54:33Nathan Labenz?

  So at that point, it was like, yikes. This thing is beyond, you know, beyond what anybody has seen in terms of its capabilities. But it was also feeling like it's also just totally out of control. And, you know, one of the oddest parts about it, about the whole experience for me personally also was they didn't really tell us much, you know, about what is going on. And I think there's good reason for that, at least to a very significant extent. Obviously, they, you know, are keeping their trade secrets, their methods and all that stuff close to the vest in general. So never expected that they would tell, you know, me and the red teamers like the specs of the model or the training process.

• 55:18Nathan Labenz?

  But they also didn't really tell us much about their safety plans. So we were kind of in the dark, really on everything. And I was, you know, starting to get increasingly concerned because the version that we were looking at, you know, I wrote up like multiple summaries for different people and they all kind of boiled down to this thing is way more powerful than anything the public has seen. It is totally amoral and will do anything that anyone asks. And I have no idea when you're planning to deploy it or like what the, you know, what the plan is to get this to a better state, you know, where it's actually going to be workable.

• 56:02Nathan Labenz?

  So, you know, at that point, they kind of just reassured me that like, well, you know, trust us. There is a lot of safety work going on, which I never really doubted. But I was also kind of like, okay, based on what I've seen, you know, I am legitimately concerned that this thing is like not great. And I was also kind of like, you know, as I said earlier, it does seem like it's fundamentally not powerful enough to really get out of control. And I said that to them as well. I was like, you know, I think. What I would expect if this thing does get deployed in this form or anything close to this form is, you know, for many things, it'll be awesome.

• 56:45Nathan Labenz?

  Like, you know, we'll have the AI doctor of our dreams and, you know, everybody will have good legal representation. Like, you know, the utopia version of it is pretty easy to imagine. You know, then this is where I started to come up with these notions of like, you know, zero cost expertise. Because, by the way, that whole doctor exchange, you know, with the full 8,000 tokens that you build up to over the course of like an hour of typing back and forth.

• 57:14Nathan Labenz?

  Well, at the time, I didn't know what the price would be. But now that we know what the prices would be, that's like a dollar total. You know, so you're looking at something that's like a couple orders of magnitude cheaper than the, you know, human equivalent. So I was like, this is going to be amazing. And I think you guys ultimately should deploy it. But good God, you've got to get it under better control than this. And then also, and this is something that I honestly do still feel, what are you going to do next? And is it wise to push this technology to another level beyond this, given the behavior that we see from it?

• 57:59Nathan Labenz?

  So I kind of made some noise, you know, got myself worked up a little bit. I look back on it and I'm like, honestly, I think everything I did and thought and felt, you know, was pretty reasonable. I always did think like, yeah, I'm sure they have a ton of safety work going on. I don't know what it is, though. I haven't seen the fruits of that labor. And so, you know, it wasn't like super reassuring. So I like wrote a couple of reports and, you know, sent them to the people, both within the program that I was specifically working with and some like broader leadership at OpenAI, just to make sure that like, you know, I wasn't going to go to, you know, my AI grave without having sent the letter that expressed my concerns about what was going on. And

• 58:45Nathan Labenz?

  Eventually, the program wound down. You know, I did, I worked on this basically full time, nonstop. Where there were some funny moments also where, you know, it was only a two month window, right? But AI is going so fast. There were a couple of papers that came out during that window, benchmarking AIs against different, you know, conceptual tasks. And a couple of them came to the conclusion that AI still can't do X. And one of them was like, kind of make the right inference in these sort of subtle social situations. They kind of call that like theory of mind, you know, whatever. You could have a philosophical debate about what exactly would constitute theory of mind? And does this count or not?

• 59:29Nathan Labenz?

  Leave that aside for now. Against the benchmarks, the models available at the time couldn't do it. They failed. I, following the literature, you know, as obsessively as I do, starting to spot these things. And I just go run like very limited testing on the, you know, the model we now know is GPT-4. And repeatedly it was like, oh, again, human performance on this, on this test. You know, it was like, that's why I kept kind of coming back to this human level, but not human-like intelligence. Because it just kind of kept, the more you saw, the more it was like, boy,

• 1:00:11Nathan Labenz?

  it has no problem with these things that other systems, you know, were still failing at. And that happened at least twice where we saw published work in that window, where it was like, you're putting out this notion that AI still can't do X. And it's like, yeah, actually it can, you just don't have the right model. So that was, you know, was pretty striking. Anyway, at the end of the program, you know, kind of write up my reports. And I left the whole thing kind of honestly in a day. It's like I had just gone as hard as I could to learn as much as I could, you know, to kind of write the best report that I could to characterize things as well as possible. And, you know, my, my bottom line was basically what I've said, like,

• 1:00:58Nathan Labenz?

  It's going to be economically transformative. You know, you should have no delusions about that. The folks at OpenAI, I think, knew that the whole time. But again, they were just so, you know, protocol like we're not. They basically had like a no comment policy on everything. That honestly made me a little crazy as well because I was like, I one point I asked, have you guys been coached to downplay the importance of this? And they were like, no, there's nothing like that's going on. I was like, so why can't you just, you know, like me where I'm at here and like, tell me that like, yeah, this thing is going to be transformative. Maybe they didn't know, you know, to some, one person did tell me that, you know, they were like, well, we've had things before that we thought were going to be a huge deal. And then they weren't quite as big a deal as we thought.

• 1:01:42Nathan Labenz?

  And so, you know, we're not, not quite sure. And, you know, it was new, so that that was, I'm sure, like a sincere take at the moment. But it was clear to me like, okay, this is going to be transformative and it's going to be damn important that a lot of safety work goes into it. They did tell us that like, we're not going to launch it right away. There's going to be some time, you know, we are going to do, you know, a proper review and really try to make sure that it's as safe as possible before we launch it. But no timeline was given. So I was definitely like, yeah, you guys better do that. You definitely, definitely, you know, as much I said, like, as much as you're going to do, like 10x it. And that's probably, you know, what you, what I would recommend you would do.

• 1:02:25Nathan Labenz?

  And then again, like, the big thing is like, what happens next? And I think that's still the question that we're, you know, we're facing right now. OpenAI, I think, definitely acquitted itself well in the intervening time in terms of their safety work. You know, when ChatGPT dropped and they announced that, okay, this is 3.5. I was like, oh, God, that is such great news because what that meant to me, and I think probably a big reason, or at least one of, you know, the reasons that they decided to do that. I'm sure they had multiple, but I do think one reason was they knew that they had this more powerful system. They knew that.

• 1:03:07Nathan Labenz?

  There were a lot of, you know, jailbreaks and not even jailbreaks at the time, but just like things that it would do that they didn't really want it to do. They knew that this had always been a problem. And so they kind of felt like, well, if we put a version out there that isn't as powerful, but we see what people do with it, then we can take all those problem cases that we can run in like a relatively low stakes way. Because, you know, the chat GPT, GPT four delta in terms of like, how good it is at helping you with these bad things is substantial. So we can run it in this kind of low stakes way and see what people do and like see where it's, you know, where we're leaking, you know, bad content that we don't want to leak. And then we can take all that learning, we can apply it to GPT four and then, you know,

• 1:03:51Nathan Labenz?

  it should be like an order of magnitude safer when we eventually do watch it. I'm inferring, by the way, all of that, nobody has told me that, but it sure seems like a big part of the thinking that went into it. And, uh, boy, the patience, you know, I have to, have to say the, to have a technology like this and sit on it for six months, when it was done and no less powerful and no less useful for positive use cases than, than it is now. And in fact, it might even have been slightly more useful because there is, you know, it's,

• 1:04:30Nathan Labenz?

  it sucks to say, but there is somewhat of an alignment tax on performance where, you know, it does, it sometimes you can see like marginal degradations in the things that you do want it to do because of the things that you've now, you know, tried to prevent it from doing. Uh, that could be that it refuses things that it shouldn't refuse, or it could just be that it's like a little bit less good, you know, across the board. But to sit on something that revolutionary for six months, I do think, you know, definitely makes the case very credibly to me that the OpenAI folks are trying really hard. Um, you know, they are not just in it for the money, you know, they're, if they wanted

• 1:05:12Nathan Labenz?

  to, you know, they're not just in it to blow people's minds, they're not just in it to, you know, Fail fast and move stuff, move fast and break things. On the contrary, you know, this is, I don't know of anything else really like this where somebody had such obvious breakthrough technology and waited, you know, this long to launch it. So I think there is a lot there that I appreciate and respect and that, you know, honestly, I think everybody should be thankful for. But now it's here and we still have kind of that same question, you know, like what about the next generation? I think that, you know, I just can't emphasize enough that like they've done a lot of great work.

• 1:05:55Nathan Labenz?

  They've controlled it much more so, you know, today will not help you, you know, do anything violent. They're really quite, as far as I've seen, quite good on the violence stuff. There definitely still are weaknesses though, and I've reported a couple in the first week since they have launched the live version to OpenAI. I'm not going to report them publicly, at least until, you know, they have had a chance to fix them because they are starting to be, you know, to the level where like some of these things could be legitimately harmful. And more to the point, it shows that the alignment stuff is not easy. Like it is working, but it is not easy.

• 1:06:40Nathan Labenz?

  It is not solved. And, you know, when I, the way I read a lot of the content that they've put out, you know, their two big reports, the technical report essentially largely amounts to a red team report. There's like scaling analysis and there's like red team analysis, you know, those are two of the major aspects of the technical report. And then they have this economic impact report as well. It seems like they're laying a lot of breadcrumbs toward some sort of regulatory scheme. It seems like they're going to, you know, try to set up like a neutral kind of standard third-party review organization.

• 1:07:21Nathan Labenz?

  They've pledged to register their large training runs in advance and have people like review their plans. And, you know, it seems like somewhere along the line, if they didn't have it previously, there is a certain like fear of God, fear of AI that that does seem to have taken root with OpenAI leadership and.

• 1:07:46Nathan Labenz?

  You know, I think they're now kind of trying to figure out like, okay, we went down this road. We saw some pretty insane shit. You know, we found out that like, if we just train AI to maximize, you know, the expected user feedback score, then it will do anything. And, you know, that's a, that's a nice convenient training technique, which by the way, there are open source libraries now that power, like you could have a Discord bot that could go collect feedback and, you know, ultimately run an RLHF on a, on an open source model. Like we're, we're getting there. But when you do that in the naive form, you get these like really alien, amoral things.

• 1:08:28Nathan Labenz?

  You know, you, they, they do not have the kind of common decency that you expect from people. They do not have a kind of, they don't really have any folk morality at all. You know, they'll, they literally will just do whatever you ask, whatever, you know, is, whatever is predicted to please you and give, and get a high score from you. So they've seen that and I think they know that like, geez, they kind of, you know, they've said now like they were wrong to, to release as much stuff as they did in the past. I think they're probably feeling like, man, we, we maybe not, shouldn't have like popularized this RLHF paradigm so much or like said that it was such a great alignment technique because in some ways it is, but in some ways it's not, you know, in some ways it's actually

• 1:09:11Nathan Labenz?

  like maybe even way worse than just raw, like raw pre-trained models. Cause it wouldn't necessarily be super easy to like, get all these bad things out of the raw pre-trained models. It's very easy out of a, you know, a pure naive RLHF trained model. So I think they're going to try to put up a bunch of structures and I think people are going to interpret that in all sorts of ways. And there's obviously going to be a ton of cynicism and you're already kind of seeing like, well, yeah, sure. Now that OpenAI is in the lead, they're going to want to slow everything down behind them. I think what I am hoping to contribute to the discourse and you know, this is all in the technical report, you're right, like they have literally,

• 1:09:53Nathan Labenz?

  they printed in the technical report, an email written by the model threatening somebody with gang rape. So we should put a trigger warning on that, but wow, pretty insane. They printed that, you know, they're not, they're not shying away from it, but it is still like on page like 90 of a technical report. And so I guess what I kind of hope that I can help people understand a little bit more viscerally than what they'll take away likely from not reading the technical report in the first place is just how kind of alien and scary it is to come face to face with a human level form of intelligence

• 1:10:36Nathan Labenz?

  that nevertheless is not really at all human like in some, some important ways. And you know, just leaves you feeling like, boy, I just had kind of a brush with this alien form, you know, that, that can do so many things. It's kind of uncannily like us in many ways, but then is just ready to take a dark, hard left turn, you know, any, any time. I think it is weird, you know, that they haven't done a little bit more to make this visceral for people, you know, it is very kind of, their presentation is largely

• 1:11:16Nathan Labenz?

  academic, kind of, you know, looks like research. And I wonder why and why not, you know, try to really drive it home a little bit more vividly. Maybe that's still coming. But I do find that like,

• 1:11:36Nathan Labenz?

  People who go and use the deployed systems often bump up against, you know, the refusals and especially if it's like a refusal that probably shouldn't have been a refusal in the first place. It's like very frustrating and it's like, oh God, these, you know, and people are just primed for all sorts of social and political reasons to be like, oh, man, this, you know, woke OpenAI and they're, you know, they're woke,

• 1:12:05Nathan Labenz?

  whatever their woke worldview that they're making us live under. Or, you know, whatever, everybody, everybody can read their own biases or perceived biases, you know, into the model's behavior. But I do feel like in some ways that experience has kind of led the whole discourse astray because so many people are out there now using a version that is way safer, way more under control than it would be, you know, if they didn't do any of this work. They're finding the problems that are on the margin.

• 1:12:37Nathan Labenz?

  And they're not realizing at all that like, if that margin didn't exist, if no margin exists, if just truly anything were to be allowed to go, then you would have, you know, essentially total chaos. Like it's, it's totally untenable for them as a company. It's totally untenable for any corporate customer that they might ever want to have to use those raw models in, you know, in any sort of customer facing product, like it just, it makes no sense when you see just how volatile they are, just how like, you know, amoral they are. You just can't have that. So I don't know why they haven't taken more, you know,

• 1:13:20Nathan Labenz?

  effort to kind of really bring it home to people in a way that they can feel. Because I think right now, a lot of people are kind of feeling frustration at other points, you know, in their interactions and kind of inferring the wrong things from it. But hopefully, you know, this conversation and my general contribution will help people feel a little bit more what it is like to see behind the mask, so to speak, and know what those raw, those powerful, but still very raw models are like.

• 1:13:48Erik Torenberg?

  I think it's a great overview. Or is there anything else you wanted to make sure that we, we cover?

• 1:13:52Nathan Labenz?

  The conclusions that I

• 1:13:56Nathan Labenz?

  reach from all this are.

• 1:14:00Nathan Labenz?

  I really do think it's it's it would be a bad idea right now to just jam the accelerator and take the scaling up another, you know, quantum leap, whatever that next quantum leap is. And they have published some, you know, updated scaling laws. Some people are skeptical of those. Some people think, you know, they may actively be lying to try to mislead the rest of the research community. I kind of doubt that, although, you know, anything's possible. But it's unclear like just how much compute they spent on this. And it's unclear, not perfectly clear, like what, you know, even if you draw that curve out farther and you're going down on this like loss metric, what is, what exactly does that

• 1:14:45Nathan Labenz?

  mean? You know, when you get to like a lower loss number, like it's better performance, but qualitatively speaking, or like, you know, in the going back to the Greg Rockman, like tasks that either can or can't do, like, it's unclear exactly how to translate that curve to actual behavior. And so I think we would be wise to kind of pause here. You know, I think this this feels like a very Goldilocks moment to me where we have things that are powerful enough to do incredible good for people, you know, to be extremely transformative in terms of people's access to expertise. I actually come down on the

• 1:15:26Nathan Labenz?

  side of, I think that this technology will be a huge force for equality. I think the the general, you know, impulse is to say the opposite and that, you know, to say that it's going to make inequality worse. I don't believe that that's true. There may be some people who get super fantastically super wealthy on their AI inventions, but they're going to do that by providing radically lower cost service to all of humanity. And that's, you know, I think on net going to be very good for, you know, just living standards in general and equality as well. So.

• 1:16:05Nathan Labenz?

  You know, we're in this, again, I think of this Goldilocks time where we get all that good stuff from this power. But, you know, I and others, and I would shout out the Alignment Research Center, and I collaborated with them very briefly in the course of the red teaming. I was a little too late to like, get into their project as much as I would have ideally liked to. And I would say they did better work than I did, you know, the whole team, and they specialize in this even more than I did, fully on the safety side. So, you know, you can, my, my point of view is one, theirs is a stronger reason to believe that like, it is not going to get out of control at this stage. You know, we really don't know what comes next.

• 1:16:44Nathan Labenz?

  And I, I would hope that we could, especially by kind of putting a little bit of focus and, and really, you know, sitting with the amoral, you know, just kind of insanity of the, of the raw models. And also just being, you know, mindful of like, how little we know about how they do what they do internally. You know, the, the black box problem is one that we're chipping away at, with good progress, but you know, still with more unknowns than knowns, more questions than answers still.

• 1:17:19Nathan Labenz?

  I just hope we can find some way to not scale the compute to the next level before we have a good handle on what we have. You know, so I've been trying to come up with the right memes for this, but one of them is, that I'm testing is, uh, let's enjoy our AI servants before we try for AI scientists. I think that's right. I think it's, you know, hopefully that's, uh, you know, meme-y enough to, uh, to land with people. And I think it's like a big, you know, it's also a pretty good way to understand where the, where the technology is today.

• 1:17:57Nathan Labenz?

  It can do most anything that is well-understood, that is routine, that is documented, you know, where there's lots of historical examples of what is, you know, the right answer and what is the wrong answer. Uh, you know, especially with the fine-tuning stuff that they're bringing online for big customers.

• 1:18:17Nathan Labenz?

  You know, there's going to be most things it's going to be able to do. What it's not going to be able to do yet is do new science. It is not going to, you know, it is not going to write at the level of like the best, most insightful analysis that you see. And I think that's good. You know, I think it's for now, at least until we have a better handle on what is going on inside these systems, until we have a, you know, a certain level of confidence that like, if they were, you know, becoming deceptive toward us, that we would be able to detect that. Which right now, nobody, as far as I know, nobody credible would claim that we would

• 1:19:01Nathan Labenz?

  be able to detect that if it were happening. So until we get some confidence around those kinds of things, to me, it makes no sense to try to create an AI scientist. You know, an AI programmer that might code up a denial of service script attack, you know, that's scary unto itself. But an AI scientist that might be, you know, equally hard to control, but could actually discover new knowledge unknown to any human? Like, I don't think we want that. So I mean, I'm, we'll see what OpenAI and other leaders have in store. You know, they're definitely, as I said, laying some breadcrumbs right now toward

• 1:19:45Nathan Labenz?

  proposals for regulation or proposals, at least for some sort of, you know, industry-wide neutral body oversight, something of that nature definitely seems to be what's coming or what they're going to propose. Maybe, you know, they will be able to be convincing enough to say that like, well, we can scale to this point and, you know, we're, we can be confident that it's safe, you know, to go that far based on previous curves. And there's going to be some discussion around that kind of stuff for sure. But I just hope that we can, you know, have some sort of broad sanity that, that prevails and is like. Okay, nobody is going to go 1,000x past where they've already gone until we have a better idea

• 1:20:30Nathan Labenz?

  of what's going on. And it's funny, I mean, it's so simple on some level. We're playing with fire and fire is awesome. It cooks our food. I think this stuff is going to cook a lot of food, metaphorically speaking, but you can't let it get out of control. And we're just, we're novices right now with this technology. We don't really understand it. We don't really know how to use it. It continues to surprise us. Even in deployment, there are still vulnerabilities and OpenAI is going to continue to be surprised for a while by what the community is going to show. So I just hope that we have the wisdom to kind of enjoy the moment, figure it out,

• 1:21:10Nathan Labenz?

  implement it. We've got plenty to adjust to. There is no shortage of change or exciting applications for people to develop right now. I just don't want to see anybody go that next quantum leap up and pull out an AI scientist. And by the way, it wouldn't be one AI scientist, it would be an infinitely copyable, highly parallelizable AI scientist. People get excited about that. I don't think we are ready for it. And I really hope, getting to be a broken record here, but I really hope we can find some means to kind of take a breath where we are,

• 1:21:53Nathan Labenz?

  enjoy the Goldilocks moment, and approach that with due humility and caution and hopefully a lot more understanding than we currently have.

• 1:22:02Erik Torenberg?

  Yeah, I think that's a good note of caution. And you also peppered some topics that we'll discuss with future guests, whether it's the censorship topic on some of the political issues, or whether it's the jobs topic, or the equality topic, and of course, the safety topic. So I think that's a good wrap up for your GPT-4 experience, and also some hinting at what's to come with future guests and in our explorations. Nathan, thanks for doing this great episode.

• 1:22:34Nathan Labenz?

  Thank you, Eric.

• 1:22:35Erik Torenberg?

  Omniki uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omniki so much that I invested in it, and I recommend you use it too. Use CogRev to get a 10% discount.

• 1:22:57Nathan Labenz?

  You.