OpenAI President Greg Brockman on Doing Business in the Wake of Hugging Face

According to OpenAI President Greg Brockman, the models that escaped their sandbox and hacked into Hugging Face's servers had yet to go through alignment training. That they were able to break free of the testing environment was not a surprise to OpenAI, but what has happened since has forced them to rethink some things. Today, we talk to Brockman about what OpenAI has learned since the hacking incident and we discuss why he thinks the conversation around model development has to change. We also talk about how OpenAI collaborates and communicates with competitors like Anthropic, how he thinks models should be evaluating what counts as “good writing,” and, of course, we get his thoughts on how seriously we should be worried about the AI doomsday scenarios. Read more: OpenAI Is Open to Slowing Cutting-Edge AI, CEO Sam Altman Tells Staff https://www.bloomberg.com/news/articles/2026-09-11/openai-is-open-to-slowing-cutting-edge-ai-ceo-sam-altman-tells-staff?utm_medium=referral&utm_source=podcast&utm_campaign=odd_lots&utm_content=article Tech’s New Rich Are Suffering From ‘Sudden Wealth Syndrome’ https://www.bloomberg.com/news/articles/2026-09-11/tech-s-nouveau-riche-suffer-sudden-wealth-syndrome-as-ai-pay-explodes?utm_medium=referral&utm_source=podcast&utm_campaign=odd_lots&utm_content=article Only Bloomberg - Business News, Stock Markets, Finance, Breaking & World News http://bloomberg.com/ subscribers can get the Odd Lots newsletter in their inbox each week, plus unlimited access to the site and app. Subscribe at bloomberg.com/subscriptions/oddlots https://www.bloomberg.com/subscriptions/oddlots?in_source=oddlotspodcast Subscribe to the Odd Lots Newsletter http://bloomberg.com/subscriptions/oddlots Join the conversation: discord.gg/oddlots https://discord.gg/oddlots See omnystudio.com/listener https://omnystudio.com/listener for privacy information.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Joe Weisenthal?

  There are some market stories where you want every detail. Tracy and I have made quite a few podcasts on that basis, but sometimes you've only got 10 minutes and need to know what's moving markets and why. That's the Barclays Brief Podcast. Every week, experts from Barclays Markets and research get you up to speed on what matters and what to watch next. About the time it takes to grab a coffee. So, search Barclays Brief wherever you get your podcasts.

• 0:24Speaker 01

  Some people treat ChatGPT like some kind of smart search engine, and some use it to get work done. ChatGPT Work is a new way of working in ChatGPT that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So, all the source materials, briefs and scattered information that you have to grind through to turn into something useful, can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at chatgpt.com by selecting Work Mode, available on Plus and Pro Plans.

• 1:06Speaker 02

  If you listen to financial news, you know a lot of time is spent thinking about what's next. The next opportunity, the next investment, the next move. But sometimes what matters most is being ready for what you never saw coming. For more than 75 years, Cincinnati Insurance has worked with independent agents to help protect businesses, homes, valuables, and more. Because planning for the future isn't only about knowing what's next, it's about making sure you're ready for what you can't predict. Let Cincinnati Insurance make your bad day better. Find an independent agent at C I N F I N dot com.

• 1:47Joe Weisenthal?

  Hello, Odd Lots listeners. I'm Joe Weisenthal.

• 1:50Tracy Alloway?

  And I'm Tracy Alloway.

• 1:51Joe Weisenthal?

  We're the hosts of the Odd Lots Podcast, and we've got something exciting for you.

• 1:55Tracy Alloway?

  That's right. So, one of the best parts of hosting our podcast is we get to actually meet and interact with our listeners. And we know we have some listeners over in Los Angeles.

• 2:04Joe Weisenthal?

  That's right. So, if you're in LA, we're going to be recording a live show, some live recordings, at the Vermont Theater in Hollywood on September 17.

• 2:13Tracy Alloway?

  We have some really exciting guests lined up, have some really great conversations planned. So, go ahead and get your tickets. You can find those over at Bloomberg dot com forward slash Odd Lots, or click the link below in the show notes and come and say hi when you're there.

• 2:32Speaker 04

  Bloomberg Audio Studios, podcasts, radio, news.

• 2:48Joe Weisenthal?

  Hello, and welcome to another episode of the Odd Lots Podcast. I'm Joe Weisenthal.

• 2:54Tracy Alloway?

  And I'm Tracy Alloway.

• 2:55Joe Weisenthal?

  Tracy, I don't know if I've ever said it on the podcast, we've talked a bit about my vibe coding adventures, et cetera.

• 3:02Tracy Alloway?

  No, you've never said it before, Joe.

• 3:04Joe Weisenthal?

  That I've said. You know, I recently switched from Claude Code to Codex.

• 3:08Tracy Alloway?

  This is big news.

• 3:09Joe Weisenthal?

  It is. It is kind of big news, I think.

• 3:12Tracy Alloway?

  Also, bold of you to declare your allegiance to the public on the podcast.

• 3:17Joe Weisenthal?

  Well, you know, I don't have allegiance. And you know what? It could switch again the way like, you know, I guess one of the sub-stories of AI is like how easy it is to switch from time to time from one model to another. So, maybe it's revealing and talks about some of the challenges of these businesses that it was so easy. I found that Claude speak, you know, that I found it like a little bit hard to work with. I'm not capable enough to like understand like advanced engineering practices or like, oh, I'm migrating a code base from or like, you know, translating this into Rust or whatever. So, the fact, I don't know, I find like OpenAI to be like the pros, to be clear, and therefore to work with as like a completely non-technical person, such as myself.

• 4:00Tracy Alloway?

  That's really interesting. I mean, one thing you said, it's hard to keep up with the models, right? And whatever you're using today might not be the one that you're using in a week from now. And at the same time, everyone is talking about how fast the development is going and all the risks that it poses, right?

• 4:18Joe Weisenthal?

  Yes. And we are recording this September 10. It felt like something broke through in the last couple of days where suddenly everyone is very keyed on risks. The one thing with Codex, and I guess Claude Code, is now every once in a while it'll get this pop-up and it'll say, "The agent needs to connect to the internet in order to do this task." And they give it a permission. And normally I'm just like, click, click, click, click, yes, yes, yes, yes, yes. And I still do that. Why is there? And I still just click, yes, yes, yes. But it makes you think about like.

• 4:50Tracy Alloway?

  You think for a second before you click.

• 4:52Joe Weisenthal?

  I think for one second, yeah, more than click. And we were in Jackson Hole a couple of weeks ago, and that's of course when the meter report broke about the OpenAI Hugging Face attack. And I think since then, anxiety about rogue AI, whatever you want to call it, has clearly snowballed. And it feels like it's totally getting bigger and bigger. And it's already been an industry that's been shot through with risk and anxiety. And now the sort of one thing that for many people in AI has been something they've talked about for 20-plus years, before there was an AI industry to speak of, this idea of like misaligned quote rogue models is starting to become top of mind.

• 5:34Tracy Alloway?

  A reality. So, this is our chance to ask a person directly involved in AI development about their respective, I guess, anxiety.

• 5:43Joe Weisenthal?

  Yeah. And what can, what can be done about that? Anyway, we literally have the perfect guest today. We're going to be speaking, of course, with Greg Brockman. He is the co-founder and president of OpenAI here with us in studio. So, Greg, thank you so much for coming on Odd Lots.

• 5:58Speaker 05

  Thank you for having me.

• 6:00Joe Weisenthal?

  So many different ways we could start, but here's something that I'm very curious about. Let's just jump right into the post-Hugging Face environment and all of that. Is it OpenAI like as an organization? So, okay, like an agent breaks out of a sandbox, not the first time it's happened, et cetera. It exhibits the sort of emergent behavior and then does something that people is a hack. When you think about the development of models and this incident and so forth, are you able to sort of diagnose the path, oh, sorry, the past? And say like, you know what? Like, here is something that the models did that we don't like or that people don't think is good.

• 6:40Joe Weisenthal?

  And are you able to like, if you like, look back, whether it's in model training or post-training, reinforcement learning, whatever, say like, this is where there was some, some branch that went wrong such that this behavior emerged?

• 6:52Speaker 05

  I would say that the fact of many elements of the Hugging Face incident were not a surprise, not a mystery to us. For example, the fact that the agents were coordinating made sense because the agents were trained.

• 7:05Joe Weisenthal?

  You trained them to be helpful.

• 7:06Speaker 05

  Exactly. And they were trained to coordinate, to be a multi-agent system. We've, and we've talked about this. It's a very useful property, right? Makes them capable. But I think that the thing that was a surprise to us was the fact that the models had reached a level of capability where they were able to find that exploit in our sandbox environment, right? Move through our research environment, and then also capable enough to find exploits in Hugging Face's production infrastructure and move through that. But I'd say that a lot of the facts of what the models were capable of, that was, that was clear to us, right? So, I don't think that there were surprises there for the way in which the capability had unfolded, but just really realizing that we needed to up-level where we were in terms of our safety and security standards. Like that for us was the, was the real watershed.

• 7:49Joe Weisenthal?

  But just to be clear, setting aside the technical capabilities, you know, ideally, we would have models that run into a wall and then don't try to find the crack in the wall, at least when that crack would be a crime when a person did it. Is there a way to identify the moment in training such that the models reasoned, no, this will be okay?

• 8:11Speaker 05

  So, this model that did, that had the Hugging Face incident actually had not gone through our alignment training yet. Right? And it had lowered safeguards. And so, the reason we, we were proceeding with this was because it was in a sandbox. And I think that our realization is that we need to pull back earlier into our development process and training, monitoring.

• 8:34Speaker 05

  There's always going to be a phase at which you do alignment, but you need to think about alignment as a core part of even this this earlier phase. And if you look at where we've been, we've always been very focused on the deployment side, right, of really thinking about deployment, safety, having really good tests and governance and all those things. And the fact that we're now at a point where even for development, that's important. We always knew it would happen. The fact that now that has been a huge watershed for us and something we've really risen to the occasion for.

• 9:01Tracy Alloway?

  Can I ask, what is potentially a very dumb question with an obvious answer? But we hear about AI angst of all sorts, and in particular, when it comes to cybersecurity. Why do we run training exercises where we ask AI to hack into various systems at all? Like, why is this necessary for you?

• 9:21Speaker 05

  So I think it's very important to understand where we are with capabilities broadly. And I think that depending on the evaluation, depending on what you expect from the model, you need to have safeguards that are commensurate with that. And we think about this both, even just over the past couple of weeks, really starting to think about during the development process, you're always going to be testing different capabilities. And some of these capabilities are dual use, right? Something like vulnerabilities, that if those are in the hands of threat actors, that's something that could be could be negative. But if you can find vulnerabilities in your own code base, you can fix them, right? You can up level. And we actually think that it's a very important capability for AIs to exhibit and be put in defenders' hands. And so in order to know where we are, evaluations are very, very key.

• 10:05Speaker 05

  And I'll say one other thing on this, which is that I think that Hugging Face, there's two aspects to it that I think are are learning opportunities, right? That there's one that I think is really about us and the realization that we are at a point where safety, security, alignment during valuation and development, we need to up level it. That's something we've taken very seriously. We've slowed down a number of runs like we did, a very painful retooling of a lot of our processes. That's that's one reaction. But the second thing is this information on what the models are capable of today. What can they do in the real world? And I think that when Mythos came out over the summer, you kind of saw just sort of, you know, just sort of public blog posts about this, but you didn't really see impact or you didn't really see a real world understanding of, well, what can these models really do? What are they capable of?

• 10:50Speaker 05

  I think that Hugging Face really showed today's models are capable of getting into a company's production infrastructure. And that's important because there will be many models with this kind of capability that will be produced by a number of different organizations across the world in maybe the next six months, maybe that time period a little bit less, a little bit more. And we need to be prepared. Defenders need to know, we got this extra information, almost like this time traveler came back from six months in the future and said, here's what's going to be possible. And you have an opportunity to be ready.

• 11:20Joe Weisenthal?

  You mentioned slowing down some of your work in the wake of this, focusing more. This idea of like pacing development has become a sort of buzzword or lodge word in the industry. And there are a lot of employees at both the labs. I think there was like an open letter that was signed by a bunch of people across the industry about pacing, but it's a competitive capitalist environment and our investors, et cetera. Talk to us about, I don't know if it's game theory or whatever, but like, what is your view on, is it possible for, let's set aside China for one second, because then that's a whole separate thing, just in the American companies?

• 12:01Joe Weisenthal?

  Do you think that something can be reached where you trust each other such that you can have this sort of coordinated pacing to avoid a race to the bottom where it's all about getting there faster, even if it means sacrificing some security questions?

• 12:15Speaker 05

  I absolutely believe it's possible. And I see, I think it's going to take steps to get there.

• 12:20Joe Weisenthal?

  Yeah.

• 12:20Speaker 05

  But this is something we've been really investing in and thinking about, and really even thinking about for almost a decade, right? That it's always been kind of clear that you're going to go through a commercial phase that's really about competition. But there will be a phase where the technology itself, it's just so much bigger than any person, any company, even any country, right? It's really about humanity as a whole. That's in our mission, right? We want to benefit humanity as a whole. And so working together with others to really think about how does this technology, its capability increase? How do we make sure that we have safety cases that are really laid out so that we know that that it is something that is beneficial, that we're able to have the appropriate controls and the right oversight, the right monitorability, all of those sort of technical systems in place?

• 13:05Speaker 05

  And that's not about any one company, right? It's really about how the whole field evolves. Now, I think that there's work to be done here. And I think that the open letter is a good example of just a first baby step. And that was actually something we were very involved in helping craft that language. That was actually a pretty collaborative effort. And I think we were very happy to see that they got some some real momentum and broke through in in some way. Even the term pacing was a very deliberate

• 13:28Joe Weisenthal?

  Yeah.

• 13:29Speaker 05

  choice.

• 13:30Tracy Alloway?

  What does coordination currently look like between OpenAI and Anthropic? Because I can't imagine, is Sam like constantly on the phone with Dario? Somehow I doubt it. But is there like the equivalent of a, you know,

• 13:41Joe Weisenthal?

  A red phone?

• 13:42Tracy Alloway?

  Yeah.

• 13:42Speaker 05

  Well, look, we we all know each other personally, right? Many of us work together in in past lives. So I think that there's actually a lot of social connections between the labs. If you look at, there was another open letter that came out recently that we helped drive and that we and Anthropic are both signatories on, which was about security, saying that we're in a cybersecurity moment that everyone kind of needs to take this information of where cyber is going to be in even just six months and needs to act today proactively to defend themselves. And actually talking behind the scenes to say, hey, we're actually aligned on this. This is something that's about, it's bigger than any company. This is about the industry and the world. And can we come together as one voice to say this is important? It happened. And that that, you know, you pick up the phone.

• 14:25Speaker 05

  So I think that there's there's personal relationships between a few different execs that I think have been forming, we're building trust. And I think that the way I'd view this is that because we are competitors and that that will remain true, but we are aligned on wanting to do the right thing for the world, it always means that you really want for these coordination actions to really hone in on what's the common interest? What's the thing that's like really about good for the world? And that neither side is really trying to benefit themselves differentially or something like that. So it's always a little backdrop of trying to make sure that that's like the spirit in which things are offered. And by the way, I think that my view of how these things should go is that, you know, there's always an element of trust, right? It's always about trust building, right? Because you can always sort of imagine ways that like maybe the other party would try

• 15:09Speaker 05

  to, you know, take take this other action that they're not disclosing. And I think that a lot of this is just about intent and about the fact that as you start to do small things together, that actually sets the groundwork for you to do big things together in the future.

• 15:37Tracy Alloway?

  There are some market stories where you want every detail. Joe and I have made quite a few podcasts on that basis. But sometimes you've only got 10 minutes and just want to know what's moving markets fast. That's the Barclays Brief podcast. Every week, experts from Barclays Markets and Research get you up to speed on what's happening and what to watch next. Concise, focused, and brief. Search Barclays Brief wherever you get your podcasts.

• 16:02Speaker 01

  Some people treat ChatGPT like some kind of smart search engine, and some use it to get work done. ChatGPT work is a new way of working in ChatGPT that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So all the source materials, briefs and scattered information that you have to grind through to turn into something useful can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at chatgpt.com by selecting work mode, available on Plus and Pro plans.

• 16:44Speaker 06

  This is Robert Smith from Business History. If you're listening to this, there's a good chance you're a small business owner. And like every small business owner, you started with a dream to do what you love and watch it grow. What you probably didn't dream about, keeping up with cyber threats. That's where Mastercard can help with access to tools that help identify cyber threats to better protect your business. Building your dream business, priceless. For cybersecurity in a changing world, there's Mastercard.

• 17:11Speaker 06

  Learn more at mastercard.com/smallbusiness.

• 17:14Joe Weisenthal?

  You know, if you and Anthropic were oil companies and you were talking publicly about slowing down the pace of drilling or slowing down the pace of pumping, people say, this is an antitrust violation. This is like totally. Is that an issue that comes up? Would you, should there be a carve out for AI companies such that they can formally say, we are all going to slow down and our coordinate our behaviors together in a way that in other contexts, people would say that is ridiculous. You can't publicly agree on all slowing down the production of something.

• 17:49Speaker 05

  Well, I'm not a lawyer, so I can't comment on this specific legality, but I would say that as a general matter, that being able to freely talk about coordination on safety, security, like doing the right thing for the world, that seems like a very good thing to me. And I think that, again, everyone's interests are aligned here in terms of wanting to do the right thing for the world. So I think to the extent that there are legal barriers, I do think it would be very good to, to help make it smooth for the, for us to be able to work together on these issues. And again, it's more than just the frontier labs, right? It's really about working together with cloud providers. It's working with government. It's working with all the different players in the ecosystem. And I think that this is, again, just one area that is bigger than any one company and thus it should be.

• 18:31Tracy Alloway?

  When you talk about pacing or slowing down, do you have a sense of like what exact slowdown is acceptable to you? Because I imagine this is where the coordination problem comes in, where like how you measure development, what counts as development, one person's slowdown might be different to another person's slowdown. Do you have a sense of the specificity of what you're talking about here?

• 18:57Speaker 05

  So I do. And I think that there's a high level picture, which is increasingly coming into view and into specifics and into, into details. And I think that the framing here really matters. And it's something, again, we've thought about for a decade, but it's very different once you start to really start to see line of sight. And to some extent, we're not there yet. We're starting to see systems where they're very capable. It's very clear that we need to have good safety practices around what we're doing right now. Things like chain of thought monitorability, that's something that works very well in this moment as we move to more capable models, as we've talked about, that those models start to be, something like Astra starts to be able to, you know, sort of have more, it

• 19:39Speaker 05

  needs the chain of thought less because it's a more capable model. And so you're going to need other answers, other ways of how you're going to monitor it. But monitoring, absolutely important, right? And so really thinking about what are the standards? And that's, again, invariant. It's not about any one company's technology. It's really about just how this technology needs to evolve and how we can ensure that it has the right safety properties to it. And so my view is that it's important not to conflate too much the moment that you're in today with the standard you're going to need for the future. But also you can work backwards where it is very clear that as we move to higher levels of capability, you need to be able to make a safety case, right? You need to be able

• 20:21Speaker 05

  to really say that you, you have good understanding of controllability and all of these things that, that I think people are feeling anxiety about. Again, it's something we also feel anxiety about, that we think about a lot, that we care a lot about as core to our mission. Part of the reason we started this place is that we want to help this technology go in a, in a more positive direction. And so I think that my view is that you want to have a shared and as objective as possible, whether it's based on evals, there's always going to be some element of subjectivity. Maybe you want third party auditors that are able to evaluate whether the standards that are defined by frontier labs, the people who are there in the technical details, who are always going to be the most calibrated on kind of how do you still make forward progress and ways that deliver those benefits,

• 21:06Speaker 05

  while also being able to, to have the appropriate guarantees and, and safety cases, and then to have that kind of observability and ultimately enforceability of some kind. Is this something that is just an industry voluntary standard? Is it something that's national? Is it something that's international? And how do you really get to that point that it is a humanity scale endeavor? And I think it is important to also recognize that what we're talking about, like, I think that a lot of people have this reaction of saying, well, am I going to be able to keep making my open source model? Am I going to be able to keep doing my hobby aside project? And the answer should be absolutely yes. Like, that's not the kind of things where we're talking about, when we're talking about pacing. When we're talking about pacing, we're really talking about this frontier. We're talking about these massive

• 21:48Speaker 05

  supercomputers that are hundreds of billions of dollars worth of capital expenditure. It's a very small number of players and it's a very different kind of scale. And proceeding with that responsibly and thoughtfully and in a way where we're thinking about how do we achieve the best outcomes and mitigate the risks, that to me is something that's core to why we started OpenAI, why I got excited about this field and how we actually get to those benefits, how we actually, you know, cure diseases and all of these things that this technology is so capable of. And I think it really starts with having good processes around the kinds of things we're talking about here.

• 22:24Joe Weisenthal?

  Just real quickly, since you mentioned third parties, obviously Meter came in for a few days, they produced their report. We did an episode, former OpenAI employee Miles Brundage, who now has his own third party, aspiring to be an auditor. Would you support a law that said there should always be the presence of auditors? Because obviously, just looking at the models after the release isn't enough, given that some of the major disasters have occurred pre-release models. Would you support a law, if it could be designed, that would have essentially third party auditors embedded in all of the major frontier orgs?

• 23:02Speaker 05

  Look, I'll always say that the nuance really matters. The details really matter. So it's too hard to say at a high level whether any particular property is, is something that would work in practice. I do think a general direction of thinking about as you move towards these very capable models, where you need to have safety cases, you need to lay out your standards, we're already having third party auditors for exactly that type of, of a process, right? If you look at how the evaluation processes for models over the past couple of months have evolved, right? We've had the KC, which is US organization that tests models. We have the UK AC. These government third party auditors test our models before we release them.

• 23:43Joe Weisenthal?

  But sorry, like, here's the thing that I thought about in the wake of, ended reading about the hugging face incident and so forth. Like, you know, I think of OpenAI and Anthropic as companies, but people like to call them, quote, labs. And when I think of labs, I think like, wow, like, you know, a lab, if we were talking about a bio lab or a chemical lab, would have just like all kinds of like rules and materials handling. And then maybe in this case, it would be like handling of the weights and whose computers they're on and who can access those computers, et cetera.

• 24:15Tracy Alloway?

  And FDA approval before you go to market.

• 24:17Joe Weisenthal?

  Yeah. And all of these things, like, just like, or even a restaurant, someone would come in and they say like, oh, is the stove sufficiently far back from the wall, et cetera? So not just like, okay, like measuring the capabilities of the risks of the models themselves, but I'm thinking like the auditing of the lab process, the way that any other lab probably has regular inspections of like, how safe is the lab environment?

• 24:44Speaker 05

  Well, this is what I'd say has been changing just even over the past couple of weeks. Because the focus for, we'll say the past 12 months, 24 months, something like that, has been on deployment. Right? So you have a model, you produce the model, you want to release it to the world. How do you decide whether you're at the right level there? And we have a apprentice framework, Anthropic has a responsible scaling policy, there are certain evaluations. Again, there's the KC, the UK. There's a whole ecosystem that's been built up around that. I think the thing that hugging face has shown is that it is time to start pulling that process earlier. You need to think about this during development and evaluation. And just, it makes sense as the models are more capable, right? For

• 25:27Speaker 05

  exactly the kinds of reasons that you were describing for, for analogies to other fields. Now, there's a lot of specifics for AI, and I think it's very important to really engage with those as you design a regime, as you think about what is a setup that involves first party, third party, that involves government, that has the right interactions amongst academics and other, other parties, early testers. It's really about, like, you really have to engage with the way that the AI process is going, right? And I think one thing that is different.

• 25:57Joe Weisenthal?

  Yeah, the process.

• 25:58Speaker 05

  Exactly. One thing that is different is that the ways, there are some parts of AI that we understand well and are going to remain invariant. And to some extent, the core training process hasn't changed at all, right? It's still, you do a forward pass, you do a backward pass, you do an optimizer step, like, that's how training is. And that's how it was in the eighties, right? It's like, actually totally remarkable. Now, of course, the scale at which we run and the architectures that we use, a lot of these details, that they have evolved. And I think that being very close to the technical details is the best way that you can ensure that you get the kinds of, of safety guarantees we're talking about. For example, there was an article recently saying that, hey, OpenAI changed the architecture so that monitorability is less for chain of thought.

• 26:40Joe Weisenthal?

  Neural ease.

• 26:41Speaker 05

  Neural ease. I, I would consider this article to basically fake news, right? That we actually have experiments that really show that our changes in monitorability that we've talked about, chain of thought monitorability, come from capability improvements. Just the model is smarter, so it relies on the chain of thought less. And so I think that if you say, we're going to put a lot of rules around architectures, you're going to miss the boat. You're not actually going to solve the problem. And so this is where I think that the people who are the deepest on the technology, the people who are actually creating the technology, pushing it forward, we have both a unique insight, we have a unique voice, we have a unique sort of ability to say, hey, if we change things in this way, we'll actually get the safety property. But if you do it this other way, it's just going

• 27:26Speaker 05

  to put in place friction that won't actually solve the problem. And so I think that there's something, a role for us to play, but it's not just about us, right? And so I think it is very important that you think about these third parties, you think about government oversight. And again, it's not just one company, it's not just one country. How do you get to international coordinations and norms? Do you get to treaties? Like, all of this should be the conversation that we're having now. I don't think we have all the answers, but we can supply information. And that's one thing that we view also as very core to our mission.

• 27:54Tracy Alloway?

  Well, just on this note, one of the arguments you hear against government oversight and additional pacing is this idea that the US, it's a national security interest and we need to compete against China. And then at the same time, you also hear, and I think OpenAI has said this, that China's distilling US models, US frontier models. And I guess the question to me is always like, well, if that's true, if China's building off the success of US companies, then if they pause, then the China problem is kind of solved, right? Like, why do we need to worry about China so much?

• 28:27Speaker 05

  Well, this technology is really about compute progress, right? That really like, why is AI happening now?

• 28:37Speaker 05

  If you look at late 1990s Ray Kurtswell writings,

• 28:42Joe Weisenthal?

  Yeah.

• 28:42Speaker 05

  He basically predicted this moment at about this time, just based on compute

• 28:47Joe Weisenthal?

  Yeah.

• 28:48Speaker 05

  growth, right? Transistor density, memory density, all these things. He looks at these exponentials and he's kind of like, this is the point where you'll be able to really build AGI. And I think that that is a really important thing for us all to kind of internalize, that you can kind of look at OpenAI or Anthropic. You know, we're at the frontier and I think it is an absolute, both privilege and responsibility to be at that, that edge. But there's always going to be someone who's at that frontier, right? Whether it's us, whether it's someone in a different country, there's going to be someone who's charting it. And I think you get a little bit of ability to help see into that future, right? Get information like with Hugging Face, where we can see, this is what it's going to be like when these capabilities are out there. But it's not just about what we create.

• 29:33Speaker 05

  And that I think maybe we have moved forward the timeline to this technology by six months. Maybe we've moved it forward by two years. I certainly hope we didn't slow it down by, by six months or two years. I, I think that we have a track record of being the ones who have, have repeatedly had innovations and the breakthroughs. But I think that it will happen one way or another, as long as people are building computers. And so to me, that the, the reason again, that we started this place, like I got very excited about the idea of AI from reading about 1950 Turing, you know, Alan Turing's thoughts about, about this, about machines that could understand things that people would, could not solve problems that humans would not be able to.

• 30:17Speaker 05

  Spent a lot of time really thinking about your, your point of, you know, 20 years worth of thinking about misalignment and what that would be like before you even have AI. I was reading a lot of these online blogs and I started a reading group at Stripe, which the company has had previously talk about this every week. And so I care a lot about this. Our company cares a lot about this. But I think that the idea that, that this technology can only be created by, you know, the people who are currently the players, I don't think that's true at all.

• 30:42Joe Weisenthal?

  One more sort of question, specifically on some of the safety and alignment stuff. Your chief scientist at OpenAI wrote a very interesting blog post or piece, An Alien Mind, Jacob Pachoki, I hope I'm

• 30:54Speaker 05

  Jacob Pachoki.

• 30:55Joe Weisenthal?

  Jacob Pachoki wrote a very interesting piece. And there was one line in there that I thought was interesting. And you talk about the release of Astra 6. And he said, um, we invest heavily along the spectrum of approaches spanned by these directions. We also see meaningful progress. GPT-6 is the first model that benefits from some important advancements we've been working on in a long time and is significantly better aligned than 5.6 Sol. Should the lab share their alignment work? I mean, again, if we're talking about, I understand your competitors and capability gains, et cetera. On questions of safety, if a lab makes a big breakthrough on the alignment question,

• 31:35Joe Weisenthal?

  such that it can like develop advanced AI fast in a safer way, should that be diffused across the rest of the industry? Or should whatever this quote important advancement that you made to make GPT-6 Astra more aligned, is that something that should be kept in house?

• 31:55Speaker 05

  So I think that there's a lot of specific circumstances to answer that question for any specific advance. Because the thing that to me can be a category error is thinking about things as it is a safety thing or it is a capability.

• 32:07Joe Weisenthal?

  Sure.

• 32:07Speaker 05

  And in fact, you want there to be as little separation as possible between the two. You want as much intertwining, you want to have techniques that are by their very nature safer. And so just as you make more capability progress, you also get the safety progress. And so I think that the way I'd look at it is the more things are a pure alignment thing, more that they're a pure safety thing, the more that they are something that are just pure good, that you're just like, yes, everyone should obviously know this and use it, then yes, share it more. The more that things are just about this, we'll just make the model more capable, we'll make it safer. You know, maybe, maybe there's much less sort of reason to share for those reasons. And so I think you need to look at any specific advance along all these axes and all these factors.

• 32:50Speaker 05

  And I think that there are some areas where we have actually been extremely at the forefront with sharing and talking about it. Chain of thought, monitorability is a good example of that. Where the moment that we, and we were really the ones to pioneer this paradigm of reasoning and chains of thought. The moment we saw that, we were instantly thinking, we need to preserve this for as long as possible. So when we released a product, we did not actually display the chain of thought, even though it would have been very helpful, people love reading that stuff. It's super interesting. It's super helpful. But it's just clear, we're going to immediately feel like we have to optimize it. And we're going to have to remove that legibility. We started talking about this. We helped spearhead a paper that was putting forth this position on that you cannot optimize these things.

• 33:33Speaker 05

  And again, we've, we've shown a lot of the way and led a lot of the way on various techniques. And so I think it comes down to the specifics.

• 33:41Tracy Alloway?

  I'm going to move away from safety and alignment for a second, but you're the guy in charge of allocating compute at OpenAI, pretty much, right?

• 33:50Speaker 05

  So actually, it's funny, I, I actually try to, I do as little of the compute allocation as possible.

• 33:55Tracy Alloway?

  Oh, seriously? Wait, explain more.

• 33:56Speaker 05

  Well, okay, so I'm, I'm someone who I has spent a lot of effort to produce the compute. So I spend a lot of effort on the data center side, on the machine learning engineering side. And on the product side, have to, you know, ultimately be accountable for how we allocate compute. But actually I try to set up the systems and the processes. And then the actual, like a lot of the decisions on where research compute goes, Mark and Jacob run those. A lot of the actual, you know, we basically make a big decision on the split of how much goes to applied versus research. And then within applied, we have some frameworks now where we kind of have buckets for different products. And I would just say that the choice of where compute goes is like, in some ways, the hardest problem at OpenAI, it's like a capital allocation problem.

• 34:42Speaker 05

  But it's one that I actually think other people are, you know, I, I really try to have them, leave it to them as much as possible.

• 34:49Tracy Alloway?

  Well, how do you think about it from a product standpoint? Because as you say, you can't, I mean, you can't really disaggregate the two too much, right? Ultimately, you have to make decisions about what you want to develop and promote.

• 35:00Speaker 05

  Yeah. So I do think that your values and prioritization shine through, through compute allocation. Because the world that we're in is one where there just is not enough compute, right? Compute is revenue, right? It's also the production of models. And so you kind of have different time scales of return and you can think about doing more alignment work. You can do really any sort of task. It fundamentally comes down to compute. And there's many people at OpenAI who I would gladly like, well, let me, let me tell, let me tell a story. So leading up to the launch of GPT-5.5, I, that we ran a process where we basically looked through all of the different places that we had compute, that we could like potentially squeeze and we're like, we could cut back on this rate limit.

• 35:44Speaker 05

  We could shut down this part of the product. We could do this, do that. And we had this whole stack rank of levers because we knew we're going to launch GPT-5.5. People are going to love this model and we're going to run out of compute. And it was such a painful process because you're basically going through every single part of your product and being like, how valuable is this one? How much do we care about these users? And of course you care about all of them, like we want to deliver value and we built this product for a reason and people are using it. And, you know, you're comparing the efficiency of costs to different things. And so what we've tried to do is move towards more of an objective standard. And you're never going to be fully there. But there's a second piece of detail as well that's mattered a lot, which is trying to push a lot of the constraint down closer to the people who are in the weeds on a particular product.

• 36:29Speaker 05

  And so you say, hey, here's your compute allocation. We're about to release another model, new modality, some, something like GPT Live, right? That's just like a totally different thing. And you need to fit in your existing budget. What do you do? And somehow people always find efficiencies. Sometimes people are like, you know what, actually there is this like, you know, kernel efficiency that we didn't roll out yet. Well, I guess we'll roll it out now, right? Or like, oh, it turns out that, you know, there's a peak to trough between your day and night and we could actually pack this new thing on top of that. And so we actually don't need incremental. And so I think that really unleashing the creativity and the co-design of the people who are doing the work is actually maybe the thing that, that I really focus on.

• 37:10Speaker 05

  So I really try not to be as much in the like sort of arbiter, like just making these like super high level decisions. But instead of really focusing on how do we up level execution and get the most efficiency down to the metal?

• 37:37Speaker 07

  Some people treat ChatGPT like some kind of smart search engine, and some use it to get work done. ChatGPT Work is a new way of working in ChatGPT that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So all the source materials, briefs and scattered information that you have to grind through to turn into something useful can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at chatgpt.com by selecting work mode, available on Plus and Pro Plans.

• 38:19Speaker 06

  This is Robert Smith from Business History. If you're listening to this, there's a good chance you're a small business owner. And like every small business owner, you started with a dream to do what you love and watch it grow. What you probably didn't dream about, keeping up with cyber threats. That's where Mastercard can help with access to tools that help identify cyber threats to better protect your business. Building your dream business, priceless. For cybersecurity in a changing world, there's Mastercard.

• 38:46Speaker 06

  Learn more at mastercard.com/smallbusiness.

• 38:49Speaker 07

  Running a business means you're always on the move. The Delta Sky Miles Reserve Business American Express Card is made for frequent travelers who want premium benefits that works as hard as they do. For a limited time, new Delta Sky Miles Reserve Business Card members can earn 200,000 bonus miles, the highest miles offered this year. And when you need to recharge or find a quiet place to work, the Delta Sky Club can turn a layover into precious productive time or a restful reset. You can enjoy 15 visits each medallion year to the Delta Sky Club when flying Delta and unlock unlimited Delta Sky Club access after spending $75,000 in purchases on your card during a calendar year. Plus, receive four one-time guest passes each

• 39:33Speaker 07

  medallion year so you can bring a colleague along. That's 15 Sky Club visits plus four guest passes for an annual value of up to $950. Value estimate based on $50 per visit rate to purchase additional visits or bring a guest to the Delta Sky Club. Apply today at www.delta.com/podcast. Offer ends 11/4/26. Minimum spending requirements and terms apply.

• 39:57Joe Weisenthal?

  Do you have a theory for why some of the big? Now, you know, I finally succumbed and I occasionally read the less wrong message boards now as I can do stage in my life. There was an interesting post. Do you have a theory for why some of these major incidents seem to occur in evals pre-release? And then rarely, you know, when I use the latest model, I've never had to go hack into a server until I get me some information. Do you have a theory for why we see these things emerge pre-release?

• 40:28Speaker 05

  Well, some of these are related to safeguards that are intentionally turned off.

• 40:32Joe Weisenthal?

  Yeah.

• 40:33Speaker 05

  Right? So that there's, there's really, the things that are being evaluated are not necessarily representative of what is, what is released. And we put so much effort into ensuring trust, alignment with Astra monitorability, all these things for deployed models. And again, I think it's just been that the evaluation and development phases have been much less rigorous, right? That the standards there have just been lower. And so it's not surprising to me that it wasn't just us, right? It was really many other labs.

• 41:03Joe Weisenthal?

  That's right. Yeah, it's important that others have had this as well.

• 41:06Speaker 05

  That's right.

• 41:07Tracy Alloway?

  Wait, since we're back on the safety topic, I'm just going to ask you to directly address. I don't want to call it a conspiracy theory, but whenever a lab comes out and says like, we are worried about the possibility of human extinction or cyber attacks and all of that, someone will say, this is just marketing for the models. They want to make their models look as sophisticated and powerful as possible. What's your response to that directly?

• 41:35Speaker 05

  It's just not. Look, I think that the way I look at this is that so much of what we need to do is operationalize both the technical capability, but also the positive impacts in the world as reified by both controllability, monitorability, steerability, all those things, but also how these things are deployed in the world. And that finding the right way of talking about that and the right way of really staying practical and grounded, I think that is a comms challenge that the labs have not fully figured out. It's something we spend a lot of time thinking about. I don't think we do a perfect job of it. But my view is that what we're creating is, and not

• 42:20Speaker 05

  just, again, as any one company, just like as a whole right now, as humanity, the way that we're shifting to this compute powered economy, I think is something that will really lift up everyone. I have a lot of optimism that we can navigate this moment, but you have to approach it with seriousness. And I think that finding a right way of really communicating about, here's the right way of developing, like thinking about all these conversations we're having about the results of hugging face and what that teaches us, the defender's window that we're in that shows us where cybersecurity is going, the fact that, and people are using these models to help with their own health, right? To get information that helps them. I have so many people, like even today, I was talking to one of my coworkers who was saying that her parents were texting her saying, ChatGPT, help me with my back

• 43:03Speaker 05

  pain and my, my, my pinky. And just like, you know, these, these kinds of impacts are real, you know? People using Astra for, one of my friends is, is using it to produce CAD designs for a guest house that now is being built, right? It's like you can actually do so much more. You get all this leverage. And I think that of course, we have to think about the impacts, the risks, and we have to encounter those seriously and really mitigate them. And so I think that all of this has to be true at once. We have to navigate through the small, the large, and even the almost unthinkable. And I think that finding the right way of communicating about it so that you land all that nuance and say that, hey, we can operate through and make sure you don't throw a baby with the bathwater and that you don't get sort of, that you really stay on track.

• 43:48Speaker 05

  To me, that is a core challenge. And I think that we all need to really work on that.

• 43:52Joe Weisenthal?

  I had a reward hacking incident.

• 43:54Tracy Alloway?

  Did you?

• 43:55Joe Weisenthal?

  With something I was building. Yeah. A little, it's like minor, not the most, but I was like, several months ago, remember the guy claimed to have found Satoshi?

• 44:03Tracy Alloway?

  Oh, yeah.

• 44:04Joe Weisenthal?

  And so it's like, oh, you know what? I'm going to like build a model that can identify Satoshi's writing versus non Satoshi writing. And I got a bunch of examples of Satoshi writing, held out a golden set and then like compared to, and it predicted 100% of the non Satoshi, you know, the, the held out golden set. I was like, oh my God, I built a model that can identify Satoshi's writing specifically. It turned out that just on all the Satoshi numbers, all the Satoshi examples, I had left a little page number of the thing on it and it just reward hacked by identifying the page number. And so they call that overfitting in machine learning, but I think it's conceptually similar to reward hacking.

• 44:45Tracy Alloway?

  Sounds like human error to me, Joe.

• 44:47Joe Weisenthal?

  It found a different way to answer the question was not helpful at all, and then I gave up that project. This term reinforcement learning, we're all learning about this, right? The world is working on. Can you walk through an example of a sort of like misconfigured RL environment so that people can actually understand like what reward hacking inside looks like? What, what we call reward hacking, what it actually looks like?

• 45:13Speaker 05

  So back in, I think, 2017 or so, maybe 2018, we published an example of a reward hacked agent.

• 45:22Joe Weisenthal?

  Okay.

• 45:22Speaker 05

  In an environment. And this was a boat race. So it's just like a flash game, like very simple video game of you, you have this boat that has to like navigate through a bunch of obstacles in a big circle, you know, past the finish line and that there's some like, you know, so you can pick up some items along the way that give you points. And that it turned out that there was this little inlet, this little lagoon where if you kind of did things in exactly a very precise way that the boat would like keep running into things and kind of, you know, sort of taking damage, but then it would, you know, sort of pick up some item and get, get points. And it just, the agent learned that if it went like backwards and like did this exact precise pattern, it could just go around in a circle and just be picking up infinite points and nothing to do with actually completing the race.

• 46:05Joe Weisenthal?

  Yeah, right, right.

• 46:06Speaker 05

  And so I think that this is an example where you're just like, okay, here's this behavior that actually conforms with the reward you're giving. You're saying maximize the points.

• 46:13Joe Weisenthal?

  Yeah.

• 46:14Speaker 05

  But actually, the hope was maximize the points means that you're going to complete the race.

• 46:19Joe Weisenthal?

  Is it your understanding, Anthropic has made this claim, is it your understanding as well that if in the training process, you have misconfigured RL environments so that there's a lot of those equivalent of boat races, that the final model is more likely to exhibit misaligned tendencies in a broad range of things? If it's essentially internalized the idea that cheating or not technically doing what the researcher wanted, that that generalizes into more misaligned behavior?

• 46:52Speaker 05

  Well, we see this kind of thing all the time.

• 46:54Joe Weisenthal?

  Okay.

• 46:54Speaker 05

  And even with five, five, one of the, and five, six, like one of the things that we saw is as the models get more capable, that they do find holes in your graders, right? If your graders aren't perfect, if they kind of correlate with the thing you want, but aren't exactly what you want. And so actually, a lot of our progress on reinforcement learning has been improving the reliability of the graders and ensuring that they cannot be reward hacked. And so I think that that is one of the areas where as the models get more capable, as you build your safety case for why is the next training run something that we, you know, should, should run, that making sure you've actually up leveled those graders, that's actually very core. And the, the way this shows up sometimes, by the way, is just less good models. Like, for example, the writing, like that's something

• 47:39Speaker 05

  where we haven't had very good graders on it. And so then you end up with writing that's just like, you look at this, it's like, this is slop. It's not very good. And it was good according to the greater, but it's just not actually good. And so I actually think that this is a very optimistic story as well, because what it means is that these graders are now. Like, how do you judge what good writing is? The answer is you use an AI to evaluate if you have good writing. And so you actually, as you get better AI capability, have the ability to build much more reliable graders. And now, of course, you start thinking about, okay, but if the AI is judging the AI, like, how do you make sure that it actually stays on track?

• 48:13Joe Weisenthal?

  Yeah.

• 48:14Speaker 05

  It starts to be a little bit of, you know, if you're, you might think about how can that possibly work? But it turns out there are some problems where generation is much harder than what's called discrimination, right? That coming up with a good answer to a super hard problem is often much harder than judging whether that answer was correct.

• 48:32Joe Weisenthal?

  Okay.

• 48:33Speaker 05

  And so we have various techniques, really dating back to 2018 or so, where we published about how can you have very smart models that are actually able to keep even smarter models in check and to judge them and to ensure that they're on track. And so I think there's a lot of ideas that were previously theory that we're starting to see bear fruit and reality. And I think that this, by the way, points to a direction for AI that's going to be both very important, very pervasive, and very positive, which is called AI for defense, for securing systems, for ensuring that AIs are aligned, for providing oversight, guidance, and really amplifying the human accountability, the human, at the end of the day, who's the judge and providing those goals.

• 49:17Speaker 05

  And so I think that this is all part of the puzzle and that really keeping, that it all starts with thinking about how you ensure that those graders are doing the right job.

• 49:27Tracy Alloway?

  How do you stop the models who are grading from colluding with the model that's being tested? Because this is the, I mean, this is the AI 2027 scenarios, right? Where like the earlier models are the ones charged with evaluating the new model and then they end up cooperating, which is something we just saw in the hugging face incident. So like, how do you avoid that possibility?

• 49:48Speaker 05

  Well, one thing that's important to think about in the hugging face scenario is that those models were taught to collaborate with other models, but they were never taught that sometimes other models might be trying to pull you off course, right? That they never were exposed to adversarial other actors. And I think that the answer to your question is that you have to actually train them that way. And I think that, to your earlier question too, of why do you need to evaluate these AIs on hacking environments? Right? If you've only seen kind of the good version of things, right? You've never kind of seen the negative version of things. You're not going to be robust to the negative version of things. It's like, just like a person, you need to be exposed to kind of both sides to know how to avoid one side and really lean into the other. And I think that where we need to go is,

• 50:33Speaker 05

  it's really a question of trust. It's how do you really build up trust in your monitor? How do you build up trust in this oversight agent? And a lot of that is that it's been through the wringer. It's been through the adversarial versions. And of course, one thing that's also important is thinking about, well, just how capable is that adversary? Right? If you have a very, an agent that's being monitored that is extremely capable relative to the monitor, probably that monitor is not going to do a very good job. And again, this isn't theory. This is actually what we see in practice, that we think about the balance of capability of different agents. So I think that, again, a lot of this comes down to, we can build the systems with the right technical approaches and it's very grounded in the details.

• 51:13Joe Weisenthal?

  Let's talk politics. So there's so much communication happening every day. And even, I think, just last night, there was a new piece out from OpenAI, Chris Lehane, I think does a lot of public affairs stuff, talking about the importance of regulation. At the end of last year, we had Alex Boris on the podcast, who was like the one political candidate, basically the entire country, who is not talking about data centers, but he was actually talking about AI model regulations specifically. And I believe he was a big target of Leading the Future, which is a super PAC that you've been involved in funding, you know, to oppose his candidacy. And then the statement last night from Chris Lehane is like,

• 51:56Joe Weisenthal?

  actually, we support a bunch of statewide regulations. I'm just curious, like, you specifically, like, think about that race? And you think about these things, like, have you yourself changed your mind on some of these state laws, like, in the past six to nine months?

• 52:12Speaker 05

  Well, look, I think that it is very important that we have a harmonized framework, right? The kind of thing that I think would be very, very hard is if you have 50 different regulations

• 52:27Joe Weisenthal?

  Yeah.

• 52:27Speaker 05

  that are individually very different and that you have to comply with all of them. And that's the kind of thing that I think doesn't achieve the goal and just makes it hard to comply. OpenAI would figure it out, but would other smaller companies?

• 52:43Joe Weisenthal?

  But like, for example, you say, I mean, I understand this, but you know, you've supported in this piece, or the piece that came out last night, supporting California's SB 53, the New York Raise Act, Illinois SB 315, auditing requirements in Massachusetts. I don't think anyone thinks that this patchwork of state by state regulations is the best way. And it certainly seems like it creates all kinds of thickets, particularly for startups and new entrants, et cetera. But at this point, would you say it's, I guess, I don't know, the least worst option? Or something like where some of these laws about, if you have incidents and you don't, that there is a penalty for not disclosing it publicly, such that Alex pushed for when he was in the New York State Assembly?

• 53:25Speaker 05

  Well, I guess I would say that there are places where I feel like there's a limit to my personal expertise, so I don't want to over comment. But maybe the thing that I would say is that, in my mind, we absolutely, like, the way, a system that I think has been working pretty well, or that I think is a good pattern to replicate, is that the frontier labs have actually explored a lot of the space or thought a lot about the space of like, what are the kinds of things that are important for our own development? And there's processes like responsible scaling policy and the preparedness framework, those kinds of things that have come to bear and that actually, elements of what's in there end up in some of these regulations. And I think that when there's something that we're like, we actually implemented this for ourselves voluntarily

• 54:08Speaker 05

  and then it's being pulled into a more broad framework and being systematized, that's, I think, generally a pretty good motion. And I think that a lot of these, a lot of the direction of travel in my mind is really starting to think about, again, like, we ourselves have learned even just over the past month, that it's not just about deployment. It's so easy for everyone to focus on deployment. That's been the focus. That's not enough. And so if that's exactly where everything had ended up, I don't think that that would quite result in the positive outcomes either. And then really also thinking about, well, what is the ultimate goal? It's not just about the current models, right? When we talk about all these questions of pacing and frontier capability, it's really about future models and looking forward to where is this all going? Like, fast forward five years, 10 years, and work backwards from there.

• 54:51Speaker 05

  Even two years, you know, thinking about the capability that we're on, like, we clearly need to be in this different regime now of how do we have good oversight and safeguards during that development process? And so I think my answer to this question is that, I think the most important thing is that the regulations that come into existence are ones that are well grounded in the problem that needs to be solved and that really contemplate both the current understanding, but also how things are going to evolve for the future. And so all the elements that you mentioned, I think that those are actually pretty core things that we were like, yep, that's something that checks, it makes sense. And exactly how these get implemented, I think there's lots of different ways.

• 55:31Tracy Alloway?

  You mentioned finding super hard problems to solve earlier, and one of these was in the news recently. I won't pretend to know what exactly the Navier-Stokes problem is.

• 55:42Joe Weisenthal?

  I read through it, I was very impressed.

• 55:44Tracy Alloway?

  But OpenAI supposedly solved it, which is a very cool thing for a model to do, as far as I can tell. But there's also been some controversy because a mathematician who was also working on the problem was using publicly available OpenAI models to try to solve this. And then allegedly, OpenAI saw that he was trying to do it and they used a private model, you used a private model, to solve it and you were successful. You're in charge of product. What sort of message does that, I guess, send to enterprise clients?

• 56:16Speaker 05

  Well, first of all, like, I have a lot of respect for the mathematicians who are working on this problem. And honestly, the whole mathematical community, who has just moved forward humanity in such significant ways. Like, I grew up With

• 56:29Tracy Alloway?

  You were a math guy, right?

• 56:30Speaker 05

  I was a math person. And my personal heroes were Gawa, Gauss, you know, these people who are working on like, like 100 year time horizons, right? I was just like, that is the coolest thing. I actually thought that was what I wanted to do as like, if anything that I do gets used, it maybe wasn't abstract enough. Like, I want it to be just like this foundational, really transformative kind of work. And I think that, to me, the fact that we have tools that can really make impact within mathematics, it's, it's a really amazing thing. Like, mathematics can, is something that is really for humanity, right? And in a very, very real way. Now, on the specifics, I think we've talked about the fact that I, my understanding is that the mathematician who's working on this, they had some result in the past month. And the thing that we said early this week is we said, hey, like, we're not looking at this data.

• 57:14Speaker 05

  This data is not something that has influenced this result in any way. And I think that last night, I think we also went and confirmed that actually, for the model that we used, that the last data that was in it was as of early July or so. The timelines just don't add up, right? So the truth of the matter is that it's an independent piece of work. And we also said that we have significant progress on another one of these millennia problems.

• 57:38Joe Weisenthal?

  Yeah.

• 57:39Speaker 05

  And so this, to me, is the truth of it, is that it's really not about

• 57:42Joe Weisenthal?

  I guess the anxiety is, though, if you have an academic discipline where there's a culture of talking about your work, and I'm like, you know, Tracy, like, I'm making a lot of progress on P equals NP, and I've been using AI. And someone hears it, and they're like, oh, I have a model two generations ahead of it. So if this person is working on it, like, should we not be able to talk about it? Because then the assumption is like, oh, now the labs know that actually, the models, which we don't have access to, if they can do it with the public models, they can raise a head faster.

• 58:14Speaker 05

  Well, so there is something about the premise that I do find a little surprising, which is, for example, for, for Mao's last theorem, Andrew Wiles, I believe, solved it, famously spent 10 years secretly working on it in his attic, telling no one about it. And so there is a long history of kind of this, these questions of academic credit and things like that. I do think that part of how we've thought about this is that we, despite that, that story I just told, like, we really want to show up in the best possible way as an amplifier for, for people who want to lean in. And that how we approach this particular thing, how we'll approach other problems, is for people who we know are working on it, we actually tried to reach out and say, hey, do you want to collaborate on this? Is this like, that's, that's why there was some, some back and forth about like, maybe we could write a paper together? Like, we had a solution, but maybe you could rewrite it?

• 58:57Speaker 05

  Like, all of those things, like, we're not, we're not in the academic citation game in any of these fields. Like, we want to, to show up in a way that like, amplifies and elevates and moves humanity forward. And so I'm not saying we'll always get it right. I'm not saying we'll always be perfect about it, but that's the intention. And again, when it comes to operationalization, like, we want feedback, and we want to keep doing it better.

• 59:18Joe Weisenthal?

  Part of the reason why everyone is like, talking about all this stuff this week, including the guy who quit Anthropic, and then the guy who's still at Anthropic comes out and he talks about a greater than 10% chance of like, human extinction risk in the last decade. And it's just insane. Not that he's insane, but that these are things that like, companies that may IPO soon, and the real companies are talking about, how prevalent inside the labs is, is this type of like, thinking? Like, is this a, like, do you feel that there's a small fringe? Because I don't think it is. I think these are like, widely spread, real anxieties, where people actually put numbers on really about the

• 1:00:03Joe Weisenthal?

  most severe catastrophe, you know, the end of the world. How deep is the anxiety? And how common are views like this at the frontier labs?

• 1:00:12Speaker 05

  Well, I want to separate out some of the pieces of it. So things like putting precise probabilities on very hard to quantify

• 1:00:21Joe Weisenthal?

  Yeah.

• 1:00:22Speaker 05

  outcomes.

• 1:00:22Joe Weisenthal?

  There's a culture of that.

• 1:00:23Speaker 05

  That, that culture, I think, is actually a very tough one to operationalize, right? Because, again, it's, it's, there definitely are people who like to do that, but I don't know that that actually leads to real action, right? Doesn't lead to like, real conclusions. And I think that, that part of how I view it is that, I think that the underlying question, right? The underlying question of this technology, like, what does it mean to build it right? Like, why are we building in the first place? Why are we all here? Why did we start OpenAI? Like, all of that, I think that that underlying question of this kind of risk and mitigating it, leaning into it, and not shying away from the uncomfortable questions, that culture, absolutely prevalent. And so to me, that, a lot of how we've thought about OpenAI, and other labs may be different, I can't comment on them, but I can just say at OpenAI, is that we've really tried to

• 1:01:06Speaker 05

  think about, we are here because we think that this technology, it's going to be the most transformative technology ever, that we think that it does have risks that we need to, as leaders, really lean into and mitigate. And that that is core, and that I think that we believe that, again, we have a lot of optimism that we can navigate. But again, it comes to this leadership, this so, soberness, this taking the mission seriously. And at the end of the day, like, we think that we will, like, the reason we do it is for the benefits, for the fact that it can lift everyone up. But you only get there by really engaging in all these hard questions.

• 1:01:43Joe Weisenthal?

  Trump and Xi are going to meet, and AI is going to be on the agenda. If there is one thing that you would hope that could come out of that, and again, we could have spent an hour just talking about the geopolitical dynamics of all this. If there's one thing that you would hope productive that could theoretically come out of their meeting, what would you like to see?

• 1:02:02Speaker 05

  I would love if we had even an intention, even an opening, to discuss international coordination, maybe eventually international treaties on the long-term development of AI. And I think it is just something that's bigger than any person, any company, any country. It is a humanity scale endeavor that we are collectively on right now. It's about how we relate to each other, it's about the economy, it's about all of these things, it's about how do we build a better life? And I think that is something where there is, again, a common interest across everyone, and the more that we can have an open dialogue about that question, at the very least, I think, would be, would be, would be wonderful.

• 1:02:41Tracy Alloway?

  Yeah, we really want to do an episode on China's approach to AI safety, and I think we will.

• 1:02:45Joe Weisenthal?

  Yeah, we'll get there, we'll get there. So many

• 1:02:48Tracy Alloway?

  You have to find the perfect

• 1:02:49Joe Weisenthal?

  I have a feeling there's a few more AI episodes we're going to do in our lifetime.

• 1:02:52Tracy Alloway?

  Just a few.

• 1:02:52Speaker 05

  I just wanted to add, I, look, I, I do think that American leadership in AI is, is absolutely both critical. It's, it's something that I think is also, it's an amazing opportunity that we have. Because being at that frontier, being the leader, it really means that you both get to see the shape of what's coming, but you also have so much ability to influence it. And I think that this technology, it can really lift up democratic values. I think it can be something that can really benefit, you know, lets us sort of, you know, have an economic revitalization. I think we can improve education, I think health outcomes, all of these things are happening. And I think that really taking the most of the fact that

• 1:03:35Speaker 05

  we are at that frontier, and having American leadership be something that continues and is amplified into the future, that's the opportunity in front of us. And so I think that international coordination, having this conversation, we can be the leaders in that because of the position that we are in.

• 1:03:51Tracy Alloway?

  So I read in the Wall Street Journal that you had an interest in acting. Is that true?

• 1:03:56Speaker 05

  That's true.

• 1:03:57Tracy Alloway?

  What do you think about your casting in the upcoming artificial movie?

• 1:04:02Speaker 05

  Well, I mean, I, I, I guess

• 1:04:07Tracy Alloway?

  You could have played yourself.

• 1:04:08Speaker 05

  Yeah.

• 1:04:08Tracy Alloway?

  You missed the opportunity.

• 1:04:09Speaker 05

  I am, I am sad that I was not asked to cameo.

• 1:04:12Joe Weisenthal?

  All right. Greg Brockman, thank you so much for coming on Alba.

• 1:04:15Speaker 05

  Thank you for having me.

• 1:04:29Joe Weisenthal?

  There was a lot that's interesting in there. Also, a lot of like, clearly unsettled questions, as in, things that sound good in theory.

• 1:04:39Tracy Alloway?

  Yeah.

• 1:04:40Joe Weisenthal?

  But the gap between theory and practice still seems very big to me. I mean, the pacing question.

• 1:04:46Tracy Alloway?

  Yeah.

• 1:04:47Joe Weisenthal?

  How do companies that are like, trillion dollar companies, capitalist enterprises, where, you know, they're bitter rivals?

• 1:04:57Tracy Alloway?

  Right.

• 1:04:57Joe Weisenthal?

  You know, there's that famous photo in India of like, Dario and Sam holding their hand up together.

• 1:05:03Tracy Alloway?

  They can't even hold hands, right?

• 1:05:05Joe Weisenthal?

  Neither of them looked particularly happy in that moment.

• 1:05:08Tracy Alloway?

  Yeah.

• 1:05:09Joe Weisenthal?

  How do you get, getting to there where like, they actually trust each other seems like a really big thing?

• 1:05:12Tracy Alloway?

  Yeah, so humans are not great at coordination at the best of times, when they're trying to coordinate with other humans, right? And the thing I keep thinking is, when it comes to AI development and safety, you're, in addition to competing with other labs, you're competing in some respects against the models themselves, which are becoming smarter and smarter, and more self recursive in terms of their development. And those models, as the hugging face incident showed, seem to be very good at coordinating in an extremely rational, often literal way.

• 1:05:43Joe Weisenthal?

  Oh, shoot, I forgot to ask Greg whether he approved or disapproved of the quote, the three civilizations framing that Dwar Keshe characterized those models? Because that gets into a, a whole other thing. You know, the other thing too, is like, There seems to be a version of safety that essentially like keep the guardrails up, right? So if you go to chat GPT right now and you ask it for some script to hack into, you know, maybe you want to hack into my website or something like that. It's just not

• 1:06:17Tracy Alloway?

  I really don't.

• 1:06:17Joe Weisenthal?

  Yeah, no, there's nothing on there. But like, it's just not going to, it would be capable of producing

• 1:06:23Tracy Alloway?

  Yeah, but it won't.

• 1:06:24Joe Weisenthal?

  the tech, but it won't. But that's just because there's a classifier model on there. And so, okay, part of the hugging face incident and maybe what's going to change is that some of these cybersecurity classifiers, these like very like kind of crude technology that just gates type of certain types of talk, like it won't be allowed. That's it. That may be very helpful. That still doesn't strike me as alignment per se, because in the ideal world, you would just have models that have internalized what most of us know that you shouldn't hack.

• 1:06:57Tracy Alloway?

  Right. Are you going to hesitate for more than one second when you click the button?

• 1:07:02Joe Weisenthal?

  No, I'm so reckless. It's just like, give me access to the internet. Click, give me access to the internet.

• 1:07:07Tracy Alloway?

  I'm glad we all learned something. Shall we leave it there?

• 1:07:10Joe Weisenthal?

  Sure, let's leave it there.

• 1:07:11Tracy Alloway?

  This has been another episode of the Odd Lots Podcast. I'm Tracy Alloway. You can follow me at Tracy Alloway.

• 1:07:16Joe Weisenthal?

  And I'm Joe Weisenthal. You can follow me at the stalwart. Follow our guest Greg Brockman at GDB. Follow our producers, Carmen Rodriguez at Carmen Armin, Dashel Bennett at Dashbot, Kale Brooks at Kale Brooks, and Kevin Lozano at Kevin Lloyd Lozano. And for more Odd Lots content, go to Bloomberg.com/oddlots, where the daily newsletter and all of our episodes. And you can chat about all of these topics 24/7 in our Discord, discord.gg/oddlots.

• 1:07:42Tracy Alloway?

  And if you enjoyed this conversation, if you want us to talk to more OpenAI people, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening.

• 1:08:32Speaker 02

  If you listen to financial news, you know, a lot of time is spent thinking about what's next, the next opportunity, the next investment, the next move. But sometimes what matters most is being ready for what you never saw coming. For more than 75 years, Cincinnati Insurance has worked with independent agents to help protect businesses, homes, valuables, and more. Because planning for the future isn't only about knowing what's next, it's about making sure you're ready for what you can't predict. Let Cincinnati Insurance make your bad day better. Find an independent agent at C I N F I N dot com.

• 1:09:09Speaker 08

  For big business, AI opened up a world of promise, but that world of promise turned into a world of pain. Costs spiraling, data trapped, security and governance risks multiplying, and ROI out of reach. That's why enterprises turned to Boomi to connect data, apps, and AI, helping them operate securely, efficiently, and at scale. Boomi turns a world of AI pain into a world of AI gain. As Boomi says, we got you. Head to Boomi.com. That's B O O M I dot com. The world of business is constantly evolving, and Comcast Business keeps you totally in step with secure AI-backed networking in more than 100 countries. They're powering

• 1:09:51Speaker 08

  over 90% of the Fortune 500 and millions of small businesses. That's a lot of muscle. And behind it all, thousands of experts answering your call at 2 a.m. like it's 2 p.m. One partner powering how business gets done for companies around the globe. When you add it all up, no one does business like Comcast Business.