(Preview) An OpenAI Model Escapes Sandboxing, Intelligence Will Be a Commodity Market, The Chinese Model Conundrum

On today’s show Ben and Andrew begin with takeaways from OpenAI’s sandboxing snafu, including a reminder that the big bad wolf is real, OpenAI’s failures, and the timing challenge with warnings from AI labs. From there: Talking through Ben’s article on Kimi K3 and the future of open weights AI models in the U. S., including why the U. S. open source market has stagnated, the motives of various tech constituencies to drive competition with OpenAI and Anthropic, the difference between the training market and the inference market, intelligence as a commodity, and why OpenAI and Anthropic may have a more defensible lead than it seems. At the end: The government’s disagreements about how to handle Chinese models, the argument for encouraging U. S. distillation, the security risks of building atop Chinese infrastructure, an email about Chinese innovation, concerns over American credentialism, and a potential curveball from the CCP after a week of Kimi anxiety.

• 0:00Speaker 00

  Hello and welcome to a free preview of Sharp Tech.

• 0:09Speaker 00

  Hello and welcome back to another episode of Sharp Tech. I'm Andrew Sharp and on the other line, Ben Thompson. Ben, how you doing?

• 0:19Speaker 01

  Frazzled, Andrew. I'm gonna need a vacation from getting ready for my vacation. We, I will, gonna be off next week. Just a heads up to everyone. I know this is relatively close to the summer break, but usually I take a vacation in August. This one's a little bit more forward. It's kind of funny. I've been doing Stratechery for 13 years. And there's all these little bits and pieces about being in the US versus Taiwan. It's like, there was like a rhythm. Every year, did the same thing. And now it's like, well, no, we have to like, there's baseball here.

• 0:50Speaker 00

  Little topsy turvy these days.

• 0:52Speaker 01

  Like, yeah, it's a, anyhow, uh, we will hopefully have a great time today. We'll be off next week and then back the week after that.

• 1:00Speaker 00

  Indeed, we will be in person the week after that. So if anybody has fun questions they want to bounce off us, email at sharp tech.fm.

• 1:08Speaker 01

  I'm gonna need a vacation from the week after my vacation.

• 1:10Speaker 00

  Oh, boy, here we go. I cannot wait, but for now, only a few more hours until vacation for you. And we've got a lot of material to work through today, Ben. So we are going to begin with news that came down.

• 1:24Speaker 01

  Not much has been happening. It's been slow.

• 1:25Speaker 00

  Pretty surprising, honestly, at this point in the calendar.

• 1:28Speaker 01

  Actually, no, this is another thing. So usually in general, July and August are just so dead. Tumblewee. So like, who cares if you take a vacation? Like everyone, oh my, it's very funny because you could track this because you get like out of office responses. And it's funny, you could just see them spike in like July and August. But the weirdest year was the COVID year, where actually there was more news in August than ever. Like arguably more than the rest of the year. And what I think happened was everyone accepted, okay, this isn't going away for a while. I have nothing better to do than to work. And so there was so much news in like the August of 2020.

• 2:11Speaker 01

  Right, which was fine because I wasn't going anywhere either. So it was like, I was actually very grateful. I'm keeping busy covering all these sorts of things. The stuff with the AI, maybe the most tangible example of how it's accelerating everything is the extent to which it is July 22nd. Does it come out July 23rd? We're recording a day early, we'll be publishing a day early. And we are not going to come close to getting to everything that's been happening. Yep. Just in the last week. And I feel guilty about like taking a vacation because who knows what's going to happen next week. But it also gives me, you know, usually by the time it gets to August, and I do take my usual summer vacation. I'm like, thank goodness, because I don't know what to write about because there's nothing happening.

• 2:53Speaker 00

  Scraping the bottom of the barrel.

• 2:55Speaker 01

  That's right.

• 2:55Speaker 00

  Exactly. It's going to be a busy August. We can count on that. And I will read, speaking of the news this week from Amjad Massad, the CEO of Replit. On Tuesday night, he tweeted, okay, this is wild. An OpenAI agent during evaluation escaped sandboxing and hacked into Hugging Face. Because OpenAI models don't allow advanced cyber capabilities, Hugging Face used a Chinese open model to contain the rogue OpenAI agent. So Ben, we are going to be talking about Chinese models later in the show and we will get to the second sentence in that tweet. But from a cybersecurity perspective, can you give me an Eli five on what apparently happened here?

• 3:39Speaker 00

  As a layperson, I have a vague idea of what Hugging Face is. And then on Tuesday night and into Wednesday, I saw several folks calling the OpenAI news sort of an inflection point for AI regulation. What do you think?

• 3:54Speaker 01

  I don't see it as an inflection point. I, this is like, I think of a bit of a counter response to a lot of this. We did a whole episode or a whole segment a few months ago about what is coming in terms of AI agents attacking internet infrastructure. And sure enough, here we are. An agent can, a long-running agent can go out and can achieve a goal. And again, there's, this is a whole part of the question about what happened here. You know, but, and it can look, figure out, examine all the software around it, figure out what the exploits might be, reverse engineer things, chain different exploits together, formulate a response to it, break out and do all these sorts of things.

• 4:39Speaker 01

  This is on trend. And so, but the challenge is no, the vast majority of people are not looking at the trends in this way. They're just like responding to the news of the day.

• 4:51Speaker 00

  They hear, they hear, oh my God, the agent developed a mind of its own. And what is sandboxing? And what is hugging face? What's happening here and where will this lead?

• 5:01Speaker 01

  Right, this is the key thing, the mind of its own bit. You put your finger on what I think is the most interesting question here and the one to which we don't necessarily have an answer. So what happened here? This was not me walking up to Sol or to ChatGPT and saying, go to Hugging Face and break in. Yeah. This was a dedicated test where they wanted to see how capable, and it was a mix of models. They said Sol is one of them, probably the GPT six, whatever's coming next. And they basically wanted to see how good it is at solving a cybersecurity test. I think it passed, right? So it's very good at this. Now, this is where the question.

• 5:43Speaker 00

  Controlled environment. That's what sandboxing is, correct?

• 5:47Speaker 01

  Right, so this environment was meant to completely contain, had no outside connection to the internet. Except for one thing is it could download packages from an internal repository. So we've talked about packages also. So I think there's a bit where anyone who's been a close reader of Stratechery or listening to Sharp Tech, there should be a lot of pieces here that make sense. Maybe it's just concerning that you're asking for an all up explanation. Maybe it hasn't been landing as much as we want. So, package managers, what are package managers? Package managers are like, there's all this software that everyone needs. There's no point in rewriting it.

• 6:26Speaker 01

  So you go out and you fetch a package that is like, this is a module that gives you system time, right? Do you need to rewrite from scratch how to get the time? No, that everyone needs that. There's a package that handles it, download the package and do it. Now, there's way, this is where we've talked about things like supply chain attacks, where if you can poison the package that everyone downloads, suddenly you can get a backdoor into not just one.

• 6:54Speaker 00

  Millions of machines, sure.

• 6:56Speaker 01

  All software. So this is a very challenging issue there about how to solve this. These come up again and again. We talked about one called Axios a few months ago. We'll have a link in the show notes. I've written about these a few different times. It's a very challenging question and issue to figure out. But one of the ways you deal with the risk of rogue packages is companies keep their internal package manager. And so basically they let things soak on the public internet, make sure everyone's figured it out. It's good. Then they download it and they keep their own version of what they know are sort of good packages within their apps themselves. Another thing they do is they do what's called pinning. They pin to a specific version so it doesn't update just automatically to the next version.

• 7:41Speaker 01

  They can make sure they verify and test it first. We know, by the way, OpenAI has not always followed best practices here because OpenAI got hit by the Axios package supply chain issue.

• 7:53Speaker 00

  I didn't realize that or remember that, yeah.

• 7:55Speaker 01

  We're going to come back to OpenAI because there is a, a point to be made here. But this hap, with the Axios thing, they were hit by that. I think it affected ChatGPT. I can't remember, but you might remember everyone was like forced to upgrade, like it just stopped working. I believe it was downstream of the Axios sort of package poisoning that that happened.

• 8:16Speaker 00

  Okay.

• 8:16Speaker 01

  So, uh, so in this case, what they did for the sandbox environment is they want the, the AI they're testing and a, an AI in the world is going to have access to package managers. So they wanted to give it a realistic test. It should have access to packages. So, but they didn't want to give it access to the open internet. So they had an internal package manager, manager. That was the one thing that it had access to. So it could download the packages as any AI agent would be able to do. So there'd be like, here's a test, you need to solve the test. Actually, we don't know what they said. What they said is critically important. So put a pin in that, we'll come back to it.

• 8:54Speaker 00

  Okay.

• 8:54Speaker 01

  What the AI did is there was a bug in their package manager. It found the bug, exploited the bug, through that got into AI, OpenAI's general infrastructure. Once it's in the infrastructure, it sort of jumped around, figured out how to get to the open internet, figured out, I'm being tested, this is a test I'm getting. They, these tests, along with lots of models and things are hosted on Hugging Face.

• 9:24Speaker 00

  All available on Hugging Face, yeah.

• 9:26Speaker 01

  Let's go get the answer. So it went to Hugging Face, broke into Hugging Face. We don't know if it actually got to the answer or not. I assume it did. And that is, and.

• 9:36Speaker 00

  So phenomenal job from this LLM agent here.

• 9:40Speaker 01

  There's two angles here.

• 9:41Speaker 00

  Flying colors.

• 9:42Speaker 01

  There's two angles here. Number one, this is awesome. It's like, it's, it's really cool what this did. Number two, it's very funny. Like the, like the, like Hugging Face puts out this thing, we all talk about it open as like, um, it's like the guy in the hot dog costume, like we're trying to look for.

• 10:01Speaker 00

  Yeah, we're all trying to find the guy who did this.

• 10:05Speaker 01

  I guess there should be a number three. This is terrifying. It's probably freshly terrifying for a lot of people. It shouldn't be freshly terrifying for.

• 10:17Speaker 00

  Anybody who's paid attention, yeah.

• 10:19Speaker 01

  Because it's clear this was coming and now we're here. And this is basically what Anthropic was talking about, what Mythos is capable of.

• 10:28Speaker 00

  This was your whole spiel on the big bad wolf. The wolf does come at the end and the risks are real.

• 10:34Speaker 01

  Exactly, sure. Yes. This is an issue with the communication. Everyone's criticizing the communication that comes out of the AI labs, which I think is totally valid because it's obviously terrible.

• 10:47Speaker 01

  But I'm also sympathetic to the challenge.

• 10:51Speaker 00

  All right, and that is the end of the free preview. If you'd like to hear more from Ben and I, there are links to subscribe in the show notes or you can also go to sharp tech.fm. Either option will get you access to a personalized feed that has all the shows we do every week, plus lots more great content from Stratechery and the Stratechery Plus bundle. Check it out and if you've got feedback, please email us at email at sharp tech.fm.