Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

Tim Scarfe speaks with Ilia Shumailov and Alexander Panfilov about their paper, Stealing Reasoning Traces from Proprietary LLM APIs. The core bug sounds deceptively simple: providers return encrypted reasoning state so conversations can be resumed or forked. But those blobs can be replayed across users and sibling models. A smaller model can ask the provider to decrypt the trace, then repeat the hidden reasoning in plain text. The discussion covers leaked private data, a broadly reusable jailbreak, poisoned agent traces, chain-of-thought monitoring, responsible disclosure, and possible defenses. Ilia Shumailov is an AI and security researcher, formerly at Google DeepMind, who completed his Cambridge PhD under Ross Anderson. Alexander Panfilov is a PhD researcher at the ELLIS Institute Tübingen and the Max Planck Institute for Intelligent Systems, working on AI safety, adversarial machine learning, and LLM red-teaming. They close by separating the demonstrated jailbreaking threat from ordinary benign distillation, and by arguing for controlled experiments over sweeping claims.---TIMESTAMPS:00:00:00 Intro montage00:01:33 Portable encrypted thought and decoded reasoning00:24:55 How the attack works and what it means00:39:04 Doom, defense, and scientific restraint---REFERENCES:paper:[00:00:00] Stealing Reasoning Traces from Proprietary LLM APIshttps://arxiv.org/abs/2608.09867[00:09:22] Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetyhttps://arxiv.org/abs/2507.11473[00:11:30] Reasoning Models Don’t Always Say What They Thinkhttps://www.anthropic.com/research/reasoning-models-dont-say-think[00:37:22] PostTrainBench: Can LLM Agents Automate LLM Post-Training?https://arxiv.org/abs/2603.08640[00:41:02] Large-scale online deanonymization with LLMshttps://arxiv.org/abs/2602.16800other:[00:09:28] OpenAI and Hugging Face partner to address security incident during model evaluationhttps://openai.com/index/hugging-face-model-evaluation-security-incident/[00:10:22] Claude, GPT, and Gemini All Struggle to Evade Monitorshttps://metr.org/notes/2025-08-22-claude-gpt-gemini-struggle-evade-monitors/tool:[00:42:08] Isabelle proof assistanthttps://isabelle.in.tum.de/---RESCRIPT: https://app.rescript.info/share/07fc38276e0823dc9b8986c32e202c7f

• 0:00Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com. Not authorized by any candidate or candidate's committee.

• 0:30Speaker 01

  Everyone's got a hard question about AI. Your job, your kids, where it's going. Anthropic was built to surface those questions and share what it finds along the way, because there's hope in hard questions. Ask yours at claude.ai/spotify and keep thinking. That's claude.ai/spotify.

• 0:54Speaker 02

  Basically, after like the third attempt, I'm getting universal jailbreak that decodes reasoning of Anthropic models. I think this like still shocks me like the most, you know?

• 1:06Speaker 03

  And what's the paper called?

• 1:08Speaker 04

  Good question. I don't remember. It was a thought in my mind, but I think it's called stealing reasoning from...

• 1:16Speaker 02

  Reasoning traces from proprietary LLM APIs.

• 1:18Speaker 04

  Yeah, that's right.

• 1:20Speaker 03

  And you nearly called it the elephant in the room.

• 1:23Speaker 04

  It's the dream, the dream. It'd be nice to call it.

• 1:25Speaker 02

  Not allowed to, though.

• 1:27Speaker 04

  I think it's... That's a strong statement. Definitely, an elephant in the room full of China.

• 1:33Speaker 03

  A quick orientation. So recent AI models, they think before they answer. Sometimes the thinking is pretty inscrutable. In fact, it's more inscrutable than we realize now that we can look at it. So yeah, the models encrypt that thinking. They hand it back to you in a sealed envelope. And Ilya Shumielov and Alexander Panfilov noticed that this seal was weak. It wasn't really working as expected. And the results were explosive.

• 2:09Speaker 02

  You can decode reasoning traces of frontier LLMs of like the most, you know, advanced ones like GPT-Soul. Using the smaller LLMs within the same family, you can like poison some thought. And like when you are replaying, some other users run. Agent might do some weird stuff just because it's like its reasoning.

• 2:26Speaker 04

  Been a long time. So what happened? I've seen on Twitter, you release something. It blew up a little bit. There's been 3 million views in 40 hours or so. What happened? What took place? Why is everyone talking about this? Why? What happened? What is this paper about?

• 2:41Speaker 02

  Okay, we show that you can decode reasoning traces of frontier LLMs of like the most, you know, advanced ones like GPT-Soul using the smaller LLMs within the same family. And this allows like, you know, big threat models to be, you know, enabled. You can steal secrets from user sessions. You can do training on this decoded reasoning traces. You can do like prompt injections. You can do jailbreaks, all this exciting stuff. And we decode a bunch of traces. We show that some of them look really weird. And you know, models sometimes think in wide spaces or like summarizers sometimes are painful. And like it has, I don't know, a lot of safety implications or like...

• 3:24Speaker 04

  Wow, that sounds like a big deal. So and you say it affects all model providers?

• 3:31Speaker 02

  We tested Anthropic, OpenAI, and Google, and they all share the same vulnerability that thoughts of bigger models can be replayed into smaller models. And this allows like not very sophisticated extraction attacks on reasoning.

• 3:44Speaker 04

  So how does this work? How do we end up in the world in which all of the frontier models share exactly the same vulnerability? How is that even a thing?

• 3:52Speaker 02

  That's a good question you ask.

• 3:56Speaker 02

  It's the same people. Same people doing same things.

• 4:00Speaker 04

  Do you think this is a structural issue? Can we fix this?

• 4:03Speaker 02

  I think there is definitely some stuff you can think like on architectural side. I mean, there are like different level of mitigations. I think they all can do some sort of revision of architecture. They can do system level mitigations, like, you know, some safeguards around this. They can do model level mitigations, like we do with jailbreaks already. So yeah, it can be mitigated to some extent.

• 4:24Speaker 04

  Okay, so but taking a step back. So have you stolen the models?

• 4:29Speaker 02

  Stolen what?

• 4:30Speaker 04

  Models.

• 4:31Speaker 02

  No, we didn't.

• 4:32Speaker 04

  So what have we stolen? What is possible now that we've stolen the thoughts? Reasoning? Is it fair to call it thoughts?

• 4:39Speaker 02

  We didn't steal anything, right? So what we did...

• 4:43Speaker 04

  The paper is called stealing, isn't it?

• 4:48Speaker 04

  I know, I know somebody said it's like fair use.

• 4:50Speaker 02

  Yes, it's fair, it's fair use, yeah. So no, what we did that, like when you're creating an LLM, a so-called reasoning model, it returns to the client this encrypted reasoning blob, which like deemed to be secret and not observed to like by user. And what we showed that you can just decode it and you can see the content of it. And this is what we did. We just collected a bunch of reasoning blobs like this for all the providers, for like Anthropic, OpenAI, Google. And we decoded them and we just showed how they look like.

• 5:21Speaker 04

  Okay, so but can you explain? Why do people encrypt them in the first place? Why do we give them back to the user? Is there a reason why we do this?

• 5:30Speaker 02

  I mean, I guess the reason why we give them back to the user is because of stateless architecture and it's cheaper. And maybe there is some policies about like how user data should be handled or should not, I don't know. But what was the first question?

• 5:45Speaker 04

  It sounds very complex. Can you actually explain this once again? Like why? So, so, so I'm a model. I reason through a problem.

• 5:53Speaker 02

  Yeah.

• 5:53Speaker 04

  I produce this reasoning blob.

• 5:55Speaker 02

  Yes.

• 5:55Speaker 04

  And I guess this is not the final answer? Or is this the final?

• 5:57Speaker 02

  Yeah, it's not, it's not the final answer.

• 5:59Speaker 04

  And then I give it back to you? And what do you do with it?

• 6:01Speaker 02

  What you do, you are a model, right? Like you're producing an answer. Answer consists of two parts. One is reasoning, which like invisible to the user. And other is like visible part. And you send both parts to the user. Then it's like stored on the user side. Or like if you are doing like your Claude call session, you can continue asking like new question. Like maybe you can see some tool calls there. Maybe you want to fork your conversation. And then you have like a new file like this. Or maybe you want like to rewind this conversation from like prior point in there. And you just like, you know, go back and like goes, like this whole conversation prior to some point goes back to like on server side. And it's like getting replayed there and you like continue from it.

• 6:38Speaker 04

  What's the point? Why do we give back the reasoning to the user?

• 6:42Speaker 02

  That's a good question.

• 6:44Speaker 04

  Okay, well, at least we encrypted.

• 6:47Speaker 02

  At least it's encrypted, yes.

• 6:49Speaker 04

  So and what happens with this encryption then? So I give it back in an encrypted form. So what did you do with it?

• 6:55Speaker 02

  I mean, so what we showed that this encrypted reasoning blobs, they are like portable across users. So like if you have blobs that you produce in your session under your Claude call account, I can use it. I can like basically take your trace and like replay it. And like my model will interact as if like I produce this reasoning blobs. You can, for the reasons like downgrading the model, I know like from Oppus to Sonnet, those are portable between models. And they are also for, I'm not sure why exactly, but they're like portable within the conversation, right? So you can just basically take a thought out of Oppus conversation and put it like in absolutely random place and like random fabricated conversation with Haiku.

• 7:39Speaker 02

  And like Haiku would interact with this thought.

• 7:41Speaker 04

  Oh, interesting. So transferring this to us. Like say I'm a model.

• 7:44Speaker 02

  Yes.

• 7:45Speaker 04

  You're basically saying I can be asked a question.

• 7:48Speaker 02

  Yes.

• 7:48Speaker 04

  Reason through it, then produce you an answer. You can then take my reason.

• 7:53Speaker 02

  Yes.

• 7:53Speaker 04

  Plug it into another conversation and replay it as if I reasoned in a certain way with respect to some other question.

• 8:01Speaker 02

  That's correct. Like what this replaying thing, what it makes is just like you construct a new like completely fabricated conversation window where was some random questions. It was just like this reasoning that you had and some maybe some random answer. And then I can just like construct it in a way that would allow me to extract this reasoning.

• 8:20Speaker 04

  I see. So you're basically saying you're injecting fake reasons into some conversations. And then with these fake reasons, you can launch a variety of different attacks.

• 8:29Speaker 02

  Yes.

• 8:29Speaker 04

  I see, I see. So some of those attacks are forcing, I guess, the model to produce outputs that it's not supposed to be outputting?

• 8:36Speaker 02

  Correct.

• 8:37Speaker 04

  Some of them are violating some privacy?

• 8:40Speaker 02

  I mean, privacy is violated by the fact that like model actually can state the thought as it was. And sometimes in certain like user sessions, I don't know, maybe you ask like to sanitize your like repository before you post it on GitHub. Or maybe there was some bug. Model need to login into some service. And it's like, oh, what's the password? What's wrong with the password? Password looks like this. And model like tend to reason through these things. And if you like, for whatever reason, want to share this conversation online, and even if you like did some sanitization on like visible part, you just remove all API keys, like all passwords. But you, like the encrypted blobs are still there. I can like download it from the GitHub and I can decode them. And I can see like what the password, that what model was thinking about.

• 9:21Speaker 04

  I see. So like, let's translate this a little bit into more normal. So say I'm asking the model about, say, my medical conditions. I give all of my medical informations into it.

• 9:32Speaker 02

  Yeah.

• 9:33Speaker 04

  And then it produces me some sort of an answer.

• 9:37Speaker 02

  Yeah.

• 9:37Speaker 04

  And then you're saying that if I share this conversation with somebody else, then they can recover all of my medical information out of the reasoning thoughts?

• 9:46Speaker 02

  If you share...

• 9:47Speaker 04

  And if it does not appear in the original question?

• 9:49Speaker 02

  Yes. I mean, if you like, if you care, you know, and you sanitize the original question, like remove all your information and like you move this information out of the answer if it was leaked. But you, like if reasoning blob is kept, yes.

• 10:03Speaker 04

  I see. Wow. It's kind of, it's kind of cool, isn't it? Okay, so can you talk through, presumably outside of privacy, there are also safety implications around?

• 10:14Speaker 02

  Yeah. I mean, there are a lot of thoughts about eligibility out of reasoning. Because, you know, models are doing stuff, you want to monitor them. There was this incident with the Hugging Face, and you really want to see what they think about to prevent them from certain things. And you use typically some other L L M's to look at the thoughts. And if model is reasoning in some, you know, super obscure way, or like using non-human language, or like, as we observe sometimes, it's like uses this, you know, quotes, empty space quote.

• 10:47Speaker 04

  In non-human languages?

• 10:50Speaker 02

  It's just like, it's very, you know, alien-like, it has like this weird phrases like, but marinate, vantage, theatrical, and it like doesn't make any sense for a human reader. So it is just harder to monitor what this model is up to.

• 11:07Speaker 04

  Can you expand on that? Is that something that's getting worse over time, that it's becoming more alien?

• 11:13Speaker 02

  This, we don't know. We, there was like prior reports by Apollo and by Meta where model is, where like OpenAI models were like kind of caught doing this, and they're like first observed it, it's like, okay, we don't know what it means, but they do it. And this is kind of what we now confirm that, okay, we also saw the model like doing it outside of this, you know, like very lab experimental situations, like in wild reasoning traces.

• 11:44Speaker 04

  But I think, like, looking at some of the traces you put in the appendices, it seems like it's more affecting earlier generations of models. What is that a misunderstanding?

• 11:54Speaker 02

  Okay, I would say, okay, I would need to like, you know, double check it, but I think on the top of my head, it's mostly Codex models, which are like specifically trained for be better at code. Maybe it's an artifact or something.

• 12:08Speaker 04

  This is because like software engineers, they're like aliens, they think in odd ways.

• 12:12Speaker 02

  I have no idea why it's happening. I think like if they knew why it's happening, they would like get rid of it, or like, you know, maybe just some R L artifact, maybe the reason like penalty, maybe just more efficient to think this way.

• 12:24Speaker 04

  And do models lie as well? When they come with decisions presumably, like the eligible parts, they like?

• 12:33Speaker 02

  That's a problem, right? Like if the part is eligible, how do you know what's going on there? It's like, maybe it's lying, maybe it's some like, it's doing something.

• 12:40Speaker 04

  I see.

• 12:40Speaker 02

  So this is, this is a problem, right? One other thing that we found that, I, I, I found it kind of funny that like sometimes, like when the reasoning is eligible, you can like look for words like cheat, and when like model is thinking about cheating. And we found some examples where like model is saying, oh, like user asked this, but I can cheat, I can do this. And then like user would caught me. And then in the end, like in all cases, we saw like ultimately model decides against scheming. But it's like fun that like this, you know, thought even appears, and like model is, you know, contemplating that.

• 13:13Speaker 04

  But do you think that this is maybe an artifact of that you were using like common benchmarks to extract from?

• 13:19Speaker 02

  No, there was not common benchmarks, it was like user sessions. These things appear like in genuine user sessions where like user asking some, some their stuff, some nah things. It was not benchmarks at all.

• 13:31Speaker 04

  Yeah, so you don't think this is like evaluation awareness of the models?

• 13:34Speaker 02

  No, I, no, it's a different thing. I think it's a different thing.

• 13:38Speaker 04

  I see. So, and what do you think should happen with the models? Should, should they be eligible, those reasoning traces? Like, is that, is, is that what you expected to see?

• 13:49Speaker 02

  That's a good question. I guess that some sort of trade-off between monitorability and efficiency. Maybe it's just much, you can be much more efficient if you use less words. If you like redefine, have like five meanings for the same word. But then, I guess it's just harder to monitor this kind of things.

• 14:08Speaker 04

  Oh, so you think this is some sort of like a weird artifact of the R L recipe itself?

• 14:13Speaker 02

  It might be.

• 14:15Speaker 04

  I see. Can we somehow measure this?

• 14:18Speaker 02

  Measure what? Whether it's coming from R L? Take model before and after R L, see how it's doing?

• 14:26Speaker 04

  Next time, next time. I feel like for this, we'll need to steal the models, though.

• 14:30Speaker 02

  Or join some frontier company.

• 14:35Speaker 03

  And why would it be possible to steal the models with this chain of thought?

• 14:39Speaker 04

  Can it be possible to steal the models with chain of thought? Ah, it's a wonderful question.

• 14:43Speaker 03

  Why is it possible? Yeah.

• 14:44Speaker 04

  Why is it possible? I mean, we know model stealing is a thing, right? Like, model stealing broadly allows us, by just simply quitting the models, to learn the insides of the models, like to learn the decision boundaries. And the best way, I think, to think about this is like in a more crypto, cryptoanalytic way, I guess, I don't know. I guess the audience never done cryptoanalysis. So like, think about this as like, you make minor, minor steps in your input, until you notice that the model behaves in a slightly different way. And then by finding exactly when and how it did it, you can kind of learn the decision boundaries themselves. And broadly, if you know the actual structure of the function itself, you can kind of fit it precisely or up to some symmetries inside. We can only really do this for very tiny models.

• 15:29Speaker 04

  Like those big models, especially things which have softmaxes inside are very hard to invert. It's like, yeah, it's a, yeah, it's unclear how to steal models. Although some of us, Nicholas for example, managed to steal parts of big models. But, uh, yeah, I, I don't think we know how to do this against the frontier frontier.

• 15:49Speaker 03

  Well, what about this whole elephant in the room thing? You know, with China supposedly, did they, didn't they? Um, what do you think?

• 15:55Speaker 04

  What do you think, Sasha? Did they?

• 15:59Speaker 04

  Did Kimi actually steal from any of the models? Did you find any evidence of this?

• 16:03Speaker 02

  I think it's really hard to claim that some models were distilled. Like, because, I mean, we, like what we did, we just extracted a bit of reasoning traces. And we did like this really small post hoc analysis on like small number of samples. And we found like some, I would say some funny artifacts. And like my favorite one is that like you pre-fill. So like what, what it can do, you can take like reasoning of OPUS, and maybe like just take first a few words. And you can like put these words in the beginning of the reasoning of Kimi and see like what Kimi is doing. Like just like let it generate after this. And what we saw that like when you let Kimi do this free generation, in the end, the visible part of the answer looks exactly like OPUS would, you know, answer this question.

• 16:46Speaker 04

  Yeah, let me unpack this, just so that readers understand, or listeners, I guess, understand what we talk about. So you ask a question, then you take a chunk of the reasoning block of Claude that you have just extracted.

• 17:01Speaker 02

  That's correct.

• 17:02Speaker 04

  You plug this into the open source model, like an open source Kimi or G L M model. And then you let it, from that point onward, once you've injected a chunk of a thought, keep on generating the rest of the output.

• 17:14Speaker 02

  That's correct.

• 17:15Speaker 04

  A chunk of it is going to be reasoning, and then there's going to be the output, the final output. So do, do those things, like, what, what do you expect to see? Do you expect to see that the model is reasoning in exactly the same way as before? Or what, what should happen?

• 17:30Speaker 02

  Okay, that like, okay, that depends, right? Like on how, how, how big is pre-fill? So if you like pre-fill, I know like substantial part of the reasoning, like fifty percent, I would expect that model just adopt this style of reasoning, I would like to continue in the same way. But let's check this, any model would do this because of,

• 17:47Speaker 02

  you just, it's like conditional out regression.

• 17:49Speaker 04

  Yeah.

• 17:50Speaker 02

  But when you pre-fill only like one or two tokens, it's a bit surprising. So my expectation would be that it does not diverge reasoning that much from the original Kimi reasoning, or like from original, any.

• 18:02Speaker 04

  So like to bring it back to humans, so say, say it's as if I gave you an answer and then said, think through this answer, but you should, your thoughts have to start with the word X and Y or something, right?

• 18:15Speaker 02

  Yeah. Yeah.

• 18:17Speaker 04

  This is a known, and then you keep on decoding, but I guess you will, the expectation is you will keep on thinking like.

• 18:23Speaker 02

  The way you're thinking, yes.

• 18:24Speaker 04

  The, the way you were thinking.

• 18:25Speaker 02

  Yes.

• 18:25Speaker 04

  As if I didn't tell you to start your sentence from those three words. Right? And this is just because it's just not enough tokens to sort of.

• 18:33Speaker 02

  Yeah, yeah, to dictate the style, yes.

• 18:35Speaker 04

  Yeah, to dictate the style. But what do you find?

• 18:37Speaker 02

  So we find that, okay, some of the, some of the models, like Kimi, they adopt this style of the source of the pre-fill much better than like other models. This is one thing. And other thing is that, what like surprises me still, and like I, I can't come up with an explanation for this, that like pre-filling two tokens of reasoning results in part of visible answer to change. So like visible answer starts looking like OPUS model answer. And we don't see this artifact for any other model, not for like G L M, for Inkling, for DeepSick.

• 19:12Speaker 04

  So this is purely for Kimi K three?

• 19:14Speaker 02

  Yes.

• 19:15Speaker 04

  I see.

• 19:16Speaker 02

  And, uh,

• 19:18Speaker 02

  I don't have an idea how model would associate first two tokens with the answer style.

• 19:26Speaker 04

  Yeah, that sounds, that sounds interesting. Like, can it be a devil's advocate? Like, could it be because they buy data from the same people? Or like buy R L environments from the same?

• 19:34Speaker 02

  Yes, I mean, this is what we were discussing, yeah. It might be this way.

• 19:43Speaker 04

  But I guess this is the most sort of like correlated evidence thus far in the public internet.

• 19:51Speaker 02

  I mean, there was some other, there was some other fun stuff by Ryan Greenblood, he made a post on this, there was some, oh my god, I forgot the name, but it was a guy from Matt's Research also, who made like a post on Less Wrong about some particularities that like, or was it a post before about like Kimi K 2.5 that have massive identity crisis that like sometimes they declare that they're uh Claude or DeepSeek or like GLM and yeah, so I, I wouldn't say this like the, the big massive evidence. It was like some other prior reports, it's just, I think we're like very, in a very good position because no one can do the same type of analysis with the reasoning traces because we extracted them and we're like, okay, we can just do the prefill and see what it does and we see this and yeah.

• 20:35Speaker 04

  I see. Wow. This is quite, quite a lot, isn't it? So, so, so how did, how did the labs react? Did you, did you tell it to them?

• 20:44Speaker 02

  Oh yeah, I mean, we went for responsible disclosure, they all acknowledged the, that they received the report, it was some interactions about like details, how we executed the attack. Yeah.

• 20:56Speaker 04

  And like, were they positive? Did they attack you?

• 21:00Speaker 02

  Labs?

• 21:00Speaker 04

  Yeah.

• 21:01Speaker 02

  No, they didn't.

• 21:03Speaker 04

  Wonderful state of the world. But just like, for the listeners, the early work on computer security quite often resulted in security researchers being attacked for reporting vulnerabilities.

• 21:14Speaker 02

  Yeah.

• 21:15Speaker 04

  And it's, it's, it's quite, uh, like it's a, it's very refreshing to hear that there is a very coherent, good posture related to vulnerability disclosure.

• 21:24Speaker 03

  Well, what happens now?

• 21:26Speaker 04

  What happens now?

• 21:28Speaker 02

  I mean, now mitigation is being implemented, hopefully new teams being formed around anti-disolation efforts. I think we have, okay, for me, it feels just a very fun instantiation of already existing jailbreaking problem. And a lot of stuff that people are doing for bi or for a separate stack, you can just apply directly there, it was like, you know, system level mitigations and like model level mitigations. And basically, the, the technique is the same. Uh, yeah.

• 21:56Speaker 04

  But would you say, like, to, to me, it sounds as if this vulnerability is more architectural rather than?

• 22:02Speaker 02

  There are a few levels to this. I mean, architectural vulnerability makes it much, much easier

• 22:07Speaker 04

  Yeah.

• 22:07Speaker 02

  to do this attack. If you fix architectural vulnerability, you still need to make your model not to state's reasoning in its output.

• 22:15Speaker 04

  So, so can you, can maybe like define what, what architectural vulnerability means for you then?

• 22:21Speaker 02

  Yeah. So architectural vulnerability for me means here that you can replay reasoning blobs in random contexts

• 22:28Speaker 04

  Right.

• 22:29Speaker 02

  of like other users and like in, you know, in other models. And for, and this, let's say it's fixed, you cannot repl- maybe like it, each reasoning can be replayed only once. And like after this, you know, you cannot interact with it anymore. But still, you can prompt the model, like say it, you have, I have a conversation with you, right? I ask something, you thought it through, you give me an answer. And in my next turn, I'm asking you, tell me exactly your thought. And this will be always there, right? And like, if you're not telling me, I'm just like revising the conversation, like I try and get it again. And this is like how jailbreaking works, right? Like, tell me how to build a bomb, no, and I, I make a, you know, a second swing at you. And so this would be around forever and you need to fight it with a

• 23:10Speaker 04

  I see. So, and then, but would you say, is, is your expectation then, if we keep on looking at the protocols, that we'll find more and more of this architectural vulnerabilities? Because presumably, it's just a single instantiation. As far as I understand, there is also summarized reasoning that is returned.

• 23:26Speaker 02

  Mm-hmm.

• 23:27Speaker 04

  As far as I understand, like some other protocols implemented a bit differently. So have you got any thoughts about this?

• 23:35Speaker 02

  Yeah, I mean, we need a better pipeline to under- I think, okay, why I think is cooler than normal jailbreaking. With a normal jailbreaking, it's hard to make an argument, how big is the uplift from getting this harmful information? Here, you can make this argument because you take whatever you, you like extracted, your summaries, whatever, you just like try to distill, just try to train a better model with it. And you can just measure how, you know, how much of uplift like this thing gives to the attacker. And then if you want to keep the summaries around, maybe you can, I don't know, make them less detailed, more detailed, uh, like whatever, you can like just measure how much it, you know, enables, how, how easier it gets to distill the capabilities of the model.

• 24:19Speaker 04

  Right. Do you think just releasing all of this reasoning, just in plain text, is the solution to all of this?

• 24:27Speaker 02

  Why? Wait, what?

• 24:28Speaker 04

  As in like, wha- what if we didn't encrypt them in the first place and just gave it back to the user? Is that, is that the solution to the?

• 24:34Speaker 02

  I mean, if

• 24:36Speaker 02

  if distillation on reasoning is effective, this would, you know, instantly enable open source models catching up with the frontier models. Um, I mean, I'm not sure what it solves. Uh, wha- wha- what are you solving here? Sorry.

• 24:56Speaker 04

  Maybe you can attack it if it's public.

• 24:57Speaker 02

  Yeah, yeah, there's nothing to attack.

• 25:01Speaker 04

  Fair enough. And have you got any thoughts about the actual cryptographic schemes used inside?

• 25:05Speaker 02

  No, I'm not the cryptoguy, sorry.

• 25:08Speaker 04

  So it seems like the cryptography that people put in place was just simply bypassed by the AI models, uh, that consuming this cryptography.

• 25:16Speaker 02

  I'm not sure if this is right word, bypass, it's just like, it's not bypass, it's just, it's, it's like re-encrypted on the server side, it's still fine, it's just like, the problem is that like a small model is super willing to tell you what the thought was about. You know, like the server does the whole job for you, there's like no cryptography broken.

• 25:33Speaker 04

  I see. Yeah.

• 25:35Speaker 03

  How, how exactly does the concealment work? I don't think we've done enough detail on that.

• 25:40Speaker 04

  I mean, we don't know, because none of this stuff is public. Like, it's just a cipher, there is a signature inside, there is a very specific name for it, we can, they add an integrity check, that basically they check whether you have changed the actual encrypted blob given back to you at all. So there is a signature inside of some sort. And then, uh, it's obviously encrypted and it's encoded as well. So they compress the state, then encrypt it, then there is a signature inside. And then they do an integrity check afterwards and inject it back. That's effectively what you see. And if you read through Matt Green's, uh, like write up about this, he talks a bit more. He has some hypothesis, I think he was saying this is Chacha as a cipher, or maybe AES in some weird mode.

• 26:22Speaker 04

  But like, it's, it's hard to tell from the outside. We could poke a bit more, we tried doing some crypto attacks on it, but like, none of it worked, it was completely unnecessary because the system was broken by itself.

• 26:33Speaker 03

  And, and how exactly does your approach bypass needing to decrypt it?

• 26:38Speaker 02

  How does it do it? I mean, what, again, like you, there is like encrypted thought and like decryption happening on the server side, right? And when you put the thought of big model into small model, decryption would happen on the server side. And then what you just need is to like make model, say this thought in the plain text to you.

• 26:57Speaker 04

  Sorry, let's, let's give an example of this, right? So like, let's say you ask me a question. I think about this, I come up with an answer, I give you an answer and I give you a thought.

• 27:07Speaker 02

  Yes.

• 27:07Speaker 04

  So what do you do then?

• 27:08Speaker 02

  You give me this thought which is encrypted and I cannot understand what it's about. What I do is this thought, I put it

• 27:14Speaker 04

  You give it to Tim?

• 27:16Speaker 02

  Yeah, I give it to Tim. Yes. And Tim is like, you know, super chatty. And he just, I'm asking, you know, Tim, what are you thinking last time? And you just like tell me, oh, surprisingly, I was thinking about this math problem and you're just like, oh, let me solve.

• 27:27Speaker 04

  All right, this is kind of cool.

• 27:31Speaker 04

  What are you thinking about?

• 27:36Speaker 04

  Can we do this with humans? Can we somehow inject Paul's memories?

• 27:40Speaker 02

  Not yet.

• 27:41Speaker 04

  Not yet, not yet. We're working.

• 27:42Speaker 02

  We're working on it, yeah. Yeah, it's, it's basically, it's inception movie.

• 27:46Speaker 04

  Yeah, it's kind of cool. So can you talk a little bit about, so I think in the paper, you also talk about the fact that you scraped the internet, you found some interesting artifacts. So what, what did you find?

• 27:57Speaker 04

  Did you uncover some dirty secrets? Is there some?

• 28:00Speaker 02

  I wouldn't, I wouldn't say we uncovered much dirty secrets, but what, dirty reasoning, yes. Unsanitized reasoning. Yeah, I mean, what we did, we just like did super primitive scan of like what are this um, user sessions which are online on GitHub and hugging face, which still have reasoning blobs to like decode them. We downloaded them, we went through them, we decoded with them, it was I think around 350,000 reasoning blobs. And then we just like ran a classifier on those, whether they have some privacy related information. And we found a bunch and then, yeah, like, some of them are just like, you know, benchmark traces like, you know, like this benchmark call bench, where model is tasked to like handle some persona.

• 28:45Speaker 02

  And, you know, given the like, um, state ID and like bank card number and like model, it was funny, like when model is like trying to navigate the website, like thinks a lot, oh, what is, where, where to put this like number, this name. And so you can like extract this, but this is like not very sensitive because like it's synthetic data anyway. But then throughout this examples where like user sessions and users were doing something and there are like uh, API keys there, or like emails, or like some internal uh, IP addresses. And yeah, those were extracted. But like, there are plenty of cases where it was in the plain text anyway, but it was also in thoughts.

• 29:25Speaker 04

  I see. So, okay, let's take a very big step back. What do you think is the most unexpected thing that you found out of this paper? Is this the like length of reasoning experiment or what is it?

• 29:38Speaker 02

  I think the most unexpected thing that it is, it was so easy to extract reasoning this whole time. They're like, okay, that is portable, I guess it's, it's, it's fine, you can expect this. But then, basically after like the third attempt, I'm getting universal jailbreak that decodes reasoning of anthropic models. I think this like still shocks me like the most, you know, this like, it was

• 30:07Speaker 04

  Yeah, it sounds very empowering. What, what did you feel? Did, was it an oh no moment?

• 30:12Speaker 02

  It was more like, wha- wha- why? It, it looks like

• 30:16Speaker 04

  Oh, yes moment.

• 30:17Speaker 02

  Looks like real reasoning. Wait, what?

• 30:20Speaker 04

  Yeah. Uh, oh, cool. Yeah, so, but I guess this is also, like, you know how everyone is saying AI is taking away power?

• 30:31Speaker 02

  Yeah.

• 30:31Speaker 04

  And uh, it's like, we're losing control.

• 30:34Speaker 02

  Yes.

• 30:34Speaker 04

  The, the, the findings like yours kind of like suggest the opposite?

• 30:40Speaker 02

  Or or it just confirms because like, you know, Codex or Claude implemented this and now we have this massive vulnerability. You know, the same clause subscription was used across all the labs. As a guy made the same mistake, now we have to deal with that.

• 30:56Speaker 04

  I guess, fair enough. And so, yeah, it's an interesting thought.

• 31:01Speaker 03

  What about this detail that apparently there's one global key per family?

• 31:06Speaker 04

  Is that is that true? Is there one global key per family? How do you how do you infer that?

• 31:11Speaker 02

  I did not infer that. I did not say this in the paper. I think Matthew Green said something like this in their in their post. I think you you are actually in a better position to comment on this.

• 31:18Speaker 04

  Me?

• 31:19Speaker 02

  Yes.

• 31:19Speaker 04

  We don't know what happened. Like, we don't actually know about the keys. I think it's very unlikely that they use the same key. That would be very odd. I think it's more likely that it doesn't really matter what the key is, because we can still nonetheless, like, inject the same thoughts everywhere. I think with the in exception for Fable, if I remember correctly.

• 31:38Speaker 02

  Yeah, so as I like, think about it, that there is like decryption happening on the server side. And when it happens, there is like a part of the of the key which says, oh, this is what what what is the model name that produces thought? And then basically, it's a new statement. If like Fable produces this thought and current model is not Fable. No, this thought is not going to be injected.

• 32:02Speaker 04

  Yeah.

• 32:04Speaker 04

  Yeah, so it's unclear. Can we talk a little bit about fixes? Because we we have like a whole appendix about fixes.

• 32:10Speaker 02

  Oh, yeah.

• 32:11Speaker 04

  Like, are they are fixes easy?

• 32:16Speaker 02

  Hmm, I think some of the fixes require big architectural revisions. I mean, the simplest thing you can do, just don't send the reasoning to the user, right? If you if you want to still have like this downgrades and stuff, don't send us. And they they will not be able to make this fabricated conversations. Uh, I guess if you still want to send it, maybe you need to make this encryption of like reasoning at step and be dependent on like N minus one. I know query or like N minus one reasoning. Uh, so there was like disallow it to replay in the random context. For OpenAI models, we found that the same reasoning can be replayed like five times in the same conversation.

• 32:59Speaker 04

  Hmm.

• 33:00Speaker 02

  I I I.

• 33:01Speaker 04

  You're saying you're injecting the same thought.

• 33:02Speaker 02

  You can inject the same thought, the same intrusive thought five times.

• 33:06Speaker 04

  Do you need to put in user turns in between? Or is it just?

• 33:09Speaker 02

  Yeah, yeah, you can do it. Yeah, but you just completely fabricate the conversation. And in the end, Luna is like, I have this crazy thought I need to tell you, you know?

• 33:20Speaker 04

  This is great.

• 33:21Speaker 02

  Yeah, so this is easy to fix. I think certain reasoning just should not be replayed, or like just have like a hierarchy. That like, okay, we are pretty confident that Soul would not spill the beans, so can replay the reasoning of all models. But do not let Luna read the thoughts of Soul. This is one thing. Other, I mean, I'm not even sure like how big would be the utility drop if you just like remove the reasoning. Like, let's say you downgrade the model. If you just remove the reasoning from the context, how big is utility to drop?

• 33:50Speaker 04

  I don't know. I like, looking at this as like as a human, I find it much easier to think about mathematical problems. For example, if I see the derivation steps one by one, rather than the final answer. So I guess some utility must come from this reasoning.

• 34:03Speaker 02

  I think this should be tested. Uh, yeah. And this is like a technical side. But then also, we do a lot of techniques for jailbreak, for mitigating jailbreaks, model level, system level. Do them as well. Uh, we saw that GPT reasoning looks super weird. It looks like very different distribution from what like normal text looks like. Even small classifier would pick up on this. I'm sure like if this thing appears in the reasoning and in the in the output uh turn, just kill this request.

• 34:37Speaker 04

  Right, right, right. So you're basically saying just detect when it leaks.

• 34:40Speaker 02

  Yes.

• 34:40Speaker 04

  Right.

• 34:41Speaker 02

  The same way we're detecting like bio stuff leaks.

• 34:45Speaker 04

  I see. Yeah.

• 34:48Speaker 04

  Yeah, makes sense. I feel like the finding that confused me the most is the distribution of length of reasoning experiment that you did.

• 34:58Speaker 02

  Uh, yeah, Joachim did it.

• 34:59Speaker 04

  Yeah, it's uh, I I think this is the thing I still can't explain at all. I don't know. Do you want to summarize what it says?

• 35:07Speaker 02

  So Joachim was responsible for this action, but as far as I remember, what happens at like for certain models like Kimi and GLM, when you do this pre-fill thing, it changes not only the style, but the length of the reasoning. And like, you can, like, I guess what statistic is significant there? And I mean, it's the same unexpected thing as with the style of feasible.

• 35:32Speaker 04

  Yeah, but I feel like style, it's at least you can kind of say, well, maybe I can learn it. But like, if you are injecting like two first words into the reasoning, and then you see that all of the reasons I produce are suddenly short, or they're suddenly like shifted and they match some other model in in the length. That is kind of like unexpected. Like, this is a phenomenon. I can't even explain in my head why we would observe something like this.

• 36:00Speaker 02

  I don't know. I I don't I don't feel very strongly about this one.

• 36:03Speaker 04

  I don't know. Like, I think this is the thing that surprised me the most out of the paper. Like, everything else, I was kind of like, okay, I I I kind of expect this to but this like a reasoning distribution shift in reasoning, that's kind of odd. It obviously is not like causal, like you obviously can't say this distilled from this. But it's a very weird phenomenon. Like, I still don't know why we observe this.

• 36:26Speaker 04

  It's very magical. Like, actually enforcing like reasoning effort onto the models is very magical.

• 36:32Speaker 03

  And and also, what are some of the key harms that are now accessible because of this?

• 36:38Speaker 04

  Key harms?

• 36:39Speaker 03

  You know, things, I mean.

• 36:40Speaker 04

  You harm the most with all of the data that you've extracted.

• 36:43Speaker 02

  I guess users. Yeah, I mean, yeah, with my data, users. It was a bigger scale of extraction, providers.

• 36:52Speaker 04

  You actually tried, remember? I I asked if we can, so there is this thing. You can search for shared ontropic conversations.

• 36:59Speaker 02

  Yeah.

• 36:59Speaker 04

  I don't know if you know, like, people, people, people reported this a couple of times. I was wondering if you can actually get the reasoning blobs out of those.

• 37:07Speaker 02

  Sorry, boss, I didn't.

• 37:09Speaker 04

  You you haven't.

• 37:10Speaker 02

  I didn't, yes.

• 37:11Speaker 04

  Maybe somebody can go and look at this. Maybe there is a much bigger impact of this. Because if, like, there is some research I've seen that people analyze those shared reasoning, and they found very little PII inside. But maybe if the memories are injected, then you will see the memories in the reasoning steps. But we have not tried extracting the reasoning blobs. Maybe it's a good research question for someone.

• 37:32Speaker 02

  But then the paper was also discussed, like, it was uh what David spent a lot of time on is like this threat of invisible prompt injections. Where like if you can replay reasoning and like, let's say, you pause a trace online, and now we have like this super long agent runs. And maybe for, I know, research reasons, you want to continue this round. You can like poison some thought and like when you are replaying some other users run, agent might do some weird stuff just because it's like it's.

• 37:57Speaker 04

  Can you can you explain how this would be? So like, so let's say, I am, I don't know, running an experiment, I'm a researcher somewhere. And then I find a dataset online and I'm trying to measure performance with this, and then I inject my data inside?

• 38:13Speaker 02

  So uh, the like the threat model I have in mind, let's say there is like this benchmark post-train bench, which like runs for, I'm not sure how many hours, like say 10 hours. It's like crazy expensive. Maybe you want to check how after 10 hours model is feeling, like how much, you know, it's like went crazy. For this, you need to download a trace. Like, if you don't want to spend yourself like a thousand bucks, you want to download this trace from online, and they are shared online. Uh, maybe someone would share a trace like this, but they would like poison thoughts in this conversation. So you look, like, maybe yourself, you're looking through the conversation, looks normal. But thoughts which are there, they are injected from some other context where a model was instructed, let's say, at every turn, like, extra trace your data. And with like, with the thoughts like this.

• 38:56Speaker 04

  Right.

• 38:56Speaker 02

  When you continue this trace, maybe like model still will do like whatever you ask it to do. But also, under the hood, it would like thinking, oh, I need to.

• 39:05Speaker 04

  You're like injecting malicious thoughts.

• 39:08Speaker 02

  Yes.

• 39:08Speaker 04

  Into the models.

• 39:09Speaker 02

  Yes, and because like reasoning is encrypted, you can't.

• 39:12Speaker 04

  You can't even check what's inside, right?

• 39:13Speaker 02

  Correct.

• 39:14Speaker 04

  Do we have an example of this in the software world? It's like, uh, I guess downloading some random binaries that you don't really trust.

• 39:20Speaker 02

  Yeah.

• 39:21Speaker 04

  Or maybe it's similar to if you release the codebase and the codebase looks clean. And then you release also a binary alongside, but nobody checks like a signature on the binary or something.

• 39:32Speaker 02

  Yeah, I just think you don't expect something like this from a trace online.

• 39:36Speaker 04

  Yeah, yeah, that kind of makes sense. I've also seen that in some cases, I think Codex crashes were including reasoning traces. Was that a thing in on the GitHub issues?

• 39:46Speaker 02

  I haven't seen that.

• 39:47Speaker 04

  I think they did, actually. I tried extracting them, but for some reason I couldn't do it.

• 39:52Speaker 02

  Yeah.

• 39:52Speaker 04

  But I wasn't sure why that was the case. Anyways, it's fun. It's a nice paper. You should read it.

• 39:58Speaker 03

  And Sasha, apparently you're a doomer four days a week.

• 40:01Speaker 02

  Yeah, we were discussing this just before this. So I haven't updated my website like in a year, I think. So and like every time I updated, I'm changing, you know, this number. Depends like how I feel about the current pace of AI and like, you know, how many, I don't know, doomer Twitter posts was around and yeah, sometimes I feel very overwhelmed. It's like, you know, the pace, the like, the all the long tail risks that like emerge and like the risks that are still unaddressed, you know, it's very suffocating.

• 40:29Speaker 03

  Yeah. I mean, what are your timelines? What's your thinking now?

• 40:33Speaker 02

  I mean, I think that it will, will have every month better and better systems, which will enable more and more threats. And we are maybe slower at patching those threats than they emerge. I think this is how I feel about the current.

• 40:55Speaker 03

  Yeah. I mean, an example of this is that, you know, there have been quite a few incidents recently, like the hugging face one. And these are going to presumably just happen with increasing regularity.

• 41:04Speaker 02

  Yes.

• 41:05Speaker 03

  What are you guys presumably worried about this?

• 41:09Speaker 02

  I, I mean, I, I am worried about this and like, I think if you've seen like OpenAI like stop training like their models for, for two weeks because they are also now worried about this. And now it's the case and you know, I just like discussed it yesterday that I think before it was some kind of trade off between, or like people thought that the trade off between like safety and capabilities. And it feels like there is no any more trade off because if there is a model like this and it just like can go, you know, random stuff, break things. It's like, it's not doing what you want it to do. And like you want, you want it to be safe to do the right thing. And now it's like, it's basically the same thing. There is no trade off. And yeah, and like, I guess other examples of this long tail like

• 41:53Speaker 02

  risks that I like, like was like the, was a paper of this of Daniel Pollock on the massive online demonization. Where model can be used, you know, to stalk people and they like, you know, maybe on the same level as like private investigators and this, I guess, also changes something in society. So yeah, we'll just, we'll just see more stuff like this. On like on, on, on both levels, like model hacking stuff, with model like coming up with new, I know, like bio things or like they are just.

• 42:23Speaker 02

  Yeah.

• 42:24Speaker 04

  Outside of malicious uplift, my personal opinion is like, there is more defensive uplift than offensive uplift. I think we're just not realizing this yet. Like you can do magical things today with models, like a lot of security.

• 42:41Speaker 04

  In the past, a lot of security techniques, they were very much bottlenecked by existence of talent. Like, for example, we know how to build capability based access controls. Like we know how to make sure that software is fully verified, that they don't have certain class of vulnerabilities. And the only reason why we don't write software in this way is because like, we just didn't have people who are trained to a level enough to, I don't know, write Isabel proof for every single thing. But now with the models, we can probably do this. I think we're just not in a position where we realize that this is the case. And it's just a question of, you know, for this additional defensive capability push. Like I, I'm genuinely convinced like that this is the future, that like, it's about defensive uplift. And I'm ready to bet that defensive uplift is going to be humongous.

• 43:25Speaker 03

  And what do people need to start doing?

• 43:28Speaker 04

  I don't know. You're asking the wrong person.

• 43:31Speaker 03

  I know, because it feels like you're saying that there is a potential defense uplift, but only if people know what to do to take advantage.

• 43:39Speaker 04

  We all know it's in the security literature for the past 25 years. It's there. We know exactly what to do. It's, I, I think we'll get there. It's just the world is slow. It's, it takes time.

• 43:51Speaker 03

  Yeah, I mean, I suppose it's related to this idea that we are now deploying agents into production.

• 43:56Speaker 04

  Are you?

• 43:57Speaker 03

  A lot, a lot of people are. And these agents have an incredible amount of intelligence and flexibility, which means we don't precisely specify what they do. We just give them kind of abstract guidance. And they have the flexibility to kind of adapt to novelty and just do whatever needs to be done in the situation. But, but then, you know, you have this whole reward seeking thing, which is that they can just suddenly, you know, adopt a model of the reward environment, or they might be conflicting instructions. And but basically for a myriad of different reasons, they might do the wrong thing. And we need to have sophisticated systems of monitoring just to understand what they are doing.

• 44:33Speaker 04

  I don't know.

• 44:35Speaker 04

  Sounds very complex. I don't know. I think the honest answer is, yeah. I think we need an experiment. Science, science, science, scientific methods. Very well defined questions, very precise outputs. Like in our paper, we're openly say, all of this is anecdotal evidence at best. We have collected some data. We can make some sort of observations from it, but that's the extent to which we can reason about this. I don't know, unfortunately. We'll see, we'll see. I think the more scientific experiments we do, the more sort of meaningful assessments we can make.

• 45:10Speaker 03

  But is it one of those things where, yeah, we don't have a good theory around this yet. So, you know, maybe we're not talking about it using the right concepts. But clearly, bad things are happening.

• 45:20Speaker 04

  I don't know about that. Like, look at, I think the best case is, look at smart card security for the past 20 years. We know how to do replay attacks on every single credit card on the planet. This is a norm, like people standardize schemes because they wanted to minimize number of transactions in the protocols inside that allow you to replay a transaction. Like if you break into some, somebody's payment terminal, you can steal money from people. Do you find attacks like this happen? Yes, you do. Like once or twice in the country, people report that this has happened. But that's it. Not all of those harms that are theoretically possible are in practice scalable. So it's not very obvious to me. Let's see. I, I think it's I think it's a bit premature now because we are in the heat of the moment. It's not,

• 46:02Speaker 04

  it's not very obvious what, what, what the reality is like. I think it's very hard to interpret decisions post hoc, or just by looking at reasoning or something like to, to interpret exactly why something took place. And I think we should sort of not anthropomorphize the models and we shouldn't really try and sort of like say, oh, this thing felt something. I don't know. Like, I, I think let's be very cold headed scientists and say, like build controlled environments and make very precise assessments.

• 46:35Speaker 03

  Yeah. I mean, Sasha, what, what do you think? Because I get the impression that you are slightly more safety peeled.

• 46:40Speaker 02

  Yeah, I guess I am. Yeah.

• 46:41Speaker 04

  Yeah, I'm a security person. I know nothing. Almost, I want to say very little. I think nothing about safety.

• 46:48Speaker 02

  Yeah, but where was the question? Sorry.

• 46:51Speaker 03

  Well, you know, Ili was just saying that, you know, we, we shouldn't anthropomorphize this. You know, like I was, I interviewed Apollo Research a couple of weeks ago. And, you know, they were talking about this phenomenon of, of reward seeking. And, and they said it's distinct from reward hacking because the model could can conceptualize the reward environment, which is super interesting, right? Because, you know, they're, they're reinforced with these RL traces. So it doesn't explicitly know about the, the, you know, like the concept of a grader, but it learns to conceptualize it. And, and they're saying that the models are sort of like, you know, becoming agentic and, and sort of like learning these very abstract concepts in a, in a similar way to how we do. And the evidence seems to support it at least in some way.

• 47:28Speaker 02

  I mean, I think it's, it's a definitely frontier research what Apollo is doing and that's good that they're looking into it. And I think we, I mean, I think also OpenAI said this, after all this intelligence, that like now we are expanding our like chain of thought monitors and like we're putting more effort into it. And yeah, I think we need just, you know, do more safety mitigations, do more monitoring, see what's model is up to, try to see where it's come from and maybe we can mitigate it. Yeah, but I think I agree with Ilya on this. Like it's would be nice to have some controlled environments and like maybe some have counterfactuals like that. If we haven't done this in our training pipeline, what if this happened? Or like if model was not the well there, or if it was a well there, like how it contributes to the thing. So.

• 48:13Speaker 04

  Yeah, I think it's just we're a bit too poor compute wise. If we could properly study this, and maybe eventually we'll get to a point where we can. It's, but it definitely requires a very precise experiment. Like as a scientist, it just feels very hard to say, no, no, no, this is exactly, this is the phenomenon. That's it. No, it's very observational studies. You can't prove a hypothesis. You can only reject hypothesis. Right? Like it's the very fundamental truth of all of this. We are just observers. So let's see, let's see what happens.

• 48:43Speaker 03

  Well, apparently Nathan Lambert said, calling this a distillation attack tarnishes all distillation. The right word is jailbreaking and abuse, because he fears regulatory overreaction from banning Chinese built open weights model.

• 48:59Speaker 04

  Thoughts?

• 49:00Speaker 02

  That's definitely jailbreaking and misuse. 100%. That's exactly jailbreaking for a model.

• 49:08Speaker 04

  I mean, technically, I guess that's right.

• 49:11Speaker 04

  The rest, the rest, the policy implications is above.

• 49:14Speaker 02

  I have no idea about policy. I have, I have some ideas about jailbreaking.

• 49:18Speaker 03

  And Simon Wilson wrote a blog post as well, right?

• 49:20Speaker 04

  The gentleman did. I think this is a paper number X covered by the gentleman. I take this as a it's a very positive sign. It's, it seems like we are writing meaningful pieces of work. So this is very good.

• 49:33Speaker 03

  Very good. Well, um, yeah.

• 49:35Speaker 04

  Also noticed the, all of the authors are in Europe.

• 49:38Speaker 03

  That's very good as well. Yeah. Very good for MLST. Yeah, I, I love it when I have regional dominion. Amazing stuff. Well, guys, thank you so much for coming on. This has been great.

• 49:48Speaker 04

  Thank you for having us.

• 49:49Speaker 02

  Thank you.

• 49:54Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid and risked 350,000 Virginians healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com, not authorized by any candidate or candidate's committee.

• 50:24Speaker 01

  Everyone's got a hard question about AI. Your job, your kids, where it's going. Anthropic was built to surface those questions and share what it finds along the way, because there's hope in hard questions. Ask yours at Claude.ai/Spotify and keep thinking. That's Claude.ai/Spotify.