⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

Watch the full show on YouTube: https://www.youtube.com/watch?v=CSgjaC6y6Mk

• 0:03Speaker 00

  Hi, welcome to Lanespace Lightning Pod. We have Patrice here from Datology, one of the founding team members. Listeners to the pod will remember our episode with Ari Montos. I think it was one of the top episodes of 2025. So I want to double down on the interesting work coming on Datology. I think not enough people are looking at the data. And we want to encourage engineers and researchers to think more deeply about data. And I think Datology is one of the best teams that is like basically all data nerds.

• 0:33Speaker 00

  Patrice, your bio has like 70% California, 20% Pittsburgh, 10% Delhi. You're in South Bay right now, right, at the office?

• 0:42Speaker 01

  That's correct, I'm at the office.

• 0:43Speaker 00

  Yeah, cool, cool. How else do you introduce yourself? Any other things that people should know about you before we get started?

• 0:49Speaker 01

  No, that was pretty good. Thank you so much for having me. I am basically finishing up my PhD at CMU and have been a part of the founding team at Datology for more than two years now. It's been a while that we've been building this company. I am very excited about data-centric aspects to AI and more of how we can build responsible and very high-performance systems by focusing on the data layer. Somehow, data is still one of the more undervalued research topics. And that's been the focus of my PhD and now also a lot of the time at Datology. I think you very well pointed that we're all data nerds. We love looking at data in general.

• 1:30Speaker 01

  I think one of the coolest parts about looking at data is like we can share like strange artifacts about it. In fact, we even made a channel called "Data is Weird" on our Slack channel where we keep posting screenshots and snippets of like all of these random artifacts that you see within data sets and they'll keep surprising you. The internet is so weird in so many ways that you would never expect that, oh, the depths of the internet has this and we're actually feeding these kinds of data points to our models. So I think there's like a shared passion about looking into data and like an investigative approach around the team in general.

• 2:04Speaker 00

  Yeah, okay, two follow up questions and then we can go into the paper. One, what's your thesis gonna be about? What's the overall theme of your works?

• 2:12Speaker 01

  My thesis is called responsible and efficient use of web scale data for pre-training, which is basically when you're training on this messy web data and you shove all of that into the model, how can we train efficiently, which is like only use things that are useful? But then also the responsible angle to it, which is like, how can we make sure we are attributing data to, like giving the right credit to like artists who put data online? Or how can we make sure the data is safe for models, the model behaviors? So I do a lot of work on the evaluation aspects of this as well, but then also on the efficiency and performance aspects.

• 2:48Speaker 00

  Okay, and then just in your data is weird channel. Any fun anecdotes over the last, recent anecdotes, just pull something out. No, we're going to go into like the GPT training data stuff, but I'm just kind of curious what gets posted there.

• 3:01Speaker 01

  So some of the cool things is all from like researcher curiosity. I think one of the recent things that we were exploring was that how do various models? Like for instance, you take the Qwen model and then ask it to do completions on certain questions. So we would just take like, we took some example questions from JEE, which is like an important Olympiad, like an exam in India. And we would like even ask, seed the models with the first two words of a question like the light, the light bulb. And it would like literally complete the actual questions. And so then we started comparing this across like family of models. And we consistently saw how many of these like models today are being like massively like kind of fine tuned on problems from.

• 3:46Speaker 00

  Like overset?

• 3:47Speaker 01

  Massively overset, like even like, imagine if I ask you the light bulb, what comes next in your mind? It won't be a question from an Olympiad or a practice exam, right? And these give the right question, they also give the options after that. And so I think that is one of the fascinating findings from data is weird from the last month.

• 4:05Speaker 00

  But no, nobody is benchmarking on JEE. That's not a, that's not the thing that people report.

• 4:09Speaker 01

  There's a JEE bench now, and I think it's also been repeated by various. Model labs, but like J is like one example. I'm sure this happens like across uh different exams. But it's very clear how the last stage of training for many of these models does have a massive amount of uh example or example.

• 4:27Speaker 00

  Benchmarks.

• 4:28Speaker 01

  Because the model will not like behaviorally complete exam like questions with options if they have not really seen it at the end of training for multiple epochs.

• 4:36Speaker 00

  And you're saying there's there's basically perfect memorization. And I'm wondering if you have a bladed this across different model sizes, like does Quen be do the same as thirty B, probably not, right? And I'm always curious about the memorization power.

• 4:50Speaker 01

  That's a great question. I think it's a mix of two things. One is the recency of seeing that data and the second is also the size of this model. I don't, we don't see this phenomenon like the earlier versions of like Quen one point five or even Quen two, but start seeing it in Quen three. So there's something about a more like a significantly higher weight on benchmark or like uh J or like any types of evaluation questions. During the final stages of training in these recent models. And then definitely see, we see the effect of like bigger models having memorization to a significantly higher level. Though I would say that with the coming of the M O E models, we're starting to see this uh phenomena at even like models which have a less number of active parameters.

• 5:32Speaker 01

  So a phenomenon that you would observe at a seventy two B model in the past is probably now visible at a twenty B active one twenty B M O E also. So in some way, the active parameters might be lesser in the M O E um life cycle, but still be able to like regurgitate exact memorization.

• 5:50Speaker 00

  Yeah, it's in- interesting when memorization is done in the M O E, like the router at the M O E actually ends up becoming like an index. Where you look up like, oh, here's the index of the thing that you need and then it's routed to that expert that is going to have memorized the answer.

• 6:03Speaker 01

  Exactly.

• 6:04Speaker 00

  Uh, that's that's a way to cheat. Okay, so let's uh tell my task about the seahorse emoji and G P T.

• 6:10Speaker 01

  So around October last year, I saw on Twitter, a few people had posted this very fascinating result. Where they would ask a frontier model such as G P T, like uh is there a seahorse emoji? And like, that's the only question they would ask. And what you would see is that the model keeps going on and on. And this was a phenomenon that really fascinated me. Like, imagine asking G P T a question and this answer does not stop. Which was really surprising to me because all of these models like are like, these are like frontier models. They would not really have this problem of like continually like going back and forth on a certain answer.

• 6:51Speaker 01

  If you see over here, it says yes in the beginning, then it starts to think more about it. And then it says no and then goes yes and no and yes and no. So there is like a huge amount of internal turmoil in the model. And a phenomenon where it just keeps saying this and does not really stop. And so then I tried this across models and I saw consistently across Grok and G P T and Gemini. That you were seeing this phenomena where models will like self-correct themselves quite a bit. And these were like non-thinking models, like the standard, the auto or the smallest fast model. And they would still do this. And I was like quite perplexed that how is it possible that these models which are like so strong.

• 7:30Speaker 01

  They are able to solve these I M O problems and they are like kind of just getting into this self-narrative existential dilemma or something. When you ask them about like, is there a seahorse emoji? And this was also a time when I was thinking a lot about reasoning data and like, should like models learn more self-correction behavior during pre-training or not? And my initial thought was that this behavior that we are seeing of models saying yes and no is very reminiscent of all of these back-correction and self-reflection behavior, which is becoming very predominant, right? In modern A I models and there are a lot of papers on like if you, the models learn how to correct their wrong traces or like in their data or in their post-training data, then the models become much stronger at reasoning.

• 8:15Speaker 01

  And so I was thinking like this is very reminiscent of that and somehow it feels like there is like an emergence of that behavior even when I did not ask for reasoning. So I thought like it would be very interesting to see like what really causes this behavior because we don't really understand that and maybe there has some tie in with the prevalence of reasoning data or self correcting monologues within the mid training or pre training phase of non thinking models. And so my first thought was, is this something that happened always? Or is it just like a G P T five phenomena? And because people have been noting this in October and potentially people asked this in like a year ago also, like would the models create this kind of a self correcting monologue? And the interesting thing was, and I have like put the number of output tokens on the

• 9:00Speaker 01

  Y axis on this graph and the release date of the model on the X axis. And so I like ran the Open I A P I across like models released from twenty twenty three to twenty twenty five and you would see like all the models until twenty twenty four December had very terse and short responses to the question, is there a seahorse emoji? They would either say that there exists, sometimes they would say they do not exist, but they would never go into this self correction loop where they like say one answer in the beginning and then a different answer at the end. And so something very interesting was happening because we know that in December twenty twenty four is when the O one model came, which is the first time where a model that could think and self correct itself was available to the public.

• 9:43Speaker 01

  And then four months after that, we noticed that the G P T four point one series is the first model where suddenly the response length of all the models had started to increase. And then the chat G P T four O model update that was in the month of May again had like this recursive behavior and this became significantly more prevalent by the time G P T five came.

• 10:06Speaker 00

  And so And just to insert a question here, this is specifically like the cutoff date that they have declared themselves, right? Like, I don't know if this is, is that, this is not, what, are you ordering by cutoff date or the release date?

• 10:18Speaker 01

  Uh. This is ordered by the release date and so. Release date. It's unclear what the cutoff date would be, but very likely that it has seen like example samples very close to that day. So for instance, ChatGPT four is an interesting case. Its original release was somewhere in December, like twenty twenty four. But then they had an update in May and only the May update shows this behavior, but the previous update does not show it. And so they kind of potentially post-trained the model on these kinds of reasoning traces and it suddenly started showing the self-reflective behavior. I think should be important to point out that this seahorse emoji is an interesting question. And like why it really brings out this behavior is because the existence of the seahorse emoji is kind of called like a mandala effect where

• 11:00Speaker 01

  a lot of people on Reddit are saying that, hey, I have seen the seahorse emoji. It looked blue in color and had a horse pointing to the left or something like that. And so there is like enough doubt or like enough conversation on the internet that some people say yes and some people do say no. And so it's kind of acting like a trigger to elicit that thinking response. Which is great for us because we do want these edge cases that can help us do this discovery or this investigative analysis. So by now, what we have figured out is that around the GPT four point one and then GPT five series, there is something happening which leads the models to have a lot of self-reflection. And now the question is like, what happened over here, uh, like which led to models starting to do this?

• 11:43Speaker 01

  And for me, this was interesting because I wanted to know if the frontier model labs also care about putting reasoning data in pre-training or in mid-training phases. And like you might have seen a lot of papers coming out these days which say that you should do reasoning pre-training and like RL pre-training and all of those papers which are saying that put the thinking tokens in the beginning. And as academic researchers or like people who are not training frontier models, it's always a curiosity question. Are these things really valuable at the frontier level or are these questions only interesting at the one B scale when you can move your data distribution? And so that's why I was excited about understanding like if there is relevant data in the mid-training phase for non-thinking models and something.

• 12:28Speaker 01

  Interesting that I was able to validate this was with the Almo series. So the Almo models are great for doing investigative science because they release everything. They release the data, the models, and they also have a very fantastic Almo trace now. With Almo trace, you can trace like what data in the pre-training of Almo was closest to its response right now.

• 12:51Speaker 00

  Yeah, this is, I was thinking of this when you're talking about your PhD thesis. It's like directly the tool to use.

• 12:56Speaker 01

  Exactly.

• 12:57Speaker 00

  Do you, do you agree with your approach?

• 12:59Speaker 01

  Oh, yeah, I think like, I honestly, Almo trace or the original paper which produced the Infinegram, like I think it came out last year at Colom or twenty twenty four. Uh, that was one of my favorite papers that year. I think they did a fantastic job in like being able to like actually trace data back. It's like a big engineering problem, be able to like look up so fast and they did like a fantastic job. But I've used it a lot for multiple projects in my PhD. So coming back to this phenomenon, uh, and I was like curious like, okay, now we know that this suddenly uh happens. So like this infection point happens in G P T four point one series. Can we trace it back to an open series of models and see if this is like, um, uh, the same phenomenon happens there? And so you will see for the Almo two model, which was released a year ago, like says there exists an emoji.

• 13:43Speaker 01

  It's the wrong answer, but does not really try to self-correct itself, right? On the other hand, in the three point one thirty two B series, it has the same self-correction behavior. Like it's a long response, I only have the last part of it, but it first says yes, then says, but wait, there is no one. And then, so we see the same monologue within the Almo three point one model. And if you look at the Almo trace, there is no particular example that it is really referencing to over here that leads to this phenomenon. So it's more like a capability rather than a regurgitation of the text. So that's great to know. It's not like someone has poisoned the data. That was another thought in my mind, has plenty poisoned the data on the internet? And the models are like regurgitating something from there, but that's not happening. So that's good, that's a good sanity check, that what we're seeing is a capability

• 14:28Speaker 01

  and not like a regulation from some website or like Reddit posts which might have said that yes and no. And then the more interesting thing is now we can actually trace the exact difference in the data sets for Almo two and Almo three. And the main difference for the instruct model over here, so Almo three has multiple variants with some are the thinking variants, some are the instruct variants. So with the instruct variant, they do not have any thinking data in the post-training phase. But they do mention that we are going to put like this is the line that they write, there is some intentional addition of thinking traces in the mid-training phase of Almo data. And so that's the information available from their report. And we can also see the actual data sets they use.

• 15:07Speaker 01

  This observation really well correlates with the whole finding about the OpenAI models where they potentially also did the same thing. Where even if in the non-thinking models, the instruct models, where you do not put thinking traces in the post-training phase, you do put thinking traces in the mid-training phase and that leads to models becoming better. And so this was like quite interesting for me, because what this suggests about the GPT training data is that the self-reflection data has now actually become pretty much core to the training of all frontier models. Because we're seeing that happen in non-instruct models across the board. Which is very interesting to me because for the longest time, people would always believe that the idea of foundation models is that you have trained this foundation and now you just post-train it.

• 15:51Speaker 01

  Which meant that there's a general purpose model, you don't need to really put everything that you care about in the general purpose model. And then you can post-train it to get the capability that you want. But this really shows us that even this foundation mod, foundation needs to have the ingredients that of the capability that you desire at the end. And so putting thinking traces in the foundation is actually useful for the post-training or fine-tuning of the thinking models. And so that's why they can only have one single backbone and you would actually prefer to have the foundation also represent the capabilities that you desire. And so self-reflection is a capability that we desire and it's become core to the foundation and not left as a cosmetic afterthought of just post-training.

• 16:34Speaker 01

  I think also was interesting to me that the GPT four point one series came about four months after the O one data was available. So it's kind of interesting like how fast do frontier labs move. O one reasoning traces would have been available to the pre-training team in December or the mid-training team, I should say in December. And then from December until what's like four months from there, end of April, the mid-training team would have potentially used the traces which they obviously were developing and there was a lot of talk about the Omni models.

• 17:01Speaker 00

  But not directly, right? Like it's just went into the web and the pre-training team picked it up.

• 17:07Speaker 01

  Oh no, I would be very surprised if it went into the web. I, I think this was very intentional because that's exactly how the Almost series happened, where they have an intentional addition of the thinking data. So they have added a data set which has this self-reflection behavior, where you will self-correct yourself by first inserting a wrong sequence and then the right one.

• 17:27Speaker 00

  Yeah, I mean, this is why I wanted to feature this piece because I think like the, the, you know, like how people are training reasoning into their models and when they're choosing to include it and all this. Uh, it's very, actually very important and not well known. And so this is one of the first like actual like somewhat investigative piece I've seen about it, so.

• 17:46Speaker 01

  Yeah, and you, you might have seen all these papers which show that RL is only like kind of enhancing or amplifying a capability that the model already has. And you actually need the code to have that capability. So I think this reinforces all of those things that the code actually contained the capability and then you can actually do better RL, you can do better post-training, all of that. And so I think in general, it was very nice to like be able to like see that these are not just things that are relevant to a 3B or a 7B model, but also relevant at the real frontier where people are training like the best models.

• 18:16Speaker 00

  Any reactions, criticisms or follow-up work?

• 18:19Speaker 01

  For now, like, um, in general, I think the community was pretty excited about this. Like a lot of people, uh, felt that this was either reinforcing their beliefs or a very interesting investigative work. I think that's something people always appreciate when you just like put your detective hat on and try to like dig into, uh, whatever is happening. Something that we are actually following this up, um, with, uh, and also in general, we have been working on that for a few months is a work, uh, which should be out like in a few weeks called The Fine-Tuner's Fallacy, which is basically kind of arguing pretty much the same thing, which is that if there is a core capability that you actually care about, that capability should be part of the foundation and not a fine-tuned artifact.

• 19:02Speaker 01

  So we kind of show this for various types of domains. And we're actually in general seeing adoption of like AI in a lot of like domain-specific use cases. So the old idea of you have a foundation model and you can just fine-tune it and get the desired capability is kind of like. We are past that stage now and this really means there's going to be a lot of specialized pre-training going forward. So I think like twenty twenty six and twenty twenty seven are going to be the years where different enterprises start doing specialized pre-training because the cost of pre-training really amortizes itself very fast. When you think of the fact that by doing specialized pre-training, you can train a smaller model, which is as capable

• 19:44Speaker 01

  as a much larger model when fine-tuned. And so that's the core thesis that we have been working towards in the past few months and also very relevant to data in general.

• 19:53Speaker 00

  Okay, got it. I may need to go some because I'm being sort of chased out of this thing. But this is secretly a data logic pitch I'm realizing, you know, because everything you're saying is like exactly the pieces of data, which is very fascinating. You know, you've also done other work. You are one of the authors on Beyond Web. I don't know anything else you want to sort of plug or feature, you know, in terms of the stuff that's going on?

• 20:19Speaker 01

  Yeah, I can quickly share something about Beyond Web and how it's been like making waves. And I also want to talk about the same idea of having the core capability that you care about at pre-training in something called safety pre-training and maybe like a few minutes over there. So let's go to Beyond Web first. So this was like one of our big releases last year where we were trying to showcase how to scale synthetic data to trillion scale, like works which are like doing like hundred billion token training with synthetic data and two hundred billion training token with synthetic data. But the actual dynamics of how you work well with synthetic data really change when you

• 20:59Speaker 01

  go to like a trillion token scale and beyond. And so this work was majorly about like understanding and like giving the lessons back to the community of like, how can you do good hybrid model training where there's like a part of data that's synthetic? So like, just like as a high level like flagship numbers, this model that we released, this results were very strong. So the Nemo Tron data set is like one of the top data sets today, which is like based with a lot of synthetic data. So the and the data logic data that we a model that we release called Beyond Web is the blue line over here. As you can see that we achieved the same performance as the Nvidia model in almost like two point seven X like lesser time and then much faster than anything that hugging face or red pajama

• 21:44Speaker 01

  does. Very interestingly, our three B model is pretty much the same performance as the eight B model of Nvidia. And this is quite like strong, given that Nvidia's NEMOtron data is like literally the most downloaded open source data set. It's like being downloaded like, I think a million times every month I was seeing recently. And so that's huge. So internally, we've been doing a lot to improve synth data. So just a little bit of backstory to explain the types of synth data approaches that exist today. So for the longest, for the beginning of how synth data came into picture was through the tiny stories and the fee model family. And so

• 22:28Speaker 00

  Really?

• 22:30Speaker 01

  Yeah, so that was like, I think the big push.

• 22:33Speaker 00

  Yeah, textbooks are all you need. Yeah.

• 22:35Speaker 01

  The textbooks are all you need. And even before that, there was this paper called Tiny Stories, which was by a couple of researchers at Microsoft on telling like, how can you train small models with entirely synthetic data? And so I would bucket these approaches into something that I call the generator driven paradigm, which means you have a big model, for instance, it could be the G P T four model, and you're trying to query that model to generate a textbook or an essay or a paragraph. But all the information in your data set is coming from the generator or from the model. And so you really need that the generator is huge and massive and contains information about everything on the world so that you can train models in the generator driven paradigm.

• 23:16Speaker 01

  And so a couple of years ago, when I was actually in internet, Apple, we released this paper called rephrasing the web. And we modeled this alternate paradigm, which is called the source rephrasing paradigm. And so we said the generator driven paradigm is exciting and doing well, but it just does not scale. You need the model to be massive and generating data will be very expensive. Plus, it really depends on you prompting in the right way. Because if I ask you to generate a paragraph about the laws of motion, then my data will have it, otherwise it will not. So it really puts the burden on the researchers to make sure that the data is diverse. But on the other hand, we have the entire corpus of the internet, which has so much knowledge available. The only issue is that the data might not be high quality.

• 23:59Speaker 01

  And so we can basically rephrase all of this data into higher quality data. So for instance, the internet becomes the source of knowledge and the synthetic data generation model is only modifying that knowledge into styles that you care about. So for instance, if question answering is something that you care about, then you repurpose the knowledge of the internet into smaller styles, into the question answering styles. So this really transchanges the whole philosophy of generating data by making the cost of synthetic data generation extremely small. Because now you can use like a very small off the shelf model. You can put the actual information inside it

• 24:42Speaker 01

  and you can ask it to rephrase it into question answering. And so the capability to transform data is actually very cheap. Even a one B model or a three B model can do this very well. You do not need all the knowledge of the internet to be within this model. And so the source rephrasing paradigm has actually become the dominant paradigm in 2006. Like even the KIMI K2 model has like a long section of like how they do rephrasing of internet content. Then even Grok 4 has been using the same source rephrasing paradigm. Nvidia's Nematron data that we are benchmarking against is also using this idea of source rephrasing. And so philosophically, in the pre-training phase, we have kind of almost finalized how we do synthetic data,

• 25:24Speaker 01

  which is like we transform existing knowledge into patterns that are useful for us. And in Beyond Web, we really go beyond what we had released at Apple when I was interning over there, which was called rephrasing the web. And now we have Beyond Web, which really like multiplies the advantages of what we were doing in SynthRita work.

• 25:42Speaker 00

  For a lightning episode, this is fantastic. And it's really good to sort of get a sense. And I think like, I want to also just like use the in-space to feature this kind of work. People are gonna read the paper on their own. We're not gonna cover the whole thing, right? So, but if they could reach out to you, I'll leave all the socia ls and all there. Yeah, and probably join you as well.

• 26:01Speaker 01

  Awesome. Yeah, thank you so much for having me. And yeah, I'm very excited to see the response.

• 26:06Speaker 00

  Thanks for all the great work. Yeah, I would say like, yeah, I, it's, to me, it's amazing. Because like when Red Pajama came out, like I, I realized like that was like the start of something. It's interesting to me that every single generation is a different company, you know? Like, oh, it was Together AI, and then it's like Microsoft or whatever. And it's Apple, then it's Hugging Face, then it's Nvidia, now it's you guys. And I'm like, you know, where, where's the, the, the sort of persistence? Or like, is this such a competitive field? Or you guys keep changing companies. So it's the same, it's the same people.

• 26:40Speaker 01

  Now we have a hub for all data enthusiasts. I think it's gonna be the same for a while.

• 26:44Speaker 00

  Yeah, okay. Well, thank you so much. That was great.

• 26:47Speaker 01

  Thank you so much. Bye bye.