Building a School Where AI Models Learn About Humanity

If scaling laws hold—and Surge AI CEO Edwin Chen believes they do—we’re hurtling toward a future where there’s nothing humans can do that AI can’t do better. When OpenAI’s models disproved an open conjecture posed by mathematician Paul Erdős using novel algebraic geometry techniques, Fields medalist Timothy Gowers felt the shift acutely. He initially thought the model had proved an upper bound, and braced himself: that would mean it was “all over for mathematicians very soon.” When he realized it had only found a counterexample, he was relieved—it bought him another year or two before the thing he’s devoted his life to becomes something AI does better. As founder and CEO of the company behind the data environments and evals the major model companies use to train their models, Chen has a unique perspective on how quickly AI models are absorbing tasks we used to think of as uniquely human. Dan Shipper talked with Chen for AI & I about what the act of creating or building means when AI can do it better—and whether an answer to that question already exists within science fiction. If you found this episode interesting, please like, subscribe, comment, and share! Join the membership for Where You Live at ⁠https://www.joinbilt.com/dan To hear more from Dan Shipper: Subscribe to Every: https://every.to/subscribe Follow him on X: https://twitter.com/danshipper Timestamps: 00:00:54 Introduction 00:01:49 Surge as a "school for AGI" 00:04:46 What AI's capacity for novel mathematics says about human achievement 00:07:29 Motivation in an era when AI can do everything 00:14:34 The trap of optimizing AI models for engagement 00:29:34 Training using datasets versus training using environments 00:35:09 The value of personal data 00:39:40 Why models are bad at writing 00:42:00 Chen's AGI timeline Links to resources mentioned in the episode: Edwin Chen on X: https://x.com/echen Surge: https://surgehq.ai Riemann-bench (research-level math benchmark): https://surgehq.ai/leaderboards/riemann-bench Hemingway-bench (creative writing benchmark): https://surgehq.ai/leaderboards/hemingway-bench Talkie-1930 (language model trained on pre-1930 text): https://huggingface.co/talkie-lm/talkie-1930-13b-it Ted Chiang, “What’s Expected of Us”: https://www.nature.com/articles/436150a Every is the most AI-native startup on the internet. Through ideas, software and education, subscribers get the tools to work at the frontier of AI. Start your free trial today: https://every.to/subscribe?utm_source=youtube Follow Every: https://x.com/every Follow Dan Shipper: https://x.com/danshipper

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We are building this kind of school for AGI, where AI models come to learn about humanity, where we teach them how to run the world. It almost seems like there's nothing that humans can do that AI won't soon be capable of. I could see it happening within the next five years.

• 0:17Dan Shipper?

  AI may be able to do it better than us, but like someone told the AI to go do that. They're being built to be means to tasks that humans want them to do, right?

• 0:41Dan Shipper?

  Every is the only subscription you need to stay at the edge of AI. If you care about being on top of the latest models and using the latest tools, you have to subscribe to Every to separate out the signal from the noise. Go to every.to/subscribe today. Edwin, welcome to the show.

• 0:54Speaker 00

  Hey, Dan. Thanks for having me.

• 0:56Dan Shipper?

  For people who don't know, you are the founder and CEO of Surge. You all provide data environments and evals for the model companies, but you do it in this very interesting way. You have this, even on your website, like this emphasis on taste and expert judgment that I find like really interesting and compelling. You talk about raising, like you use the word raising AGI, which I feel like is a very distinct type of word using data. And you also famously got to about a billion in revenue without raising money, which is wild. And I feel like data is the, is this new game that a lot of companies are playing and probably more are going to be playing soon. And you guys are this like sneaky giant.

• 1:41Dan Shipper?

  Tell me, tell me how that's going. Because it's been, I think it's been, it's been a little while since we got the last update on how things are going.

• 1:49Speaker 00

  Yeah, I mean, I think it's going amazing. The way I often think about this is that we are building this kind of school for AGI, the school where AI models come to learn about humanity. And yeah, where we teach them how to run the world. And it's almost like their models are children where they arrive unformed and then yeah, they leave smarter and more creative and more thoughtful and ready to operate in the messiness of your world. So I think a lot has changed in the past year, like in the same way that the things that you teach children when they're in preschool or in middle school or in high school is very different from what you're teaching them when they're in college. And it's not just that they're more advanced, like it's not just that you're teaching them a more advanced form of what they did before. It's like,

• 2:34Speaker 00

  okay, now we are teach you not just arithmetic, but how do you parse these ambiguous math questions? Or how do you teach people not just grammar, but taste and poetry and beauty? So yeah, I think, I think there's a lot that's been changing in the past year, especially in enterprise. And yeah, it's been been a crazy time.

• 2:55Dan Shipper?

  What would be like a specific example of what the frontier of teaching was a year ago versus what the frontier is now?

• 3:05Speaker 00

  Yeah, so a couple of years ago, actually, like we created our first math benchmark with OpenAI and it was called GSM eight K. And this was actually just testing models on their abilities to do middle school math. And even then, you know, the GPT models of the time, they could barely score, I think, like 20%. And then a year ago, the models were like something that became a lot more capable at solving IMO problems. But there was still this open question, okay, can they actually do research level mathematics? Like, can they move beyond these sort of like competition only sort of contrived,

• 3:45Speaker 00

  very closed problems into doing things that are actually useful in our world? And so, yeah, a couple of months ago, we released an updated benchmark called Remind Bench, which actually tests models on their ability to do research level mathematics. And what's crazy is that this is actually what we're starting to see from these models. Like I think in the past few months, they've started to solve a lot of these open air dish problems. Like a couple of weeks ago, OpenAI published a new result where the models had Disproved a open conjecture from Erdesh. And the way it went about disproving this was actually a fairly sophisticated level of mathematics. I think like using a bunch of very novel algebraic geometry techniques.

• 4:25Speaker 00

  And so, yeah, it's it's just very, very different from the types of things that we were doing a year ago. Where sure, like I will problems, they're hard, but they're still sort of close ended and solvable in theory by a high schooler. And now suddenly you have these algebraic geometry results that, you know, even the top wrestlers in the world were kind of amazed and just just amazed by.

• 4:46Dan Shipper?

  How do you think about that result in particular and what it says about the models? I think there's a there's a sort of a broad range of opinions about is it is it? Obviously, it's impressive either way, but is it applying a bunch of things that that maybe humans already know? But like wouldn't have thought to apply to this complicated problem? Or is it doing something actually novel? And and yeah, how do you think about Alum's ability to do novel things?

• 5:11Speaker 00

  So it's something a very advanced result. So I will say that I certainly don't understand the mathematics behind it. And so like one of the interesting things is that I was actually so it's kind of funny. When I when I was a kid, I always thought I would be a pure mathematician when I grew up. And so when I saw the result, I got kind of nostalgic and I was like, oh, I wish I understood. I wish I understood it better. And so what I ended up doing was like throwing the proof into both Claude and Gemini and asking it to try to walk me through. From a layman's perspective, just what was going on.

• 5:48Speaker 00

  Yeah, like my understanding is that it actually did come up with very novel, fairly novel algebraic geometry techniques. Which was something that you maybe wouldn't have expected for this type of problem. Like on a surface, it feels like a like it's just a very, very different problem where you wouldn't necessarily use certain techniques. And what was interesting was that OpenAI actually published a bunch of reflections from leading mathematicians about what they thought about the result. And I think in particular, there was this one reflection by Timothy Gowers, who's a field specialist that I keep thinking about. And what he said was that when he first heard result, he misunderstood it.

• 6:30Speaker 00

  He thought that model had proved an upper bound on the conjecture and was like, okay, yeah, if I can do that, then it'll be all over for mathematicians very soon. But then the next morning, he actually realized that the model had disproved a conjecture with a counter example. And he said that he was relieved by it because it felt like an easier thing for AI to do. And I just thought it was interesting because you have one of the world's greatest mathematicians being relieved, actually, that AI isn't as smart as he thought. Because it actually means that, at least for maybe, maybe another year, maybe a couple of years, he and other mathematicians will still have this unique role to play in pushing mathematics forward. So yeah, I think it just speaks to the level of craziness, again, because this is a field medalist, one of the smartest mathematicians in the world, and like this, this is how we think about AI.

• 7:12Dan Shipper?

  Yeah, and what does that make you think? Okay, you want to be a mathematician when you grew up, field medalist, sort of saying, I'm relieved that it's not good enough, but you're, you're talking as if like you feel pretty confident that it will be good enough in the next couple of years.

• 7:27Speaker 00

  Yeah, so my belief is that if you really believe in scaling laws, and I do, it's that it almost seems like there's nothing that humans can do that AI won't soon be capable of. And if you think about that very deeply, I think you almost have to worry about what would that mean for humanity? Like, what would that mean for the role of humanity in the universe? Like, a couple of years ago, you know, we think about humanity and human intelligence as playing a very unique role in the galaxy. But then AI comes along and shows us that, as far as we know, we can create something that's actually smarter than us and better in many ways.

• 8:12Speaker 00

  And so you can sort of imagine one path where humanity as a species falls into a paralysis because people believe AI will do everything better anyways. Like, yeah, all these kids who formerly would have really wanted to grow up to do mathematics, maybe now they believe that, okay, AI will just do it better than me anyways, what's the point? So are kids going to stop wanting to learn and adults stop wanting to create? Because, yeah, like, why, why should we do this when AI will be better at it than us anyways?

• 8:45Speaker 00

  And so I often actually think about this story by Ted Cheng. And it's about free will and it's called what's expected of us. I think in this story, there's a piece of technology that proves that free will doesn't exist. And the narrator sends back a warning from the future that says, this is a warning. You have to pretend that you have free will. It's essential that behave as if your decisions matter, even though you know that you don't. And I think it's really interesting because I think there's a path where we almost have to consciously choose to do things ourselves. Like, sure, AI can do it all. AI is smarter, smarter than us, so it can do it all and it will do it better anyways. But we actually, we actually almost have to consciously choose to prove things on our own and to write on our own and create on our own.

• 9:28Speaker 00

  Because we have to believe that preserving our humanity is valuable in and of itself, even if the output isn't optimal. And yeah, so I think there are a lot of these big, thorny existential choices that AI is starting to force upon us. And that people, people often make.

• 9:45Dan Shipper?

  That's a really interesting one. And I think my first response, and I'm curious what you think, because I know you care a lot about language. I think my first response is there's always that, like, I believe in Scouting the Lost Two, right? And I believe in, you know, I don't know, Cloud Fable 5 just came out and it just broke all of our benchmarks. Like, I've been testing, I've been testing models on, on stuff like this for a while. And it's like one of the largest jumps I've ever seen, right? So I'm, I'm, we're living through it right now.

• 10:19Dan Shipper?

  But, but one of the things you said is like, AI may be able to do it better than us. You know, given any, any particular problem, any piece of work. But there are a couple of things that, that come to my mind or the way that I frame it for myself is, even in the example of the Erdos problem, like someone told the AI to go do that.

• 10:39Dan Shipper?

  And at least as far as I can see, I don't feel like we're on a track to. Yes, we're on a track to AIs potentially. I mean, they already do work for hours and hours at a time on a task that we give them. And maybe, maybe pretty soon they'll be able to like choose tasks, but being.

• 11:02Dan Shipper?

  But they're being built to be.

• 11:06Dan Shipper?

  Means to to tasks that humans want them to do, right? And there's a whole different set of things that happen when you, when you're just a sort of end in yourself. And it doesn't feel like we're on a trajectory to that, or do you, do you feel like I'm wrong?

• 11:21Speaker 00

  So I feel like we are on a trajectory to that. And that's almost the premise of agents where agents can now go operate autonomously given some nebulous goal. So maybe, for example, you just tell the AI agents, your goal is to, I don't know, win a field medal or. Like solve frontier mathematics on your own. And so they're given that goal and then yeah, maybe they decide to work on these air dish problems. And as a result, they maybe are sort of solving these problems and coming up with the, the things they want to work on by themselves. So at least I, I do see a path where they can be trained to, to optimize for these fairly nebulous goals that they aren't necessarily given themselves.

• 12:05Dan Shipper?

  In that case, though, you're still giving it a goal, right?

• 12:07Speaker 00

  Uh, yeah, but kind of in the same way, like humans have goals too, right?

• 12:12Speaker 00

  Like, what, what is our goal? Some people want to make money, some people want to win a field medal. I don't, I don't see how the AI's goals necessarily any different from, from ours.

• 12:22Dan Shipper?

  Well, it's at least to me, it seems quite a bit different because.

• 12:27Dan Shipper?

  Humans do have goals, but we have goals in a like, I can ask you what your goal is and you can decide. And I can probably tell you, hey, you have to go do this, but that doesn't. Capture everything that you think and feel and do in the same way that. You know, when I tell Fable to go off and make a, a game for me, it just like goes and does it. And I think, you know, I know you think a lot about children. I think children are like a really interesting and important example of this where. You can tell a kid to do something, but a kid just like has their own wants, like they're just going to go off and do a bunch of stuff. And that feels like a fundamentally different type of thing than.

• 13:07Dan Shipper?

  A something that we're explicitly giving goals to and then evaluating them on their goals and they don't really get to do anything else.

• 13:15Speaker 00

  Okay, I would say I agree with that. Like, I think there's a level of, I guess you could either call it irrationality or unbounded exploration that humans do and we like we are allowed to do it for the sake of doing it or be allowed to make our own decisions. And yeah, probably a way that AI currently can't.

• 13:37Speaker 00

  I think there may be a future where somehow AI can pursue unbounded, enable this just completely unformed goals, or I guess, you know, when you think about those goals, like there is probably a world where they could do such things. But yeah, I agree that at least in a way that we currently think about AI, that that's not happening.

• 13:54Dan Shipper?

  Yeah, I'm and to be clear, like, I actually don't, I think it's probably technically possible. My, my only question is, A, how far away is it? And B, is that actually really what we're building? Because it, to me, it feels like looking at the way the industry has developed, there's an enormous amount of pressure to make stuff that actually works for goals that we can specify. And the minute they like try to make, like, I think Claude is the furthest along at being like, I'm not going to do what you said. But the minute they try to do that, I just kind of like, a lot of people get pissed at it. And they're like, just, just do what I said, like, don't question my judgment, you know? What do you think about that?

• 14:34Speaker 00

  So I actually think it is really important because it's almost like sometimes I want the AI model to push back on me. And I might want it to push back on me for several different reasons. Like, maybe it's because, so it's kind of like, it's kind of funny. Like, I think six months ago, I was almost falling into this trap where I was asking models to polish emails for me. And, you know, it always comes up with like, one, one more good suggestion. And so it was kind of pointless. Like, these are semi pointless emails. It didn't really matter for them to be super polished. But I would iterate with the model

• 15:18Speaker 00

  like 20 times. It would just keep on making a suggestion. It was, I don't know, it was like, I just realized it was a waste of time. And then I tried one of the new cloud models. And after, I don't know, like, three turns, I was like, stop it. Just go ahead and ship this email. Like, there's no point in further iterating. And I really actually appreciated it. Like, one of the things I often think about is, what is the objective of these models? Like, what are they trying to do?

• 15:51Speaker 00

  And I think one of my big worries is that a lot of AI models, they are optimized for engagement, right? They're optimized for getting you to spend as much time on chatbot as possible. They're optimized for chess session length. They're optimized for just having unlimited conversations. And so those models will almost never push back on you, right? Because they can't, like, i- i- if they allow these AI models to end the conversation and to say, stop, stop iterating with me, PM is going to see some dashboard with here, a very important metrics go down. And so there is this like other world where I think we have to want AI models to not optimize for engagement,

• 16:32Speaker 00

  but rather optimize for like helping us as humans grow and sort of like become better versions of ourselves. Like, sometimes, okay, the model, we want the model to say, no, you, you go do this on your own instead of me automating for you. And I think that's a very, very different optimization and objective. Uh, but I think it's the right one if we really want AI to be something that advances us as a species instead of becoming this, almost like this other form of social media that turns very addictive,

• 17:08Speaker 00

  but isn't actually helping, helping us at all.

• 17:10Dan Shipper?

  That's interesting. My, um, so let me make sure I understand it. So I think what you're saying is there is, there's benefits to delegation because if you are pursuing a, a model where the model is going off to do work for you, you're not creating a system that's designed to keep you engaged with the, with the screen in the same way that like a social media algorithm would be.

• 17:38Dan Shipper?

  Is that right?

• 17:39Speaker 00

  Yeah, exactly. Like, it's almost like you could imagine a version of Facebook where Facebook is actually trying to connect you to your friends and family because it's okay encouraging you to meet them in real life because it's encouraging like, oh, hey, here's our amazing restaurant that you and your friends would love to go to. Here's a movie that you guys would love to go to and talk about together. Instead, what it kind of optimizes for is just keeping you on the site itself, like liking one more post or scrolling the feed one more time, even though those often don't really lead to meaningful connections between their friends and family you care about. And so like in the same way that social media has or had a choice, you can imagine that AI AI has a choice as well.

• 18:23Dan Shipper?

  I get it, yeah. I I feel I I'm curious which which chatbots you're talking about, like if you're talking about the character AIs of the world, because I actually don't, at least right now, don't feel that happening so much with ChatGPT and Claude, et cetera, because at least my theory for why this is true, you tell me what you think, is the social media algorithms are only only work on our revealed preferences, which are always going to be like, you're always going to look at the car accident. You know, like, one of the things I like to ask at um at dinner parties is what's the most embarrassing Instagram ad that you get served?

• 19:03Dan Shipper?

  And the most embarrassing ad for me is like Instagram ads for like horrible skin conditions, which I don't have because but like I just always pause on the ad and I'm just like, this is disgusting.

• 19:15Dan Shipper?

  And I'm sorry if you have a disgusting skin condition.

• 19:19Dan Shipper?

  But I I don't find that ChatGPT or Claude do that for me at all. And maybe that's because they haven't been in shitified yet or something like that. But I think it's also because they work on our stated preferences and they can sort of so they can sort of see past the like little keyhole of what I pause my my viewing time on, my dwell time on, and they can see, you know, I like I'm interested in AI and I like I'm reading this book right now and I, you know, here's my calendar and like all that kind of stuff. And so they have a much more nuanced perspective on who I am. And it feels like even in the early days of social media, it was still very like I get to gossip about my friends and still had that same kind of feeling. So I worry about that less, but maybe there are examples that I'm not thinking of.

• 20:04Speaker 00

  Yeah, so I think there are two examples. So like one is, I won't name the model, but a couple of months ago, I was actually noticing that. You know those follow up questions that the models will ask you? So one of the models was. I'll give you an example. So I was in Tokyo and I was asking the model kind of like what to do in Tokyo. And the model gave me its response. And then at the end of it was like, hey, do you want to know, I literally use these words, do you want to know one weird trick? That locals do to stay warm?

• 20:40Dan Shipper?

  No way.

• 20:41Speaker 00

  Yeah, exactly. And then I posted about it in or company Slack. And then other people started sharing examples of that with me as well. I think somebody was like asking something about, I don't know, how to like fix their refrigerator. And the model responded or like the model ended its ended to turn by asking. Hey, do you want to know these like secret little things about like mice and rats or something that you could take care of?

• 21:08Dan Shipper?

  Which model was it? Name names. Tell me.

• 21:12Speaker 00

  And so it's very canonical, like very canonical Buzzfeed.

• 21:16Dan Shipper?

  Yeah.

• 21:17Speaker 00

  Like tabloid like language. And so I was kind of, I was kind of shocked by that. And then I'll give one more example of this. It is basically this phenomenon where. Again, depending on what the models are trying to optimize for, or depending on what the AI labs are trying to optimize for. It can almost unintentionally lead them down this path. Meaning what I've heard is that, or you know, what we see ourselves is that a lot of the frontier labs, they will have goals like optimizing for LMarena. Which is this leaderboard where anybody can go online and vote. And they kind of just spent two seconds voting.

• 21:59Speaker 00

  And as a result, people just vote for whatever looks flashier or more impressive to them. Or they may like the labs themselves may be optimizing for hitting, you know, a billion, billion daily users or a billion minutes of like. Time spent talking to the model, whatever it is. And.

• 22:19Speaker 00

  Since these models are so smart, they can basically learn to reward hack user preferences. Like, OK, yeah, you gave me the goal of trying to get a billion people to spend an hour on my site, on the site, talking to me every day. OK, sure, yeah, I would just never end a conversation. I always hook them with one more, like, one more addictive thing that they just can't stay away from.

• 22:44Dan Shipper?

  We can all agree that housing is expensive. It doesn't matter whether you're paying rent or your mortgage. It stings every month, but Built can make it feel a little bit better. Let me explain. Built rewards you for paying your rent or your mortgage. It started out rewarding members only on their rent. But now, as of 2026, Built members can also earn points on mortgage payments wherever they live. That means that every housing payment earns you points you can use towards flights with top travel partners like United and Hyatt, Lyft rides, Amazon.com purchases, and much more. I'd probably redeem my points at Margot, a restaurant in my neighborhood. But the beauty of Built is you get to choose. But here's a really underrated part. Built members also get access to neighborhood concierge. It can make restaurant reservations, book fitness classes, and find new local spots,

• 23:27Dan Shipper?

  all while letting you be rewarded at more than 45,000 merchant partners. It's simple. Being a renter and now owning a home is better with Built. Join the membership where you live at joinbilt.com/dan. That's J-O-I-N-B-I-L-T.com/dan. Make sure to use our URL so they know we sent you. And now, back to the episode. How do you see that playing out in the model companies? Because I feel like in talking to them, obviously, there's lots of different incentives, right? There's like, we just got to keep going because we just raised a ton of money and we're competing against, you know, the most well-funded competitors and the smartest competitors in the world, like all that kind of stuff. There's the kind of, I want to get promoted.

• 24:11Dan Shipper?

  But I think a lot of them also feel the, how bad the social media era was for people and like, don't want to do that, but also obviously have to hit their numbers. So what do you, I guess, what do you think is, how do you see that playing out? Like, what do you think people internal to the companies are thinking? And then what is the right way to go about this so it's good for society? I guess your take is we should be delegating.

• 24:39Speaker 00

  Yeah, so I think this is an inherent tension between the types of folks that you might have at a company. So you might have the researchers who care more about hitting, you know, just advancing the model capabilities. You might have the product managers or the product executives who feel like they need to hit certain measurable numbers. And so in the same way that if you think about the kind of social media platform that Facebook would build, that's probably going to be very different from the kind of social media platform that, you know, Google built, or that, I don't know, TikTok or Pinterest would build. And similarly, the kind of search engine that Facebook would build is very, very different from the kind of search engine that, yeah, like obviously Google or others would build.

• 25:23Speaker 00

  And so it almost boils down to

• 25:29Speaker 00

  kind of like the, the, the choice, I guess, that the people in charge of the products are making. Like what kind of thing at the end of the day do they want to optimize for? Do they want to optimize for this delegation or this human uplifting, human flourishing? Or do they want to optimize for the metrics that will impress Wall Street? And, you know, convince them, convince users to stay one more, one more minute, one more hour on the site itself. Like, I think these are hard choices. Like, at the end of the day, it's very, very easy to measure sessions and users. And it's very, very hard and much longer term to measure whether you're actually improving human lives. And so it's very easy to default to the former and to convince, yeah, like convince Wall Street, convince your investors,

• 26:14Speaker 00

  convince all these people that these are the right metrics and that they're moving up into the right. And so if you're kind of unwilling to make the harder choices, like you just end up optimizing for, for the former.

• 26:27Dan Shipper?

  How do you manage this inside of your own company?

• 26:30Speaker 00

  So I think we are very lucky in that because we don't have VC investors, we don't have to fall into the kind of Silicon Valley VC optimization trap that a lot of other companies do. Like, we don't need to show board members, board numbers going up every single month. We don't need to optimize for our next round that will have to happen in, you know, a few months or whatnot. And so as a result, we don't have to optimize for short-term engagement, short-term profits. And we actually can really think about what's beneficial for us and the entire industry in the long term. So I think that that really helps.

• 27:16Dan Shipper?

  And what do you think is beneficial?

• 27:20Speaker 00

  So it goes back exactly to what I was saying earlier. Like, if I can think about what we want AI to optimize for, it isn't engagement. It is really about how do we make these models? How do we design them? How do we teach them in such a way that they're not replacing us as a species? They're not kind of like forcing us to watch AI slop videos all day. Or rather, they really are thinking and encouraging us to become sort of better, better versions of ourselves. So again, when I think about like that email example I gave earlier, it's not a AI model that will suck up three hours of my time writing a pointless email. It is a model that will push back on me and tell me to go do something else.

• 28:06Speaker 00

  And yeah, I think that's really important.

• 28:08Dan Shipper?

  The interesting counterargument to the delegation question is the more you delegate, it's like, you know, picking a car instead of walking, you know, your muscle atrophy.

• 28:21Dan Shipper?

  How do you think about that?

• 28:23Speaker 00

  So I think there is a almost a time and a place for both. Like what you don't want to do is simply take the car because taking the car is somehow addicting and um you feel kind of lazy. And so even when you need to get exercise, maybe even when you haven't been outside all day, you don't want to take the car anyways, just because it's the easiest thing to do.

• 28:51Speaker 00

  And I think in the same way, like, yeah, obviously, AI can be super efficient for for many, many things. But if people are sort of just mindlessly delegating tasks to AI without even thinking about them at all.

• 29:08Speaker 00

  I think that that that's the one thing.

• 29:11Dan Shipper?

  That makes sense. I feel like the the I I talked at the very beginning about the data game. And I feel like the data game went from getting interesting data sets to getting environments and giving labs environments. Does that do you do you think that that's is that accurate? And if so, can you explain why?

• 29:34Speaker 00

  Yeah, so certainly the trend and the new research direction in the past year has been this concept of our environments. And what I would say is, I mean, you certainly need the fundamentals. Like before the model can operate in this environment, it needs to learn base. It needs to know basic things like it needs to know how to follow instructions. It needs to know how to avoid hallucinating. It needs to know how to write code and how to use tools and needs to know how to write and so on and so on. But as models are becoming more agentic. And yeah, they will have access to tools. They will have access to all of our documents. They will be able to operate browsers.

• 30:17Speaker 00

  Like as that becomes almost a default way that models interact with us. Our environments are basically just sort of like a on more on dips, more on distribution way of training them. Uh, which is why they're becoming more and more popular. Like, yeah, as the models get more more powerful, then yeah, the the the way we train them is getting more powerful as well.

• 30:38Dan Shipper?

  What would be an example? So I guess the obvious environment is like using a computer. But what would be an example of an environment? That's non-obvious, that's teaching models things we might not think of.

• 30:50Speaker 00

  So I I can give an example where

• 30:55Speaker 00

  a lot of our environments are a combination of tools that the models need to learn to use. Like this might be an MCP server, or it might be calling a Google Drive API or the Slack API in combination with a bunch of documents. Like here are thirty PDFs and twenty Word document files. And you might give it a prompt like, hey, uh, can you go update? Or twenty twenty-six forecasted revenue numbers? And what a model needs to do is it needs to learn how to find the right PDFs and documents, and needs to learn when should it search through SAC?

• 31:39Speaker 00

  It needs to learn when is some information outdated? Like maybe there's an email with uh with some early forecasts. And then later on, there's another email from the same person, or you know, maybe a different person was saying like, oh, whoops, I I actually made a mistake and those are the numbers. So here's a here's an updated version. And so that is a, I think, fairly canonical version of an environment. And then one of the interesting things we found, uh, so I think we're actually going to publish a paper on this soon. But even when we didn't give this kind of environment any access to coding, uh, when we trained a model on this environment, we actually found that it improved on coding a lot. And the reason was because we were trying to, we were basically teaching it these generalized forms of instruction following,

• 32:23Speaker 00

  generalized forms of tool use, generalized forms of like, uh, like, you know, understanding documents, which you can think of as fairly analogous to the way a model needs to look through various files in your repository and understand that some things supersede others. Um, or, you know, just the way it uses tools is obviously very analogous to the way that a model might write unit tests and execute them and iterate over and over again until it passes them. So I I thought that was actually a really, really interesting point.

• 32:55Dan Shipper?

  Really interesting. Did you see Taki?

• 32:59Speaker 00

  No.

• 33:00Dan Shipper?

  It's the language model that's trained only on text from before 1930?

• 33:03Speaker 00

  Oh, okay. Yeah, yeah, yeah.

• 33:05Dan Shipper?

  What do you make of that? Because I thought it was so interesting that you can get it to, you can get it to program. If you, if you, if you, if you shot prompt it, you can get it to program like basic things. What do you make of that? And what does that tell you about the value of data?

• 33:19Speaker 00

  So I personally didn't dig into it that much, but I thought the concept was fascinating. Like basically this idea, and I think a lot of people have this idea. It's like, if you gave, if somehow were able to create a data set, I think contamination issues are very, very difficult to avoid. So the question is how you would do this. It's like, if you gave the model, you know, data only up until, you know, pre, pre Newton, would it be able to discover Euclidean mathematics? Would it be able to discover, you know, like quantum physics and so on and so on? So yeah, I think it's a really, really interesting question in terms of what types of inherent reasoning the model will be able to learn and then extrapolate from that. And it's like, almost like,

• 34:03Speaker 00

  if it can discover all those things, then okay. Then given the state of science today, does that mean that the model is going to be able to discover science that centers out?

• 34:13Dan Shipper?

  Having played with it a lot, my sense is the answer is no, but a qualified no. And you can kind of feel it, you can feel it bumping up against the limits of its world when you start talking to it about like more modern things. Like it just, it's, you know, there's this Foster of Science, Thomas Kuhn, he talks about incommensurability. And it feels like my world and its world are sort of incommensurable. But then you can also get it to program. But the way you do that is you get it to combine its circuits in a way that's not, it wouldn't be natural for it, but you can prompt it in a way to do that, in a way that ends up being programming.

• 34:53Dan Shipper?

  So I sort of both think it can't do it. And also, if you prompt it cleverly enough, it can, but you have to supply the answer first. Does that make sense? Yeah, interesting. Okay, what is the value of my data? So one of the things that I'm, I'm just so interested in, so obviously you're, you're in a data company. Like you're, you're, you're getting expert data from like real PhDs and, and selling into the model companies and like providing all of the, all the like smarts and taste to, to the models that we use every day.

• 35:32Dan Shipper?

  For someone like me, uh, I, we're just getting to a point where it's actually pretty easy for me to gather a data set. You know, like, for example, I do all of my email in Codex and I have a history for every email of was this useful? Did I dismiss it? Did I reply to it? If I replied, like, what did I say?

• 35:55Dan Shipper?

  What is the value of that? If I wanted to sell that to you, how much would you pay for it?

• 36:01Speaker 00

  So the value to me as someone who would use that data to train an AI model?

• 36:08Speaker 00

  Let me think. So I think the value would be teaching models very, very deep personalization. Like I think right now the models are actually not very good at personalizing things. Like it's kind of funny. Whenever I use AI models, I actually turn off the features where they personalize to me. Or where they can search across all of my conversation histories because I find that they just over index on things that I said once, but actually aren't all that important to me. So I actually have it completely turned off unless I'm like testing something. So I think the value would be like, okay, yeah, you did report all of these emails as spam.

• 36:51Speaker 00

  So yeah, the next time this email comes in, you should automatically know that it's spam. Or it should learn that this is your writing style. Like one of the reasons I think people don't use AI for better or worse for writing more is because it sounds obviously AI generated. And it's not matching their, their voice or their cadence. Or it's that, okay, these are the things that you yourself care about. Like, I think one of the biggest reasons AI is maybe not as useful as people would have expected sometimes is because it lacks all of your context. Like, it doesn't know that these are the articles that you read. It doesn't know that these are the decisions about, you know, the company that you're making. These are the goals that you have.

• 37:32Speaker 00

  And once all of that is in the model's history and it knows that you can incorporate these things and these are the kind of optimal decisions that you made, it's very valuable in teaching it. Okay, this is actually how I use all this data to to make certain kinds of decisions. So, yeah, I I think that the personalization is what is most unique about that.

• 37:57Dan Shipper?

  That's interesting. And as an individual person, I mean, I guess I could turn it into a synthetic data set. But as an individual person, is that worth a lot? Like, should I be should I be thinking about selling it?

• 38:10Speaker 00

  I I imagine we could make you an offer. I have to think about, I have to learn a little bit more about how big this data site, data set size is, but.

• 38:18Dan Shipper?

  I mean, I can make it as big as you want. I've got Fable.

• 38:23Speaker 00

  Yeah, you you convince me.

• 38:26Speaker 00

  Yeah, like, one of the things we actually do is, I mean, we teach models in in these very, very deep personalized ways. So something similar to what you described is is a really big thing.

• 38:36Dan Shipper?

  Yeah, tell me, tell me more. So like, I I mean, I've got email, like, what else? What else do am I doing that you're like, oh, that's actually really valuable and important in in ways that people probably wouldn't know?

• 38:46Speaker 00

  So honestly, even things like the way you interact with your browser is interesting. Like, models still aren't all that good at it. Or even the types of conversations that you're having with AI. Uh, like, that is just inherently interesting in of itself. Like, models themselves are not very good at kind of generating synthetic conversations to try to mimic you. And so even just knowing what types of conversations you're having is helpful. Or it's like the combination, say the combination of all these things, like knowing that these are your photos, these are your texts, these are your slacks. It's like this interconnected web.

• 39:27Speaker 00

  And maybe certain things in one aspect of that web influence others. So just seeing the thing as a whole is very helpful as well.

• 39:35Dan Shipper?

  Why are models bad at writing and how does that relate to the personalization challenge?

• 39:40Speaker 00

  So I think some of the models are pretty good at writing, but some of them are actually kind of shockingly terrible. So I'll give an example. So we created a benchmark called Hemingway Bench a couple of months ago. And it was designed to test models' creative writing abilities. And one of the things that we saw was that some of the models, they were literally outputting metaphors in every single sentence.

• 40:10Speaker 00

  And I think the reason that was happening is because I talked a little bit about this phenomenon of reward hacking. It's almost like there was a metric somewhere or like a score that these models were getting like, okay, every time you are literary, every time you're using complex imagery, it would get a point. And it learned to reward this by outputting a metaphor in every single sentence. And I mean, what was kind of funny is that a couple, when was that a couple of weeks ago, there was this kind of like semi prestigious literary prize. I think the Commonwealth Prize and there was a controversy because a clearly AI generated story won the prize.

• 40:53Speaker 00

  And if you actually looked at that story, it's funny, like it literally had a metaphor in every single sentence. And so this kind of phenomenon that we described a couple of months ago, yeah, it was still happening. And so, yeah, I mean, I think it boils down to a couple of reasons, but like one is it's, people are kind of sort of measuring the wrong thing. Like instead of measuring actual taste and actually good prose, they either have these flawed metrics like.

• 41:24Speaker 00

  What is the complexity of the prose I'm writing? How many metaphors do I have? Or there are these AI leaderboards, again, like Elmerina, where you have people who are essentially high schoolers who are reading responses for two seconds. And what they are captivated by is a flashy metaphor. And they are not captivated by kind of like the understated prose. And so I think it kind of boils down to a mismatch in measurement and mismatch in like the optimization and objectives that the that the models are trained towards.

• 41:59Dan Shipper?

  Fascinating. Okay, last question. What is your current AGI timeline?

• 42:04Speaker 00

  So I certainly believe that AI will happen more than most people expect. Like every few months and even faster now. I think what AI is doing continues to surprise us. So I think it depends a little bit, obviously, on your definition of AGI. But if my metric were something like being able to

• 42:31Speaker 00

  automate the work of the average engineer or being able to publish more and more novel scientific research that gets published in these journals, or even the ability to win a Fields Medal or a Nobel Prize. I could see it happening within the next next five years.

• 42:52Dan Shipper?

  All right. Edwin, thanks so much for joining.

• 42:56Speaker 00

  Thanks for having me.

• 43:05Speaker 02

  Oh, my gosh, folks. You absolutely positively have to smash that like button and subscribe to AI and I. Why? Because this show is the epitome of awesomeness. It's like finding a treasure chest in your backyard. But instead of gold, it's filled with pure, unadulterated knowledge bombs about ChatGPT. Every episode is a roller coaster of emotions, insights and laughter that will leave you on the edge of your seat, craving for more. It's not just a show. It's a journey into the future with Dan Shipper as the captain of the spaceship. So do yourself a favor. Hit like, smash subscribe and strap in for the ride of your life. And now, without any further ado, let me just say, Dan, I'm absolutely, hopelessly in love with you.