He Built an AI Audience Simulator. It’s the Future of Customer Research. - Ep. 49 with Michael Taylor

Michael Taylor has perfected the art of getting AI to speak in tongues. He’s taught it to mimic the voices of your customers—so you can see how they would respond before you ship. Michael is the creator of Rally, a market research tool that lets you simulate an audience of AI personas. He built a simulator that lets us A/B test Every’s headlines on an audience that mimics the real Hacker News audience. It’s become a part of my writing workflow, and I love it because you test your assumptions quickly, cheaply, and without any of the risks of putting something out into the world. Besides Rally, Michael co-authored a book on prompt engineering for O’Reilly, and he writes a column for Every about managing AI tools like you would people. In a past life, he founded a growth marketing agency which he grew to 50 people and sold in 2020. One of the reasons I’m drawn to Michael’s work is because he has a tinkerer’s mindset. He’s always exploring the limits of what a new technology can do, and what he’s into today, everyone else will likely discover six months later. We spent an hour talking about using language models to judge your work, best practices for assessing an AI’s performance, and Michael’s flow inside Cursor. He also demos Rally live on the show, testing three different potential headlines for an Every article. If you found this episode interesting, please like, subscribe, comment, and share! Want even more? Sign up for Every to unlock our ultimate guide to prompting ChatGPT here: https://every.ck.page/ultimate-guide-to-prompting-chatgpt. It’s usually only for paying subscribers, but you can get it here for free. To hear more from Dan Shipper: Subscribe to Every: https://every.to/subscribe Follow him on X: https://twitter.com/danshipper Timestamps: Introduction: 00:01:32 AI can simulate human personalities with remarkable precision: 00:04:30 How Michael simulated a Hacker News audience: 00:08:15 Push AI to be a good judge of your work: 00:15:04 Best practices to run evals: 00:19:00 How AI compresses years of learning into shorter feedback loops: 00:23:01 Why prompt engineering is becoming increasingly important: 00:27:01 Adopting a new technology is about risk appetite: 00:44:49 Michael demos Rally, his market research tool: 00:47:10 The AI tools Michael uses to ship new features: 00:54:53 Links to resources mentioned in the episode: Michael Taylor: @hammer_mt Join the waitlist for Rally, Michael’s synthetic market research tool: https://askrally.com/ The book Michael co-authored on prompt engineering: Prompt Engineering for Generative AI The column Michael writes for Every: Also True for Humans Michael’s article on personas of thought: "I Asked 100 AI Agents to Judge an Advertisement” Michael’s article on building a Hacker News simulator: "I Created a Hacker News Simulator to Reverse-engineer Virality”

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Dan Shipper?

  You're thinking about using AI to do simulations of people?

• 0:03Speaker 01

  If it can role play as one character, why not multiple? You can have it be a whole focus group for you.

• 0:08Dan Shipper?

  So basically you typed in the prompt and then it spun up an audience and then it asked the audience that question. So like, which of these three would you click on? So that's cool. But also in the sidebar, you can see like each individual, like simulated person, like what they said.

• 0:23Speaker 01

  You can test things and kind of look at the assumptions that you're making before you actually put it out there in the world.

• 0:30Dan Shipper?

  I think people may not like even really understand the full weight of what you're saying. My workflow now is, you know, when we do an announcement or like I'm writing a post or whatever, I will figure out like what I want to tweet or what the headline should be or whatever. And you have a little MVP that I'm, I've been using and I can just like put it in there and get some wisdom of the crowds and I love it. This is so valuable. Before you'd had to like throw it out into the world and that costs time and money. And if you don't have an audience, like it's really hard. If you don't know the people who are, it's going to be going to. So yeah, I love this thing. Where people's heads might be going is like, okay, cool. You can just like automate everything, you know, you'll always have the right answer to every question and you're going to make amazing content like every single time or whatever. I think what's interesting about this is there are so many different variables to it. I think it's not going to do the thing that people think where it's like, oh, we're just going to turn into like zombies and everyone's

• 1:14Dan Shipper?

  marketing messages are going to be the same because the space of possibility is really big.

• 1:31Dan Shipper?

  Michael, welcome to the show.

• 1:33Speaker 01

  Yeah, good to be here, Dan.

• 1:34Dan Shipper?

  Good to have you. So for people who don't know, you are, you describe yourself as a recovering agency owner. So you had a, you had a marketing agency that you grew to 50 people and then you sold it in 2020 and then you just dove into AI stuff. You wrote the prompt engineering book for O'Reilly and you are also a regular columnist on every. You have an amazing column called Also True for Humans. I'm psyched to have you.

• 2:00Speaker 01

  Yeah, it's good to be here. And I've been watching a bunch of these episodes and just kind of thinking about how other people use AI. So it'd be interesting to see like how I differ.

• 2:09Dan Shipper?

  Amazing. So I'm excited to have you. I think like,

• 2:15Dan Shipper?

  One of the reasons I really loved your work is you have a little bit of a, you have a tinkerer's mindset and you're just always like playing with new things and getting psyched about them and like exploring the limits of what the current technology can do. I think that's why you're a really good prompt engineer and why you think about it so much. I think you're really good at like building little workflows to yourself to automate things. And I'm just, I'm always into or interested in what people like you are currently like playing around with. Cause I think like what you're into right now, people, other people are going to be into in like six months or a year or something like that. And the thing that you're working on that I'm most excited about is you're thinking about using AI to do simulations of people. And I wonder if you, let's start there.

• 3:00Dan Shipper?

  Lay the groundwork for me, explain what that is and why you think that's interesting. And then let's talk about what you're doing with that.

• 3:05Speaker 01

  Yeah, so I studied economics and you know, then went into a career in marketing, but always was interested in this idea of like being able to predict behavior. I think that's what got me into economics, because with microeconomics, you could somewhat predict behavior. And then get into growth marketing that way, because you could run A B tests and you could see how to predict behavior, you know, would they click on this ad or that ad? And I guess this is kind of natural that when I got into AI, I would start messing around that stuff too. And I use role play a lot, right? Like, and that's pretty obvious, like that's the number one prompt engineering tactic that everyone tries first is like,

• 3:46Speaker 01

  you know, as a, you know, a researcher in this field, like give me an answer, right? And it actually really does change the answer. And because the training data is so wide, it can kind of role play as almost any character, right? And the logical leap there is, well, if it can role play as one character, why not multiple? And you can have it. You know, be a whole focus group for you. You can have it be an entire audience. And then you can test things and kind of look at the assumptions that you're making and see in a risk free environment, whether your idea would work or not, before you actually put it out there in the world.

• 4:22Dan Shipper?

  Right. I want to, I want to stop there because I think people may not like even really understand the full weight of what you're saying. Like, there are a lot of studies that show if you tell a AI to like, adopt a certain personality, it will like, make the same, a lot of the same decisions, a very high overlap as like, a real person who has that personality. So, um, one thing that I did early on, this is maybe like a year ago ish, uh, it was like when GPT four was still good, which should tell you like, it feels, that feels like forever ago.

• 4:52Speaker 01

  Yeah, it's like a hundred years ago now.

• 4:55Dan Shipper?

  But one thing that I did was, uh, cause, cause I was sort of, I sort of knew about this research too. And I was like, kind of into it. And I, um, I got GPT four to look at my tweets. And then based on my tweets, write a personality profile for me. And then I gave that personality profile to another GPT four. And I was like, be this person and adopt this personality. And then I had it take a personality test as me. So I took the personality test, then it took the personality test to see like, what the overlap in the scores were. And the overlap is extremely high. And then I was like, I wonder if like, someone who knows me really well in my life would be able to do what GPT four just did,

• 5:37Dan Shipper?

  um, just based on knowing me. And so I had my girlfriend at the time, and my mom, um, pretend to be me and take a personality test and try to think about what I would pick. And GPT four is better than both of them, just from my Twitter account, which is, it's crazy. It's really wild.

• 5:52Speaker 01

  Yeah, yeah. Maybe they don't read enough of your tweets, I guess.

• 5:57Dan Shipper?

  Just, yeah, it's, it's really not that GPT four is good. It's just indictment of my ex-girlfriend.

• 6:03Speaker 01

  No, I, I mean, yeah, it's, I, I've done the same thing. Like, uh, very early on, I was doing a lot of writing and I was like, why don't I just automate my entire job? Um, and, uh, you know, and then actually I did do it and, um, people couldn't tell. Uh, and then people would ask me like, hey, Mike, I really liked that thing you posted on LinkedIn. And I was like, you're gonna have to remind me what it was.

• 6:24Dan Shipper?

  I'm too busy and important and popular now to even know what I tweeted or what I put on LinkedIn.

• 6:28Speaker 01

  Exactly. Must be how like famous people feel when they have someone managing their account. But yeah, like, you know, I always, I always thought like, oh, there's actually surprisingly good at this. And, and then I saw it like, like you said, I saw a bunch of these studies. There was one that jumped out at me. I'll have to find the link, but they basically said that if you do a two hour interview with someone and you take the transcript and then create a personality profile, you get 80% accuracy in big five personality traits, like personality tests, how they, with the decisions they make in economic games, you know, how they vote as well. And the crazy thing is that like 80% is already pretty amazing, but it's actually as accurate

• 7:13Speaker 01

  as if you interview the same person again, a second time. So, so there's always drift. Like when you interview, like if I interview you today and I interview you next month, next month Dan is only going to agree with 80% of what Dan said today. So virtual Dan is like as good as that.

• 7:29Dan Shipper?

  I contain multitudes, Mike.

• 7:36Dan Shipper?

  Yeah. And I think, and where you're going with that is like, you know, if you're a founder and you want to like interview your customers or know what your customers want. And it's, you know, it's in an industry that you're not as familiar with. Like a good place to start is just like talking to chat GPT or talking to Claude, which I've done. It's, it really works. It's like, it's crazy how well it works just from like basic prompting. If you want to like try to, you're thinking about, I want to build a product in X industry. Like, let me go figure out what that, what that kind of person, how that kind of person would think and what their day is like and all that kind of stuff. It's like really, really good, but you're taking it a step further. Tell us about what you've been building behind the scenes.

• 8:15Speaker 01

  Yeah, yeah. So this all kind of stemmed from a post I did for you guys, actually. Personas of thought. My last one in the column. Where I had like a prompt I was using, or I said, okay, let's generate a bunch of people who are relevant personas who could answer this question first and then query and then like kind of fill in the blanks of like what that person would say in answer to the question. So I had that. I was doing that all in one prompt. But then I wanted to automate it. So I wrote a script. And then I think we were talking, you were like, help me solve a debate. I think you liked the, you were trying to describe.

• 8:53Dan Shipper?

  Is every, what, how should we describe every, a meta media company or a multimodal media company? Was the, was the question and Kate, our editor in chief was like, I don't like meta. And I was like, but I like it. So we were like, well, let's test, let's test like the wisdom of the crowds. Like, let's see what would happen. And then you built the script, you put it into the script.

• 9:14Speaker 01

  Yeah, I ran the script and then the thing that kind of convinced me to get working on this a bit more was that you changed your mind. It's like, I, I, it's like very rare that I see a CEO change their mind about anything.

• 9:29Speaker 01

  I was like, okay, this might be something here, you know, and, and then I was, I, I, I was using it for everything. I was like, oh, I'm, I'm actually changing my mind to. And it was an unexpected result. Like I didn't, I didn't expect, I thought people would be really skeptical of the AI responses, but I think because each AI is like answering itself, it gives the reasons. And then when you summarize those reasons, then you can kind of look at them and go, oh, I agree with those reasons and therefore I can change my mind without being, you know,

• 10:00Dan Shipper?

  Totally.

• 10:01Speaker 01

  Too stuck in my ways, you know, I don't know, like, what, how was that? Like, is that kind of a similar thought you had with the time or?

• 10:06Dan Shipper?

  Basically, I mean, like, I think.

• 10:10Dan Shipper?

  You know, my basic way of being as a CEO or writer or whatever is like, as I talk, I'm like paying attention to how people respond. And I'll like get a sense for like, ooh, this little thing I just tried like worked really well, you know? And then that's how I write headlines or that's how I figure out ideas and all that kind of stuff is I'm just like constantly trying stuff. And, um, and so I'm that feedback that was very important to me. And, um, that's why I love, you know, using X or whatever is because like I can just like put something out and like lots of people will interact with it or not. Or like doing sales meetings, the same thing, all that kind of stuff. And so this feels like an extension of that, um, where I can take way more risks, you know?

• 10:54Dan Shipper?

  Um, and I can also like, it's a little more quantitative. It's a little bit more like I can pit one thing versus another. Whereas in real life, like people remember what you just said. So like, you can't just wind back the clock and be like, what if I had said it another way? How would you respond? So it's like hard to do like a really like a fair test, you know? And so I was immediately like, wow, that makes a lot of sense. And also like, I do think like having the thoughts of the model is like really helpful because they were like, I think they were, I think they were like meta. It just sounds like Facebook to me or whatever. And that's what Kate was saying. And I was like, I don't know, Kate, like, I'm, I'm the CEO, like, I don't think of meta when I think of it or whatever. But I, you know, other people were thinking, and I was like, okay, well, that, that makes a lot of sense.

• 11:39Dan Shipper?

  And then, and then I changed my mind and, um, and yeah, like it is, I, I'm, I will literally, I will do anything to get this to be part of our bundle. Because I love this product. I think it's so cool. Like my workflow now is, you know, when we do an announcement or like I'm writing a post or whatever, like, uh, I will, you know, use spiral, which is our content repurposing, uh, platform to, uh, to figure out like what I want to tweet or what the headline should be or whatever. And then I'm, I'm just like pinging you or you have a little demo that I've, a little MVP that I'm, I've been using. And I can just like put it in there and, and get some wisdom of the crowds and I love it. And I, what's interesting to me too is, um,

• 12:21Dan Shipper?

  I think where people are going, like where where people's heads might be going is like, okay, cool. You can just like automate everything and like, you know, you'll always have the right answer to every question and you're going to make like super, super like crazy, amazing content like every single time or whatever. And I think what's interesting about this is there are so many different variables to it. So like, for example, one variable is what is the audience? And you can spin up different audiences with different like demographics and different viewpoints and personalities or whatever. So we have an every audience and then we have like a hacker news audience that we can test and all that kind of stuff. And then also the results that you get are extremely dependent on the question that you ask.

• 12:58Dan Shipper?

  And so like, if I'm pitting one thing versus another, it's not like going and finding like, what is the, what is the most, what is the best possible way to phrase this? It's just like pitting one thing versus another. And like, maybe there would be a way to like automatically like,

• 13:13Dan Shipper?

  try tens of thousands of messages or whatever. But even then, like, you start to realize that the, the kind of like space of possibilities is so huge. It's so huge that anything like you can only ever test a small portion of it. And different people are going to start in different places in the landscape. And so, yes, these tools are like super powerful and like it, it makes my process way more efficient to like have it available. But I think it's not going to do the thing that people think where it's like, oh, we're just going to turn into like zombies and like, and everyone's marketing messages are going to be the same because like the space of possibilities are really big.

• 13:50Speaker 01

  Yeah, exactly. Like there's, there's two things combined, right? Like the space that you're testing it in and like the personas you're using. And then like, cause like a hacker news audience is going to be very different from your, your Twitter audience or whatever. But, but yeah, like that, that was like something that we did a lot of in the agency. Like we were testing tons of creative ideas with Facebook ads and you could just never have like enough budget, even the biggest brands like don't have enough budget to test everything. Right? So, so yeah, like this kind of expands the space in which you can play in, but it's still your duty to play. You know, like you still have to come up with good ideas and, um, you know, AI can help you come up with good ideas too. Like I, I tend to use it in the brainstorming phase and then I use it in the testing phase, but I'm in the middle, you know?

• 14:36Speaker 01

  I see quite often it's, it's, it's better at judging than it is at like finalizing the copy. And this is something I saw a lot of like as a prompt engineer, like working over the past few years is just that like LLM is actually pretty good at judging the results of tasks, even if they can't do the task very well themselves, which kind of led me down this path in some respects as well.

• 15:00Dan Shipper?

  That's really interesting.

• 15:04Dan Shipper?

  I want to push you there because I've had experiences with like Claude, for example, where I'm like push, where I'm like putting in like an essay or whatever and I'm like grade this essay and it just like always gives you a B plus the first time or an A minus the first time. And then like if you make some revisions, it always moves up to an A or whatever. So like, where are the tasks where it is good at judging and where are the tasks where it's kind of going to give you like that kind of bull like, oh, it's an A minus or whatever. And here's, you know, and the next turn it's an A, you know?

• 15:30Speaker 01

  Yeah, it's really bad at grading. So you can't, you can't grade things on a like scale very easily. Like you can't give it like stars out of five. It tends to always overestimate the middle. So like it's almost like being too nice to you. Everything's like a four out of five.

• 15:49Dan Shipper?

  Yeah, I'm like mean.

• 15:52Speaker 01

  I know. Yeah. And there are tricks for getting it to be more mean, but, but, but actually what I use it for a lot in terms of grading is I'll, I'll have like something that is a one or a zero. So I'll have some criteria of like, for me, a good article is one that is like as concise as possible. And so I'll have like a one and zero on the concise scale. Is this concise, right? I'll have another one, which is like, does this have a compelling hook? And I'll kind of build up maybe like 20 of these for any task that I'm automating. And then I'll run my testing suite and each one is like an LLM judge that just checks that zero, one. And when I combine that score together, it just

• 16:37Speaker 01

  kind of gives me, you know, an aggregate score that, that is much more reliable. When you run it again, it's not as, you know, it's not changing as much.

• 16:46Dan Shipper?

  That's interesting. So what about like, so first of all, I like the idea of breaking it up into subtasks, but then like within a specific subtask, like the hook. Is it still only going to be able to do like zero or one like good hook or bad hook? Or is it, could you do like zero to five and would it be able to do that?

• 17:02Speaker 01

  Yeah, it's not as good when you go zero to five. So what I would try and do is break that up into like, break that classification down into subclassifications, right? So like, I would, I would think about, okay, what is a good hook? You know, a good hook is like, it grabs your attention and a good hook, maybe name drops or references something famous or credible. A good hook does this. And the way I get these, by the way, is I use an LLM judge for this too. I think the other thing it's really good at is comparison. So you can give it two different articles that you've written. You say, what is different between these two articles? And then you look at the differences and then some of those will then become judge criteria.

• 17:45Dan Shipper?

  That's a whole other tool that would be super useful is like just recursively make it like, recursively creating a template classifier thing that like, takes a bunch of examples of good and bad and then uses that to create like the most detailed possible rubric for another LLM to use, you know?

• 18:01Speaker 01

  Yeah, yeah, I did think about doing that. I worried it is like way too far down the rabbit hole.

• 18:07Dan Shipper?

  It might be.

• 18:08Speaker 01

  How much, I don't know how much people actually care about the quality of their writing for most places.

• 18:13Dan Shipper?

  Yeah, maybe not for writing, but like in general, I think like there's, I feel like there's got to be use cases for that, but it may, it may not be like a today thing. Like it might be like a, in a year, people will be more sensitive to it kind of thing, you know?

• 18:25Speaker 01

  Yeah, I was doing it a lot for prompt engineering, like with my clients. And then, and then once you have like a list of these criteria, then you can build a metric and then, and then you can use like a prompt optimization library. And that goes pretty deep into the weeds like DSPY or something like that. Because then you can improve the accuracy of the classifier. And then once the classifier is much better, then you can improve the accuracy of the generation task as well. So it depends on how deep you want to go on this, but this is like something I spend a lot of time thinking about.

• 18:56Dan Shipper?

  I want to go deep. This, I mean, the thing I'm thinking about, like, we're talking about making evals, basically. And I'm just thinking about Quora, the, our email tool and like, trying to improve those summaries and then trying to figure out like, what is a good summary? And like, and we're basically doing this, where like, every week I like go in and look at the summaries that Quora generates for me with Kieran, who's the GM of Quora. And I like, Literally like rewrite my inbox for him.

• 19:25Speaker 01

  Yeah.

• 19:26Dan Shipper?

  Personally.

• 19:27Speaker 01

  That's good. That's usually what I try to get the clients to do.

• 19:30Dan Shipper?

  Yeah. And just sort of hope that over time, like we'll have enough examples and enough rewrites and enough pulling out all those principles that it starts to work. And it's really fun. It's really interesting and it's also really hard. And I love, I love all of the like,

• 19:47Dan Shipper?

  mapping all those little things that I know that are not explicit, that I just sort of know, you know, because I'm like, this summary is wrong or like, here's how I would summarize this, but like, and I'm, I can do it, but I can't explain it until I look at it. And then I'm like, well, this is the rule I'm following, you know?

• 20:02Dan Shipper?

  And it's, uh, it's really fun.

• 20:04Speaker 01

  And sometimes you don't know the rules you're following as well.

• 20:06Dan Shipper?

  You mostly don't know the rules you're following. That's part of the fun.

• 20:09Speaker 01

  Yeah. Yeah. Once you see it, then you're like, oh, I get it. And so I was trying to make like a Tinder like interface when I'm doing this, uh, where I'm like one to zero, one to zero.

• 20:17Dan Shipper?

  Like the eye doctor.

• 20:18Speaker 01

  Yeah, exactly. Yeah.

• 20:20Dan Shipper?

  Better or worse, you know?

• 20:21Speaker 01

  Yeah. But, but so the, I'll walk you through how to make a quick evals for, for this, right? So, so if you, uh, if you take that big list of all the emails where you have the one that it generated and then the one that you rewrote, then what you want to do is, uh, run a big analysis. Um, uh, you like, you can do this in Egyptian notebook or get the engineers to do it. Or do it manually. It'll take a long time. Um, but you just get, uh, you compare, just ask, uh, say Claude or, or GPT-4, um, what, what is the difference between, uh, the rewritten and the original?

• 20:57Speaker 01

  Um, and then, and then you, uh, take all of those differences and then summarize and say, uh, what are the main differences between a good, a good email and a bad email? And then, and you ask it for bullet points and then you have each bullet point then becomes your evaluation criteria. Yeah, and then, and then, yeah, and then you can, then you can look at the accuracy of the evaluation criteria, because you have, now you have, like, for every one of your rewritten emails, you have, whether it scored one or zero on that evaluation criteria, whether it was present or not. So then you can do like a, you know, look at the false positives, false negatives,

• 21:35Speaker 01

  like, this example, I said was to, you know, didn't have a good hook. But, but like, my LLM judge said it had a good hook, so it was wrong. Right? So that's like a false positive. So, so that would mark the score of the classifier down. So, yeah, that's what we do.

• 21:54Dan Shipper?

  And you would do this in a Jupiter notebook? Or like, what's the format that you keep all the emails and examples in?

• 21:59Speaker 01

  Yeah, I would love to use a tool to do this, and I have tried a lot of them. And I find the abstractions change so often with AI that I keep going back to just, you know, no framework, no tool, just Jupiter notebook. And then, like, I mean, I'm doing it in cursor, and I'm just letting Claude, like, Yolo, like, create the, create the interface, create the, you know, create, like, save everything locally in CSVs and stuff. And it's probably not the best way to do it. But, yeah, I'm like, you know, I'm like the, you know, the sort of guy that just doesn't really care that much about the elegance of my code. And just want to kind of get, get the job done.

• 22:39Dan Shipper?

  I've always been like that. And now, like, everyone is catching up to us, because, like, now everyone's just like, well, I let the LLM do it, and I press accept all. And I've always, I've always basically, like, Yolo'd all my code. Just, I used to have to handwrite it, and now I don't.

• 22:50Speaker 01

  Yeah, I was Yoloing code back when it was Stack Overflow, you know?

• 22:56Speaker 01

  And just copy someone else's code in there and just see if it runs.

• 23:00Dan Shipper?

  Um, you know what this also reminds me of is, like, I frequently, all the time, like, we run a media company, so like, all the time, I'm giving feedback to different people that's like, the same on their writing, or like, the videos they make, or the tweets, or whatever. And like, I just want someone to, like, make a rule thing, and then, like, make a bot that just, like, grades this stuff and gives all that feedback, because it's, like, so repetitive. And what's been really interesting is,

• 23:28Dan Shipper?

  We have this writer, Alex, who started writing with us over the last month or two. And he writes Context Window, which is our Sunday email. And it like, he writes all of the like, here's what happened in AI. Here's all the new releases. Here's some analysis or whatever. And then also sometimes like when new models come out, like I always write a big piece. So like, you know, OpenAI launched Deep Research this week. And so like, I wrote that and we collaborated on that. And the first time we collaborated, he's like, he's, you know, he's a junior writer, right? And so he's got a ton of great ideas, but like the first draft he did of that, like,

• 24:06Dan Shipper?

  of the first piece we did together, which was, it was some other, it was some other new model release. I can't remember, but it was the first draft he did, like, I was like, oh, this is, this is not good. Like, no shade to Alex, but like.

• 24:19Speaker 01

  It doesn't feel like you, right?

• 24:20Dan Shipper?

  Yeah. And not, not just not like me, like, because, you know, his bylines on it and stuff, but it had a lot of, it had a lot, it had a lot of the information that was supposed to be in there, but it was also like, it had a lot of information that shouldn't be in there. And then like the actual sentences themselves, it just weren't that good. And there's a lot of, there's a lot of reasons and a lot of mistakes and a lot of whatever. And so what I did was like, after he, he gave me the draft, like, we didn't have a lot of time. So I just like rewrote it and like, honestly, like that kind of thing where it's like a new product release. Like, it doesn't take me that long to write, like, it took me like 45 minutes to write the whole thing. Once all the information is there. And then what I had him do is take the, his draft and then take my draft and throw it into O1 Pro and like, be like, what's the difference?

• 25:06Dan Shipper?

  And like, the next time, like when we wrote this deep research piece, I had him write the first draft and it was like a thousand times better the first time. And I was like, what the [bleep]? Because, because like before all of these tools, to get from where he was to what I saw him deliver would have taken like a thousand articles. He would have to write a thousand articles or like a year or two of like really grinding. And it was just like the next, the next time it was great. And I was just like, this is crazy. It's absolutely crazy. And it's because like he's sort of, every time I edit him, he like throws in the difference and then like makes a little rulebook and like is constantly, as he's writing, going back with the O1

• 25:48Dan Shipper?

  or, or, you know, any of these models and having it remind him of some of these things and write little portions that he struggles with and all that kind of stuff. And it's so much better. It's crazy.

• 25:58Speaker 01

  Yeah, and I think that's the thing that people are missing is that, yes, AI is like a threat to people's jobs. And, uh, you know, nobody really knows what's going to happen as we get like, you know, PhD level AIs doing everything. But, um, but then at the same time, uh, they're, they're also helping us learn a lot faster. So if you're actually using these tools and you have, um, you kind of these, like you're keeping rule books, right? Of what is a good article, like, uh, according to the every style guide, right? Then, uh, then you have that, that opinion, uh, that strong opinion, um, based on, you know, kind of the collective wisdom of the publication. Right. And, and then that, that is a prompt, right? Like, it's not, I feel like prompts are kind of converging where you'll be using the same prompt for people and for AI.

• 26:43Speaker 01

  But, um, uh, but yeah, it's, uh, it is essentially like what I did at the agency as well. Like we would write these SOPs, we called them like standard operating procedures. And we had like a Google Drive full of hundreds of them. And it was like, here's how I do this, here's how I do that. Yeah.

• 26:57Dan Shipper?

  That's the thing that like, you know, I don't know, a year ago, like the really popular thing to say was like, oh, like prompt engineering is going to be dead or whatever. And that the opposite has turned out to be the case, like it has become more and more important. And I think what, I think the mistake that people made when they said that is, um, they, what they were saying is the gap between what you intend and what the model does is going to get smaller and smaller. Like, so the gap between your objective and what it does is smaller and smaller. But like the thing that people miss is that, um, everybody has different objectives. Like objectives are these big, high dimensional things that are really hard to express.

• 27:37Dan Shipper?

  And, um, and people who are better at detailing that, that their particular objective are prompting. And that's like what it is, you know? And so, uh, like detailed prompts are not going away because people have very detailed, like things they need to get out for the, to tell the model what to do. And I just think that that's, that's like a really interesting.

• 28:04Dan Shipper?

  I love, I love the ways that we tend to misjudge new technologies and, and this is, I think, a really great example.

• 28:11Speaker 01

  Yeah, I think it's, it stems from the fact that a lot of the people who are early in AI come from machine learning or engineering backgrounds or data science. And, and while obviously that's a huge benefit and, you know, that they did really well, even very early on in AI to like get some of these systems working. The, they only really worked on things that were well specified.

• 28:36Dan Shipper?

  Yeah.

• 28:38Speaker 01

  Being engineers, like the, the tool is already briefed, right? Like, like the product manager took the messy world outside and like crammed it into a PRD for you, right? So, so like you don't, you don't realize how messy the world is, I think, when you're an engineer. And, and I say that as someone who, you know, was a business person and now is a full-time engineer. And I can see both sides of it, but, and, and, and so they are completely right. That like for well-specified tasks, there's, you don't need to do prompt engineering anymore. You can use DSPY, but, but like so much of the world is completely unspecified. And like people didn't even know what the specification might be, like we talked about earlier. Like, you know, you don't know what is a good article until you see it.

• 29:21Speaker 01

  And then you're like, oh, that's something that I feel allergic to, you know?

• 29:25Dan Shipper?

  Totally. And I think that's, that's the really interesting thing is so much of the world is, is, is not specified. And it's really interesting to see these companies like OpenAI going down the like RL route with getting better at tasks where you can specify the end result in every step, verify, verify the end result in every step, which is, which will do some interesting things, but like

• 29:48Dan Shipper?

  so much of the world is outside of the realm of what can be specified. And I think, I think people just like sort of miss that or assume that like eventually it will all be specified. And, and I think that it won't. Specifications tend to be too low dimensional to express like all the stuff that's going on. There's some, some things always have to be sort of inexplicit is, is my, my sort of philosophical view. And I think the beauty of language models is. It is the first time that we have had tools that can deal with things that are inexplicit. We've never had that before. It's always any, any kind of computer you have to, you have to be able to specify it has to be exact, has to be like mathematical and logical,

• 30:32Dan Shipper?

  basically. And L L M's are like, here's an example, just like do just, you know, classify it based on these examples or like follow this, follow the examples or whatever, which is a good way to talk about something or point to something that that is inexplicit. Like, why is that example like a good example? You can probably boil it down to rules, but there's something there that is still kind of unexpressed, or, or there's a lot more richness in an example than the, the rule that you use to describe the example. And I just, I love that. I love all that stuff. It makes me so happy.

• 31:05Speaker 01

  Yeah, like I, I mean, I do wonder like how far it will go, like how, if we're gonna keep seeing this exponential improvement, or if it's gonna be an S curve. Like even if it, even if it does tail off, I think, you know, we'll have enough to keep us busy for 50 years. You know, even with the models just stopped improving today. So I'm not worried about that. But, but yeah, like I always think about this, this, this thing, people talk about billiards or snooker or pool. You can, you can calculate where a ball is going to hit for like the first bounce, but like by the third or fourth bounce, you'd need like more time than has ever passed

• 31:46Speaker 01

  in the entire universe to calculate where it's gonna hit. And I think it's, there's something in that, you know?

• 31:51Dan Shipper?

  You know, I first read that in, in one of the Nassim Taleb books. I can't remember if it's Fool by Randomness, or The Black Swan, or whatever. And I love that because the reason why is because you have to start to take into account the pull of gravity of every like object in the universe in order to like keep, because everything pulls on everything else. And so like, After one hit, it's not that much pull, but after the fifth, it's like a lot, you know?

• 32:15Speaker 01

  Yeah.

• 32:16Dan Shipper?

  Same thing for personality prediction. It's like, if you can get 80% of what I would do, you might be able to predict the next thing I do. But like, once we get five moves out, things are wildly diverging. I want to take one minute away from this episode to introduce you to our sponsor, LTX Studio. I think storytelling is one of the most essential skills of the AI age. You can bring your stories to life with just a few words, complete with a cast, a storyline, settings, all according to your style and your specifications. LTX Studio is helping storytellers visualize their stories in entirely new ways. Two of the most magical parts of LTX are in their character generation and storyboarding. Here's how it works. If you type in a description and maybe add a headshot, LTX generates unique, dynamic characters, each with their own distinct look and personality.

• 33:00Dan Shipper?

  Remember the old days of struggling with storyboards? LTX makes it simple. If you need to map out a bustling debate in ancient Greece, or a castle under the sea, LTX lays out and expands on your vision shot by shot. Better yet, it suggests new angles and shots you might not have considered. First-person perspectives, wide angles, close-ups, you've got it all. I can switch between six different rendering styles for the characters and settings. Whether you want ultra-realistic characters or cartoon-style art, it's all just one click away. The AI revolution is just starting. But if one thing's clear, it's that it's not replacing human creativity, it's expanding it. So if you've ever had a story in your head with no way to bring it to life, start with LTX Studio. It might just be the creative partner you've always needed. Check out the link in the episode description for more details.

• 33:45Dan Shipper?

  And now, back to the episode.

• 33:46Speaker 01

  Yeah, but essentially, you're not going to fit the gravitational pull of every object in the universe in the prompt.

• 33:53Dan Shipper?

  Yeah, exactly. Probably not.

• 33:55Speaker 01

  Yeah, you can't, it doesn't matter how good the models get. You know, like, it doesn't matter how good we get at pre-training or whatever, you know? Like, it's never, it's never going to be able to completely specify. Like, it's always going to be a simplification of reality, right? It's always a model.

• 34:12Dan Shipper?

  And then I think, to your question about like where things go, and like, obviously, like, I don't really know either, but my feeling about this stuff is...

• 34:22Dan Shipper?

  We tend to collapse our view of ourselves and the world to reflect the tools that we have at our disposal. So as an example, when we only had like watches and telescopes and calculus, like, we tended to think of the human mind and the universe as sort of like a mechanical clock, you know?

• 34:49Speaker 01

  Yeah.

• 34:49Dan Shipper?

  And sort of operating according to Newtonian mechanics, which it sort of does if you look at it that way, but also if you look at it a different way, it does not work like that at all. But we were completely convinced that like for hundreds of years, basically, that that's how everything worked and we just needed to like, we just needed to do a better job of specifying or finding the like underlying rules or principles by which the clock of our minds or the clock of the universe worked. And then Einstein came along and he was like, well, it's not really like that at all from this other perspective or maybe this larger perspective or whatever. And I think something similar happens with language models where like, you look at a thing that can do a lot of tasks that you're

• 35:33Dan Shipper?

  used to being able to do and you're immediately like, it's going to do everything I do. Because like, you just sort of shrink your sense of self and sense of the world and sense of what you can do to like that thing. At least initially, it tends to like hide all the complexity of who you are and what you can do and like all that kind of stuff. But I think over time, what we will discover that there's like a lot of things about us that it's not able to do, even if it's operating at PhD or post PhD levels. We just can't like see what that is right now, but like, I think that it's there. And that's another example of like ways that new technologies cause us to like see things differently and mistakes

• 36:17Dan Shipper?

  we tend to make. Yeah, I really think, I said this before on the show, but like, I think there's this like, there's this false peak thing with AI where every time you get to a new level, you're like, I can see the peak right there. And once I get there, that's going to be it. It's going to take over everything. And then you get to that peak and there's, there's another horizon that opens up. And I think that that's true too. Humans are way more flexible and powerful than I think we give ourselves credit for in a lot of ways. And not to say that this technology is not amazing and powerful, and I totally think that too. But there's, you know, it's, I think, holding both is important.

• 36:54Speaker 01

  Yeah. Do you know the Bull**** Jobs book? Like the hypothesis that some like forty percent of all the jobs in the economy are basically like unnecessary?

• 37:06Dan Shipper?

  No.

• 37:06Speaker 01

  Like admin jobs. So he uses the example of like, you know, a parcel being delivered to a military base, right? And fifty years ago, the guy who was delivering the parcel, like a package from Amazon, whatever it is, you know, would would just like take it straight up to the soldier and then the soldier would sign for it, right? And like today, the guy has to like go and like put it in a holding space. And then there's a admin, like a manager, office manager of the holding space who signs for it, fills in a form. And then they take it, there's someone else who goes and then they sign it off. And then they have like HR kind of training and basically when you kind of add all these extra people in, the cost is inflated by forty, fifty percent.

• 37:50Speaker 01

  But like the same task is still being done. There has been no improvement in the efficiency of parcel delivery.

• 37:58Speaker 01

  And he, I don't know if I fully agree with his thing, but but like the interesting germ of a thought in my head when I, when I read that book was, oh, like, we, we already have like universal basic income. It's just that, it's just that like, you know, they're like admin jobs. And maybe that's controversial, but like, I do, I do think, you see what like, say, Elon is doing and some of his companies are in the government now and like, like ripping out a lot of the, these jobs that like actually don't seem to make a measurable difference. I mean, we'll, we'll see, right? Like what happens if he still keeps getting away with it. But, but like, yeah, I mean, it's, it's an interesting hypothesis.

• 38:42Speaker 01

  So like, if you believe that, then essentially all of the gains from the internet and all the productivity gains essentially went to creating this extra forty percent float. And then AI might just make that eighty percent or ninety percent.

• 38:56Dan Shipper?

  That's interesting. I don't think I agree with that fully. I think the interesting devil's advocate, not that you're, I think you're definitely not arguing with you. You agree with everything about, about that, that book, but I think the interesting devil's advocate is like, why? Like, one way to think about it is like, um, where we make all those sorts of rules because like, we just wanna waste things, basically, but, but usually there's like, there is at least some organizational reason why. It's like, there's some like risk management thing where it's like, well, if the parcel gets delivered directly to the soldier, then like XYZ bad thing happens. And so we need to like, create more process or whatever. And, and usually like, I think the, the interesting things about,

• 39:38Dan Shipper?

  the interesting thing about human organization is that we've needed process or rules in order to facilitate collaboration between large groups of people. That like system of rules just creates a lot of bureaucracy and middle management and, and like things that like everyone sort of complains about and doesn't really like, even the people who are doing it, they don't really like it. But like, we need that in order to coordinate masses of people. And I think like, one possibility for AI is that it, it removes the need for so much, so many like layers and rules and processes because like the AI can do a lot of the like coordination stuff that usually would require lots of middle managers or, and that kind of stuff. I think another way that I like,

• 40:22Dan Shipper?

  sort of agree with, with this, um, one of the conclusions of, of this sort of like hypothesis is like, when I go to, you know, get my, get my shirt tailored in Brooklyn, like they don't take credit card, you know? It takes a long time for people to adapt new technologies. Even, even we are not getting in as much out of this stuff as we possibly could. Like, there's so much stuff that I'm sure is like latent in, in the tool. And so like, um, pure productivity, pure efficiency, like, um, it, it takes a really long time to filter through society and it's, it's not like, one of the things that I always say when I talk to people at, at big companies is like,

• 41:08Dan Shipper?

  I think AI is like a really good test of what would happen if we had magic powers, like we would just basically do nothing like maybe, maybe we would create like a working group where people would like, some people at the company would like spend some of their time like figuring out what to do with our new magic casting, magic spell casting abilities. But like, we probably then wouldn't even, it would take them a year and they wouldn't actually figure anything out. And so, yeah, I think, I think we tend to feel or believe that just because we have a new capability, that it just sort of like gets integrated like super fast in society. And that's definitely not true.

• 41:44Speaker 01

  Yeah, I think where there is some kind of insight here is that a lot of the way that we organize people is based on kind of like organizational scar tissue, I guess, is how I think about it. Like, you know, one, one person, one time did something really stupid. And therefore, now we have a form, right?

• 42:06Dan Shipper?

  My dad is always like, one guy tried to blow himself up with his shoes in like 2006. And like, now we have to take our shoes off like every time. And it's like, yeah, that is, that is true. Like the shoe bomber, he ruined it for everybody. And he didn't even succeed.

• 42:21Speaker 01

  And we're like afraid to tear down the fence in case something comes through it, you know? So like, I think it does serve a purpose, but I think it's like an emotional purpose. And, and like a sociological purpose, maybe like, we, it makes us feel good that we have these protections in place, even if they're not actually doing something measurable, right?

• 42:40Dan Shipper?

  Yeah.

• 42:41Speaker 01

  So, so I actually think that AI will increase them. Because the only reason we have, like to take your example, the only reason we have the TSA scanning your bags at an airport, but not at a train station, is that like, it's too inconvenient to do it at a train station. Like, you couldn't, you couldn't do this in the subway in New York, right? The whole thing would shut down. Maybe they should. Some issues recently. But, but, but like, as it becomes more convenient to make you fill in forms and to do administrative tasks, I feel like eventually, that's just going to balloon. And then you'll have like, my AI talking to the government AI, and like, checking that everything's okay, you know, across like, 1000 different forms.

• 43:25Dan Shipper?

  As long as I don't have to do it, I'm fine.

• 43:28Speaker 01

  Yeah, that's true. Still win, I guess, if, if it's automated.

• 43:31Dan Shipper?

  I think one of the things you're saying that I agree with is like, Actually, the limits of the technology right now are not even the limits of what's technologically possible. Like, there's, you know, you can't, like, GPT-4, GPT-3.5 could have taught you how to make a nuclear bomb, but it doesn't, because, like, OpenAI made it, so it wouldn't do that, you know? And so, um, there's all sorts of ways in which the technology is limited by our, like, our tolerance for risk, and everyone's tolerance for risk is different. So every organization has a different approach to, like, what risks they're willing to take, or whatever. And, you know, to some degree, that's what makes baseball. Like, I think that's actually good.

• 44:12Speaker 01

  Yeah, they can find the right answer, like, collectively, by saying, oh, that feels wrong. Like, when they've released it completely openly, like, they get backlash, or, you know, or like, if they're too conservative, then they fall behind, and it becomes like a real race.

• 44:26Dan Shipper?

  Exactly. And it also provides opportunities for startups, because, like, what OpenAI can do is just different from what, like, you and I can do. Every government in the world is, like, watching them like a hawk, you know? And, um, and we're just, no one's really watching us, because it's small enough that, like, no one cares. And so, we can take more risks and, and find out more good things to do with it than, than, you know, a bigger company can. So, you know, another, another way in which I think people were wrong at the outset of, of AI stuff is to be like, oh, incumbents are just gonna win. And it's like, incumbents always [bleep] it up. And, and, and it's about risk-taking. And it's not their fault. It's not like they're stupid. It's just, it's just massively difficult to take real risks when you have big, like, tons and tons of customers and a big organization,

• 45:11Dan Shipper?

  and, like, all this pressure. Like, it's just, it's just not a good environment for that.

• 45:16Speaker 01

  Yeah, it's, it's really rare that they manage to do something good. And, and it's not through lack of trying, right? You know, they've, they've all read, you know, the disruption stuff, right? Like, they'll see it happening, and you can see them trying their best to do it. But, you know, I would say, actually, they're doing, I think the scorecard is pretty good. Like, I know people talk about Google and falling behind and all this stuff. But, but I would say, uh, big companies are innovating faster today than they ever have.

• 45:47Dan Shipper?

  Yeah.

• 45:48Speaker 01

  And, like, there was one example that really jumped out in my mind, where I went to I I I went to like a like a conference. I think it was like it was it was two months after ChatGPT came out, right? It was like a like a B to B like enterprise, like it was all like Fortune 500 companies and like they had like NASA there and talking about AI and stuff like this. And and and they they said, who's working on like generative AI projects? And like 80% of the hands shot up. And I was like, this is like two months after ChatGPT. It's pretty amazing. And just contrast that with like, I was working in growth marketing, like growth hacking with my agency. That was a growth hacking agency. We started in 2014. And it was it took it took like four years to for

• 46:33Speaker 01

  before our first Fortune 500 client like Googled the term growth hacking. Like, we were number one for growth hacking on on Google. And and yeah, it was like four years before we got our first enterprise client.

• 46:46Dan Shipper?

  Yeah, yeah. Or like mobile, you know, it took years or even the Internet, you know, years and years and years and years. Yeah, that's interesting. I do think that they're getting better at it because they've been they've been learning their lesson in a lot of ways. Yeah. So I want to talk about Cursor. But before we do that, I want to bring this whole conversation back around to Rally because we haven't shown people Rally. And so if you feel like showing a demo, like I think it'd be really fun. I just think it's the coolest piece of software.

• 47:10Speaker 01

  Yeah, so we talked about the script earlier about how I was A B testing your your headlines. You know, the multimodal media versus meta media. And what I've done in the past like couple of weeks is put like a basic user interface on it. And I have that live now, actually, with a waitlist. But it's it's at askrally.com. And this is like the main thing I'm working on now, just to see how far we can push it. Because I'm getting like surprisingly good responses when I when I run this. And I've got like maybe 20 of my friends checking it out now. So yeah, we could we could we could try having a look on the on the call as well.

• 47:52Dan Shipper?

  That's great. Yeah, let's do it. So basically, for people who are listening instead of instead of watching, so like, the Rally screen is at the top, you see something that says like your audiences, and you can pick an audience. So like millennials, manufacturing CEOs, there's an audience in there, general population, there's an audience in there that's like Hacker News audience, Doglovers in Dallas.

• 48:12Speaker 01

  That's my that's my test audience I always use because it's the first thing I think of.

• 48:15Dan Shipper?

  Perfect.

• 48:18Dan Shipper?

  Let's do Hacker News. So like if we select Hacker News. And then it's like, what would you like to ask the audience? There's a text box. So it's sort of like, who wants to be a millionaire, you know, but like, but like, you know, for real, anyone can do it. You don't have to know Regis Philbin. So do you have a question? Or should we use? I mean, I can come up with one. Should we use the meta media one?

• 48:38Speaker 01

  Yeah. Well, why don't we do like, maybe do like a headline test? Do you have like a couple of headlines for the latest posts?

• 48:46Dan Shipper?

  Yeah, totally. We just did a new launch where we said like every now includes Quora, which is our email, email app. And basically as part of the every bundle, like you pay one price, you get access to Quora and all the other stuff we make. So the current headline is the every bundle now includes Quora, which I don't really like that much. The every bundle now includes Quora.

• 49:07Dan Shipper?

  One thing I've been thinking about is like whether or not we should call it a bundle or just like in the every subscription or every, people don't say like the prime bundle. They just say prime.

• 49:16Speaker 01

  Yeah. Membership, maybe like, yeah.

• 49:18Dan Shipper?

  What?

• 49:19Speaker 01

  Membership, like every membership.

• 49:21Dan Shipper?

  Yeah, membership is an interesting one. So like, maybe a B would be like, every now includes Quora, manage your emails with AI.

• 49:37Speaker 01

  And then you like the capitalizations on each one?

• 49:42Dan Shipper?

  Yeah, we usually do all caps, basically, but like, I don't think the caps like make that much. That's a really, that'd be a good AB test is do the caps make a difference on what people click on?

• 49:51Speaker 01

  Yeah.

• 49:52Dan Shipper?

  We can do that.

• 49:52Speaker 01

  Yeah, I used to test stuff like that in the agency.

• 49:57Dan Shipper?

  And then the next, the last one would be like, every now includes Quora, the most human way to manage your inbox.

• 50:05Speaker 01

  So every now includes Quora, most.

• 50:13Speaker 01

  Even way to manage inbox. Yeah, my, my wife always complains about the loud sound of my typing because I'm like a heavy typer. You're probably hearing that now.

• 50:27Dan Shipper?

  There's that George Clooney movie up in the air where she says like, I type with purpose. I always think about that.

• 50:34Speaker 01

  There you go. I need to use that line. Cool. So, so, you know, we have the three different examples. And this is just like sending a, a prompt to chat GPT. So we were going to kind of tell, tell it like what we want out of this. And in this case, you're going to say, you know, which of these headlines would you click on?

• 50:57Dan Shipper?

  Probably like, I want to know if they would open it like, because it's going to go to their inbox.

• 51:01Speaker 01

  Yeah. Okay, these are email subject lines for the publication. Every, which of these headlines would you open?

• 51:17Speaker 01

  Okay, so we just keep it simple there. And then I'm going to put it in voting mode, which just basically just gives it like a tally at the top as well.

• 51:25Dan Shipper?

  So cool. I don't think I've seen voting mode. That must be.

• 51:27Speaker 01

  Yeah, I just added it. Yeah.

• 51:30Speaker 01

  Yeah, thank you, Claude.

• 51:32Dan Shipper?

  That's the other thing I love about this like era of AI tools is like, everyone's just shipping so fast. You're just like, oh yeah, I added that like in an hour, like two, two, two days ago. I had the idea and I just did it, you know, it's like, I love that.

• 51:43Speaker 01

  Yeah, yeah, no, it's definitely really appealing. It's the problem is like when you have to refactor everything.

• 51:49Dan Shipper?

  Yeah. And then the UI gets messy and the whole thing is like, yeah. So, okay, so basically you, you typed in the prompt and then basically it spun up an audience. And then it asked that, asked the audience that question. So like, which of these three would you click on? And then what you can see basically return the results and the results are every now includes core. The most human way to manage your inbox is by far the most clicked on or opened one, which is really interesting. And the most human way to manage your inbox is the one that we've been going with. So that's cool. But also in the sidebar, you can see like each individual, like simulated person, like what they, what they said.

• 52:29Dan Shipper?

  So like the phrase, the most human way to manage your inbox stands out to me because it's just a more personal approach, which resonates with my values about technology and its impact on our, on our lives. So like, that's what one of these AI. People said about why they clicked on it. And then you also have this like summary thing where it's like it summarizes what everyone says. So the feedback indicates a strong preference for the subject. Every now includes the core of the most human way manager inbox. The choice resonates due to this emphasis on balancing technology with a human touch, which aligns with the values of the individuals. So like, I think that's really interesting and I think we would get different results. For example, like if we, if we change the audience or another thing that I know you're building,

• 53:11Dan Shipper?

  which I really want is like, I don't want it to be testing them against each other. I want them tested in the context of someone's inbox and the kind of emails they usually have to see like which, which one of them has clicked more or opened more.

• 53:26Dan Shipper?

  And, and I'm sure I know that that's coming, but like, yeah, this like, this is so valuable. It's like, before you'd had to like, throw it out into the world and that costs time and money. And if you don't have an audience, like it's really hard. If you don't know the people who are, it's going to be going to you, like whatever. So yeah, I love this thing.

• 53:45Speaker 01

  Yeah, exactly. And then once you, once you have like a sense of whether you're asking the right questions or, you know, if you, if you really disagree with a result or you really agree with it, you can then go and validate with real tests, right? So, you know, it just kind of gives you more options and more, you know, room to navigate these different creative decisions.

• 54:05Dan Shipper?

  Right. Obviously like disclaimer, this could be wrong. It's just, it's another piece of information, you know?

• 54:11Speaker 01

  Exactly. Yeah, it's another opinion. And, um, you know, this is like still pretty early as well. So like, we're working on getting all the evals set up and, you know, doing the rigorous testing against like real human responses. So, you know, that's, that's coming too. But, um, but yeah, that's like a long road. And, um, yeah, there's a, there's a lot of, there's a lot of really cool stuff you can do. Like, um, like the thing I'm working on right now, I showed you earlier today. It's like a hacker news simulator as well. So you've got like all the hacker news links from today. And then you just like, whoop, like insert your headline in there and just see if they, if they click on that one.

• 54:48Dan Shipper?

  I love it.

• 54:49Speaker 01

  So yeah, we'll see how that goes.

• 54:50Dan Shipper?

  Yeah, that's the best. So. You're using Cursor for all this stuff. Like, tell me about that.

• 54:57Speaker 01

  Yeah, so I've tried a bunch of them, right? Like, I tried Windsurf and, you know, heavy, like, copy and paster initially, you know, early adopter of, you know, GitHub Copilot. And the reason I like Cursor and I keep coming back to it is just that you can hop quite quickly between very low level and very high level. So what I, my typical flow is I'll actually like talk through a, an idea. Like, I'll, I'll, I'll record myself, just kind of like blabbing about the thing I want to build, because I don't really know what I want to build yet.

• 55:36Dan Shipper?

  Like in a voice note, or?

• 55:38Speaker 01

  Uh, yeah, like, quite often it would be, um, yeah, just record it. Like, I'll, I'll, I'll use, like, D-Script. Uh, just because it's like what I have on my computer for, for recording videos. So I'll just record, it doesn't really matter what you record it with, but then, then I'll, I'll transcribe that. So, you know, you can upload that to, to Google Gemini, like the AI studio, and it's pretty good at transcription. So I tend to do that. And then I'll, and then I'll have like a transcript of me blabbing. And then I'll stick that into, usually Claude and kind of get a PRD, like a product requirements document, kind of, I'll, I'll, I'll edit that quite a lot. And that's where I think a lot of the prompting.

• 56:18Dan Shipper?

  Claude versus O1?

• 56:21Speaker 01

  Yeah, I tend to use Claude still over O1. I think O1 Max is, I'm sorry, they changed all the names, but like the, the, the $200 version. I, I do use that. I'm using that more and more, but I still find Claude is like more creative.

• 56:37Dan Shipper?

  Yeah.

• 56:38Speaker 01

  Is that the only way I can describe it? I, I use O1 to settle problems. Like, it's like the, the big daddy that comes in and just like fixes the problem. Like, when, when Claude is like getting in a loop and doesn't know what to do. So, so like, I'll, I'll, I'll take that PRD and I'll, I'll run, uh, like Cursor Agent and let it just completely go nuts and create any files at once, like, you know, do, do whatever it likes. And then I'll kind of look back and see what I have and if it works and and I'll, you know, change a few things based on the general vibe. But when it gets stuck, which is actually pretty often, then that's when I kind of jump out to O1 in some cases and say like, hey, here's the

• 57:20Speaker 01

  context, here's where it's getting stuck, like here's what we've tried. And I'll just kind of copy all that and then and then get the answer back.

• 57:27Dan Shipper?

  But you say you're, you're, you're editing the PRD that comes out of Claude before you put it into Cursor, like you're not just YOLOing it. Tell me about, about that. Cause I like, one of my failure modes when I do this stuff is sometimes I just YOLO it because I'm just like too lazy, you know, and it, it kind of gets [bleep] and yeah, tell me about that.

• 57:46Speaker 01

  Yeah, I, I found that quite a lot. Like, it's actually similar to what we talked about earlier with the evals where what I'm looking for is like, what do I strongly agree with and what, what am I allergic to? So I'm scanning mostly. I'm not like fully reading in a lot of detail, but I'm just like looking for things. It's actually similar to like if, if an employee sent me something and I'd be like, I'm going to scan it and just see if there's anything that really upsets me. And if not, then it's like, fine to go ahead. But, but quite often, especially when it's a new feature that I haven't really thought through, then I'm like, oh, I see that it's tried to do this and that feels really wrong to me. So I'm going to change my whole, whole approach. Actually, that was like why DeepSeek was really good, actually,

• 58:30Speaker 01

  is I like, I used that a fair bit over the past, past week. But, you know, just the ability to kind of see the thinking and just see like, oh, it's thinking wrong. Like I must have got something wrong in my prompt. Like it's trying to create a new file, but I already have that file. I just needed to put it into the context. So, so it's really about refining, like it's just me figuring out what I actually want. And then, and then once I've, once I'm fairly like happy with that, then, then like it's much easier to build the stuff.

• 58:58Dan Shipper?

  Interesting. What do you think of O3? Are you using it at all?

• 59:01Speaker 01

  I did try O3 Mini a bunch today and I would say it's like an impressive model. It's surprisingly fast, which was like my first first impression for a thinking model.

• 59:13Dan Shipper?

  Which makes a big difference. Like it's not trivial that it's fast.

• 59:16Speaker 01

  Yeah, I find I lose. I don't know if you find this, but like I lose a lot of, you know, productivity because I like just dump my thoughts into O1 and then just say like, oh, go have a coffee. But like, there's only so many, so many coffees you can have in one day. So then I'm like browsing Twitter and I'm like talking about politics. So, oh, God, this is like, I look up like 30 minutes later, like, oh, yeah.

• 59:41Dan Shipper?

  I have the same thing. Like I did that this morning with deep research. Like I posed it this like really big research question. It was like a page of stuff. And then I like went out and like bought like got coffee and like, and just, I knew it was working. It was so weird.

• 59:57Speaker 01

  I need to find something to do in that, in that thinking time, you know, like I did this.

• 1:00:02Dan Shipper?

  Maybe just enjoy yourself. How about that?

• 1:00:05Speaker 01

  I did hear someone say that the other day. They were like, oh, it's, it's really nice to like have regular breaks now with AI. Whereas before I was thinking the whole day. But no, I mean, I, it reminds me a lot of like my first like internship. It was in the UK government and like the benefits, like government benefits, like social security is like the equivalent. And we, we were writing analysis code, basically in SAS, which is like a very old language. And we'd, we'd write the, the code and then we'd send it off to a central server and it would take a couple of hours to run. So I basically did no work that whole internship. It was just, I was like, sorry, my code's running, you know?

• 1:00:46Speaker 01

  And it's like, I think there's like an XKCD comic that makes fun of this as well, like my code's compiling. But yeah, it feels like we're back in those times again, you know?

• 1:00:54Dan Shipper?

  Back to the future.

• 1:00:55Speaker 01

  Yeah. Yeah, I gotta find a way to be productive. I was thinking about maybe I should start um double teaming cursor. So like I'll have like, you know, I'll have like um my my my blog uh writing in cursor like on on one uh one tab and and I'll be creating blogs and like talking about those and then and then like while while you know everyone's working in the background and the other thing.

• 1:01:17Dan Shipper?

  I like that. I've I was doing that a lot with Devon when I was using it. Um like I would have four Devons at once doing different things and um and then I would be like, you know, every like every if you have four of them every minute like something it needs help on there. And so you're just like constantly kind of like tending it. And it reminded me a lot of um when I was in college, I used to play online poker. And like I was never that good, but like good players can like play four tables at once or whatever. And so like sometimes I would do that because like whatever I was in college and I was an idiot, but like it felt like that where it's like you're you're doing all these different things all at once, but uh you don't it it each one only requires divided attention. So you can get it all done. And it's I think that's so cool.

• 1:01:58Speaker 01

  Yeah, it's like those people use the chess grandmasters you see in uh Washington Square Park and they're playing like four games at once.

• 1:02:04Dan Shipper?

  Yeah, yeah, yeah.

• 1:02:05Speaker 01

  Yeah, it does feel like that. But but yeah, I wonder how much we're really getting done when we do that, you know?

• 1:02:10Dan Shipper?

  I don't know. Well, I mean, to some degree, like running a company is like that. It's very similar, you know, um where I'm constantly just like I'll spend like three minutes like looking at something and and go go on to the next thing or whatever. So I just think more people are going to be working that way. With all of its benefits and all of its trade offs. So we're gonna have to like figure out how to manage that, you know?

• 1:02:33Speaker 01

  Yeah, yeah, like what I really want to try and get good at is uh doing it while I'm like on a call, you know, like obviously not this call. But like um I I find um quite often, maybe this is just me, uh but but like I I'll I'll be on a call and I like it could have taken five minutes, you know, like the actual relevant part of it was five minutes. And um I don't even need to take notes anymore because AI is listening and taking notes from me. So uh that's that's the thing. I I like haven't quite got got good at it yet. But uh but yeah, that's the the dream is like I can I can be like managing cursor in the background while I'm like also, you know.

• 1:03:11Dan Shipper?

  Also, it's just like I'll have my agent call your agent and like we don't even need to get on the phone, you know?

• 1:03:17Speaker 01

  Yeah, I do you know, one of the things, one of the really out there ideas that I would love is maybe I'll just do it for fun on the weekend, but like I'd love like a negotiation, like settlement kind of agent where I give I give it like my red lines and it's like a trusted third party and you you give it your red lines and say like, this is the price I I'm willing to pay. This is why, you know, want to argue for and and then it just goes back and forth for our like virtual negotiation and then it comes back and says like, we've settled at this price. It's the best you could get.

• 1:03:49Dan Shipper?

  I think it's cool. And then and then it's like, you have to also decide how much you want to fund the agent, like, how many compute cycles does the agent get to get to a result, you know?

• 1:03:58Speaker 01

  Yeah, yeah, yeah. This was a big thing with the early agents like Agent GPT. I don't know if you ever played with that back in the day. I say back in the day, it's like two years ago.

• 1:04:09Speaker 01

  It's still going, but but yeah, like I I remember seeing a talk by those guys and they had, it was like, it really stuck out to me because they had LLM judges in their CI pipeline. So like whenever they pushed code to that repo, they had AI agents like checking the code and making sure it didn't break and running tests. And at the time, that was kind of radical. Now there's like products that do that. But but they they said they were spending like $200 every time they pushed a commit.

• 1:04:43Dan Shipper?

  That's crazy.

• 1:04:44Speaker 01

  That is like, that's crazy, but also like very, you know, like now that probably that same commit probably costs like $2, right?

• 1:04:50Dan Shipper?

  Totally. You probably lost, yeah.

• 1:04:52Speaker 01

  Yeah, I do think it makes sense. There is like a case to be made that you should just be spending as much money as possible in AI because we're almost like definitely being too conservative.

• 1:05:01Dan Shipper?

  Totally. Agreed. Well, that is a great place to leave it. This is a great conversation. I'm so glad we finally did this. I'm so excited for Raleigh. I'm so excited to get to work with you. If people are looking to find you and your work online work and they find you.

• 1:05:17Speaker 01

  Yeah, on Twitter, I'm Hammer underscore MT. So it's probably easiest place to find me. And then the the book that I did for O'Reilly was Prompt Engineering for Generative AI. It's on Amazon. It's got like a an armadillo on the front, so it's easy to spot.

• 1:05:32Dan Shipper?

  Awesome. Thanks for joining.

• 1:05:33Speaker 01

  Cool. Thanks, man.

• 1:05:42Speaker 02

  Oh my gosh, folks. You absolutely, positively have to smash that like button and subscribe to AI and I. Why? Because this show is the epitome of awesomeness. It's like finding a treasure chest in your backyard. But instead of gold, it's filled with pure, unadulterated knowledge bombs about ChatGPT. Every episode is a roller coaster of emotions, insights, and laughter that will leave you on the edge of your seat, craving for more. It's not just a show. It's a journey into the future with Dan Shipper as the captain of the spaceship. So do yourself a favor. Hit like, smash subscribe, and strap in for the ride of your life. And now, without any further ado, let me just say, Dan, I'm absolutely, hopelessly in love with you.