A.I. Is Changing Everything. Does That Include You? | Freakonomics Radio | Episode 556

For all the speculation about the future, A. I. tools can be useful right now. Adam Davidson discovers what they can help us do, how we can get the most from them — and why the things that make them helpful also make them dangerous. (Part 3 of "How to Think About A. I.") This episode was originally published September 6, 2023. For a full transcript, resources, and more, visit: https://freak.ws/3qW03Uh Photo credit: Audrey Bernstein FOLLOW FREAKONOMICS RADIO: YouTube: https://freak.ws/3yIl6dl Apple Podcasts: https://freak.ws/3yAvQh0 Spotify: https://freak.ws/3TsdCmV ABOUT FREAKONOMICS RADIO: Discover the hidden side of everything with host Stephen J. Dubner, co-author of the Freakonomics books. Each week, Dubner speaks with Nobel laureates and provocateurs, intellectuals and entrepreneurs, and various other underachievers to uncover things you always thought you knew (but didn’t) and things you never thought you wanted to know (but do) — from the economics of sleep to the future of education, from markets for marriage to the surprising utility of wolves. ABOUT THE FREAKONOMICS RADIO NETWORK: Freakonomics began as a book, which led to a blog, a documentary film, more books, a pair of pants, and in 2010, a podcast called Freakonomics Radio. Hosted by Stephen J. Dubner, it’s one of the most popular podcasts in the world, with a reputation for storytelling that is both rigorous and entertaining. Its archive of more than 500 episodes is available, for free, on any podcast app, and the show airs weekly on NPR stations. Freakonomics Radio is now the flagship show of the Freakonomics Radio Network, which includes the podcasts No Stupid Questions, People I (Mostly) Admire, and The Economics of Everyday Things. FREAKONOMICS RADIO NETWORK PODCASTS: Freakonomics Radio: https://freakonomics.com/series/freakonomics-radio/ No Stupid Questions: https://freakonomics.com/series/nsq/ People I (Mostly) Admire: https://freakonomics.com/series/people-i-mostly-admire/ The Economics of Everyday Things: https://freakonomics.com/series/everyday-things/ Special series: https://freakonomics.com/topics/ CONNECT WITH US: [email protected] https://twitter.com/freakonomics https://www.instagram.com/freakonomics/ https://www.facebook.com/freakonomics/ https://www.tiktok.com/@freakonomicsradio https://www.linkedin.com/company/freakonomics-media Subscribe to our newsletter: http://eepurl.com/bKm0cf

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Adam Davidson?

  Hey there. It's Stephen Dubner. You are about to hear the third and final episode in our three-part series called How to Think About AI. The guest host for this series is Adam Davidson, one of the founders of NPR's Planet Money. Here's Adam. Can you just tell us who you are and what your job is?

• 0:23Speaker 01

  This is a question I dread at parties, but my name is Anna Bernstein. I am the prompt engineer at a company called Copy AI.

• 0:34Adam Davidson?

  Have you heard of this new job, prompt engineer? It's a job that could only exist right now, a job that satisfies a need that almost none of us even knew we might have until the last few months. So what is it exactly?

• 0:50Speaker 01

  The prompt engineer essentially is an expert in being the linguistic intermediary between user input and AI output.

• 1:02Adam Davidson?

  Please note how precise Anna Bernstein's language is. That is her superpower, being really, really precise about language. If you've played around with AI tools like ChatGPT or Google's Bard, you've seen it. They respond to the precise words you type in. They can't figure out a vibe or a hidden intention. Also, unless you happen to work at an AI company, you don't ever interact with the raw model itself. Whatever you type in is put through a filter, a filter that people like Bernstein design. The filter between you and the raw AI model is there for a bunch of reasons. The first is to avoid the ugly stuff.

• 1:43Adam Davidson?

  Since the AI models were all trained on an unfiltered mass of text from the internet, they can easily call up truly horrific, offensive words and ideas if left unfiltered. But another reason for the filter is that the AI in its raw form is not always great at understanding what human beings want. Or put another way, human beings are not always that great at telling the AI. AI precisely what we want. We might type a quick prompt like, write some marketing copy for my new blog. The AI has no way of knowing what length you want, what audience you are targeting, what voice or writing style you're looking for.

• 2:25Adam Davidson?

  That's the kind of context Bernstein's work provides. She gives the details that most people don't think to type in. In the previous episode of our series on AI, we talked about how all new technology destroys some jobs and creates new ones, jobs nobody could have imagined. Well, prompt engineer is one of those new jobs, and most people have still never heard of it.

• 2:49Speaker 01

  Honestly, most of the time I would just say, I make the AI talk good, and then that will get a laugh and everyone would move on, hopefully. It's a difficult job to define.

• 2:59Adam Davidson?

  As far as you know, do you think you might be the first person to have this job?

• 3:02Speaker 01

  As far as I know, if someone is like, nope, I had it before that, I'm not going to fight them on it. But of who I know, yes.

• 3:09Adam Davidson?

  You are the first one I've heard of, and you are the person from whom I learned that this is a job. But I now see that it's an in-demand job, right? There's a lot of listings out there for prompt engineers.

• 3:21Speaker 01

  Yeah, I think a lot of companies and a lot of industries are still figuring out how and whether they get a lift from prompt engineering, which is also very tied into the question of whether AI will be relevant to them at all, and how it's going to be relevant to them.

• 3:40Adam Davidson?

  For a job that barely existed a few years ago, prompt engineering has become hot. I've seen dozens of prompt engineering gigs on jobs boards. Most pay more than $100,000 a year. I saw one that was more than $500,000. This is just one early example of a new job created by AI. Talk to leaders in medical research, engineering, finance, healthcare, education. Almost all the folks I speak to say roughly the same two things. AI is going to change everything, and we have no idea what those changes will look like. That's what we're talking about. About today on the third and final episode of our series on AI here on Freakonomics

• 4:25Adam Davidson?

  Radio. What does the world look like when AI is everywhere? When AI is just assumed at your job, at school, in your personal life? How will it change the way you live? And how will you have to change?

• 4:51Speaker 02

  This is Freakonomics Radio, the podcast that explores the hidden side of everything with guest host Adam Davidson.

• 5:08Adam Davidson?

  When Anna Bernstein talks about making the AI talk good, she is speaking of this new generation of AI programs called large language models, known as LLMs. These LLMs take in massive amounts of data. We don't know exactly how much, but it's some huge percentage of all of human knowledge, meaning just about every publicly available book, blog, news article, as well as, well, whatever Twitter and Reddit and comments on YouTube videos represent. And then the AI does something fairly simple. It assigns probabilities to words. We do this too. If you heard me say, I saw a huge school when I was out on that

• 5:52Adam Davidson?

  boat, you're probably able to do a quick calculation and figure out, I mean, a school of fish. You might be using a handful of parameters to make that calculation. The word boat makes you realize I was on the water. Maybe you also know something about me and what I've been up to lately and what I'm interested in. Maybe we had just been talking about fish. Neurologists explain that our brains can actively process about three to seven different chunks of information at any one time. The AI large language models employ hundreds of billions, up to a trillion, discrete parameters. So the words AI writes are not based on a few facts, like that I'm in a boat and we

• 6:37Adam Davidson?

  were just talking about fish. They are based on trillions of data points. There's an old saying, "Knowledge is knowing a lot of facts and wisdom is knowing which facts matter." In its raw state, an LLM has almost all of human knowledge and almost no human wisdom. There are a handful of LLMs now. There's OpenAI's GPT-4, which fuels ChatGPT, as well as Microsoft's Bing. Google's LLM is called Palm 2. We've heard so much about these LLMs, I started to wonder, what are they exactly? Are they one big box or some network of computers? Are they in one place or just some vague capability in the cloud?

• 7:20Adam Davidson?

  Who creates and manages these LLMs?

• 7:23Speaker 03

  The number of people required to do it has gone up over time. When I was at OpenAI doing GPT-3, it was basically three of us who trained the original model.

• 7:32Adam Davidson?

  That is Dario Amade. I find this amazing. He says that basically three people did the original work on this LLM that eventually changed the world. Amade quit OpenAI, the company that created ChatGPT, and has a new job now.

• 7:48Speaker 03

  I'm CEO of an AI company called Anthropic.

• 7:52Adam Davidson?

  Before he was CEO of Anthropic, when he was at OpenAI, Amade was VP of Research there.

• 7:59Speaker 03

  Then around the end of 2020, I and a number of our colleagues left to start this company called Anthropic. That was really going to focus on, A, scaling up AI systems, but B, really thinking about the safety and controllability components of it, more so than we felt other folks in the field had done so far.

• 8:20Adam Davidson?

  And why did they want to do this?

• 8:22Speaker 03

  Things are moving very, very fast and we want to move fast too, but we want to move fast in a way that's good.

• 8:28Adam Davidson?

  Anthropic has raised billions of dollars from venture capitalists and from Google, which owns a reported 10% of the company. Its goals are not small. It wants to win the race to be the dominant large language model in the world, surpassing OpenAI by being bigger, smarter and more aligned with humanity. They call their LLM chatbot Claude. There are around 35 people working on it today. I asked Amade, what does it take to build one of these LLMs?

• 8:59Speaker 03

  The first stage is actually surprisingly simple. You just take this large language model and you feed it a whole bunch of text. People typically crawl the internet. So, you know, this would be something like trillions of words of content, just wide range of stuff you can find on the internet, from news articles to Wikipedia articles about baseball to the history of samurai in Japan. And you basically tell this language model to look at document after document. And for each document, look through the document up to a point and always predict the next word. So look through the first three paragraphs. What's the first word of the fourth paragraph? Then what's the second word of the fourth paragraph? And you can always tell whether it's guessing right or wrong because you know what the truth is.

• 9:43Adam Davidson?

  And my understanding is early on, it's basically random. It's totally wrong.

• 9:48Speaker 03

  Oh yeah, it starts totally random. I mean, it's like a newborn baby, right? You know, they can't move any of their limbs or really understand the world. They can't really even see effectively. And then what you see is that different things form. First, it gets a basic sense of how to spell words, right? So, you know, at the beginning, it isn't even spouting useful words. And then it starts to get the grammar right. It learns the idea of subject, verb, adjective. The grammatical structure of sentences are correct, but it's still not really making sense. What's the famous sentence? Furious green clouds sleep furiously or something? It's a grammatically correct phrase that doesn't mean anything. It's total nonsense. Then it gets to the point where the sentences start making sense.

• 10:32Adam Davidson?

  Let me just say it's colorless green ideas sleep furiously.

• 10:35Speaker 03

  Colorless green ideas sleep furiously. So just piece by piece, it starts to put together an actual picture of the world and some things it learns later, like takes a long time for it to learn about arithmetic or math or how to program. And some things it learns earlier. It tends pretty early to cotton on to kind of basic facts and basic ideas. Like it learns that Paris is the capital of France pretty early.

• 11:01Adam Davidson?

  When you're doing this work, like if it's trillions of words, it would take, I don't know, 100,000 people their entire life or something to read all that.

• 11:09Speaker 03

  We've, we've no ability to do any of this manually.

• 11:12Adam Davidson?

  So you're not sitting there going, oh, cool. It just learned Paris.

• 11:14Speaker 03

  We're seeing statistics about how good it is at predicting the next word. And every once in a while, we just ask it to generate things. Right. We just talk to it and you know, you can ask it like, what's the capital of France? And when it's a week into training, it seems like it has no idea. It says, you know, clouds or something, and it doesn't even understand the type of object. And then it says, you know, Madrid. And then you're like, okay, well, it's starting to understand that it needs to be a city, but it's not getting it right. And then it's like Marseille and you're like close, but not quite. And a little more training, it's like Paris. And you're like, okay, well, it's learning the concept. So you can check these things, although what the model knows as a whole is so vast, right? It's like asking a child a question at different ages. You can't understand everything that they're learning

• 11:59Speaker 03

  that's going on in their brain, but you can check one particular thing if you want to, if they've learned it or not.

• 12:07Adam Davidson?

  This, by the way, keeps coming up. People who work on AI talking about the models like they're children. Some talk as if they are newborns, listening to language and slowly figuring out what words mean, the rules of grammar. Although large language models often learn dozens of languages. Others talk about the models like they're older kids, though still naive about the world. I sometimes think of those Amelia Bedelia books about the girl who takes everything literally. You ask her to plant a bulb and she puts a light bulb in the dirt. Here's prompt engineer Anna Bernstein.

• 12:42Speaker 01

  The analogy I often use is it's like I'm picking up a cup. And teaching a toddler to drink from the cup. And I like bring the cup to my mouth and drink from it. And the toddler picks up the cup and brings the cup up to their mouth. And I'm like, yes, yes. And then it brings the cup over the floor. And I'm like, no. And then it just drops the cup. And I'm like, OK, we were close.

• 13:04Adam Davidson?

  I found this surprising. The main work, the hard work at these AI companies is not building the original large language model. That's a big project. It costs more than $100 million. It takes several months of intense, although fairly basic, computing. But the hard work comes once all the training and the math are finished. And you have a large language model and it's sitting there, knowing everything, understanding nothing, equally capable of a romantic sonnet or a racist diatribe. OpenAI spent months getting GPT-4 to stop being so offensive, so violent, so ugly, so useless. That requires a ton of human intervention.

• 13:47Adam Davidson?

  With Anthropic's Claude, they intervene using a method they call Constitutional AI.

• 13:53Speaker 03

  The idea in Constitutional AI is that you write an explicit set of principles, a constitution. Claude's constitution is about five pages long. It has some principles drawn from the UN Declaration on Human Rights, some from Apple's Terms of Service and some that we kind of came up with ourselves. And then you essentially train the model to do things in line with the constitution and another copy of the model to tell the first copy of the model, hey, is what you just did in line with the constitution. And so we kind of feed the model back on itself and teach it to be in line with the constitution. And the great thing about the constitution is, although I think this question of what should

• 14:37Speaker 03

  the values of the model be, how should it interact? I think that's a very hard question. But I think an important advance here is if we're able to make this five-page document and we can say, this is what Claude is attempting to do. It might not be perfect about it, but Claude is attempting to respect human rights. Its aim is not to have political bias in any direction. And then we can point to the constitution and we can say, look, we're not perfect at this. But like, the training principles are right here. You know, we've not secretly snuck anything into the model.

• 15:10Adam Davidson?

  This might be a weird question, but if I just Googled around, I could find neo-Nazi white supremacist stuff. And I don't like that that stuff exists, but I'm a journalist. Sometimes I want to know, like, what are those folks saying? Why not just let the thing, you know, it represents the collective view of humanity and that's what people should have.

• 15:30Speaker 03

  There's a certain amount of sympathy I have for that. And one place I have sympathy for it is when the model is originally gobbling up all the data. I don't want to block it from reading almost any content. My view is models like humans should learn about these things, but I'd want the model to use its knowledge of history, including the bad parts of history, to understand what it means to respect human rights. That's what I think of as a healthy human being, right? They've seen the good stuff. They've seen the bad stuff. They understand why the bad stuff is bad. My hope is with things like the constitutional AI that we can do the same thing with AI systems.

• 16:06Adam Davidson?

  This is a big concept Amadeus proposing, that one of the things we should teach AI is to tell right from wrong. Anthropic is not the first team that's tried this. It has been a major hurdle for researchers. Amadeus cautiously optimistic that Claude's constitution will help it clear that hurdle. And he thinks there are other reasons to feel optimistic about AI's potential. Before he started working on AI, he got a PhD in biophysics.

• 16:35Speaker 03

  When I think of the places we've made progress and the places where we failed, the diseases that we've done a good job of curing are those that are fundamentally simple, right? Viral diseases are very simple. There is a foreign invader in your body. You need to destroy it. Same with bacterial. The ones we haven't succeeded that well at yet, even though we've sometimes made progress, are cancer. You have A billion different cells in your body that are out of control and rapidly self-replicating. Each of them has a different cocktail of like crazy mutations that causes it to do a different crazy thing. How do you deal with that? It runs into this incredible complexity of number of cells, number of proteins within

• 17:19Speaker 03

  each cell, all these incredibly complicated regulatory pathways. And my thinking on it as I went from biology to AI was, oh my God, this is beyond human comprehension. I'm not sure that humans can completely solve or understand these problems, but I have this feeling that maybe machines can, with still some help from some parts that need to be done by humans. But all the facts, all the incredibly complicated proteins, regulating proteins, tens of thousands of RNA sequences, like this is raw data. This feels like machine language, not human language. I mean, curing cancer has almost become a joke, right? The idea of totally curing a disease like that to most people in that field and maybe

• 18:02Speaker 03

  to most people, it's been said too many times and the promise hasn't been delivered. But I actually think AI is a technology that could do it. I mean, as worried as I am about the downsides, I think the upsides are incredible and we're in a period where science has not progressed as fast as we like, but I think AI could unblock it.

• 18:23Adam Davidson?

  But what about those downsides? Amadei testified in Congress about his fears of bad actors using AI to develop biological weapons. He points out that today it takes enormous resources, typically the resources of a state, to develop most biological weapons, which is scary enough. But what if AI puts those tools in the hands of terrorists and psychopaths?

• 18:46Speaker 03

  It just stands to reason that if you can do wonderful things with biology, you can also do horrible things with biology. And I'm very concerned about this. The truth is, the scary sounding stuff and the stuff you can get on Google, that's not really the stuff that makes the core experts afraid. What you should think about is there's a long process to really do something bad. I'm not going to talk about it in a public setting and a lot of things I don't know about. But there are certain steps in that process where the information is really hard to get and the tasks are really hard to do. And what we looked at is, is that information being implicated? And the concern is that we're getting to the beginning of where it is. We're not there yet, but two to three years is our best guess. I don't know what's going to happen, but that's our best guess for when we'll get there.

• 19:30Adam Davidson?

  That's scary. There's a lot of people with really bad ideas and the world puts up barriers to them because they don't have knowledge and resources. And so it's hard to think, how do we lower barriers for good stuff and maintain barriers for bad stuff?

• 19:45Speaker 03

  I think that's the hardest question. The term we use is differential technology development. How do we develop the good stuff ahead of the bad stuff?

• 19:54Adam Davidson?

  These debates are essential. Is AI good? Is it bad? How can we encourage the goodness and try to head off the worst of the badness? Recently, Anthropic, OpenAI, Microsoft and Google announced they were forming an oversight group to self-police their industry. It's called Frontier Model Forum. Amade and other AI executives have also called for the government to develop regulations for this new technology. There's obviously an inherent tension here. Amade says AI could be very dangerous and should be regulated. At the same time, he's running a company whose very mission is to make AI more powerful.

• 20:35Speaker 03

  I think of myself as someone who's trying to do the right thing, but I can't say that my company has all the right incentives here. You're kind of relying on me to go against the company's incentives, and that's true for the other companies as well. Somehow, the government needs to play a watchdog or enforcement role while leveraging the expertise of the companies. There's probably a role for nonprofit organizations, too. And so there needs to be some kind of ecosystem where the strengths of each component help to ameliorate the weaknesses of the other

• 21:08Adam Davidson?

  How to regulate AI is a big, important issue that's in the midst of being figured out. In the meantime, AI is here. We can't stop it. A lot of us are using it not to develop bioweapons, but to beef up our resumes, inspire ideas for a dinner party, or just to play around. So it makes sense to try to figure out what it's good at. And getting to know AI is kind of like getting to know a stranger. It takes time.

• 21:33Speaker 01

  I mean, at this point, I've spent, I don't want to know how many hours with these models.

• 21:38Adam Davidson?

  After the break, we'll get to know AI a little better. I'm guest host Adam Davidson, and you're listening to Freakonomics Radio.

• 21:55Adam Davidson?

  AI, like any new technology, will create winners and losers. For now, Anna Bernstein is one of the winners. When she started this work, the job didn't have a name. She and her bosses barely knew it was a job.

• 22:08Speaker 01

  They hired me on contract for a month to fix tone at the time. They were like, if we can really nail, you know, friendly or professional or formal or informal, that'll improve the product. And I kind of figured that out for them, and so was hired full time. And yeah, it's been, it's been a really wild ride ever since.

• 22:28Adam Davidson?

  I find Bernstein's experience so interesting because she's an early pioneer into this new world of large language model AI. What she has learned in copywriting will, I think, eventually apply to people in lots of other fields. After all, whatever we do with AI in biological research, or aerospace technology, or whatever, human beings will need to figure out how to communicate their goals to this computer brain. What Bernstein learned, and what she taught me, is that learning to talk effectively with an AI LLM requires you to spend some time thinking about how you talk to human beings.

• 23:06Speaker 01

  It's definitely opened my eyes to how much of human communication is inference, is under this tent of context where, when we speak to each other, we think we're speaking in a nuanced and precise way. And we are, but we're doing that through relying on context cues and even in written communication, and not just talking about body language and tone, but that as well, but relying on the shared context with the person we're talking to, this sort of fuzziness or like slack they'll give us. And you have these models that are trained for both really, really precise and literal instruction that human beings would struggle with because

• 23:48Speaker 01

  they are very intricate and it's a lot of instruction at once. And at the same time, this same model is supposed to be good at this sort of fuzzier communication with people. And those two really, at times, are at odds with each other.

• 24:04Adam Davidson?

  Think of saying something like, "It would be fun to get together soon," to another person. That phrase could mean what it says and could be followed up by an email asking to schedule a date. But it could also mean, and maybe it's more likely to mean, that it would very much not be fun to get together soon. Maybe I'm just saying that to be polite and to get out of this awkward encounter. That's a great and sometimes very confusing aspect of language. The very same words can mean a specific, precise thing or the exact opposite. For me, at least so far, this has been a bit maddening when I use AI tools. I'm thinking of something I want them to do. I type out the words, telling it to do the thing I want, and they don't quite do what I was hoping. I look at the words I

• 24:48Adam Davidson?

  use and realize I hadn't put in enough context to guide the AI to what I want. For someone like Anna Bernstein, though, that's not maddening. That's the fun.

• 24:58Speaker 01

  Now, at the intersection of those two things is actually something I really enjoy, which are really, really well-crafted prompts, even like simple prompts where when you get just the right wording, it does exactly what you want. If you can describe exactly what you want, it gives you exactly what you want, where you really Hit the nail on the head for describing the type of copy. And that can be so powerful.

• 25:20Adam Davidson?

  My early experiences with AI were frustrating. I type in a prompt and get a very boring and generic response. Generate a scientifically backed meal plan to lose weight. Give me a list of movies that an 11 year old boy and his parents might enjoy. These are pretty generic. I didn't tell AI what I like to eat, or if I have any dietary restrictions. I didn't explain what my son is into, what I'm into, what my wife's into. If you don't tell AI, it won't know. So while it has been exposed to so much, it'll just give you a blah, middle of the road answer. The good news is that it's not hard to improve the AI results. Just be more specific, give it more context and learn what kinds of prompts lead to better results.

• 26:03Adam Davidson?

  Anna Bernstein has a YouTube video that I found helpful. It's called Master the Art of Prompt Writing: Six Tips to Writing Better Prompts. If you want to check that out. So what's Bernstein's advice?

• 26:15Speaker 01

  It's going to sound really basic, but just getting the right wording, trying synonyms, trying different syntax variations on the same theme can really unlock capabilities in the AI. You can also pile synonyms on there. If you're trying to like, get it to use a very enthusiastic voice and just enthusiastic isn't quite enough. You can pile on like excited, hyped up and just like use all of those at once.

• 26:37Adam Davidson?

  That has been a huge realization is that often the problem is not the AI or not just the AI. The problem is me. I haven't actually thought through, wait, what do I want? You know, one thing I did is write an essay about, and then it was the subject. In my case, I'm working with a friend of mine who teaches at Harvard. He's into serologists. So write a paper about ancient Assyria. And it would just be a really generic paper. He actually said it probably could have gotten like a B minus at Harvard. But it forced me to realize, well, what do I actually want to do? So use a lot of synonyms, but also use the right words and particularly use the right verbs. Use powerful verbs.

• 27:21Adam Davidson?

  Can you explain that?

• 27:23Speaker 01

  The paper is a great example because you could maybe change that to something like explore ancient Assyria in the form of a paper, et cetera, et cetera.

• 27:33Speaker 01

  And that might give a more interesting dynamically written results that does what you want more, which is to explore the topic rather than just sort of rote-ly recurgitate facts about the topic. So you really want to think about the purpose of what you're doing and imbue the verb with that.

• 27:53Adam Davidson?

  One of the ways in which it's really different from us is you have the rule down as no negativity allowed. But the idea that it really doesn't perform well when you tell it not to do something.

• 28:06Speaker 01

  It depends slightly on the negative command, but as a general rule, if you tell it, you know, don't be formal, don't be stiff, it may get the message, but it'll at the same time be like, oh, interesting. I know what those words mean. I'm thinking about them as I'm doing this task and it'll just influence the task. So at best you'll get no effect and at worst it'll be a lot more formal and stiff than when you started.

• 28:37Adam Davidson?

  With AI, it's quite effective to just use normal language. It works well to just write down your thoughts or even better, speak them like you'd be telling them to a person. A friend told me to think of talking to an intern, a capable, super eager, but hopelessly naive young intern. Use regular language, but add as much context as you can. So don't just say, I want to go to Montreal for the weekend. Give me a list of fun things to do. Give way more context. I want to go to Montreal for the weekend. My wife and I love trying new local foods in out of the way restaurants. Our 11 year old son loves anything to do with sports. We love to go on long walks in regular neighborhoods

• 29:18Adam Davidson?

  and we don't like touristy spots. You get the picture. Give a lot of context. A few other tricks or tools I find helpful. Tell the AI to ask you questions. So I might give it a prompt to do something and then I'll add, ask me any questions that might help you fully meet my needs. Another trick a friend told me, give it an instruction and then write, "Do not do anything yet. First, tell me what you think I'm asking you to do, and let me know what you find confusing." For what it's worth, I sometimes find this process pretty fun, like I'm learning how to communicate with an alien. And I sometimes find this process maddening, like I'm communicating with an alien.

• 29:58Speaker 01

  I mean, I love it. I have been on like meetings with coworkers where they're like, "This prompt isn't working. Can you help me?" And I start fixing the prompt and they're like, "Sorry about making you do this tedious thing." And I'm like, "What? What tedious thing?" I think it also comes from a bit of a poetry writing background where at least my process for writing poetry involves like really concerningly obsessive editing, where I am resetting the same line break over and over again and giving it time and rereading it. I had this poetry professor who was like, "The next right word is usually your first thought or like your 153rd thought, but it's really like your second thought or third thought."

• 30:42Adam Davidson?

  I've been focused on avoiding the hype, the dystopian and the giddy. And truth be told, for all the attention, AI is still in such an early stage that most of the ways I use AI and that I see others use it are pretty banal.

• 30:58Speaker 04

  I think that it helps to have a utopian vision here.

• 31:01Adam Davidson?

  That's Ethan Mollick. He's a professor of management at the Wharton School at the University of Pennsylvania, where he studies entrepreneurship and technological change. And I'm not aware of anyone who is having more fun with AI than Ethan Mollick.

• 31:15Speaker 04

  I've been playing around with it for a while. I've been AI-adjacent for my whole career. I am not a computer scientist, but I've been thinking about uses for it for a long time.

• 31:23Adam Davidson?

  Mollick publishes a weekly newsletter called One Useful Thing, which provides a stream of things you can get AI tools to do. One area that Mollick finds AI especially helpful with is entrepreneurship, which is what he teaches at Wharton.

• 31:37Speaker 04

  America's a nation of entrepreneurs in waiting. When we do surveys, a third of people have had an entrepreneurial idea in the last five years they wish it could execute on, and almost nobody does anything.

• 31:45Adam Davidson?

  What does that look like? AI helping would-be entrepreneurs actually pursue their dream? After the break, Ethan Mollick and I go into business together. I'm guest host Adam Davidson, and this is Freakonomics Radio.

• 32:10Adam Davidson?

  I'm Adam Davidson, sitting in for Stephen Dubner on this, the final episode of our series on AI. I know for me, one of the best ways to get going on something new, a new business, a new project, a new exercise regimen, is to get a partner, a buddy who knows things I don't, who can share the journey with me. But it is really hard to find good partners.

• 32:31Speaker 04

  AI is a general purpose technology. It screws up in some areas. You have to use it the right way. But it's really exciting to have a generalist co-founder with you who can give you that little bit of advice, a little bit of encouragement, push you over the line, and that makes a big difference in people's lives.

• 32:46Adam Davidson?

  Ethan Mollick teaches entrepreneurs how to start businesses. He's at the Wharton School, one of the top business schools in the world. He also studies what leads to success and what leads to failure when people start businesses. And he says that a major barrier to success is not some fancy formula cooked up in the Ivy League. It's so much simpler. One big reason people fail is that they stop too soon. Many stop before they do anything. They have an idea and they never pursue it. Others pursue it for a while, but hit some blocks, and that's when they stop. AI is far from perfect. Mollick will certainly agree with that. But it's always ready to push another step. It never just gives up.

• 33:32Speaker 04

  If you ask the AI, especially GPT-4, the most advanced model you can get access to, and you say, what should I do next to do this? The steps that it will give you are perfectly reasonable. Are they the best steps possible? No. We'll make mistakes, probably. But overcoming that inertia, getting a little push about what to do next is really helpful. And then it will help you actually do those experiments. I'm forcing all my students to actually interview the AI as an actual potential customer. That's not because it's as good as interviewing a customer. You absolutely have to interview potential customers, but it actually gets you partway there. It greases the wheels a lot of ways to doing that initial testing, and overcomes those barriers where you might otherwise say, I need to hire someone to do this, or I don't know what to do next, because it can help you overcome that inertia.

• 34:14Adam Davidson?

  I find that really exciting. That feels really cool if every person in the world has an entrepreneurship engine in their home.

• 34:22Speaker 04

  I absolutely love the phrase entrepreneurship engine because most entrepreneurship is not like big E entrepreneurship. It's not trying to launch the next Twitter. It is trying to start a small business in your town.

• 34:33Adam Davidson?

  We've all had these fantasies, right? Of opening a lovely business in our town. Here's mine. I live in a small town in Vermont. The other day I was two towns over in Bristol, which has the perfect tiny little small town Vermont Main Street. I was with some friends eating lunch and across the street I saw a sign announcing that the local stationery store was for sale. I started to fantasize, telling my friends how much I would love to buy that place and turn it into my dream stationery store. I love stationery. Pens and paper and stamps and the whole thing. I love fancy artisanal fountain pens and I also love big stacks of regular old copy paper and a giant wall filled with big pens. But of course, it was just a silly fantasy, some idle conversation.

• 35:18Adam Davidson?

  But talking to Ethan Molik, I started to wonder, what if I had used an AI tool to flesh my idea out a bit? See if it was at all viable?

• 35:26Speaker 04

  So you say you have an idea, right? But I still think ideation is its own phase. Because a lot of ideation is just about combining ideas together with variation. And that's something that AI does really well because it finds connection between ideas. So I might start off with, I want to launch a stationery store in Vermont. Give me 20 different variations on stationery store ideas that could be great. Give me 100 ideas. And then I would probably do some constrained thinking. Let's say it unlimited money. Give me 20 ideas with unlimited money. So generally, when you prompt the AI, you want to tell it who it is, what context it's operating under.

• 36:01Adam Davidson?

  And let's slow down there for a second because that was a real big game changer for me is that you literally just tell the AI what it is.

• 36:08Speaker 04

  You're not actually like magically invoking Warren Buffett when you say you're Warren Buffett. What you're doing is putting it in a context, right? Because it's absorbed all of sort of human knowledge, right? So what context is it operating in? Who's its audience? So I would start with that idea of generating a lot of ideas. Ideas. And by the way, develop the ideas you like. Tell me more about idea two. Give me more variations on idea six. What would the steps be at idea 12? So you're interacting with it. You're not just typing prompt in and getting a query back. It's not Google. The more you interact, the better. And you can also give it your context. You could say, I'm really interested in stationery as a lifestyle business. I want to do more than break even. The more context you give it, the better off you are. So I would be experimenting with that. And again, you'll find out what works for you.

• 36:51Speaker 04

  And I would also be super careful. If you know nothing about stationery, you're going to get lied to by the AI sometimes.

• 36:57Adam Davidson?

  I'm actually tempted. Can we just do this? Why don't we do this? Here, I'm logging into my OpenAI account. I'm gonna click on GPT four.

• 37:09Speaker 04

  Yes.

• 37:10Adam Davidson?

  All right. So what should we start with?

• 37:12Speaker 04

  Well, we could just say, you're very good at coming up with creative ideas. Let's start like the most basic kind of approach.

• 37:17Adam Davidson?

  So what are 20 ideas for a stationary store in Bristol, Vermont? That would, what should I say? That would attract people from all over the world, something like that?

• 37:32Speaker 04

  Yes, let's do that. Yep. And I will not know until you type this in about how useful this will be. So the point is you can't just bounce off at once, right? We're going to get this working right. So let's figure out what we get out of this. It may be the wrong direction. So the great thing is we can nudge it back into shape.

• 37:48Adam Davidson?

  Right. All right. So it's saying I could do an eco-stationary store, products made from recycled and sustainably sourced, historical stationery.

• 37:57Speaker 04

  That's actually kind of neat.

• 37:58Adam Davidson?

  I like that one, actually. Bespoke stationery, custom-designed stationery. That feels more like a virtual business for me. Vermont-inspired stationery, I'm not quite sure that works. Worldwide stationery. Maybe locally-made artisanal stationery. That one doesn't feel as exciting somehow, because I think that exists. Stationery subscription business, high-tech stationery, ephemera store, vintage stationery. A cafe where people can relax, write their sketch while enjoying a good coffee or tea.

• 38:27Speaker 04

  And by the way, I would expect none of these to be perfect ideas, right? Hopefully, though, I'm seeing, I'm watching your eyes here, right? And you're hopefully making connections in these cases where it's like, oh, that's actually, I hadn't thought about the vintage angle from that way.

• 38:38Adam Davidson?

  It surprised me to see just how fun and useful this process that Moloch calls constrained ideation could be, even when the ideas were silly or terrible. It got my brain working in new ways, thinking new thoughts. And since the AI tools are so fast and easy, you can zoom in to explore the most ridiculous ideas.

• 39:01Speaker 04

  Let's do something that's like stationary that would be useful to astronauts or something like that.

• 39:08Adam Davidson?

  Right, got you. Okay, so let's see.

• 39:12Adam Davidson?

  Astronomy themed stationary.

• 39:16Adam Davidson?

  Stargazing, space tech accessories. I would definitely drive a few hours for space tech accessory store, but I'm both a stationary nerd and a space nerd. So it's got me in two ways. Astral memoirs and notebooks, spacecraft blueprints, glow in the dark. All right, I'm liking like the theme, but I'm not seeing why that would be in Bristol, Vermont.

• 39:38Speaker 04

  Absolutely. But you know, it's costless, right? So the other thing we don't quite realize is like you just completed idea generation session in the minute and a half we're talking, right? This would have been a whole process before. There's no reason not to throw 50 constraints to the system and you spend a half hour and you'll have a list of ideas that you're super interested in. And again, dig into anything, like give me five variations of that one.

• 40:01Adam Davidson?

  Yeah, give me five variations on a stationary store that sells historic stationary. And it's just immediately jumping in, right? Ancient scripts, stationary store.

• 40:15Speaker 04

  It gets kind of obsessive, right? Like you're like, ooh, I could keep getting it to generate stuff. Like my mind is spinning now. Like I've got ideas for a stationary store to compete with yours one town over because this sounds great.

• 40:27Adam Davidson?

  This was starting to get fun. Now, I don't want to open a space themed stationary store. That's not the point. The point is that constrained ideation can get your brain open to all sorts of possibilities. I actually started to give that historical ancient stationary idea some serious thought. I'm so fascinated by old ways of writing. I would love a place where I could buy clay and a stylus to practice cuneiform writing, get some vellum and dip pens, or some real papyrus. I was just reading how in the ancient Middle East, some people wrote by scratching into pliable lead. Is it just me? Or would you love to go to a store that had all of that? And with Malik's guidance, before long, I had a halfway decent business plan and market

• 41:10Adam Davidson?

  research proposal. Now, I'm not actually going to open a historical ancient stationary store, although, man, I do kind of want to. But the point is, the process was fun. And in 12 minutes, I accomplished what would probably normally take months. And who knows? I like playing around with different business ideas, and maybe someday I'll actually do one of them. A key point that Malik showed me is that this is very much not a passive thing. I used to have that idea that AI meant giving over all the thinking work to some computer, that using AI was cheating. And it can be, like all the reports of people getting AI to write their school assignments for them. But this Malik approach of using AI as a thought partner, pushing me forward, playing around with

• 41:55Adam Davidson?

  goofy ideas, fleshing out semi-formed thoughts into more rigorous ones, it felt very active. And it got me closer to what I actually think and want. It made my own thinking clearer to me.

• 42:09Speaker 04

  In every case where we're using this, you should be able to push the AI to a point where you're getting to a back and forth interaction with it, where you as a human are adding a lot to that interaction, and it's helping you by giving you the immediate gratification you need, variation ideas that you need. And if you can get there, then I think this becomes really magical.

• 42:26Adam Davidson?

  I caught a bit of Ethan Malik's infectious optimism. It started to help me see that AI doesn't necessarily have to replace us. It can expand the range of human possibilities, allowing us to do far more than we could before. Ethan Malik's wife, Leelach Malik, is also at Wharton, where she leads their digital learning programs. They both weren't sure at first how AI would impact education. There are a lot of fears, reasonable fears, that it will hurt education. Instead of learning, students will just use AI to pretend to have learned. But the Malics have been experimenting with AI as an educational tool and have found that it can be quite helpful.

• 43:06Speaker 04

  So, for example, as much as we hate them, tests are one of the most powerful ways to learn because they not only test your knowledge, but they actually increase your future recall. Writing tests is hard to do. This writes tests for you. Educating people is the key to unlocking everything. If we could do that scale, what does that mean? That's incredibly exciting.

• 43:27Adam Davidson?

  We started this series of episodes asking if AI is more good or more bad for humankind. We've made clear throughout that nobody really knows. This thing is so new, so weird, so fast changing, that any prediction at all in any direction is quickly overtaken by a surprising reality. So I'm not going to predict the future. I do think we can say some things with some confidence about right now. And here they are. Number one, the current iteration of AI is not worthy of the hype we've seen. It's a tool. Yes, it's a big deal. Pay attention to it, but it's nowhere near mature. Two, the future of our lives with AI is not yet written. Whether it makes life on balance better or worse

• 44:12Adam Davidson?

  for most of us is up to us. It will be how human beings use AI. Set the rules for AI that will determine its impact. Three, AI will almost certainly transform your job and your company and your entire industry. You will probably still have a job, if you have one now. You'll be doing work, but it will be done differently. And in some ways, better and in some ways, worse. And four, this is my one recommendation. Don't sit it out. Get to know it. You can hate it. You can love it. You can have all sorts of mixed feelings. But the more you understand it, the better prepared you'll be. Just as I was finishing this episode, a friend told me about something that had

• 44:53Adam Davidson?

  just happened to him. My friend was in India because his mother-in-law was in the hospital, 84 years old, on a ventilator. The doctor said she would pass within a day or so. Her husband, who is 92, was distraught, for all the obvious reasons, but for another one, too. He wanted to tell her how much she had meant to him, how wonderful their 60-plus years of life together had been, but he didn't know how to say that in words. As it happens, his granddaughter, my friend's daughter, works in AI. She guided her grandfather through some AI prompts, asked her grandfather some questions, and entered them into ChatGPT. It produced a poem, a long poem. He said it perfectly captured his feelings about

• 45:36Adam Davidson?

  his wife, and that on his own, he never would have been able to come up with the right words. He sat next to her, reading the poem, line by line. She died soon after, and he said it allows him to know he told her everything.

• 45:54Adam Davidson?

  I want to thank you for listening. It has been so fun to work on this show, a show I've loved for as long as it's been around. Thank you for letting me sit in for Stephen, who will be back next week.

• 46:13Stephen J. Dubner?

  This is Stephen, and I want to thank Adam Davidson for taking us on this lovely, leisurely stroll through what AI is and isn't, and what it can be. If you missed an earlier episode in the series, go catch up on any podcast app or on Freakonomics.com. Just look for the series called How to Think About AI. Coming up next time on the show, the union that represents NFL players recently conducted their first-ever survey about workplace conditions. I would never have thought to ask, are there rats in your locker room? And they gave a letter grade to each of the league's 32 teams.

• 46:49Speaker 06

  This is really about, are we giving you the inputs you need to be as productive as possible?

• 46:54Stephen J. Dubner?

  The NFL is the richest and most successful sports league in history. Each team is worth at least $4 or $5 billion.

• 47:03Speaker 07

  Nobody wants to be known as the cheapskate. Before, when it was rumored you were the cheapskate, it was harder to prove. Now there's data.

• 47:12Stephen J. Dubner?

  And what does the data say? That's next time on the show. Until then, take care of yourself and if you can, someone else too.

• 47:24Stephen J. Dubner?

  Freakonomics Radio is produced by Stitcher and Renbud Radio. You can find our entire archive on any podcast app or at Freakonomics.com, where we also publish transcripts and show notes. This series was produced by Julie Kanfer and mixed by Eleanor Osborne, Greg Rippon, Jasmine Klinger, and Jeremy Johnston. We also had help this week from Daniel Moritz Rapsen. Our staff also includes Alina Coleman, Daria Klenert, Elsa Hernandez, Gabriel Roth, Lyric Bowditch, Morgan Levy, Neil Carruth, Rebecca Lee Douglas, Ryan Kelly, Sarah Lilly, and Zach Lepinski. Our theme song is "Mr. Fortune" by The Hitchhikers. The rest of our music is composed by Luis Guerra. As always, thank you for listening.

• 48:05Adam Davidson?

  I don't know that ChatGPT would ever tell me, "By the way, don't do this. This is a dumb idea."

• 48:10Speaker 04

  Entrepreneurship is this alternating thing between like, "I'm the master of the universe." And, "Oh crap, this was terrible. I can't believe I did this."

• 48:21Speaker 08

  The Freakonomics Radio Network, the hidden side of everything.

• 48:29Speaker 01

  Stitcher.