GPT-5 Arrives, and We Try the New Alexa+

“Sam Altman in his remarks said that this is a major upgrade,” but he also said that “we’re not at A. G. I. yet.”

• 0:00Speaker 00

  Modern enterprise is made up of a lot of moving parts. And Comcast Business helps you orchestrate it all. With SD-WAN working at scale to keep 150 hospital locations connected and working as one. Plus, SASE and Zero Trust Security protecting financial data across a bank's 2,000 branches. And AI-powered networking that optimizes traffic across five continents. No one does business like Comcast Business.

• 0:30Speaker 01

  I had a first this week.

• 0:31Speaker 02

  What's that?

• 0:31Speaker 01

  I had my first experience with smelling salts.

• 0:35Speaker 02

  Wait, did you faint?

• 0:36Speaker 01

  Yes. So I had to get a blood draw at the doctor. And I am a big baby when it comes to getting blood taken. And so like half the time when I get blood taken, I pass out. And this time, I not only passed out, but I vomited.

• 0:54Speaker 02

  Oh, no.

• 0:55Speaker 01

  And had to be brought back with smelling salts, which

• 0:57Speaker 02

  Heaven!

• 0:58Speaker 01

  Casey, if you have never experienced smelling salts, they're not messing around.

• 1:04Speaker 02

  They are not. And I, okay, so I cannot believe that I'm just learning this information about you because I am also a fainter.

• 1:11Speaker 01

  You're a fainter!

• 1:12Speaker 02

  I am a fainter.

• 1:13Speaker 01

  We are legion.

• 1:14Speaker 02

  In 12th grade, we went to go see a cadaver for my AP Biology class. And intellectually, I was like so fascinated, you know, by all the systems of the body. And so me and all the other kids are standing around the cadaver. And the person is sort of explaining, well, you know, this is the liver and this is the spleen. And then I just got a whiff of something, I don't know if it was embalming fluid or formaldehyde or something. But it was like something triggered in my brain and it was like, this is against nature. Like, you should not be this close to an opened up dead body. And I truly, I spun around, I took a header off a whiteboard that was like, you know, against the wall. And I wake up and I'm staring at the ceiling. And the first thing I hear is my AP Bio teacher, Ms. Oliver, saying, "Do we have an emergency contact for this kid?"

• 1:58Speaker 02

  Obviously, like, I don't want to tell people that they should like faint, but it is one of the most amazing, crazy experiences you can have in the human. Like, do you know what I mean? The moment when you're like, your consciousness just sort of like leaves you

• 2:08Speaker 01

  Yes.

• 2:09Speaker 02

  crazy.

• 2:09Speaker 01

  And when I was brought back with the smelling salts, it felt very like Victorian, you know? It was like, I was like

• 2:15Speaker 02

  No, the vapors, the vapors!

• 2:19Speaker 01

  Call Mr. Darcy!

• 2:26Speaker 01

  I'm Kevin Roos, a tech columnist at The New York Times.

• 2:28Speaker 02

  I'm Casey Noon from Platformer.

• 2:30Speaker 01

  And this is Hard Fork.

• 2:31Speaker 02

  This week, give me five.

• 2:33Speaker 01

  GPT-5.

• 2:34Speaker 02

  We'll tell you all about OpenAI's latest frontier model. Then Kevin and I get access to the new Alexa Plus. We found a few minuses and we're bringing in Amazon's VP of Alexa to talk about it. Alexa, prepare my interview questions.

• 3:01Speaker 01

  Well, Casey, it has been another busy week in the world of AI.

• 3:04Speaker 02

  Boy, has it.

• 3:05Speaker 01

  And because we are going to talk about OpenAI, I should add my disclosure that The New York Times Company is suing them and Microsoft for copyright violations.

• 3:14Speaker 02

  And my boyfriend works at Anthropic.

• 3:16Speaker 01

  So we've gotten a bunch of new AI releases and announcements this week. We're not going to go through all of them, but some of the highlights. We got something called a world model from Google DeepMind. Genie 3 has this kind of interactive game engine where you can just sort of like describe a game that you want to play and it can sort of build it in real time. Pretty cool. We can't use that yet, so that was just a demo or research preview. But that was early in the week. And then we got a new Claude version. Opus 4.1 is out, so I've been playing around with that. Not too different, but sort of a newer update from them. We also got open source models from OpenAI, putting the open back in their name.

• 3:59Speaker 01

  They released two open source models this week. And Casey, have you played around with either of those?

• 4:03Speaker 02

  I have not yet downloaded them. Have you?

• 4:05Speaker 01

  I have not. One of them is apparently small enough that you can run it on a MacBook. Another one, you kind of need a dedicated GPU for it. But these are basically OpenAI's first open source models since GPT-2 many years ago. People have been hounding them. You guys are sort of betraying the founding spirit of OpenAI by not making these things open and accessible through open source. And they said, well, here you go. Here are some models. They're not their sort of top of the line. Line models, but people are sort of finding various uses for them. And this is sort of designed to compete with the open source models coming out of China with companies like DeepSeek.

• 4:44Speaker 02

  Yeah, and the early word on these is that they're pretty good and that they're competitive with the O3 Mini, O4 Mini, which are more proprietary models. But yeah, the early reviews I was reading of the open source models were that they were powerful and good.

• 4:56Speaker 01

  Yeah. So those are some of the announcements that we got earlier in the week. But the big one is that this week OpenAI released GPT-5, their long awaited flagship model. People have been asking Sam Altman about this, including us, for many months now. This was long awaited. There was lots of hype and rumors flying around about this. And we just got off a press briefing, a sort of Zoom call with Sam Altman and some of the other leaders of OpenAI. And Casey, what did we learn?

• 5:25Speaker 02

  Well, it probably won't surprise people to learn that what they told us during this briefing was that GPT-5 is their best model ever.

• 5:36Speaker 02

  You know, Sam Altman, in his remarks, said that this is a major upgrade. He called it a significant step along the path to AGI. But he also said that we're not at AGI yet, among other things. For example, he said, look, this model does not continuously learn. And in his view, AGI will continuously learn. So I actually thought it was cool that he said that because now we have sort of one thing to hang on to where, well, maybe when a model can do that, we'll feel like we really are getting close to AGI. But you know, the one other thing that he said that struck me that I thought was kind of funny was that he said that after they had sort of put GPT-5 together, he went back to using GPT-4. And he said, quote, "It was quite miserable." He said he never wants to go back to using GPT-4 ever again.

• 6:21Speaker 02

  That's how good he says GPT-5 is, Kevin.

• 6:23Speaker 01

  Yeah, so he sort of compared it to the previous models. He said GPT-3 felt like talking to a high school student. GPT-4 felt like talking to a college student. And GPT-5, he said, is the first time it feels like talking to an expert, someone, you know, who has a PhD in a subject. So I think we should caveat this all by saying that as of our taping this week. GPT five had not yet been rolled out and we hadn't been able to sort of put it through its paces. But it will be rolled out this week, including to free users of chat GPT, who have not previously had access to their top of the line models.

• 6:58Speaker 02

  Yeah, and I think that that's important because OpenAI's best models at the moment have been reserved for paying users. So the chatbot that I use the most is O3, which is a reasoning model that OpenAI makes. That's not accessible to people who are on the free plan. So I do think it's really notable now that even free users, which I think is going to include a lot of high school and college students out there, are now going to have access to what at least they are saying is PhD level intelligence and reasoning.

• 7:26Speaker 01

  Yeah, so one of the most annoying features of chat GPT for years now has been this model selector, where you go in and it sort of gives you this little dropdown menu and it defaults to GPT 4.0 now, but you can kind of pick your own model if you want something more powerful than that and you're a paying user. But for GPT five, OpenAI is doing away with the model picker, or at least making it sort of less necessary, because they have built this, what they called a router, that will essentially analyze your request and how much sort of computation it needs to be able to answer that request and whether it's a simple query or something more involved, and it will kind of direct it to the correct model.

• 8:07Speaker 01

  And I think for a lot of people, this is going to be their first experience with a reasoning model, because OpenAI does not make that the default right now in chat GPT. And so I think that will be a big update for people, regardless of whether GPT five is actually better than previous models, just the ability to kind of use these reasoning models for free seems like a pretty big deal.

• 8:27Speaker 02

  Yeah, getting rid of the model picker, I think, could cut both ways. And we should say that all of the big labs have a model picker, so Gemini has one, Anthropic has one, and Claude. And sometimes I will, you know, maybe ask an easy question of one of these models that is maybe set to reasoning mode, and then I'll think, oh, gosh, you know, I probably didn't need that much computational power. On the other hand, I do sort of feel like it sets up an incentive for OpenAI, which wants to sort of save as much money as it can to just always try to route you to the sort of the absolute least compute that you need. And so I will just be curious to see if I feel like that is affecting the quality of my experience now that I, you know, maybe can't go in and actually say, hey, let me use the good stuff.

• 9:09Speaker 01

  Yeah, so on this briefing, OpenAI said all the sort of expected things about how GPT-5 is, you know, better at everything than previous models. But they also spent a lot of time talking about what they called the vibes of the model, which they believe are quite good.

• 9:25Speaker 02

  They also gave a series of demos and one that I thought was interesting introduced this concept of what I believe Sam Altman called "software on demand." So that GPT-5 can instantaneously create a piece of software for you. In the demo that we saw, one of the employees there built a tool to let his girlfriend learn French, and it did this in some sort of fun ways. In one case, it sort of created a series of flashcards for her. In another, it created like a little snake game with like a mouse and a piece of cheese. So every time like the mouse caught a piece of cheese, it would like show her a new word to learn.

• 10:06Speaker 02

  And he was able to do all of that just via a text-based prompt, and it actually looked like pretty good. Yeah, I mean, I think, you know, five years ago, if you turned that in in an intro to computer programming class, you probably would have gotten an A.

• 10:18Speaker 01

  Yeah, it's pretty impressive, but those are also things that other models can do today. So I'm gonna need to like really drive this thing myself to figure out what it can do. And I'm gonna put it through my usual bevy of tests known as Rooster Bench and see how it does.

• 10:33Speaker 02

  Yeah.

• 10:34Speaker 01

  I confess, during this briefing, I zoned out a little bit. I've been to a bunch of these. Everyone says their model is the latest and greatest, and it's so good at coding, and it's all got all these agentic capabilities. And it all starts to sound a little bit like marketing hype to me. For me, the interesting question to ask about new models these days is not how much better is it, or how does it score on these benchmarks? It's like, what is possible for me now that wasn't before?

• 10:59Speaker 02

  Right.

• 10:59Speaker 01

  And I still don't have a really good answer to that from GPT-5, although I'm gonna investigate.

• 11:05Speaker 02

  Yeah, and you know, that might be a good point to bring up two of the questions that got asked of the GPT-5 team during the briefing that we're on that I think would be of interest to our listeners. One was, somebody asked, hey, like, are you starting to run into the limits here, right? Are the scaling laws holding? And Sam Altman said, quote, "They absolutely still hold, and we keep finding new dimensions to scale on." He said that they're still finding new paradigms that will let them scale in new ways. So he very much tried to give the impression that like, no, we are not struggling at all to figure out how to build better models.

• 11:44Speaker 02

  I suspect, though, that that might get some pushback as people start to use this thing and just sort of observe that like, yes, it is like clearly better in a handful of ways. But to your point, Kevin, can you really do anything that you couldn't do before? That doesn't like seem like it's the case. It's just sort of that it can do what it used to do a little bit better. I'm curious what you made of that.

• 12:01Speaker 01

  Yeah, I think that's reasonable. I mean, I wonder if they are starting to finesse their definitions of the scaling laws to sort of account for the new reasoning models. Because people, you know, for months now have been saying, well, the models in the pre-training phase may have gotten as good as they're going to get, or about as good as they're going to get. But the way to get them to be more intelligent is through this post-training phase, is through these reinforcement learning cycles, these sort of reasoning environments that they're trained on and put into. And so I suspect that when they say that the scaling laws have not broken, they're also sort of referring to this kind of reinforcement learning reasoning approach as well.

• 12:42Speaker 01

  And I believe them. I've talked to people who say, you know, they think there's still a long way to go on that. But this was a really big model. We don't know exactly how big. We don't know exactly how many GPUs it was trained on or how much data it was fed. But it's safe to assume that they did everything they could to sort of max out the scale of the model, at least in pre-training. And, you know, from what we saw in the demos, it doesn't look like it's like that much smarter. I mean, maybe it's a little better at some things, but it did not, you know, come out of the box, uh, super intelligent or anything like that.

• 13:12Speaker 02

  Yeah. The other big question that I'm always interested in when these big Models come out is what was the safety testing experience like? And is this model going to be sycophantic? And what sorts of very intense relationships are people going to form with it? And Nick Turley addressed that one. He noted that earlier this week, OpenAI put out a blog post, which actually wrote about in Platformer this week, that is all about their approach here. They say they're working with physicians, really trying to bring in a lot of outside expertise to help them understand how people are interacting with these models and make them safer. And he said that they are absolutely not optimizing for engagement here. They just want to sort of make a useful tool that sends you on your way. And that essentially they're going to have more to communicate about this soon.

• 13:56Speaker 02

  So we didn't get a ton of detail there, but they have said that at least in some ways, they think that they have improved these models to make them less sycophantic. In addition to that, they said they did 5,000 hours of red teaming. They shared, I believe, these models with some external experts for advice on that. So they did say that they rate this model as high on the scale of could it be used to create novel bio risks? So they're building in a bunch of protections around that. You know, but that doesn't seem great. But anyway, that was the sort of safety report that we got in advance of the launch.

• 14:32Speaker 01

  So they also said that in addition to all these new capabilities, GPT-5 is much more reliable than previous models. They claim it hallucinates less. And it does this interesting thing called safe completions, where basically if a model doesn't want to, you know, accommodate some requests or carry out some tasks because it's sort of against the guidelines, instead of just refusing it, it will sort of make up a safer version of the request and complete that instead. So it would be interesting to see how people use that. But yes, this is the claim they make. It's more reliable, less deceptive, and gives these kind of safe completions.

• 15:12Speaker 02

  Well, and that actually gets into something interesting though, Kevin, which is like, what is it that makes OpenAI say this is GPT-5? The the sort of like big number releases. This amazing marketing power now, I think because the leap from GPT two to GPT three was so big. And the leap from three to four was also pretty big. And so that creates a lot of expectations for five. But in the background, OpenAI is just trying a bunch of things, building a bunch of new models. And then sort of stapling various things together. And then eventually they get to something and they say, we're going to call this one five. But like, it's not quite as linear as it looks from the outside, right?

• 15:56Speaker 01

  Yes, and all of the labs have had experiences where they thought they were training like a a new model and then it didn't quite work out for as well as they wanted it to. And so they sort of assign it some lower number, you know, that that's happened at at least a number of big labs that I know about. It's happened at OpenAI, you know, they had a previous big model that they were building, which ended up becoming four point five, I believe was supposed to be GPT five at one point. And it just didn't turn out as well as they'd wanted it to. So yes, they are playing games with the numbering of the models and sort of the marketing around that. But I think calling this GPT five just signals that they they want this to be viewed as a similar step in capability that people saw from GPT three to GPT four.

• 16:37Speaker 02

  Yeah, to me, that is one of the most interesting things about this release is like whatever GPT five turns out to be. This is the thing that they thought was kind of the next big step forward. And I think we should like evaluate it on those grounds.

• 16:51Speaker 01

  I think it's also like the big picture here is that OpenAI is trying really, really hard to stay at the head of the pack. This is a company that has been sort of racing very hard toward AGI or something that they can claim is AGI. And they are, they are still going. I find that actually their execution to be quite impressive. Like this is now a very large company. They've got a lot of different competing teams and priorities. And they had all the this board drama. And I think, you know, it was reasonable to expect, and I certainly expected that in the wake of all that they would sort of slow down and maybe allow some competitors to catch up. But they showed this week that they are not slowing down, that they are.

• 17:35Speaker 01

  Are in fact accelerating and they want to get here before anyone else.

• 17:38Speaker 02

  True, although they have also experienced a lot of poaching in recent weeks and months. And I think one thing I'll have my eye on over the next several months is are they able to continue iterating very quickly or have some of the losses that they've experienced over the past few weeks really hurt them.

• 17:57Speaker 01

  Yeah.

• 17:57Speaker 02

  You know, incidentally, Kevin, I'm told that in response to the GPT-5 launch, inside Meta headquarters, the superintelligence researchers have moved their desks even closer to Mark Zuckerberg. So that's how seriously they're starting to take this over there.

• 18:10Speaker 01

  They are now sitting on top of Mark Zuckerberg.

• 18:12Speaker 02

  There are now two researchers who are sitting at Mark Zuckerberg's desk with him. And we'll have to see how that plays out.

• 18:19Speaker 01

  Yeah.

• 18:19Speaker 02

  Yeah.

• 18:20Speaker 01

  Those are some of our initial impressions, but we are going to actually come back tomorrow after we have had a little time to play with the model and give some first impressions there, too.

• 18:31Speaker 02

  Let's travel to the future now, Kevin.

• 18:34Speaker 01

  All right, Casey, it is now Thursday. GPT-5 has been officially released for a few hours. I still do not have access to it for some reason, but I gather that you do. So give me your day one vibe check. What are you seeing? What is GPT-5 like? And what do you make of the reaction to it?

• 18:52Speaker 02

  Well, this is a very significant moment in the history of hard fork, Kevin, because for the first time I'm having a conversation with you while vibe coding something.

• 19:02Speaker 01

  What are you vibe coding?

• 19:03Speaker 02

  Well, ChatGPT is currently hard at work building a to-do list app for me. I said I wanted it to have the aesthetic of the Fantastic Four First Steps movie that just came out. I didn't love the movie, but I did love the production design. So I was like, make me a to-do list app that looks like that. Let's see how it goes.

• 19:21Speaker 01

  I can't believe we're building giant gigawatt data centers for your stupid to-do apps. This is so wasteful. God.

• 19:30Speaker 02

  Listen. You need to send over Roost Bench, your proprietary suite of evals, so I could really put this thing through its paces. But look, let me, let me give you some high level notes, Kevin, on what I'm seeing and on what others are seeing. The headline here is that this does seem like a really meaningful improvement to ChatGPT. And I think in particular, if you are a free user of ChatGPT, you're going to have a great day, right? Because for the first time now, in addition to the kind of, you know, standard ChatGPT model, you're going to have some reasoning capabilities. So essentially, if you're cheating your way through high school, you're just going to have a lot easier time of it now because this thing can do some really extended work on long problems.

• 20:15Speaker 01

  Yes, you can now cheat your way through an entire semester with just one press of a button.

• 20:20Speaker 02

  Exactly.

• 20:21Speaker 01

  This is good. Yeah, I mean, the way I saw people talking about it online was that they thought that OpenAI had not raised the ceiling of the AI frontier by a lot with GPT-5, but they had raised the floor. Essentially, now all the free users who previously got defaulted into the less powerful models are now going to be using the more powerful models, which could be a big perceptual shift, if not a sort of one about the frontier capabilities.

• 20:47Speaker 02

  For sure. And I, and I do think it does have some things that are not quite capabilities, but still will meaningfully affect how people use these AI systems. For example, this thing really is just a lot faster than its predecessor. So just over the past couple of hours, I took some editing work that I sometimes ask ChatGPT to do. I know about how long it takes using the O3 model. I put it through GPT-5 and sure enough, yeah, it blazed through it. It did just as good of a job as it had done before. And so if you're the sort of person who's using ChatGPT a lot, I think that's really going to stand out to you.

• 21:22Speaker 01

  Yeah. What about the pricing? I saw some people saying that GPT-5 was much cheaper than they expected it to be. That would not for cheaper for the sort of ChatGPT subscriber. Those subscription prices are staying the same. But for developers who are building on top of it, my understanding is it's a lot cheaper than other models from other AI labs.

• 21:39Speaker 02

  That's right. It came in at $1.25 per 1 million input tokens, which is the same as Google's Gemini 2.5 Pro. Google has, of course, also been pricing really aggressively to try to box out the competition. What makes that figure interesting, I think, Kevin, is that that number is a lot smaller than Anthropic's Claude for OPUS API, which comes in at $15 per million input token. So I think some of these really well-capitalized AI labs are taking this moment to say, hey, we're going to put a lot of pricing pressure on some of our competitors.

• 22:15Speaker 01

  Yeah, it very much reminds me of the moment like 10 years ago when like Ubers were $4 because the venture capitalists were just subsidizing the artificially cheap prices. We're in sort of that moment for AI tokens now.

• 22:29Speaker 02

  Yeah.

• 22:29Speaker 01

  What else can we say about GPT-5 in the couple of hours now that it has been out?

• 22:34Speaker 02

  Yeah, well, you know, I sometimes like to joke that the worst insult you can make to anyone who has just released a new AI model is my timelines are now longer. And it does seem like that is something that people are saying about the new GPT-5. And what I slash they would mean by that is I now think it's going to take a little bit longer until we reach AGI, some sort of very, very powerful AI system. In fact, some people are posting online screenshots of some prediction markets that until today, when asked, who do you think will have the most powerful AI model at the end of August? We're showing OpenAI in the lead. And almost instantaneously after the live stream on Thursday,

• 23:18Speaker 02

  OpenAI collapsed and Google is now ascended and is assumed to have the best model by the end of this month. So, you know, I don't want to overstate what that means necessarily, but it does seem like there was a huge contingent of people who thought. That GPT-5 was going to be this revolutionary new model. And it seems instead like a more evolutionary one.

• 23:39Speaker 01

  Yeah, that makes a lot of sense to me. One other thing that stuck out to me, and I wonder if it stuck out to you too, was OpenAI released some benchmarks and some data about GPT-5. And one of the things they showed was that hallucinations, the rate of GPT-5 just making stuff up while answering questions, has gone way down. It's now, you know, for some types of questions, it's sort of around 1% hallucination rate. And I think that was interesting to me because this is clearly something that was a problem with earlier versions of this. In fact, there was some speculation and some indication that actually newer, these newer reasoning models were hallucinating at higher rates than the sort of previous generation of models. And there was a lot of concern about that. And it seems like they have figured out a way

• 24:24Speaker 01

  to get the hallucinations under control with GPT-5. Although with everything, I don't totally trust these benchmarks. I'm going to have to see this for myself.

• 24:32Speaker 02

  Yeah, everyone's mileage is going to vary on this one. I will say I have already caught it hallucinating a couple of times, like somewhat disappointingly. So as always, don't trust these things for anything mission critical. You're always going to want to double check your facts.

• 24:49Speaker 01

  Yeah, only use it to build stupid to-do apps with the Fantastic Four aesthetic on them.

• 24:56Speaker 02

  The Fantastic Four have a very cool aesthetic, and I think you need to open up your mind a little bit.

• 25:00Speaker 01

  OK, that is our day one vibe check of GPT-5. And we will continue to play around with this and tell you anything cool or interesting or strange or upsetting that we find.

• 25:10Speaker 02

  Sounds good.

• 25:14Speaker 01

  All right, that's enough about GPT-5. When we come back, we'll talk about another AI system we got our hands on this week, Alexa Plus.

• 25:35Speaker 03

  Support for this podcast comes from Dell Technologies. It's time to put AI to work. Is your business ready? The challenge isn't getting started with AI. It's scaling beyond pilots while maintaining control of your data, governance, infrastructure, and costs. Dell AI Factory provides the integrated foundation you need to scale AI securely and efficiently with one trusted partner. Dell, AI on your terms. From idea to impact. Learn more at dell.com/yourwaytoAI. That's dell.com/yourwaytoAI.

• 26:09Speaker 04

  Building and scaling AI today can feel pretty chaotic, mostly because teams are juggling too many fragmented tools at once. That's where OutSystems comes in. OutSystems gives you one unified Agentic Systems platform to manage your entire portfolio of agents, apps, and workflows. You get to innovate at the speed of AI, but with the security, governance, and control your enterprise actually needs. And the results are already there. A major manufacturer used OutSystems to build a multi-agent system that analyzes blueprints, and a global bank cut their development time by 60%. Own your agentic future today at outsystems.com/nyt.

• 26:45Speaker 05

  Your sales team is at risk of missing quota. Don't panic. Just ask Rippling AI. Built on your real-time people and business data, Rippling AI can pull metrics from Rippling and your CRM into a meeting-ready dashboard showing quota attainment, headcount trajectory, and revenue to quota by region. In seconds, you'll see what's behind your quota risk and fix it before it's missed. Head to rippling.ai/hardfork to get the only AI built to give you full visibility and take complex actions across your whole business. That's r-i-p-p-l-i-n-g dot ai slash hardfork. Sign up today.

• 27:20Speaker 01

  Now Casey, are you an Alexa user?

• 27:23Speaker 02

  I have been an Alexa user for a long time. I still have one of the original Amazon Echos in my house. And to Amazon's credit, it still works.

• 27:31Speaker 01

  The Pringles can.

• 27:32Speaker 02

  Yeah, I have the big ol' sort of Pringles can Echo.

• 27:34Speaker 01

  Yeah, me too. So I am a heavy user of this product. I have probably five of them in my house, in various rooms. And so I'm very excited for our conversation today, which is going to be about the new AI-ified Alexa Plus. And before we get into our experiences using this thing and our interview with the guy who runs it, we should make a couple disclosures. One of them is that the New York Times company has recently agreed to a licensing deal with Amazon that will allow Amazon access to Times content for its AI platforms, including Alexa. So we just thought you should know that. We have nothing to do with that, obviously, but that is going on in the background in another part of the company. The second thing we should say is that if you have

• 28:17Speaker 01

  an Alexa device, it is going to be going off constantly during this segment unless you go over right now and hit the little button that mutes it. So sorry in advance to Alexa owners, but we'll give you a little bit of time right now to pause this, go over, hit the mute button on your Alexa and come back.

• 28:33Speaker 02

  Or alternatively, just find the circuit breaker in whatever house you're in right now. You just shut them all off. Run on battery power for the rest of this episode.

• 28:40Speaker 01

  Alexa, order 14 bags of dog food. I wonder if that actually works.

• 28:46Speaker 02

  Wait, and I should also probably disclose that my boyfriend works for Anthropic because I'm pretty sure that Anthropic is providing APIs that are being used in Alexa Plus.

• 28:54Speaker 01

  Wow, we've got so many disclosures today.

• 28:55Speaker 02

  Yeah.

• 28:56Speaker 01

  Okay.

• 28:56Speaker 02

  All right.

• 28:57Speaker 01

  Let's get started.

• 28:58Speaker 02

  Okay.

• 28:59Speaker 01

  Alexa, Alexa is one of the most puzzling technology products that I have ever encountered. And like you, I have been an Alexa user since the very early days. People don't realize, this is, this product was released in 2014. Alexa is 11 years old.

• 29:15Speaker 02

  Yeah.

• 29:15Speaker 01

  And when it came out, I was very excited. I thought, I'm going to put this smart speaker in my house and I'm gonna, you know, ask it to do things for me and it's going to be like having a little assistant right there on my kitchen counter. And Alexa has added

• 29:30Speaker 01

  Dozens of features, maybe hundreds of features since 2014. And I use zero of them because the three things that I use Alexa for are setting timers, choosing music to play in my house and telling me the weather before I leave for the day.

• 29:44Speaker 02

  Absolutely.

• 29:45Speaker 01

  Are those like similar to what you use this for?

• 29:46Speaker 02

  Those are the exact three things that I use Alexa for. Have I tried to use it for other things? Yes, but the experience frankly has just never been that great. So I always come back to those three.

• 29:55Speaker 01

  Yes, those are the big three in my house, same use cases, same limitations. But when generative AI started to get good a couple of years ago, I think people naturally started to ask, well, when is Alexa going to start using this new generative AI technology? It's sort of built on this older, more deterministic kind of system. But it seemed like a natural thing to expect that Alexa would start to incorporate some of this technology to be able to answer maybe more open ended questions, to give longer, more detailed responses, to do more than just set timers and tell you the weather.

• 30:28Speaker 02

  Yeah, I mean, once OpenAI released voice mode for ChatGPT, it just immediately seems so much more interesting and powerful than Alexa and Siri, which is Apple's very similar system that it makes for its devices. And so, yeah, I think both of us were like, okay, well, when are we going to get that OpenAI style voice mode in these smart devices that we have in our homes?

• 30:47Speaker 01

  Yeah, and so it's taken a while. We should say that like it has not been a smooth or a simple process. And part of what I'm so excited to talk with Daniel Roush, the VP of Alexa about later in this show is just why it's been so hard to sort of shove an LLM based generative AI technology into this pre-existing assistant product. But we should just talk briefly about what Alexa Plus is and then our experiences with it, because both you and I have gotten to try this over the past few days.

• 31:13Speaker 02

  Yeah, so Kevin, tell us a little bit about Alexa Plus.

• 31:17Speaker 01

  So Alexa Plus is the name for the most recent overhaul of the Alexa virtual assistant. It's powered by generative AI. We don't know exactly which model or models, but it seems to be sort of a mix of Amazon's proprietary AI models. And then maybe some of Claude, which it has a deal with Anthropic. And so it's been using Claude inside of its AI product. For a number of months now and Amazon claims that the new Alexa plus is able to do much more. It's able to be much more conversational, more personalized. It can do things like book reservations at a restaurant or order you an Uber. It can answer questions that sort of aren't just pure lookups where you're looking for, you know, what time is the baseball game tonight?

• 32:01Speaker 01

  It can actually do more complex things for you. It can control the smart devices and appliances in your house and it can purchase things for you online. So this new Alexa plus is not out to everyone yet. They've been rolling it out slowly. They are now in what they call the early access period. But we were able to get this on some new devices that we ordered. It also doesn't work on every kind of Echo device. You have to have sort of one of the newer ones to be able to run it.

• 32:29Speaker 02

  And Kevin, when you say that this has been rolling out slowly, like it has been rolling out extremely slowly. Like it was only on June 23rd that Amazon said that 1 million people had Alexa plus across presumably hundreds of millions of Echo devices out there.

• 32:45Speaker 01

  Yeah, so you and I both got the new Echoes that can run the Alexa plus early access program and turned it on and set it up. And a few things stick out to me right away. One is the voice on this new Alexa is just way better than the old Alexa.

• 33:01Speaker 02

  I would agree with that.

• 33:02Speaker 01

  It is way more fluid. It sort of sounds like more like something you hear out of chat GPT voice mode. They have managed to sort of overhaul the actual voice part of the voice assistant. So it sounds much more like a human.

• 33:14Speaker 02

  And there are a bunch of different voices. I think I saw eight of them inside the app. Half of them are masculine, half of them are feminine. So yeah, you can change that to your liking.

• 33:23Speaker 01

  Yeah. The other big difference I noticed right away is that the new Alexa plus does not require you to say the wake word like Alexa between every question and answer pair, right? With the old Alexa, if you wanted to ask a follow up question, you had to say Alexa again. With the new one, you can just kind of leave it and it will sort of intuit or pick up that you have a follow up question and it will sort of listen for a while longer. So you can actually have these more extended. Multi-turn conversations.

• 33:51Speaker 02

  Yeah, and that lets it do different kinds of things. Like, one of the first things that I did with Alexa Plus was it said, "Hey, would you like to try to solve a riddle?" And I thought, "What are you, the Sphinx?" But I said, "Well, sure. What the heck." And, you know, it gave me a series of clues. And within, you know, three clues, Kevin, I was actually able to solve the riddle.

• 34:10Speaker 01

  Wow, good for you.

• 34:12Speaker 02

  Yeah, I felt really smart.

• 34:12Speaker 01

  I'm so proud of you.

• 34:14Speaker 02

  Thank you. Thank you so much.

• 34:16Speaker 02

  So, yeah, what else were you doing with this thing?

• 34:18Speaker 01

  So, another thing it can do is just give you longer answers. Like, the original Alexa was sort of limited to a sentence or two. Maybe you could ask it to look something up on Wikipedia, and it would sort of spit out a few sentences. But it was really limited beyond that. But I can now ask it to make up a story and read it to my kids. So, we had some fun doing that the other night as a family. You can ask it to suggest a recipe for dinner based on, you know, what's in your fridge, and it will sort of help you with that. I used that last night. So, these are some of the new features that I was excited to try. I also tried some of their integrations, like, they have an integration with OpenTable and with Uber and a bunch of other companies.

• 34:55Speaker 02

  Tell me about this, because I set this up, but I did not actually use it. So, how did that work?

• 34:59Speaker 01

  So, basically, you scan a little QR code on your phone and link your Uber account or your OpenTable account to your Alexa account. Takes, you know, a minute or so. And then you can just kind of say, like, "Order me an Uber from this place to this place." Or, "I want a table at a restaurant in downtown San Francisco, near the Ferry Building, for two people at 6:30 tomorrow." And it will sort of pull up a couple of options, and you choose what you want, and then it can go book the table for you. So, that, I thought, was cool.

• 35:27Speaker 02

  And that actually worked when you tried it.

• 35:28Speaker 01

  So, I did not actually follow through with the booking, but I did order an Uber for myself, and it did work.

• 35:33Speaker 02

  Okay, cool.

• 35:34Speaker 01

  Yeah.

• 35:34Speaker 02

  I mean, that actually seems like truly useful. Just be like, say to your thing on your desk, "Hey, I need an Uber to the airport," and it pulls up one. That's great.

• 35:42Speaker 01

  Yeah, and it can do other cool sort of multi-step things, too. Like, I was able to say I needed a new thing for my kitchen, like a box grater, and I was able to kind of go to Alexa and say, "Hey, look up on Wirecutter what the best-rated box grater is," and add it to my Amazon cart.

• 35:58Speaker 02

  Now, can I guess why you needed a new box grater?

• 36:00Speaker 01

  Why's that?

• 36:00Speaker 02

  You used it to grate ginger, and it dulled the edges.

• 36:03Speaker 01

  No.

• 36:03Speaker 02

  Okay, what was the reason?

• 36:04Speaker 01

  I left it in an Airbnb.

• 36:05Speaker 02

  Okay.

• 36:08Speaker 02

  I should have seen that coming. Anyways, go ahead.

• 36:09Speaker 01

  So anyway, those were some of the good things about this product, but we have to talk about some of the limitations as well. Casey, what was your experience with Alexa Plus?

• 36:19Speaker 02

  Okay, so I have to say, I did not have a good experience with this thing. First of all, I bought an Echo Show 5. There's a big banner on the, on the page that says it works with Alexa Plus. And so the thing shows up at my house. And basically what I've come to understand is that an Echo Show is a device that just constantly invites you to spend money with Amazon. And I found it honestly infuriating because I plugged this thing in. And when you set it up, it's like, what, you know, what kind of background do you want? I was like, show me some art, you know, that's one of the options. And I would say for about four seconds per minute, it would show me, you know, some Renaissance, you know, masterpiece or something. And then it would be like, hey, do you want aspirin? You want paper towels? You want to buy paper towels? You can actually buy paper towels right now.

• 37:04Speaker 02

  Just say, hey Alexa, buy paper towels. And it was just sort of this like forever. So I eventually just unplugged the thing because I was like, why did I just spend $90 to have a permanent rotating advertisement for household products like on my desk? That is so weird. So I was like, it just put such a bad taste in my mouth about the whole thing. And then a day later, I get the Echo Show 15, okay? And for some reason, Amazon sent me two of them. I truly don't know why I did not need two of them. And so I unbox the thing and the thing is meant to be mounted on a wall. Now, there are a lot of things I'm willing to do for a podcast, but mount an Echo Show on my wall?

• 37:46Speaker 01

  You're not willing to do a construction project?

• 37:48Speaker 02

  It's not one of that. No, I was not going to do that. And so I just, and also the thing like can't stand up on its own. So I just had like a 15 inch screen sitting on my desk for a day while I'm talking to it. And it's like, this whole thing is like very silly. So that's like the hardware side of it, okay? You may have a better experience because, you know, I don't know. You like mounting things to your wall. And so you did that. And so, you know, you're having a good time. But that was just kind of all of the. Um, you know, the, the, the precursor steps I needed to take to even be able to like engage with this thing. And so then I, you know, I finally have it set up and I start to try to put it through its paces.

• 38:27Speaker 02

  So I, you know, go through the little riddle game and I, uh, you know, it's like, uh, hey, like I could help you like to a personalized meal plan. It's all right, great. But yeah, you've set me up with a personalized meal plan. And, you know, it's like, well, you know, we could do this or that. And it showed, showed me like a row of, um, of like recipes that it could cook for me. And so I swiped through with my finger and I see a lemon pasta. And I said, okay, show me the lemon pasta. And it says, sorry, I didn't get that. And I said, Alexa, uh, the, the lemon pasta right there, could you make me this lemon pasta from this website that you're showing me right now? Dead silence. And I was like, oh my God.

• 39:07Speaker 02

  And it's like this, like, and right here, we have just landed in the exact spot that has been bedeviling Apple for the last year. That is bedeviling Alexa right now. These systems are just very hard to make reliable. Now, I will say, I, the, the device was sort of having trouble connecting to my internet. Everything else in my house was connected to the internet. It was working fine. But like I, this was just sort of every once in a while being like, you're not connected to the internet. So was that an issue with me? Was that an issue with the hardware? I'm not totally sure. Maybe that was why it wasn't able to perfectly answer my question. I, you know, I do want to say that in case this was not actually an AI issue. But oh man, I, within five minutes, I was like, get this thing out of my house.

• 39:49Speaker 02

  Like true. And like, and I, and again, I wanted to like, I was excited about it. And after five, and after like two days of ads for paper towels and one day of, I'm not going to show you the lemon pasta. I thought, what am I doing with my life?

• 40:00Speaker 01

  Yeah. I, I should say like, I have also had a bunch of like very bizarre and frustrating experiences with this thing. Okay. So, uh, we've said what we like about this thing.

• 40:10Speaker 02

  Yeah. Which remind me what that is again?

• 40:12Speaker 01

  Many of the new capabilities are quite cool.

• 40:14Speaker 02

  Yeah.

• 40:15Speaker 01

  Unfortunately, many of the old capabilities I relied on as the reason I used Alexa at all have become broken as a result of this update.

• 40:23Speaker 02

  Okay. So tell me about this.

• 40:24Speaker 01

  So one of the things you also noticed very quickly. Quickly when you're using this thing is that the latency is just like a problem. It's a little slow to respond to questions. It's not as zippy as the old sort of pre LLM Alexa. I understand that these things have to go to the cloud. They're processing more complex instructions. It's all going to take a little time. I assume that will get better. The basic things that it gets wrong now include alarms, which is actually a thing that I use Alexa for every day. Wait, so tell me how it got it wrong. So the new Alexa Plus update seems to have broken Alexa's ability to reliably set and cancel alarms, which is a core thing that I use this product for.

• 41:08Speaker 01

  And so, for example, this morning I woke up on my own a little bit earlier than my alarm, like 10 minutes before my alarm was supposed to go off. And so I said to Alexa, Alexa, cancel the alarm. Silence, nothing. This is a command that I have issued probably 1,000 times. And Alexa Plus is a little smarter now and she's giving you the cold shoulder. Yes, and she's saying, actually, I'm going to wake you up anyway in 10 minutes. So that was not good. I also experienced some like hallucinations when I would ask it questions about like things happening in the world, things happening in the news. I asked it about a tennis tournament that's going on right now. I said, who's the top seed in this tennis tournament?

• 41:51Speaker 01

  It gave me the name of a player who's like not even playing in this tournament.

• 41:56Speaker 01

  And it also like has trouble orchestrating the different tasks. So one of the things that would happen is I would like tell it to, I gave it a research project for a dinner playlist. I was looking for some new, new music to put on our dinner playlist. And instead of doing that research project, it just started searching on Spotify. Like it routed the query to Spotify within the Alexa thing and started playing the music. When what I asked was like, do some research for me. So it seems to have a little trouble figuring out like what exactly the user wants and like orchestrating the commands. That case seems like a little borderline to me. I can imagine some people.

• 42:37Speaker 02

  Like asking for that, you know, maybe being happy if it like played some music. But I had this almost opposite issue where again, as I'm sort of going through, okay, what can this thing actually do? And it's, you know, and it's like, you know, ask me what I can do. So I asked it and one of the things it said was, um, I can help you explore Gen Z music trends, which there was just like something funny about the way it said it to me. So I was like, yeah, sure. Why don't you help me explore Gen Z music trends? And, uh, you know, thanks for a second. And then it goes, well, I found some podcasts about it on Amazon Music. And I was like, I sort of assumed you were either going to tell me something about Gen Z music or you were going to play Gen Z music. But now you're trying to sell me Amazon Music, which I feel like is very consistent with how Alexa Plus handles everything,

• 43:20Speaker 02

  which is could we sell you a service right now? Could we sell you a product? So, you know, Kevin, I want to say two things. One is I have not used this product all that long. And so I don't want people to think about anything I'm saying as anything other than first impressions. Like I have not truly had a chance to do the amount of reviewing that I would like to do. Two, I'm very confident that lots of other people are probably having much better experiences with this thing. Cause I think if most people were having experiences bad as me, I just would have heard about this before now. But all of that said, Alexa Plus did not make a great first impression on me. And the Echo family of devices that are just little windows that let you send money to amazon.com, they're not for me.

• 44:01Speaker 01

  Yeah, I had, I think a slightly more positive experience than you. I did actually enjoy some of my, uh, interactions with Alexa Plus, but it just seems like it is not quite there yet. And I think Amazon knows this, which is why it's in this early access program. If you open it up, it says, you know, Alexa may make mistakes. So they're sort of like doing all of the careful rollout that you would expect from a product that is not fully baked, but like some of the features just don't seem to work. There's another feature that I tried where you can like email, uh, a document to this email address and it will sort of ingest it into your Alexa and then you can have it summarize it. So I was very excited. I was like, I can like, you know, I can learn about new papers in AI while I'm like doing the dishes.

• 44:44Speaker 01

  And so I emailed the paper to the Alexa email address and I say summarize the paper I just sent you. And it says, I did not receive a document.

• 44:54Speaker 01

  I'm just like, I think they need to spend a little more time in the kitchen cooking this one, but I think the the overall, my overall impression is the like, the Alexa Plus that you have now in this early access program, it is a little like having a kind of GPT 3.5 class model inside of a smart speaker, which I think is a valuable thing and one that I would like them to continue to build on. But it is not the state of the art in either the language model or the kind of basic tasks. And actually, it seems to be regressing on some of the basic tasks. So I would say this is like two step forward, one step back.

• 45:31Speaker 02

  I think the most powerful thing that the new Alexa Plus has done for me is it has made me forgive Apple for not shipping anything with the new Siri. I get it now, Apple. I talked a lot of mess about you on this podcast about not shipping this thing, but now having used one of your close rivals attempt to do the same thing that you're doing, like, I get it now. I think the finest minds in the world who are working on this stuff actually don't know how to do this yet. That's my big takeaway.

• 45:59Speaker 01

  Yeah, I think what's happening with Alexa and Siri right now is sort of a, it's like a symbol of what's happening in the American economy writ large, which is like, we are trying to jam these like new AI technologies into these legacy systems and processes. And it's just kind of a messy fit. Like, these things are weird. They are not deterministic. They are not reliable in the ways that like a sort of older, more rule based thing could be. And they have these amazing capabilities. But when you try to like make these hybrid Frankenstein things with like the old system with the new brain, it just doesn't really work. And I think that's like happening, not just at these virtual assistants, but

• 46:44Speaker 01

  like in a lot of places throughout the economy.

• 46:46Speaker 02

  Absolutely. I also just think that when I'm using a chatbot on my laptop and it gives me something that's like 80 or 85% right, that's much more useful to me than like an Alexa response that's 85% right. Because on a chat, like in a chatbot. Setting, I can just sort of take what I need. I can edit or modify it. I can maybe ask the same question of another chat bot and see if I got a slightly different or better result. Like, I feel much more in control of my own destiny. I can take the stuff that works and leave behind the stuff that doesn't. When you're doing this thing with a smart speaker, if it doesn't work, you say, yeah, why did I spend 90 bucks on this piece of junk? You know? And it's like, and, and I think I, what I learned about myself was I have so much

• 47:31Speaker 02

  less patience for this sort of thing. When it is a piece of hardware in my home that has made some really big promises about how it's going to help me with all my routines and everything. Well, if it's kind of hard to set up and it doesn't work like the vast majority of the time, it all just kind of feels like a waste.

• 47:47Speaker 01

  Right. So I'm, I'm very glad we got to exchange first impressions of Alexa Plus. We should also have a conversation with someone at Amazon who has been involved in this overhaul of their flagship voice assistant. So when we come back, we're going to be joined in the studio by Daniel Rousch. He's the Vice President of Alexa and Echo at Amazon, and we're going to get into all of this with him. And you can ask about those ads. Oh, I'm gonna. But first, some ads.

• 48:33Speaker 03

  Support for this podcast comes from Dell Technologies. It's time to put AI to work. Is your business ready? The challenge isn't getting started with AI. It's scaling beyond pilots while maintaining control of your data, governance, infrastructure, and costs. Dell AI Factory provides the integrated foundation you need to scale AI securely and efficiently with one trusted partner. Dell, AI on your terms. From idea to impact. Learn more at dell.com/yourwaytoAI. That's dell.com/yourwaytoAI.

• 49:07Speaker 04

  Building and scaling AI today can feel pretty chaotic, mostly because teams are juggling too many fragmented tools at once. That's where OutSystems comes in. OutSystems gives you one unified agentic systems platform to manage your entire portfolio of agents, apps, and workflows. You get to innovate at the speed of AI, but with the security, governance, and control your enterprise actually needs. And the results are already there. A major manufacturer used OutSystems to build a multi-agent system that analyzes blueprints. And a global bank cut their development time by 60%. Own your agentic future today at outsystems.com/NYT.

• 49:43Speaker 06

  Today's AI builders need infrastructure that scales with their ambitions. Irene is a vertically integrated AI cloud platform, delivering data centers, compute, and software for AI training and inference, underpinned by its expansive portfolio of land and grid-connected power in renewable-rich regions globally. Irene's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. Irene, AI cloud for the next big thing.

• 50:24Speaker 01

  Daniel Rausch, welcome to Hard Fork.

• 50:26Speaker 07

  Thanks so much for having me.

• 50:27Speaker 01

  So Casey and I have both spent the past few days playing around with the new Alexa Plus. And I'd like to just start by asking about the technology that powers this thing.

• 50:37Speaker 07

  Yeah.

• 50:38Speaker 01

  How much of it is a new LLM-based system versus the old, more deterministic model that powered the old Alexa?

• 50:46Speaker 07

  Yeah, well, from an AI and model perspective, everything is entirely new. There are some legacy deterministic systems downstream, but really, it's a complete rearchitecture of everything that you would say Alexa is, from the way you have a conversation and engage with the experience at a very basic level, you know, all the way through Alexa acknowledging you or just maintaining a chat. So. There's a lot of new under the hood.

• 51:11Speaker 02

  Yeah. Talk about the challenge of moving to the, from this deterministic system to something that is very powerful, but also much less reliable.

• 51:23Speaker 07

  Yeah, I would say, well, hopefully you're not seeing much less reliable. I would say, you know, we've got some edges to sand and we're in early access. I'm sure we'll get to talk about the nature of the rollout, but.

• 51:33Speaker 02

  I just mean like in general, LLMs are not as reliable as a deterministic system.

• 51:37Speaker 07

  I get it. So, you know, we want to capture all the benefits of that non-deterministic, we'd call it stochastic system in the space. It has the elegance of really engaging in human conversation, but we want the predictable outcomes. Now, large language models don't support interfaces out of the box to classic systems. So getting those capabilities to interface, we would talk about it as APIs across these interfaces for other systems. It's quite hard. They speak natural language. APIs don't speak natural language. They speak clunky computer science language, but it's very predictable and it gets a lot of things done. So I would say, if you had to list the technical challenges, you would say, well, the many millions of things, we stopped counting at some point, but the many millions

• 52:21Speaker 07

  of things that, you know, original Alexa could do, marrying that with the power of LLMs is definitely the first and most prominent on the list.

• 52:29Speaker 02

  So take us back to when LLMs are first coming out. You guys are starting to play around with them. It's sparking ideas for you of, gosh, if we could like marry this to Alexa, we could have something really cool. What are some of the uses that you're thinking about? Like, what are, what are the kind of dreams that you have for this model that you're hoping you can bring into reality?

• 52:51Speaker 07

  I mean, I think it's, we think of the capabilities in two buckets, I would say. Take everything that Alexa, the original Alexa can do and just make it way better. You know, just picking up from what customers are already doing with Alexa. Then you start brainstorming and where I think you were really headed was what are all the new things that we can do? And the depth of conversation that you can have with the new Alexa experience just opens whole vistas of new kinds of things we can get done. We can help you plan a trip and then follow through on it. We can watch for concert tickets for you. We can, you know, not just help you brainstorm about cuisine, but either.

• 53:29Speaker 07

  Pick a recipe and get some groceries and invite the neighbors or let your partner know it's date night and we're going out and book a table. So I think the kinds of journeys and the kinds of tasks we can get done for customers are just so much more expansive.

• 53:42Speaker 01

  So Casey and I have spent the past couple of days trying out Alexa Plus and we have some feedback which we can share with you now or later. We've talked about it on the show just before this. I think it's fair to say we both had some things that impressed us about the new Alexa Plus and some things that were challenging, including some of the basic stuff that Alexa seemed to be very good at before, or at least that I know how to get reliable performance out of Alexa for that no longer seemed to work as well. But what I actually want to know is like, why has it been so hard to do this? Because back in 2023, when Amazon announced that it was going to revamp Alexa, sort of give it this brain upgrade with these new AI capabilities, they said this was going to

• 54:24Speaker 01

  be ready in 2024. And then that got pushed back a couple different times. So walk us through kind of the the journey that you all have been on over there trying to sort of shoehorn this new technology into this existing products and maybe some of the challenges that you encountered along the way.

• 54:40Speaker 07

  I'll tell you, you know, we should definitely get some of the feedback. We can cover as much as you like here on the show. So if you rewind the tape, actually, you were asking about this too, as we're starting to experiment, what can we imagine doing? If you go back to 2023 and the models that were available, then the state of the art, very little instruction following reasoning, low ability to execute on these interfaces with other systems, we announced something called Let's Chat, which was sort of a mode of Alexa. So think about, you know, flipping a switch on Alexa and turning on a chat interface so that you can do some basic question and answer and have a discussion about a topic, mostly about

• 55:21Speaker 07

  knowledge native to the model's training data versus bringing something in at runtime the way that, you know, modern chatbots answer questions by going out on the internet. I think what we mostly learned from that announcement and the customers that we rolled out to is that we just had to increase our vision and do something more audacious, basically. Customers really wanted, and we all really wanted, to pick up from where Alexa is and was and extend all of those capabilities. That is many millions of things that Alexa can do. And when you count the tens of thousands of services and devices that are integrated with Alexa and the space of the interfaces and the systems that you need to integrate

• 56:03Speaker 07

  with, it's incredibly large. That's the first sort of technical challenge I mentioned before. It's sort of the first and probably most important bucket. Second is really grounding it in authoritative sources. I think, as all of us know, you can sit there and fiddle with a chatbot long enough to press it into being smarmy or responding in ways that we don't believe are the way Alexa might act, for example, or press it to give you wrong information that's from some unauthoritative source or a mistake in its training data online shifts back to its native training. So getting Alexa to speak confidently in her personality with authority and answer questions right, it's another key challenge.

• 56:44Speaker 07

  Personalizing an experience of this depth so that Alexa's always learning from her interactions with you and extending your interactions so they get more delightful over time. I think this is something you probably wouldn't have seen in a weekend's worth of fiddling with the experience. You'll see it gets more personalized. That's another big technical challenge because the surface area is so much bigger. So those are a few of the reasons why it sort of took so long. And I think if you rewind the tape to 2023, it's really about learning how big a project Alexa Plus would be and then starting to put one foot in front of the other, really inventing the space of creating those integrations because it just hasn't been done.

• 57:23Speaker 01

  What's an example of some early failure mode that you all had to overcome? I mean, I've heard some stories from folks who have worked on Alexa or worked with suppliers who provide models to Alexa. They would tell me stories about, you know, you'd ask Alexa to set a timer for you and it would like write you an essay about the history of timers. It was just sort of misunderstanding the request in the way that a large language model might. So tell us some of those stories.

• 57:47Speaker 07

  Well, that's a good one. I mean, verbocity was definitely an early issue and, you know, these-

• 57:52Speaker 02

  It continues to be an issue on our podcast, by the way. We still haven't solved it.

• 57:56Speaker 07

  I've got some training ideas.

• 58:01Speaker 07

  You know, verbocity, these models want to give you an extensive answer. Customers don't want an extensive answer read out and they certainly don't want a disquisition on the nature of timers, right? What they want is an interface that sets a spaghetti timer.

• 58:16Speaker 01

  And how do you get them to do that? Is it just as simple as putting in the system prompt? Like if a customer asks for a timer, like don't give them an essay on the history of timers or be concise or what? How do you actually solve that?

• 58:26Speaker 07

  I would love it if it were that easy. You need a set of models. There are over 70 models in Alexa Plus. It's a vast space. There are different models specialized in different tasks. There are different corpuses of training data we use on different models to get them to complete instruction sets for us and really follow the rules of the road in interfacing with something. You always need to loop back to central systems that are maintaining context in the conversation and picking up from references and pronouns that you've used to refer back in time, right? And sort of cascade those forward. But the amount of work that went into just the interface between a large language model

• 59:09Speaker 07

  and the downstream systems that complete tasks is, I mean, it's the biggest body of work that we've put in and there's, it would be too much to even try, without a whiteboard here, it would be too much to even try to explain to you and your listeners, I think, the technical depth that went into it. We've got a great team working on it and it's hard.

• 59:26Speaker 01

  Of those 70 models in Alexa Plus, how many are Amazon's own in-house models versus models like Claude that you all get from external companies?

• 59:34Speaker 07

  There's a mix. So the best way to know what models in our Alexa Plus is just go to the Bedrock webpage and look at an update on the latest in there. We use the best tools that we have available to us for the job and we've got great partners over in AWS helping make sure we've got the right best tools for the job. Most of our traffic does flow through Amazon Nova models. We have the most control over how those get trained and tuned and post-trained. I think it's over 80% of traffic on sort of the main big inferences within the system flow through Nova models, but there are many different reasons to use many different models. I think you guys know better than most that models are special, you know, they specialize in different things, so we use the best tool for the job.

• 1:00:18Speaker 01

  Can you give us a sense of like how big the team is that's working on Alexa? Like how big of a priority is this within Amazon?

• 1:00:24Speaker 07

  It's thousands of people.

• 1:00:25Speaker 01

  Okay.

• 1:00:25Speaker 07

  Yeah. Now, that's building hardware, it's building Alexa Plus, it's integrating with all those systems, it's adding new integrations and new things that Alexa can do. It's a pretty, it's a pretty vast scope, so it takes, it takes a big team.

• 1:00:43Speaker 01

  Yeah. There was a former machine learning scientist at Alexa AI, Mahal Eric, who did a long post on X last year, sort of his version of a postmortem or retrospective on what was happening with Alexa. And he wrote that Amazon had, quote, "all the resources, talent and momentum to become the unequivocal market leader in conversational AI." But then he said that Amazon and Alexa had fumbled the ball because Alexa was, quote, "riddled with technical and bureaucratic problems." Sort of made it seem like the problem was not just that the technology was an uneasy fit, but that there were also some organizational and bureaucracy problems that had to be solved. Can you talk a little bit about that?

• 1:01:25Speaker 07

  I won't comment on that post in particular. Honestly, I don't remember it. But there is definitely, I would say, sort of a startup culture transformation happening within the Alexa team. I think, you know, the life cycle of any product that's been around for 10 years, right? It has ups and downs. But I think our rate of innovation had slowed down. And I think coming through for customers on integrating these new powerful tools is something that's really quickened and inspired the team. I don't identify with the bureaucratic comment. Maybe it's, maybe it's a comment about me, so maybe I won't identify with it. I don't know. But I do think the team is inspired, it's inspired by the vision, executing at an unbelievable

• 1:02:08Speaker 07

  pace and really, really creating a lot of invention, because it's, there are a lot of really hard problems.

• 1:02:15Speaker 01

  I'm curious where the new Alexa sits in relation to Amazon's overall AI ambitions. You know, this is a company that has offered a lot of AI models through AWS. Big, you know, market share in cloud-based AI. Also recently started an AGI lab at Amazon that is going to be pushing towards something like an artificial general intelligence. Is Alexa part of that overall effort to create and serve more capable AI systems? Or is this sort of a consumer targeted sort of spinoff of those efforts?

• 1:02:55Speaker 07

  I would say we do believe, and I share this belief, the leadership team at Amazon has that this generation of generative AI is going to transform every customer experience we have. And that means, I mean, we have a lot of different types of customers. You mentioned AWS, we have enterprise business customers, we have consumer customers. We offer a very big landscape of services. I know that at some point within the last year, we counted and there's over a thousand different AI efforts going on with consumer applications alone. So if you sort of look at the scale and scope of what Amazon does and, you know, assume our belief that every experience will be transformed with generative AI, it's as big

• 1:03:39Speaker 07

  as Amazon is at that point. I would also say just internally, it's part of how we work now. You, to be as productive as you can be in this day and age and get as much done for customers as we aspire to, you have to build AI into how you're working. You both do this, I know that, and I'm sure many of your listeners do too, but it's certainly part of what's going on at Amazon as well.

• 1:04:01Speaker 01

  Yeah. Okay, well, Daniel, we have some product feedback for you.

• 1:04:04Speaker 07

  Let's do it.

• 1:04:05Speaker 01

  As they say, feedback is a gift.

• 1:04:06Speaker 07

  Always.

• 1:04:07Speaker 01

  So we'd like to give you some gifts. And it's Christmas!

• 1:04:10Speaker 07

  Casey, why don't you start?

• 1:04:12Speaker 02

  All right, well, so, let's see. The, I feel like most of my feedback is less about like Alexa Plus as an AI than it is about like Alexa Plus in the actual like hardware that I got. I first started with the Echo Show 5, which does say on the website that it is Alexa Plus enabled. But then some of your folks were like, no, like to get the full experience, you should get the 15. So I sort of had the two experiences.

• 1:04:40Speaker 07

  Okay.

• 1:04:40Speaker 02

  On the 5, my first observation was that after I told it I would like to see art, I feel like every time I looked over at it, it was asking me if I wanted to buy paper towels or Advil or something. That was a little bit less the case once I got the 15. I don't know, you know, why that might have been. But I felt like the Alexa Plus AI thinks of me primarily as a person who might send more money to Amazon. If you just sort of gave me a few more ideas for how I might do that. And what I would love is if it evolved to treat me like a person who isn't like constantly looking to buy paper towels. You know what I mean?

• 1:05:19Speaker 02

  So that I think is actually my biggest piece of feedback was I wanted fewer ads, fewer reminders that Amazon Music exists, fewer reminders that Amazon Prime Video exists, just like get to know me as a person a little bit. That's my big feedback.

• 1:05:33Speaker 07

  Subject line.

• 1:05:34Speaker 02

  Yeah.

• 1:05:35Speaker 07

  Enough with the paper towels.

• 1:05:36Speaker 02

  Enough with the paper towels. If I say I want to see art, I really mean it. So I get it because you, you want to show everything that your hardware can do. You worked very hard on it. It can do many things. You want to like showcase all of those things. But I do think it comes across as a kind of insecurity in the device. Like if we're not constantly showing you everything that we've built into this thing, you'll never discover it and you'll put this thing in a drawer. I understand the pressures that you're under and I understand why it has evolved this way. But I, I have to say, when I unplugged it, I felt more relaxed because it wasn't giving me a list of things to do. And I didn't feel that way about my original Alexa, which is like great at the things that it does. So that's, I know, I know that. That's a lot, but those were my emotions.

• 1:06:21Speaker 07

  The first one to me, the Echo Show 5 feedback sounds like a bug. I don't know what state it got into, but just if you asked for artwork and that's not what it was showing, dude, that one sounds like a bug.

• 1:06:31Speaker 02

  Okay.

• 1:06:31Speaker 07

  The latter part, it might just be you have a different reaction than most of our customers do to the to the onboarding experience is maybe what it sounds like, or you're just looking for more diverse things. I will be curious to follow up with you in a week and find out if your use has helped shape the nature of what we're showing you.

• 1:06:49Speaker 02

  Yeah.

• 1:06:49Speaker 07

  That is certainly our intention is that when you're onboarding to the new experience, the types of things you're asking for are the types of things we're showing you. And that could be anything like one of my most delightful. We have a new element called For You, which is a place where we post little notifications about things we think you might be interested in. And I had been helping my daughter study the periodic table for part of her chemistry final. And I wasn't ever great at remembering in particular the elements that like you need a mnemonic for like lead or, you know, they don't match.

• 1:07:21Speaker 02

  PB.

• 1:07:21Speaker 07

  Right. PB, very good. So you were good at chemistry, obviously.

• 1:07:25Speaker 02

  Yeah, nailed it. Very, very low latency on this one.

• 1:07:27Speaker 07

  You don't need the mnemonics. But I had done that the night before and when I came in in the morning, my For You said, you know, should we, should we make a chemistry quiz for Ellie or something like that? I was like, write a chemistry quiz for Ellie. And with the generative content capabilities of Alexa Plus, I just said, yeah, let's try that. Like, let's make, can we make a sheet of all of the elements that aren't intuitive?

• 1:07:49Speaker 02

  Now, did it also ask if you wanted to buy lead?

• 1:07:53Speaker 07

  It didn't ask me that, I think, which, you know, it's a product safety thing. So I'm glad we ticked that box. We will have to just look and see what, like the extent of the Amazon services being, being shown to you. But, you know, I will tell you that the body of feedback that we get from customers. Doesn't accord with that specific version of it. Definitely customers want to learn what they can do. That is one of the biggest things that we hear from customers. I want to, I want to come back to what you said about, you know, unplugging the device and plugging it back in. We made the Alexa Plus experience incredibly easy to get out of and get back into and get out of, which is not true for sort of like an OS update, right? It's very hard to go backwards. And we worked very hard to try to make it possible because we knew there would be so much change.

• 1:08:38Speaker 07

  The very high 90% tile of customers stick to the new experience.

• 1:08:43Speaker 02

  That makes sense to me. I mean, it's clearly much more capable, it can do more stuff. And I know it's going to evolve and presumably improve over time. So, yeah, no part of me was like, I want to go back to the old experience. I was just like, wow, this is like very intense. I honestly, I think the bigger shift that I experienced was going from like just a pure speaker to something with a screen. Like that actually feels bigger than the change.

• 1:09:06Speaker 07

  I understand that.

• 1:09:06Speaker 01

  To piggyback on Casey's question, I think this is one of the big questions about the Alexa business model is whether you see this as something that is going to make money on its own or whether this is primarily sort of a way of increasing the amount of money that people spend on Amazon. You know, I spend an ungodly amount of money on Amazon.

• 1:09:25Speaker 07

  Thank you for your business. Thank you for your business.

• 1:09:28Speaker 01

  A large fraction of my income is spent on various things on Amazon. And so I'm well aware of the many products that exist on amazon.com, the website. I do not need like ads to be cascading on my, on my screen, telling me to buy more stuff on Amazon. But it does seem like this is primarily going to be an ad-supported product. Andy Jassy recently said on the earnings call for the most recent quarter that you all were trying to bring more advertising experiences to Alexa Plus. So talk to us about that and like, are we just going to inevitably be more annoyed at the number of ads that are showing up on these devices?

• 1:10:05Speaker 07

  I definitely don't think you'll inevitably be more annoyed. I would say advertising is definitely part of the business plan, but it's not the biggest part. It's actually probably the smallest part. The most important decision we made on the business side with Alexa Plus was bringing it into Prime. And putting it into Prime sort of brings together both all of a customer's Prime benefits, because you might watch a video or listen to a song from Amazon Music or use your Amazon Photos benefit, which is awesome with an Echo Show and review your family photos. I use that all the time to look back at the kids in particular. But you have this long list of Prime benefits. Alexa is a great place where they come together.

• 1:10:49Speaker 07

  And putting in the value of having this world's best personal assistant into Prime just turns the Prime flywheel. And we know every time we've added a benefit to Prime, customers use their Prime benefits more. It's stickier to them. It provides them more value and it turns into a great business. That's the goal.

• 1:11:06Speaker 01

  Okay, so Casey's feedback was about advertising. Mine is about some of these new features that don't work and some of the old features that actually don't work either. So some of the more complicated stuff that I tried with Alexa Plus, such as setting up routines that involve multiple steps, such as emailing documents or research papers to the Alexa email address and trying to have it summarize these things. These things just didn't work for me. The routines didn't run. The papers didn't show up to be summarized. I assume this is just sort of growing pains and beta testing bugs and things like that. What I found more frustrating that I wanted to ask you about, because I'm actually not sure why this happens, was that some of the basic features that Alexa had previously

• 1:11:50Speaker 01

  been good at and reliable at for me were less reliable with Alexa Plus. So this morning, for example, I tried to cancel an alarm that was about 10 minutes from going off and Alexa just didn't listen, didn't hear me. The alarm went off anyway. So help me understand why that is. Is that like a hallucination of the model? Is that a problem that is sort of related to the orchestration of the various tasks and sending it to the right place? What is going on there?

• 1:12:16Speaker 07

  Honestly, we'd have to dive deep into each of those to figure it out. Early access is here as a program to cover off on these kinds of issues and to make sure customers know that they can opt into Alexa Plus, they can opt out if they want. Again, the very vast majority of customers stick to it.

• 1:12:36Speaker 07

  The key challenges probably in everything you said are that interface between the large language models and these more predictable, rule-based systems that communicate through APIs. Something like canceling an alarm, making sure we find out the exact intent of what you were looking for, translating that into a set of commands and then issuing those commands to an API. Sometimes it does fail. It's rarely, at this point, because of hallucination, we've got so much going on to monitor for model hallucinations. It is sometimes because of just incorrect use of an API or just misunderstanding exactly where to send those commands. So that's more likely the case in each of these cases.

• 1:13:15Speaker 01

  Got it. I will give you one more piece of feedback, which is actually not from me, this is from my three-year-old son, who is our house's most active Alexa user. He talks to Alexa all the time, probably more than talks to us. Should I be concerned about that? Maybe, but we'll save that for a later episode. But he was doing storytime with this because he wants to, you know, constantly wants more stories about various vehicles, you know, various dinosaurs. And so we were doing a storytime about Super Tow Truck that rescues cars from the water. And he asked for another one, and it gave him a totally different set of characters. So if there's some way for kids to have like a kind of, you know...

• 1:13:58Speaker 02

  Their own private cinematic universe?

• 1:13:59Speaker 01

  Persistent cinematic universes for Super Tow Trucks. I know at least one three-year-old would really appreciate it.

• 1:14:05Speaker 07

  I got it. Excellent product description, by the way. I like that for sure. I agree that children, as they explore, it doesn't even have to be an imaginary friend, but they do love themes and they love to continue them. So it's great. That's great feedback. We'll take that to the team.

• 1:14:18Speaker 01

  Yeah. For all of our feedback, I actually am very glad I got to try this. I'm gonna keep testing it. We are very active Alexa users in my household, so we'll keep sending you our feedback.

• 1:14:30Speaker 07

  That's awesome.

• 1:14:31Speaker 02

  Yeah, we like trying new things around here.

• 1:14:32Speaker 01

  Yeah. Daniel, thanks so much for coming.

• 1:14:34Speaker 02

  Thanks, Daniel.

• 1:14:34Speaker 07

  Really appreciate your time, guys. Thanks a lot.

• 1:14:36Speaker 02

  Oh, wait.

• 1:14:38Speaker 01

  Did you just set off your Alexa?

• 1:14:39Speaker 02

  Siri, stay out of this! Gosh, she's got a lot of nerve coming into this podcast recording. Wow.

• 1:15:04Speaker 03

  Support for this podcast comes from Dell Technologies. It's time to put AI to work. Is your business ready? The challenge isn't getting started with AI. It's scaling beyond pilots while maintaining control of your data, governance, infrastructure, and costs. Dell AI Factory provides the integrated foundation you need to scale AI securely and efficiently with one trusted partner. Dell, AI on your terms. From idea to impact. Learn more at dell.com/yourwaytoAI. That's dell.com/yourwaytoAI.

• 1:15:38Speaker 04

  Building and scaling AI today can feel pretty chaotic, mostly because teams are juggling too many fragmented tools at once. That's where Outsystems comes in. Outsystems gives you one unified Agentic Systems platform to manage your entire portfolio of agents, apps, and workflows. You get to innovate at the speed of AI, but with the security, governance, and control your enterprise actually needs. And the results are already there. A major manufacturer used Outsystems to build a multi-agent system that analyzes blueprints, and a global bank cut their development time by 60%. Own your Agentic future today at outsystems.com/nyt.

• 1:16:14Speaker 06

  Today's AI builders need infrastructure that scales with their ambitions. Iren is a vertically-integrated AI cloud platform, delivering data centers, compute, and software for AI training and inference, underpinned by its expansive portfolio of LAN and grid-connected power in renewable-rich regions globally. Iren's AI Cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. Iren, AI Cloud for the next big thing.

• 1:16:49Speaker 01

  Well, Casey, we've got some good news and bad news. The bad news is that our hat promo is coming to an end. So as of this week, the limited time offer to get a free Hard Fork hat along with a new annual New York Times audio subscription is running out. This is your last chance to get this very cool, limited edition hat as a thank you for subscribing to New York Times Audio. And these Hard Fork hats are only available to subscribers in the United States. The good news is that we are going to have more hats available for sale. A different hat with a slightly different design is going to be available in the New York Times store, we're told, pretty soon. So look out for that if you already subscribe to New York Times Audio or you just want the hat without the subscription.

• 1:17:33Speaker 01

  We are told that the next wave of Hard Fork hats will be available internationally as well.

• 1:17:40Speaker 01

  Hard Fork is produced by Whitney Jones and Rachel Cohn. We're edited by Jen Poyant. We're fact-checked by Caitlin Love. Today's show was engineered by Chris Wood. Original music by Marian Lozano, Diane Wong, Rowan Nemusso, and Dan Powell. Video production by Sawyer Roque, Pat Gunther, Jake Nickel, and Chris Schott. You can watch this whole episode on YouTube at youtube.com/hardfork. Special thanks to Paula Schuman, Pui Wing Tam, Dahlia Hadad, and Jeffrey Miranda. You can email us at [email protected] with a story about when you fainted.

• 1:18:38Speaker 00

  You can't undo everything in life, but you can undo prediabetes. More than 2 in 5 adults in the US have prediabetes. Most don't know they have it. But with early diagnosis and action, you can delay or even prevent type 2 diabetes. Visit doihaveprediabetes.org to take the 1-minute risk test and start undoing the things you can. Brought to you by the Ad Council and the Centers for Disease Control and Prevention.