AWS CEO Matt Garman on Amazon's Big AI Chips Bet, Working With OpenAI, and Nuclear Energy

Matt Garman is the CEO of Amazon Web Services. Garman joins Big Technology Podcast to discuss Amazon's AI strategy and future plans. Tune in to hear Garman's insights on AWS's AI chips, partnerships with leading AI companies including OpenAI, the ROI of generative AI, and how AWS is helping customers leverage AI while managing costs. We also cover AWS's investment in nuclear energy, the current state of the cloud market, and Amazon's culture challenges Hit play for a wide-ranging and in-depth conversation with the executive leading the world's largest cloud computing operation. Transcript: https://www.bigtechnology.com/p/aws-ceo-matt-garman-talks-amazons If you like what we're doing here, please rate the show five stars on Apple Podcast and Spotify. Thanks for listening! Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Alex Kantrowitz?

  AWS CEO Matt Garman is here to talk about AI, the state of the economy and Amazon culture. That's coming up right after this. This episode is brought to you by Genesis. What does it actually take to put Agentic AI to work across an entire enterprise? I recently spoke with Genesis Chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, Head of Enterprise Business Solutions at Voya Financial, to go inside a large scale Agentic CX transformation, including the business decisions, trade offs, and lessons along the way. You can watch both conversations now on my YouTube channel today.

• 0:44Alex Kantrowitz?

  It's never been more challenging to get frontier compute. How do you know where to look? Who to trust? Or how much to pay? Akash is an open cloud that provides easy access, flexibility and peace of mind. That's how companies like Razor were able to cut their inference costs by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation. Welcome to Big Technology Podcast, a show for cool headed nuanced conversation of the tech world and beyond. We are here in Las Vegas, Nevada, at Amazon's Reinvent Conference with the CEO of AWS, Matt Garman. Matt, so great to see you. Welcome to the show. Yeah, thank you for having me. Great to be here.

• 1:29Alex Kantrowitz?

  Great to be here with you at your event. Let's talk a little bit about infrastructure. You're the kings of building data centers, right? There's no one that does it better than AWS. But there are headlines in the AI world. Elon Musk took 122 days to build a 100,000 plus GPU data center to train Colossus, which is his latest AI model. Does this show that scaling data centers is now core to competing in AI? Is it validation? What do you think about it?

• 1:59Speaker 01

  Well, look, we've been building data centers for almost two decades now. And so this is something that we spend a lot of time on. And it's less that we were out there kind of bragging to the press about. But what we do is we provide-

• 2:10Alex Kantrowitz?

  You can brag. What's your size of yours?

• 2:12Speaker 01

  Well, more is, what we do is provide infinite scale for our customers. And so our goal is for largely the customers not have to think about these things right now. So we want them, across their compute, across their storage, across their databases, to be able to score, to be able to scale to any number of size of like- And so take something like S3 as an example. It's incredibly complex, very detailed system that keeps your data, keeps it durable, and scales infinitely. And customers largely just put data in there, don't have to think about it. And so today, S3 actually stores 400 trillion objects. Because it's an enormous number that's hard to even get your head around. But it's just something where we just keep scaling and we keep growing for our customers. As you think about AI now, these are power-hungry massive data centers for sure.

• 2:56Speaker 01

  And AWS is adding tons and tons of compute all the time for our customers. Largely what we think of though, is less about, you know, how fast can you build one particular cluster? The absolute size of AWS is dwarfed by any other particular cluster out there. But we're focused on how do we deliver the compute the customers need to go build their applications? Take somebody like Anthropic as an example. Anthropic has the, what are widely considered to be the most powerful AI models out there today, and their Claude set of models. We're building together with them what we call Project Rainier. And so it's using our next-generation Tranium 2 chips. And this cluster that we're building for them in 2025

• 3:38Speaker 01

  will be five times the size, the number of exoflops that they use to train the current generation of models, which are by far the most powerful ones out there. It's going to be five times that size next year. All built on Tranium 2, delivering hundreds of thousands of chips in a single cluster for them so they can train the next generation. That's the type of thing where we work with customers, understand what's interesting for them, and then help them scale to whatever level they need. And that's just one of our customers, of course, right? We have hundreds and hundreds of other customers as well.

• 4:03Alex Kantrowitz?

  Here's my point. You're so good at this. Look at what you just talked about in terms of Anthropic. Being able to help them scale the way that you are.

• 4:13Alex Kantrowitz?

  That would lead me to believe that Amazon would have its own cutting edge, state of the art model, one that would lead and be better than the OpenAI's and the Anthropics. This is your core competency and this is what makes these models run. So why hasn't that happened?

• 4:25Speaker 01

  Our core competency is about delivering compute power for all of the people that need it. And for a long time, we've been very focused on how do we build the capabilities to let our customers build whatever they want. And sometimes there are areas that Amazon also builds and other times they're not areas that Amazon builds. And so you think about whether it's in the database world or you think about in the storage world or you think about the data analytics world or you think about the ML world. We build this underlying compute platform that everybody can go build upon. And sometimes we build services that compete with others out there in the market. Think about a Redshift competing with a Snowflake, who's also a very important partner of ours and a big customer of ours, somebody that we do a lot of partnering together on.

• 5:08Speaker 01

  And then there's other times where there's applications that people build on top of AWS that Amazon doesn't go and build. And so we operate across that whole swath of area and sometimes we'll build and sometimes we don't. But that's the kind of the beauty of AWS is that our goal is to build that infrastructure so that sometimes we can build those, sometimes we won't build them. But we want this platform that everybody can go build the broadest set of applications possible out there.

• 5:32Alex Kantrowitz?

  But I'm thinking about it for AI specifically and the world that you play in.

• 5:35Speaker 01

  Yeah.

• 5:35Alex Kantrowitz?

  You have Google, they have their own model, they sell cloud services. You have Microsoft, okay, they don't have their own models necessarily, but they have this deal with OpenAI.

• 5:42Speaker 01

  Yeah.

• 5:43Alex Kantrowitz?

  Pretty sure that OpenAI is exclusive on Azure. Now, this is where a lot of the growth is coming from.

• 5:49Speaker 01

  Yeah.

• 5:49Alex Kantrowitz?

  And so.

• 5:50Speaker 01

  And I think it's a mistake, actually. So interesting thing there is, and this is where a lot of people started. And I just think it's fundamentally the wrong way of thinking about it. Just a lot of times people are thinking about there's just going to be this one model. And I want to have the one model that's going to be the most powerful and the one model to rule them all. And as you've. Transition, as you've seen over the last year, there isn't one model that's the best at everything. Like, there's models that are really good at reasoning. There are models that are great for, um, that provide open weights, so that people can bring their own data and fine tune them and distill them and create things that, and create kind of completely new models from that that are completely custom for customers. And on that, you may want to use a Llama model, or you may want to use a Mistral model. There's customers who really want to build the, the world's best images and they might use something like a Stability, or they might use something like a Titan model. There's customers that need really complex reasoning and they might use an Anthropic model.

• 6:36Speaker 01

  Uh, there's a whole ton of these offering out there and, and our goal is how we help customers use the very best. It doesn't have to be one thing. It's not just one. And we don't think that there's one best database. We don't think there's one best compute platform, uh, or, or processor. Uh, we don't think that there's one best model. Uh, it's across that whole set and, and that's been our strategy. And customers have really embraced that strategy. As you, as you get to them and they're thinking about how they go building, uh, uh, production applications, they want the stability, the operational excellence and the security that they get with AWS. But they also want that choice. It's incredibly important for them. And I think choice is important for customers, no matter if they're building generative AI applications, no matter if they're picking a database offering, no matter if they're picking a compute platform.

• 7:18Speaker 01

  They want that choice and that is something that AWS has from the very earliest day has really leaned into. And I think it's an important part of our strategy. And it's maybe not the strategy that others have. Maybe others say it's just this one and this is the one that we're going to lean into. But it's not the strategy that we've picked. Our choice is around choice. And it's part of why we have the broadest set of partner ecosystem as well. It's why, as you walk the halls here in Reinvent, it's filled with partners who are building their business on top of AWS, who are leaning in and helping our joint customers accelerate their journeys to the cloud, their modernization efforts, their AI efforts. It's because of that, I think that, that is a lot of what makes, uh, AWS special.

• 7:52Alex Kantrowitz?

  Okay. And I'm going to move off this in a moment, but I, the reason why I'm asking these questions is because you do have at least a bet that big foundational models are going to matter. That's the four billion you just invested in Anthropic. And I think that the strategy that AWS has makes a lot of sense, right? This Bedrock strategy, there's a lot of different models in there. People have their data in the cloud. They're going to build with, uh, their data they have within AWS using Bedrock picking models. But you also are limited in the fact that OpenAI is not there. I don't think Google's there. So wouldn't it make sense in parallel to the bring your own model strategy? To also use this capacity that you have to scale infrastructure to get in the game yourself?

• 8:30Speaker 01

  Look, what I will say is I'll never say never, right? I think there's an interesting idea and then we'll, we'll, we never close any doors. I think we're always open to frankly, a whole host of things. We're always open to, to having OpenAI be available in, in AWS someday, or having Gemini models be available in AWS someday. And, and maybe someday we will spend more time focused on our own models for sure. I think, I, you know, I think all of that is open and part of what I think makes AWS special is we're always open to, you know, take our announcement earlier this year about partnering deeply with Oracle, about making Oracle databases available in AWS. Lots of people had said, oh, that's never going to happen. It's against your strategy. Our strategy is to embrace all technologies because we want anything that customers can use, we want them to be available and to be able to use it inside of AWS. And look, sometimes it happens today, sometimes it happens tomorrow, sometimes it happens

• 9:14Speaker 01

  weeks from now, months from now, years from now. But, but that is our goal is to make all of those technologies available for our customers.

• 9:20Alex Kantrowitz?

  Okay, I'm going to parse your language a little bit because you said that you're always, you're, you might be open to having OpenAI on a bedrock within AWS.

• 9:29Speaker 01

  Sure.

• 9:30Alex Kantrowitz?

  Are you talking to them? Would you want to ask them to come on?

• 9:33Speaker 01

  There's, there's nothing to announce there today, but I'm saying if customers want that, that's something that we would want and we'd love to make it happen at some point.

• 9:39Alex Kantrowitz?

  Okay. Well, maybe they're listening and they want to make that move.

• 9:43Speaker 01

  Yep.

• 9:44Alex Kantrowitz?

  But let's speak to the one that I think is the biggest challenger to them, the one that you have all this money in, which is Anthropic.

• 9:49Speaker 01

  Yeah.

• 9:49Alex Kantrowitz?

  So what does the $4 billion that you just invested in Anthropic get you?

• 9:54Speaker 01

  Yep.

• 9:54Alex Kantrowitz?

  And how does that make you differentiated from other cloud providers?

• 9:58Speaker 01

  Well, I have a couple of things I would say. One is, um, you know, we, we, we make the investments in Anthropic because we think it's a, it's a good bet. They have a very good team. Uh, they, they have, um, they've made some incredible traction in the market and, um, we really like what they're, where they're innovating. And so we, you know, we thought that's a good investment.

• 10:15Alex Kantrowitz?

  And we're, we're definitely cloud heads on the show.

• 10:17Speaker 01

  Yeah, it's, it's a fantastic product, right? And that, and Dario and team are, are very good and, and they continue to actually attract some of the best talent out there in the market today. And the other thing that we get from that is a, a deep collaboration on Trainium. And, uh, you know, we're, we've made a big bet on Trainium as a, an additional option for customers, you know, the vast majority of-

• 10:38Alex Kantrowitz?

  We should define it. It's the chips that, oh, sorry, go ahead. It's me, I'll just say the chips that people can, that companies can use to train their own models with.

• 10:44Speaker 01

  That's right.

• 10:45Alex Kantrowitz?

  At AWS.

• 10:45Speaker 01

  That's right. And so today, the vast majority of, of AI processing, whether it's inference or training is done on Nvidia GPUs. And we're a huge partner of Nvidia. We will be for a really long time. And I think that, and they make fantastic products, by the way, and they continue to do that. And when when black hole chips come out, I think people are very excited about that next generation platform. But we also think that customers want choice. And we've seen that time and time again. We've done that with general purpose processors. We have our own custom general purpose processor called Graviton. And so we actually went and built our own AI chips. The first version was called or they're called Tranium. And we launched Tranium one a couple of years ago, in 2002, and are just Yang Tranium two here at Reinvent.

• 11:29Alex Kantrowitz?

  So that's news that's happening this week.

• 11:31Speaker 01

  It'll be announced in my keynote.

• 11:33Alex Kantrowitz?

  Yes.

• 11:35Alex Kantrowitz?

  We're going to release the transcript as your keynote hits and the podcast the day later.

• 11:39Speaker 01

  Great.

• 11:39Alex Kantrowitz?

  But this is brand new news.

• 11:40Speaker 01

  Brand new news.

• 11:41Alex Kantrowitz?

  Press off the press, folks.

• 11:41Speaker 01

  Press off the press. And so we'll have Tranium two, and Tranium two gives really differentiated performance. We see 30 to 40% price performance games versus our instances that are GPU powered today. So we're very excited about Tranium two, and customers are really excited about that. And what Anthropic gives us, back to your question, is a leading frontier model provider that can really work deeply to build the very largest clusters that have ever been built with this new technology where we can learn from them, right? And just learn what's working, what's not, what are the things you need accelerated, so that Tranium three and Tranium four and Tranium five and Tranium six can all get better as we as we continue to go. And and the software associated with GPUs gets better, or the accelerators gets better as well. I think

• 12:23Speaker 01

  that's one of the things where people who've tried to build accelerated platforms before have fallen down is the software support has not been as good as Nvidia's software support is fantastic. And so that's a big area where they're helping us as well as we help iron out the creeks and the kinks and try to figure out how we make sure that developers can start to use these Tranium two chips in a very seamless way and high performance. So we learn a lot from them as big users that are really leaning in and help us learn. And they get benefits from they get that scale and and cost benefit of running on this price performance platform that gives them a huge win. And and we think then from that investment, we can both benefit as they deliver better and better models over time.

• 13:04Alex Kantrowitz?

  There's an interesting thing that happens when I speak with people who are working in cloud or working to train models, are working to build their own chips. There's always a preface, we love working with Nvidia and we're also building chips that compete with what they do. So how does that relationship work out? They don't get upset that you're trying to build the same, I mean, they have a supply issue, but yeah, how does it work with them?

• 13:25Speaker 01

  Oh, no, I have a great relationship with Nvidia and Jensen and and this is the thing that we've done before. We have a fantastic relationship with Intel and AMD and we produce our own general purpose processors. And and it's a big world out there and there's a lot of market for and for lots of different use cases. That's not one is going to be the winner, right? There's going to be use cases where people are going to want to use cheap use. And there's going to be use cases where people are going to find training to be the best case. There are use cases where people find that our Intel instances are the best choice for them. They're ones where they find that the AMD instances are the best choice for them. And there's increasingly a large set where they find Graviton, which is our purpose built general purpose processor is the right fit for them. And it doesn't mean that we don't have great relationships with Intel and Nvidia or Intel and AMD.

• 14:09Speaker 01

  And it means we'll continue to have a great relationship with Nvidia. Because for them and for us, it's incredibly important for Nvidia processors and and and GPU powered processors to perform great on AWS. And so we are doubling down our investment to make sure that Nvidia performs outstanding in AWS. We want it to be the best place for people to run GPU based workloads. And I expect it will continue to be for a really long time.

• 14:32Alex Kantrowitz?

  What's the buying process like with Nvidia? Because you want as many chips as you can get. I would imagine you have Elon who buys and buy the truckload. You have Zuckerberg who has been buying lots and I think he wants to power them with a nuclear submarine or something like that. So do you have to jostle with the other companies to get Nvidia chips? Or do you get every exact quantity you want?

• 14:53Speaker 01

  Nvidia is very fair about how they they go about it.

• 14:57Speaker 01

  I mean, you can ask them about how they internally allocate. That's not really a question for me, it's for them. But but they're they're they're very fair in dealing with us and we give long term forecasts and and they tell us what they can supply. And we all know that there's been shortages. In the last couple of years, specifically as as demand is really ramped up. And they've been great about ensuring that we get you know, enough to support our joint customers as much as possible.

• 15:20Alex Kantrowitz?

  What about your inference chips in French here?

• 15:23Speaker 01

  Yeah.

• 15:24Alex Kantrowitz?

  Because last time I heard you speak, you said that the activity within AI right now, Gen AI is 50% training, 50% inference. Does that ratio still hold? And how are you going to put the chips out there to allow companies to be able to do cheaper inference? Because that's the issue with generative AI. It works well, but it's so expensive that companies take proof of concepts and only one fifth actually make them out into production.

• 15:48Speaker 01

  Yeah, it's absolutely the case. And I think we're, you know, we're still probably seeing about that ratio of 50 50. I think if more and more, it's more inference than than training. And increasingly, we'll see more and more of the workload shift that way. It costs is a super important factor that many of our customers are are definitely worried about and thinking about on a on a daily basis. And, you know, if you think about where a lot of people were, they went and did a bunch of these Gen AI capability or tests, right? Where they did proof of concepts and they launched hundreds of proof of concepts across the enterprise without really paying attention to like, what was the value going to be or anything like that? And now they're looking at them and they're saying, well, our ROI is not really there. They're not really integrated in our production environment. They're just kind of these POCs that I'm not getting a lot of value out of, and they're expensive, as you mentioned.

• 16:32Speaker 01

  So two things that people are thinking about is one, how do I lower the cost? So that I make that the cost much lower to run. And that's the point about cost inference. And two, how do I actually get more value out of that? So the ROI equation just completely shifts and makes more sense. And it turns out it's probably not all hundred of those. It's probably two or three or five of those that are are really valuable. And so there's a couple of things. Number one is on the cost side. As there's a few things that we're doing to help people lower costs. Number one is I think training them to will be a material impact there. And as these models have gotten bigger and bigger, you mentioned in Ferencio. Originally, we had a small chip called Inferencia that would run really fast, lightweight inference. Now, as you're running models that have billions, tens of billions, hundreds of billions,

• 17:14Speaker 01

  trillions of parameters, they're way too big to fit on these small inference chips. And effectively, they're running on the same training chips, like they're all the exact same things. And so you run inference today on H100s, H200s, or you run inference today on training twos or training ones. And so we may come out over time with other inference, Inferencia chips, as you will, but they're really using a lot of that same architecture and they're still really large servers. And so we actually expect that training two is going to be a fantastic inference platform. Our naming is not necessarily always our suit as to what these chips are for, but it's going to be a fantastic inference platform. We actually think it'll be, as you think about that 30 to 40% price performance benefit the customers are going to get. Now, if you can run inference at 30 to 40% cheaper compared to the, the, the leading

• 17:58Speaker 01

  GPU based platforms, that's a pretty big price decrease. And then there's a couple, there's also announced at Reinvent, we're launching automated model distillation inside of Bedrock. And what that makes, lets you do is you could take one of these really large models that's really good at answering questions. You can feed it all your prompts and things for the specific use case you're going to want, and it'll automatically tune a smaller model based on those outputs and kind of teach a smaller model to be an expert only in the area that you want with regards to reasoning and answering. So you can get these smaller, cheaper models, say like a Llama 8B model as opposed to a Llama 405B model, cheaper to run, faster to run, and you can still get it to be an expert at the narrow use case that you want it to be. And so that combined with a cheaper infrastructure, we think is one of the things that is really

• 18:43Speaker 01

  going to help people lower their costs and, and be able to do more and more inference and production.

• 18:49Alex Kantrowitz?

  Yeah, those small models seem to be the cost solution. Sounds like you're a believer.

• 18:52Speaker 01

  That's right, absolutely.

• 18:54Alex Kantrowitz?

  One more question about Nvidia. You've tested the new Blackwell chip. Is it the real deal?

• 18:59Speaker 01

  We have, you know, they're, they're, they're working on getting the yields up and getting it into production, but we're excited about that. And, and also at Reinvent, we're going to announce that the P6, which is the, the Blackwell-based instance that's coming early next year. And we're excited about that. I think customers, I think we are expecting about two and a half times the compute performance out of a Blackwell chip that you get out of an H100. And so that's a, that's a pretty big win for, for customers.

• 19:25Alex Kantrowitz?

  So you're in with Jensen's, the more you spend, the more you save?

• 19:28Speaker 01

  That's right. That's the, you know, that's that they, they've, that team has executed quite well and they've, they continue to deliver huge improvements in, in performance and we're happy to make those available for customers.

• 19:40Alex Kantrowitz?

  Okay. Should we talk about ROI?

• 19:42Speaker 01

  Sure.

• 19:42Alex Kantrowitz?

  All right. Two year anniversary of ChatGPT.

• 19:46Alex Kantrowitz?

  All these companies have rushed to put generative AI in their products.

• 19:49Speaker 01

  Yeah.

• 19:50Alex Kantrowitz?

  To this point, there's a couple of things that I've heard that have worked well.

• 19:53Speaker 01

  Yeah.

• 19:54Alex Kantrowitz?

  AI for coding.

• 19:57Alex Kantrowitz?

  AI that is a customer service chatbot with a little more juice.

• 20:00Speaker 01

  Yep.

• 20:01Alex Kantrowitz?

  AI that can read unstructured documents and make a little sense of them.

• 20:05Speaker 01

  Yep.

• 20:06Alex Kantrowitz?

  Those are the three big ones. I haven't heard much more outside of that.

• 20:08Speaker 01

  Yeah.

• 20:09Alex Kantrowitz?

  We're talking about something that's had a trillions of dollars potentially to public company market caps, something that has had the largest VC funding round, and then probably the subsequent

• 20:18Speaker 01

  Yeah.

• 20:18Alex Kantrowitz?

  three after that.

• 20:19Speaker 01

  Yeah.

• 20:20Alex Kantrowitz?

  Is there, are the three examples that I listed enough to make this worth the money?

• 20:25Speaker 01

  No, definitely not. But they're super valuable right now and they're just the tip of the iceberg. And that's the thing. It's like, you just have to look at the rest of the iceberg to

• 20:31Alex Kantrowitz?

  Yeah. So where are we going?

• 20:33Speaker 01

  The opportunity is. And, and on those three, look, I think those are actually massive opportunities by themselves. We're, we have a number of announcements here at Reinvent around Q developer and making developers and their whole lifecycle more valuable. If you think about the first generation, just using this as an example, the first generation of even developers was just code suggestions, right? It's in like code suggestion, super valuable, actually. It made developers much more efficient being able to code. It turns out also developers on average code about one hour a day. The rest of their day is spent doing documentation, it's spent doing, writing unit tests, it's spent writing, doing code reviews, it's spent doing, you know, going to meetings, it's spent doing, upgrading existing applications, doing all that stuff.

• 21:08Alex Kantrowitz?

  It's in writing code.

• 21:09Speaker 01

  Maybe something, yeah.

• 21:12Speaker 01

  And so as part of that, we're actually launching a bunch of new agents that do all of those things for you. You can just type in slash test and it'll actually automatically write unit tests for you as you're sitting there coding. You can have a Q developer agent build, write documentation for you as you're writing code. And so you can have really well documented code and you're done and you don't have to go think about it. It'll even do code reviews and look for where you have risky parts of your code, where you maybe have open source or your parts that you want or you should go look at and think about what the licensing rules are around. How you think about where even from deployment, where you may want to think about how you're deploying stuff. Things that you would expect out of somebody doing a code review for you before you go do a deployment. Q can now do all that for you. Same on the contact center side, right?

• 21:53Speaker 01

  We're doing a ton of announcements around connect, which is our, our, our contact center in the cloud offering. Making it much more efficient. So for customers to get a ton out of that contact center. All powered by generative AI. And to your point, that's just it, it, you know, those, so you, those use cases, I think get more and more valuable as you add more capabilities. And, you know, I think, um, if you think about where things are going, it is a lot more about, you know, if you, if you think about how I, how I talked about code generation, moving to a bunch of the value, it's adding agents in there so they can do a bunch of these things, right? Now it's not just giving you code suggestions, it's actually going and doing stuff for you, right? It's writing documentation for you. It's helping you identify and troubleshoot where you have operations issues.

• 22:36Speaker 01

  And it says, ooh, you have an operations issue and it can look and understand your whole environment. You can interact with it and you and Q together can go and look and say, ooh, it looks like some permissions over here were broken. And if you go fix those, maybe this is something that you can automatically, you know, it'll fix your application. So saving tons of time across that whole development lifecycle. And I think that that's where as AI gets to be more integrated into the core of what a business is, the core of what you do, and you really have to learn it, that's where you get the value. There's a startup, but there's a number of these doing it. There's a startup that we work with called evolutionary scale. And they use AI to try to discover new proteins and molecules that may be more applicable to solving certain diseases.

• 23:21Speaker 01

  Now you think about not AI is not just generating stuff or it's doing, but it's actually sitting there instead of being able to find tens or hundreds of new molecules a year, you can now find hundreds of thousands of different proteins and test all of these and figure out which are the most likely to be successful and get drugs to market much faster. And, and that's a huge amount of additional revenue. So if you think about models and capabilities that can do that, whether it's in health, care and life sciences, whether it's in financial services, whether it's in manufacturing automation, every single industry in our view is going to be completely remade by generative AI at its core. And, and that's where we think that that's where you get that, that huge value.

• 23:59Alex Kantrowitz?

  Other question about this. I was speaking with a developer friend that said, yes, AI can code. AI can do all these things, probably looking at the different things that these agents can do.

• 24:07Speaker 01

  Yeah.

• 24:08Alex Kantrowitz?

  The problem is, and this applies probably across the board, when you trust things to generative AI, something breaks and then you've lost the skill set to go in and fix that because you've relied so much on the artificial intelligence. What do you think about that? Isn't that a problem?

• 24:22Speaker 01

  Well, it's three times four, 12. Yeah, you still have Excel, but you still know how to multiply. Like I would say that, like maybe, but like, you know, you're able to.

• 24:28Alex Kantrowitz?

  But this is, this is different than multiplication. This is.

• 24:30Speaker 01

  Well, it's different. Again, it's, it's different, but I think that the key parts of coding are not the semantics around writing language, right? The key about cards, parts about coding are thinking about how you break down a problem, how you creatively come up with solutions. And I think that doesn't change, right? The tools change. You can make you more efficient, but you're the developer, the core of what the developer actually does is not going to change. You're going to want to think about, you know, there's not a lot of developers today that know a lot about garbage collection. That's just true. They don't, right? Cause they Java just does it for them and they just don't have to worry about that. Doesn't mean that all of a sudden, like if it breaks, people don't know how to do garbage collection. They can go figure it out. They just don't do it as part of their daily jobs and because it's not fun and it's not value added and they can focus more on that writing code, right? This is what new languages have done.

• 25:12Speaker 01

  But and so increasingly, I think developers are gonna get to do the things that are exciting. They're gonna do good, do the creative work. They're gonna get to figure out how to go solve those interesting problems. And they're gonna be able to move much faster because they don't have to worry about writing documentation. And someday, if it breaks, they probably will know how to write documentation and will figure out how to face that. It's not rocket science. It's just things they don't necessarily want to do today.

• 25:32Alex Kantrowitz?

  Okay. So you're a believer in reasoning. I know that AWS has some news also this week that you're gonna have automated reasoning tests where it checks for hallucinations before an answer goes out.

• 25:47Speaker 01

  Yeah.

• 25:48Alex Kantrowitz?

  Is this something that's sort of, because like another issue when it comes to ROI is again, how can I trust it? It always comes out with wrong answers.

• 25:54Speaker 01

  Yeah.

• 25:55Alex Kantrowitz?

  So talk a little bit about your announcement this week and how reasoning can solve some of these issues.

• 25:59Speaker 01

  This is a, it's a different reasoning than you might be thinking about too. So automated reasoning is a form of artificial intelligence that it's been around for a while. And it's a thing that Amazon has adopted pretty significantly across a number of different places. And, and we use it. It's actually, what it does is it uses mathematical proofs to prove that something is operating as you intended. Okay, that's the historical way. And, and an example of that is we actually use it internally to make sure that our permissioning system is actually, when you change permissions, that it's actually behaving as expected. And so we have a, it's a, it's the, this AI system has this mathematical proof that can go say, okay, all the places that permissions are applied across the surface area that's too large for you to actually go check everything.

• 26:41Speaker 01

  It can prove that they're applied in the way because it knows how the system is supposed to operate. And it can go kind of mathematically prove, yes, your iron permissions mean you can access this bucket or you can't access this bucket.

• 26:53Speaker 01

  We took that and we said, can we apply that to AI to eliminate hallucinations? And so it turns out, not universally you can't do it, but for, for, for selected use cases where it's important that you get the answer right, you can. And so what we do is say an example, like you're an insurance company, right? And you want to be able to answer questions about people that say, hey, I have this problem. Is it covered? Right? And you don't want to say yes when the answer is no or vice versa. Right? And so that is the one where it's pretty important to get that right. Okay. What you do is you upload all your policies and all your information into the system and we'll automatically create these automated reasoning rules. And then there's a process you go through that's a couple of minutes, kind of 10, 15, 20, 30 minutes, where you, as the developer, answer questions of how it's supposed to interact, right? You and you tune it a little bit, say, yep, that's how you'd answer that type of question,

• 27:36Speaker 01

  or no, or that's what this means, it'll ask you questions and you kind of interact with it. And then it goes, okay, now I have a tuned model. Now, if you go ask it a question, you say, hey, I, you know, I ran my car through my garage door, like, is that covered by my insurance policy? It'll go and it'll actually produce a response for you. And then it'll tell you that, yes, this is provably correct, that the answer is yes. And here are the reasons why and the documentation I have and why I feel confident in that. Or it'll tell you, actually, I don't know the answer. Here's some suggested prompts that I recommend you put back into the engine to see if you can get the answer correct. Cause I can't, I can't tell you, I came up with yes, but I actually don't know for sure that it's the right answer. Change the prompts and it'll give you kind of tips and hints on how you can re-engineer your prompts or ask additional questions to come back until you get an answer that's a for sure answer,

• 28:21Speaker 01

  that's provably correct by automated reasoning. So by that, by this kind of mechanism, you're like systematically able to actually mathematically prove that you got the right answer coming out of this and completely eliminate hallucinations for that area, right? It doesn't mean that we've eliminated hallucinations all along.

• 28:38Alex Kantrowitz?

  But just for that area.

• 28:38Speaker 01

  Yeah, if you go ask it, then, you know, who's the best pitcher on the Mets? It may or may not answer your reasonable question.

• 28:43Alex Kantrowitz?

  That's a tough one. It's a tough question, right? Maybe there is no correct answer to that one. Although, pretty good season this year. Yeah. But let me ask you, what you're talking about also is very similar to what Marc Benioff talked about on the show last week, where he said that because companies have large stores of information within his platform, agents will be able to go in and pull it out and then present it and sort of help create the linkage to go from step A to step B. And it was interesting to me because I had always thought agents are going to be something that may be built by Anthropic, where it's my individual agent that goes out into the world and does what I need. And I think both you and Benioff, and correct me if I'm wrong, have this idea that the agent is going to be something that I'm going to interact with

• 29:25Alex Kantrowitz?

  when I'm speaking with the company, or actually is going to perform tasks at work. Maybe that's going to happen before consumers get them.

• 29:32Speaker 01

  Yeah, I think that that's right. I think that agents are going to be a really powerful tool, actually. Another thing that we're launching this week. Is, you know, one of the things that agents today are quite good at doing relatively simple tasks, right? And, and you can have an agent that goes and, and what they're very good at actually is tasks that are pretty well defined in a, in a particular narrow slice and go accomplish something. And so what a lot of people are doing is starting to launch a bunch of agents, right? One that's very good at going and doing, you know, one particular task, another one that's good at another task, another one's a good at another task. But increasingly, you actually need those agents to interact with each other, right? So we have an example in my keynote where we talk about, if you're thinking about, should I launch a coffee shop? And you actually, you're, say you're a global coffee chain, you want to say, I'm going to launch in your location here.

• 30:15Speaker 01

  You might have an agent that goes out and investigates what a, you know, what the situation is at a particular location. You might have another agent that goes and looks at what are the competitors in that area? You may have another agent that goes and, and does a financial analysis of that particular area, another one that looks at the demographics of that zone, et cetera. And that's great. So now you have like half a dozen, dozen of the agents that go and do a bunch of these things. Saves you some time, but they, they're actually kind of interact with each other, right? Like the demographics may imply, like they may, and they may change your financial analysis as an example. And so that's super hard. And then if you want to do it across a hundred different locations, see where the best one is, that's also hard to do. Like, and it's super hard to coordinate because actually those also may be interrelated too.

• 30:57Speaker 01

  Like, uh, you know, putting a coffee shop here and then another one two blocks down may interact with each other. They can't be independent. So we launched a multi-agent collaboration capability where you basically have this kind of super agent brain that can actually help collaborate across all of them, break ties between them, help like, uh, pass data back and forth between them. And so we, we think that this is going to be a really powerful way for people to.

• 31:19Alex Kantrowitz?

  This episode is brought to you by AppPoint. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data and most teams have no single way to see it, secure it and prove it's under control. That's exactly what AppPoint does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. Avpoint, the unifying trust layer for AI.

• 32:03Alex Kantrowitz?

  Head to avpoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast. Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a wide glove enterprise security service, they protect your people with continuous personal data removal,

• 32:46Alex Kantrowitz?

  proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwell.com/bigtechnology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwell.com/bigtechnology. Learn how to stop online threats before they become real world attacks. The link's in the show notes.

• 33:19Speaker 01

  Really accomplish much more complicated things out there in the world. With, again, there's a fundamental model under the covers that's driving a bunch of this reasoning and breaking these jobs into individual parts. And then the agents go and actually accomplish a bunch of this work.

• 33:33Alex Kantrowitz?

  Okay, I'm just going to say before we go to break, I appreciate how much news that you're thinking into this.

• 33:37Speaker 01

  Yeah, sure.

• 33:37Alex Kantrowitz?

  This is the ultimate number of keynote announcements that have been. Introduced into a podcast, so thank you for that. All right.

• 33:45Speaker 01

  Welcome to Reinvent.

• 33:45Alex Kantrowitz?

  Exactly. All right, we're going to take a quick break and come back with Matt Garman, the CEO of AWS. And we're back here on Big Technology Podcast with Matt Garman of AWS. Let's talk about some broader, some, you know, more earth centric topics and starting with nuclear.

• 34:03Speaker 01

  Yeah.

• 34:03Alex Kantrowitz?

  You have invested 500 million in a company called X Energy to do nuclear. You're also part of, I would say, a wave of companies that are reanimating nuclear energy in the United States.

• 34:14Speaker 01

  Yeah.

• 34:14Alex Kantrowitz?

  And part of that is because these nuclear plants just didn't, they had excess capacity that they needed to get off their hands.

• 34:21Alex Kantrowitz?

  I just want to ask you a broad question about, should we really believe in this moment for nuclear? Because on one hand, it's for the moment cleaner than fossil fuels. On the other hand, we don't really know what happens with nuclear waste. We can't get rid of it. It has to sit in silos that could be damaging for the planet over time. So is it really, and this is sort of a sensitive one, but is it really an improvement to go to nuclear? And how can we be sure because of the long-term effects here?

• 34:47Speaker 01

  Yeah, like I think nuclear is a fantastic option for clean energy. It is a carbon zero energy that has a ton of potential. And as you look about the energy needs over the next couple of years and really the next couple of decades, whether it's from technology or broadly in the world, whether it's electric cars or just the general electrification of lots of things in our world, we're going to need a lot more energy. And it's, you know, we, the Amazon are one of the biggest investors in renewable energy in the world. In the last five years, we've done 500, over 500 renewable projects where we've added and paid for new energy to the grid, whether they're solar, wind or others. And so, you know, and we'll continue to continue to invest in those projects.

• 35:30Speaker 01

  I think they're, they're super valuable and there's probably not going to be enough of those soon enough for us to really get to where we want to get from a clean energy perspective. And so I think nuclear is a huge portion of that. You know, look, there's always the fear mongering from like back in the sixties and seventies of what nuclear used to be. Nuclear is an incredibly safe technology today. It's much different today. It turns out technology has changed in the last 50 years. It's improved a lot. And so there is a ton of improvements in that space. And we think that it is a both a very safe, very eco-friendly energy source that is going to be critical for our earth. If we're going to keep for our world as we keep ramping our energy needs. And we think that as part of that portfolio, right? You're going to have solar, you're going to have wind and you're going to have other. But nuclear is going to play an important role in that.

• 36:14Speaker 01

  And we're excited about what that potential looks like. You mentioned X energy. We do think that, you know, over the next, probably starting somewhere in 2030 and beyond. These small modular reactors, which is what X energy builds are going to be a huge component of this. And so one of the and there they'll be part of that portfolio of offerings. But today, all these nuclear plants that people build are really large implementations. Like they're multi billions and billions of dollars to go build these energy plants. They produce lots of energy, which is great. But they're obviously all that energy is in one location. And then you have to invest in a ton of transmission to get the energy to the actual place you need it to go. And they're big projects. These small modular reactors are much smaller.

• 36:57Speaker 01

  You can actually produce them almost like you produce gas turbines, like in a factory type setting eventually. And you can put them where you need them, right? So you can actually put them next to a data center where transmission is not going to have to be an important factor. And so we think that that's a great solve for a portion of the world's energy needs as we continue to evolve over time. And it's one of the components of an energy portfolio that we're very excited about.

• 37:21Alex Kantrowitz?

  Okay, so we'll be watching that closely.

• 37:22Speaker 01

  Yeah.

• 37:23Alex Kantrowitz?

  On the state of the economy, AWS had a few quarters of stagnant growth. It was still impressive growth, but it flatlined for a moment. And part of that was because customers.

• 37:33Speaker 01

  Quite flatlined, but it was down from where it had been.

• 37:35Alex Kantrowitz?

  Okay, but that's, I'm just talking about the percentage of.

• 37:38Speaker 01

  Yeah, yeah.

• 37:40Alex Kantrowitz?

  Part of that was because the economy was in a rough moment. Everybody was looking for efficiency. And so what you did was, I think you made some deals with customers to help get their bills down, or help get them the most out of what they were doing so they could, you know, effectively live that efficiency.

• 37:55Speaker 01

  That's right.

• 37:55Alex Kantrowitz?

  Motto. What does it look like right now? Is the economy back or people still in efficiency mode?

• 38:01Speaker 01

  Yeah, I'd say, and by the way, it wasn't even just deals. We went and proactively jumped in with our customers and helped them figure out how they could reduce their bills. And we looked about where they could consolidate resources, where they could move to cheaper offerings, where they could maybe do more with less. And we were really proactive about helping customers reduce those costs because we thought, from our view, one, as important for them is they thought about how they got their economics in the right place. And it was the right thing to do for them and built that long-term trust. Now, customers, I think, number one, a lot of them have been optimized, right? And there's only so much you can kind of squeeze into an optimized place and customers are still looking for optimizations.

• 38:41Speaker 01

  But a lot of that work has been done and they're using some of that optimization to help fund some of the new developments that they want to do. A lot of that is in the area of AI. Much of that is in the area of migration and modernization where they're moving from on-prem into a cloud world. And so some of those optimizations they did are helping them fund some of that work. That's moving more of their workloads to the cloud, that's moving and letting them go and build new AI experiences in AWS. And so that is where you've seen our growth start to come back up as a percentage basis. Some of that is customers leaning into those new experiences and doing some of those more of those modernization migrations.

• 39:17Alex Kantrowitz?

  Okay, I want to wrap on a culture question.

• 39:19Speaker 01

  Okay.

• 39:20Alex Kantrowitz?

  Andy Jassy recently emailed the company and he said that as a consequence of scale, and I'm going to get it exactly as he said it. He says, there have been pre-meetings for, for pre-meetings, for the decision meetings. A longer line of managers feeling like they need to review a topic before it moves forward. Owners of initiatives feeling less like they should make recommendations because the decision will be made elsewhere. Was that going on within AWS and what is the process to change that?

• 39:49Speaker 01

  Yeah, I think it's across Amazon, so it wasn't specific to the rest of Amazon. It was definitely inside of AWS too. And I think, look, it's kind of a natural evolution. Like, we have these leadership principles inside of Amazon. And I think one of those ones that's important for us, a couple, we have a couple that are things like being customer obsessed and really understanding the customer. And in order to really understand the customer, you've got to be close to the customer. And so a flatter organization, the more layers you have, the more removed you are from customers. And so we just kind of fundamentally, as we were growing, and then we went through an area of explosive growth of just the number of people and the size of the company and the size of the business. And so throughout that, we just didn't always have the organizational structure exactly right. And so it's, you know, we believe that a flatter organizational structure is better.

• 40:33Speaker 01

  The closer you are to the customers, the better decisions you're going to make, the faster decisions are you going to make. And you really want ownership to be pushed down to the people who really are making some of those decisions. And when you have a very kind of hierarchical organization where people don't feel like they have that ownership to make decisions, you go slow. And for us, speed really matters. And so I think Andy was just highlighting some observations we had where, and I think he's incredibly thoughtful on these points and which I appreciate. We've had a lot of debate here where it's nothing is broken, but you could see like really early warning signs or stress around it. And for us, culture is so important and doing the things in the right way, being that customer obsessed, being ownership, like having the right level of ownership is so important for what makes Amazon so special.

• 41:18Speaker 01

  And so kind of getting ahead of there being any problems, it's not like there was any burning problem and we obviously could have just said, done nothing and kind of let things go for a while. But for us-

• 41:27Alex Kantrowitz?

  It's not the Amazon way.

• 41:28Speaker 01

  It's not the Amazon way. And so we were just being proactive identifying that like, hey, look, this is super important for us. And so let's just be aware of it. Let's be like, be upfront about it, think about it and be very intentional as we think about organizational structures and things about where we can land. And I think all of that has been received really well because it turns out not many of those things customers complained about. They were really focused on ownership. They love being customer obsessed and most of that's been quite well received.

• 41:57Alex Kantrowitz?

  So you can be a big company, but not have big company culture?

• 41:59Speaker 01

  That's right.

• 42:00Alex Kantrowitz?

  Um, okay, last one before we go. You've said that less than 20% of all workloads have moved to the cloud so far. What is the max number that that can, can that be 100% in time?

• 42:13Speaker 01

  I was gonna say 100.

• 42:14Alex Kantrowitz?

  No, but what's realistic?

• 42:15Speaker 01

  Um, yeah, you know, I, I think, um, it's a good question. I think if you think about how many workloads are out there, um, I don't know what the max is. I'm very bad about picking the maximum size.

• 42:27Alex Kantrowitz?

  Can I be honest with that? They have some sort of total.

• 42:28Speaker 01

  But I do think, and I actually think that at a minimum, um, I think that that percentage can flip and it could be 80, 20 versus 20, 80 where it is today or even less. I think there's a massive number of applications that just haven't moved. And if you think about, um, you know, line of business applications, as you think about workloads that are in telco networks, if you think about workloads that are running inside of hospitals, if you think about like, it's not even just traditional data center workloads, but there's a lot of these other workloads that'd be, that would be much more valuable. They'd be much more connected. They'd be much more, um, able to take advantage of advancements in AI if they were connected into the cloud world and, and running there. And so I think that there's a huge opportunity for us to continue to expand what it means to be in the cloud and to, and to continue to migrate many of these workloads that are, that just haven't moved.

• 43:11Speaker 01

  And so, um, there's a massive opportunity, I think.

• 43:14Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman, Whitman's votes drove up costs, cut Medicaid and risked 350,000 Virginians healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com, not authorized by any candidate or candidate's committee.

• 43:43Speaker 03

  In this life, you can't win.

• 43:45Speaker 04

  When the world gets you down.

• 43:47Speaker 03

  No!

• 43:47Speaker 04

  Escape to Pluto TV.

• 43:49Speaker 03

  This is objectively fantastic.

• 43:51Speaker 04

  Relax into the movies and shows you already know and love, all for free.

• 43:54Speaker 03

  Shwing!

• 43:55Speaker 04

  Even our news is uplifting.

• 43:57Speaker 01

  You stay classy, San Diego.

• 43:58Speaker 04

  Watch School of Rock, Crazy Ex Girlfriend, Anchorman: The Legend of Ron Burgundy, The X-Files, Wayne's World and so much more.

• 44:04Speaker 03

  We're not worthy!

• 44:06Speaker 04

  The best things on TV really are free. You just have to know where to look. Escape to Pluto TV. Stream now, pay never.

• 44:13Speaker 01

  You know, I think kind of flipping that percentage over time could be an interesting opportunity for us. And the size of the pie is getting bigger too. I think that's the other exciting thing about generative AI is that the total amount of compute workloads are actually significantly accelerating too.

• 44:26Alex Kantrowitz?

  And timeline to flip?

• 44:29Speaker 01

  Decades. So I still think we're still ways out for the whole thing to flip. There's just a massive amount of workloads out there, but we'll keep working on them and keep going as fast as we can.

• 44:38Alex Kantrowitz?

  Mike Arvin, great to meet you. Thanks so much for coming on the show.

• 44:40Speaker 01

  Yeah, thank you.

• 44:41Alex Kantrowitz?

  All right, everybody. Thank you for listening. We'll be back on Friday breaking down the news and we'll see you next time on Big Technology Podcast.