Amazon's Longterm AI Vision — With Matt Wood

Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. --- Matt Wood is the VP of AI Products at Amazon Web Services (AWS). Wood joins Big Technology Podcast to discuss the current state and future potential of AI, according to Amazon. Tune in to hear insights on how customers are adopting AI, the importance of model choice and specialization, and the evolution of AI in the near-term. We also cover AWS' AI platform, Amazon's Alexa assistant, and the cultural shifts needed for organizations to successfully leverage AI. Hit play for an engaging and informative conversation on the cutting edge of AI with one of the industry's leading experts. --- For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Want a discount for Big Technology on Substack? Here’s 40% off for the first year: https://tinyurl.com/bigtechnology Questions? Feedback? Write to: [email protected] Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Alex Kantrowitz?

  The VP of AI Products at Amazon Web Services joins us to discuss what people are actually building with the technology and whether it's worth the investment. All that more is coming up right after this. This episode is brought to you by Genesis. What does it actually take to put Agentic AI to work across an entire enterprise? I recently spoke with Genesis Chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, Head of Enterprise Business Solutions at Voya Financial, to go inside a large-scale Agentic CX transformation, including the business decisions, trade-offs, and lessons along the way.

• 0:43Alex Kantrowitz?

  You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. Well, a year later, we have Matt Wood back with us today. He's the VP of AI Products at Amazon Web Services.

• 1:26Alex Kantrowitz?

  Last year, we spoke at the AWS Summit in New York City, all about Amazon's AI strategy. And we have a great opportunity now to talk a little bit more about where the AI field is heading. Matt, welcome back to the show. Great to see you.

• 1:41Speaker 01

  Good to see you, too. Thanks for having me back. This is awesome. And congrats on the growth of the show. It's been amazing. I listen every week, so it's a pleasure to be here.

• 1:49Alex Kantrowitz?

  Thanks so much. That's awesome. So you'll have some context here. So let me ask you, I think, the most pressing question that I have first, which is, when we spoke last year, you said, "I wouldn't be surprised if just the AI part of our cloud computing business was larger than the rest of AWS combined in a couple of years." So I'm actually curious where it is today, but before- Before we get into that, here's the sort of disconnect I have. So obviously, we spoke last year, there was all this potential with AI. We've been talking about it on the show a lot. And yet, I was just speaking with a colleague of yours who referenced this Gartner study that said only 21% of AI proof of concepts. So the different programs and products within companies actually go into production. That's a one in five

• 2:33Alex Kantrowitz?

  rate, which is, which is not great, given how much effort and money it takes to get these things going. So talk a little bit about like where the potential and the state of AI building today, why there's that disconnect and where we might be heading.

• 2:47Speaker 01

  Yeah, happy to talk through it from, from my perspective. I've been very fortunate over the past year or two to talk to literally hundreds of customers in every single industry. And I have honestly not seen this level of energy and enthusiasm for any technology, probably since the advent of the cloud from customers. Most customers are investing very diligently. They're making good progress. There is a group which is moving slightly faster than the average, which is somewhat counterintuitive. And that group is actually the regulated industries. And so it's folks like in financial services and insurance and healthcare and life sciences and manufacturing.

• 3:28Speaker 01

  And they're able to move a little bit faster in part because all the regulations that have been, they've had to comply with over the past 20 years. That probably felt at the time like a bit of a headwind, have actually driven the right set of behaviors for that group to be successful with generative AI. And so they have, you know, all of the governance of their data figured out. They understand the quality of their data. They understand which data can be used where, by whom, and for what purpose. They have very, very large amounts of private text data, exabytes of the stuff in some cases, which are market reports or clinical trial results or insurance document, life insurance documents, those types of things that the models have never seen before,

• 4:11Speaker 01

  but are really good at looking at and reading and summarizing and connecting the dots and finding disconnects. And that just earlier They're in their kind of digital transformation journey. And so they've probably looked across and felt like they were kind of sitting to the side. As you know, other areas like retail and transportation and hospitality and media kind of went through this very aggressive digital transformation over the past 10 years or so, driven by the web and driven by mobile and a lot of other factors, including the cloud. And these organizations are looking to not just use generative AI to catch up, but to actually leapfrog ahead using the data that they have, which is privately held.

• 4:53Speaker 01

  So that's one area that I think is probably a little counterintuitive. I don't think I would have guessed, you know, even a year ago or two years ago, that a 160-year-old life insurance companies would be in the vanguard of really delivering value through generative AI. But they have these very, very large document stores of 90-year-old life insurance documents, which are probably going to pay out in the next decade or so. And they've been scanned at some point, but no one's ever read them. And they're not sure what level of risk is associated to those documents inside their business. And so they're able to use generative AI to be able to, you know, piece that risk together and understand it more completely. And so I think.

• 5:34Alex Kantrowitz?

  So I see that you're going to say like the structured, the companies with structured data who have it very organized and have this partnership already are going to be the ones that are going to benefit the most, which is, that makes sense. But then there's also like some of the more glaring issues. Change management is difficult. The models still cost too much to run. They're not quite good enough yet, like last year, and we're going to get to it. But last year we were talking about agents and all these other, you know, advanced use cases. And they've clearly not hit the way that they are supposed to. So what do you think about these limitations? Aren't they the main things that are holding back the field versus just like getting their data?

• 6:10Speaker 01

  Well, I think there are limitations to the technology today. And part of being successful with the technology is understanding those limitations at a deep level. And you referenced, I'm not familiar with the details, but you referenced kind of 20% of prototypes, you know, going into into production. Honestly, that sounds pretty good to me. Like, if you think of just the amount of experimentation that is happening inside organizations around generative AI, just the number of experiments that are being run on AWS for different companies in the regulated industries and all the other industries that I mentioned. You know, Bedrock, which is the service that we make available to customers to build generative AI applications.

• 6:55Speaker 01

  That's one of our fastest growing services ever. And all up, AI and machine learning at AWS is already a multi-billion dollar business in terms of ARR. So there is a lot happening, and I think that 20% is actually pretty good because the denominator is absolutely massive. And when technology shifts happen, you really do want customers to be able to innovate, to be able to experiment really safely, really quickly with that technology to find out what works and what doesn't work. And we're dealing here with a technology which is, just in its very earliest days, it's much more like a discovery than it is an invention. We discovered that if you build these very sophisticated mathematical models,

• 7:36Speaker 01

  that there is emergent behavior within them that resembles reasoning, that resembles intelligence. And we're applying that in some places, and some of those applications will turn out to be successful. And it is no surprise to me at all that some of those experiments turn out not to be successful because if you're experimenting in the right way, a lot of those experiments are going to fail. And so it's why customers in part turn to AWS for running some of these workloads, the majority of these workloads, because they're able to broadly democratize the way that these applications are built using generative AI. And they're able to validate the ones that work really, really quickly. And then, when they find that 20% that works, they're able to take it into production

• 8:19Speaker 01

  very quickly and at a very, very large scale with the right cost structure around it as well. And so I think that the 20% is a little misleading if you think the denominator is small. But that denominator is massive because there's just so much experimentation happening. And we see it on AWS and inside Amazon as well.

• 8:38Alex Kantrowitz?

  Right, right. And look, this is where I always kind of get tripped up because we talk about these, you know, these big emergent, big things like emergent behaviors and models being able to do reasoning and how it's a discovery. And then we talk about, okay, so what practically are they doing? And it's like, well, they're combing through insurance documents. You know, shout out to all the folks working in insurance, and I'm sure we have some listening to the show. But I'm like, man, if we made this, I mean, if, if people in the tech field made this discovery, that models are intelligent and can think for themselves. And like, the thing, that's, that's like, one side of this. And then, but then we ask when it's applied. And it's like, well, insurance adjusters are a little bit

• 9:20Alex Kantrowitz?

  more efficient. And it's like, can that, because we've, the market has valued and the industry has sort of started building around these like, discovery and use cases, the reasoning, the emergent behaviors. But then we ask practical, and it's like, the most boring applications you could possibly imagine. So, is that going to change?

• 9:40Speaker 01

  It's interesting you say boring, because boring workloads are boring because there's so freaking many of them. They're just everywhere. And so, yes, I absolutely believe that there will be large step function changes in a significant number of industries that are going to drive, you know, orders of magnitude improvements for the organizations that work on them and society at large. One example is just computational biology. And we can talk about that in more detail, but the work that's going on there, in terms of using generative AI, the likes of the Dana Farber Cancer Institute, or Genomics, Genomics England, or Pfizer, or the work we've done with a startup called Evolutionary Scale,

• 10:21Speaker 01

  to be able to use generative AI, to be able to design entirely to entirely new molecules, to design entirely new antibodies, that are manufacturable, that can go on and find new drug targets. Like, that is a major opportunity and step function. It's early, for sure. The company just went out of stealth. They just published their paper, which is a great paper. Recommend everybody read it just for background on what's happening in that field. But I absolutely believe that There will be many different step functions forward in multiple different industries of that of that format. I also think that there is a huge number, some of it's going to be long tail, but just a huge number of, you know, what you call boring workloads that are going to be completely reimagined through the use of generative AI.

• 11:06Speaker 01

  And that's okay. You actually want a lot of that boring work to be automated. You want a lot of that work to be improved. You want to be able to channel the boring work, which has maybe inside some organizations is seen as a bit of a, just as a cost center. And to be able to turn that on its head and and channel it into something which drives invention and drives growth. And this is exactly what we saw with cloud computing in the early days as well. Now, I I literally could have said that sentence. In fact, I think I did, you know, with the advent of of cloud computing. That there is, you know, a huge number of workloads inside many, many enterprises that can take advantage of not just the cost savings in the cloud, but can take advantage of the agility in the cloud.

• 11:50Speaker 01

  And take something which is traditionally considered a cost center, building out data centers, which offer no undifferentiated value, and turn it on its head and drive the right cost structure and the right agility to be able to use that infrastructure to drive new product creation, new invention, and reimagination of all of these different products. And so what we consider boring today is going to be rechanneled, in my opinion, into, you know, much more a high-leverage growth opportunities for many organizations.

• 12:22Alex Kantrowitz?

  And there's such a big change management component to it as well, right? We talk about the models, right? There's a cost, there's a capability of the model. But also, you know, one thing about trying to reimagine how boring work is done is there's a lot of people who are sort of used to that work. What percentage of the workplace do you think or the workforce do you think is really ready? To like, let's say this AI can can revolutionize the way they do work. What percentage of the workforce do you think is ready to take advantage of it?

• 12:51Speaker 01

  It's a good question. I'm not sure I would peg it as kind of ready. I suspect that whilst there will be these step function changes over the long period, I think in the shorter term, in the shorter outlook, it's going to feel a lot more incremental than we're probably used to. There's an old adage of a story that folks tell that when we finally discover that there's life on another planet in another galaxy. Yeah, we all have this idea that this will be a huge, you know, societal shifting event for the planet, that we discover there's life on another planet.

• 13:30Speaker 01

  But in reality, I suspect there's just going to be lots and lots and lots of small iterative announcements. And that when the NASA press release comes out that there's life on another planet, it will seem really obvious at that point. And it'll, from now to when that eventually happens, yeah, that's a really big jump. But incrementally, we'll get there incrementally, not in one big shift. And I think the same thing will apply here. There will be over a long term, like big incremental shifts in how we deliver products, in how we deliver technology, and how we interact with data and information and each other. But it'll probably appear kind of incrementally. And having patience and having a long-term view allows you to drive more of that value incrementally

• 14:15Speaker 01

  and allows you to experiment more. And it allows you to have big goals and kind of, you know, iterate yourself to iterate your way to greatness. And having that long-term view, I think, is, to go back to your question, one of the most important cultural shifts that organizations will need to make. You're going to need to have the right teams for sure. You're going to need to have the right talent. You're going to need to have the right technology. You're going to need to partner with the right organizations to be able to drive that technology. But having the ability to be able to take a long-term view so that you can allow Those creative inventive builders

• 14:51Alex Kantrowitz?

  Right.

• 14:52Speaker 01

  To be able to use that technology to be able to iterate and improve and experiment and invent. That requires discipline from a leadership perspective. It allows, it requires you to set up kind of small blast radius experiments. And it requires the organizations to be very tolerant to that failure. Because experiments failing, you've learned something there if you've set it upright. And that learning is disproportionately valuable at this point in the kind of technology cycle. And so that cultural element that you outlined is absolutely critical. I'd actually say it's more like 50% technical, 50% cultural. In terms of the weighting of the elements of investment, they're going to be required to be successful.

• 15:34Speaker 01

  So I'm not sure exactly what percentage right now is kind of ready. I would guess if I had to put a number on it, I would say it's probably 25%, 35% in most large sized enterprises. But over time, you know, if you look three years out, five years out, 10 years out, whatever it might be. With that long term horizon, my guess is going to be 100%.

• 15:55Alex Kantrowitz?

  Yep. Okay, I'm going to ask a follow up on that, but first, you believe in aliens?

• 16:01Speaker 01

  I think, I think you have to believe in aliens if you understand just how big the universe is. It just seems incredibly unlikely that we have hit the absolute only magical sweet spot in the whole universe to encourage, you know, carbon to be able to animate and dance around as we do as humans every day. So the probability of it just being limited to Earth seems very, very unlikely. Although I acknowledge the paradox that if there's life out there, you know, where is it? So that's why I kind of like that.

• 16:39Alex Kantrowitz?

  Or, I mean, yeah, they could also all be dead. And we might be like right now, like the only living. I mean, I think there probably are some sort of life forms out there that have existed in the universe. You know, either before or will come after, but to have them exist concurrently is.

• 16:54Speaker 01

  Yeah, I agree.

• 16:55Alex Kantrowitz?

  That's the question. All right, let's, let's, yeah.

• 16:57Speaker 01

  Yeah, go ahead.

• 16:58Alex Kantrowitz?

  I just want to get back to the AI stuff. I guess we could do another show on aliens.

• 17:02Speaker 01

  I would love it.

• 17:04Alex Kantrowitz?

  Goodness. All right. So you're what you're saying about patience, incrementality, you know, 25% of the organizations being ready and replacing the boring stuff. That all sounds good, but it also makes me wonder if we're going to end up in a sort of trough of disillusionment with this technology because there's been so much hype and so much money that poured into it that are demanding almost a revolution now. And what you're describing isn't a revolution or isn't a quick moving revolution. It might be a slow moving incremental sea change, but not something that happens immediately. It's not something that, you know, the Wall Street types, for instance, will be like thrilled to know that it's just

• 17:48Alex Kantrowitz?

  going to take a while because they think in quarters. So do you think we're there's a risk here in like within the next few years, sort of the public perception of this technology turning a little bit because of the incremental incremental nature of it?

• 18:02Speaker 01

  I think it would be a possibility if, and it's a huge if, if the technology wasn't poised to improve. So if what we if you believe that what we have today is pretty much what we're going to have to work with, with only incremental, small improvements over the next three, five years, you know, then I suspect that, you know, folks will feel like, you know, the the promise on this occasion hasn't been delivered on. But, you know, technology tends to follow an S curve over time. And, you know, you you get to the top right hand corner of that S curve, and you end up with the technology with the capability, and you get these, you know, just decreasing improvements over time. You never really

• 18:46Speaker 01

  know where you're at on the S curve until you're looking backwards. And so it's kind of hard to judge where we're at. I think most people would think we're probably in that kind of middle section, high gradient piece, just because there's so much happening, and there's so many, yeah, so many improvements. There's new models and new techniques and new technologies from academia and the public sector, private sector. And I have no doubt that by the time we finish this conversation, there'll be another technique out there that is worthy of our attention. But my guess is that it's probably more likely that we're at the bottom left-hand corner. I don't think we've hit the kind of hockey stick inflection point yet of what this what this technology is capable of. It's still very, very, very early.

• 19:30Speaker 01

  So at some point, we're going to hit that hockey stick inflection point. And it always happens with different technology shifts. It can take more or less time depending on the shift and the speed of the technology. If you look at, you know, the and the thing that triggers the S-curve bend is different in a in a number of different ways. So if you kind of look at the maturation of the internet itself, you know, that that hockey stick inflection point, it really, I think, landed with the development of kind of SaaS-style Web 2.0 applications, whether it was whether it was web mail or whether it was finance systems, whether it's hotel booking systems, whatever it was. That capability of being able to have access to those types of services, that made and

• 20:15Speaker 01

  the fact that you could integrate those services kind of through APIs and do interesting things with them, that meant that every new service that was added to the internet made all of the other services more valuable. And that's kind of what pushes you up the S-curve in many times. You saw the same thing with kind of the mobile transformation, where we had these remarkable new devices, we had these these these applications that more and more people invested in, more and more organizations invested in, they became more and more sophisticated. And over time, the operating systems on which those applications ran allow those those those applications to interoperate and interact in interesting ways, both with the operating system and with each other. So every net new application added makes all of them more more more useful, makes the

• 20:58Speaker 01

  whole system more useful, the whole device in your pocket gets better over time without you having to do anything. And so that pushes you up the S-curve as well. And I don't think we're at that point with generative AI yet. We have a really robust set of really interesting, really powerful models, which are going to mature over time. But customers will, I'm sure, find interesting ways to combine those different models. There isn't one model to kind of rule them all. Each different model has, you know, different sweet spots. And it's my expectation that most customers will invest in, not building the foundation models, but will invest in fine-tuning and improving those individual models and customizing them in interesting ways for their own use case. And and those capabilities are interesting in isolation, but part of what will push us up the S-curve that we're seeing

• 21:44Speaker 01

  With customers at AWS and at Amazon is that combining those models together, leaning into the sweet spot of all these different models, allows you to build systems that in aggregate are have a compounding effect on intelligence. It's not additive, it's a multiplier. And so that's going to push us a little bit further up the S curve. I think another really interesting area and the one that's probably closest to SaaS applications and mobile apps is what you mentioned earlier is agents. I think agents have a good chance of being the apps for the generative AI world and the generative AI era. And that as we add more of those and we find ways to orchestrate multiple agents together, and there's already customers that are building multi-agent systems on AWS today

• 22:29Speaker 01

  that combine specialties and combine agents that can goal seek on your behalf and collaborate or contest with each other in interesting ways. That means that every new agent that's added to the system drives you up the S curve. It makes all the other agents, you know, more useful at the same time without you having to do anything. And that's a really big part of it.

• 22:48Alex Kantrowitz?

  But are agents an actual thing in production now?

• 22:51Speaker 01

  Yeah, I think so. You know, we have.

• 22:54Alex Kantrowitz?

  Because like I'm just going to say like last year we spoke about like you made you made an announcement about how agents were. Agent building technology was on its way. And it's just like a full year has gone by and I haven't seen one example of like of a realistic agent going out there and taking action for people.

• 23:13Speaker 01

  Well, I think I've certainly seen some. I use some on a day to day basis.

• 23:17Alex Kantrowitz?

  Let's hear it. Yeah.

• 23:18Speaker 01

  Yeah, for sure. I'd recommend you check out a couple of things that may be interesting to you and the audience. One is a startup company called Ninja Tech. You can check them out at ninjatech.ai. They have a assistant assistive system. You can interact with natural language as you may be familiar with. But they also have under the hood a set of specialized agents that can perform different tasks on your behalf. So they have a they have a researcher agent, they have a scheduling agent, they have a web agent, all sorts of different agents. And just by asking your question, they interpret the question and then they have an agent which looks at the response and says, hey. This looks like you're doing some research.

• 24:02Speaker 01

  Let me ask my researcher how I can best help you. And I'll set a problem to my research agent. And that research agent runs off and does its thing. And they say, oh, there may be some web data that will be useful here. I'll set my web agent off to go and collect that data for me. And so on and so forth. And it pulls back all the information together and allows you to interact with your with your calendar and with your schedule or your email or whatever it might be in levels which are much more automated than than you could do with just a standard assistive chatbot. So that's one example.

• 24:34Alex Kantrowitz?

  But if this stuff is is so useful, then why hasn't it broken out into the public?

• 24:37Speaker 01

  I just think it's very early. You know, today, agent systems or agentic systems, as they're sometimes called, that is still relatively early. But I think they are breaking out, to be fair. I think Ninja Tech is seeing remarkable growth. They have hundreds of thousands of monthly active users. We've also built some really powerful and popular agents on AWS. And so we have a an assistant for builders that we call Q, Amazon Q. And Amazon Q allows you to generate code if you're building software. It will take a question and give you answers and give you guidance on how to build on AWS and all the things you would expect. And that's useful. That gets you a bump in productivity. We've seen some customers get, you know,

• 25:22Speaker 01

  in terms of just the amount of code that is automatically generated that they accept. It's usually between thirty five and fifty percent. It's higher on Q than than any other comparable service. But the thing that drives productivity for developers is what we call the developer agents inside Q. And so with the developer agents, you don't just ask a question about what code to write or write a comment and get the function back. You actually set a task to Q. You say to Q, hey, I want to add this feature to my software. Here's Q looks at the software across across your repository. It looks at the changes that you've made inside your development environment. It understands the type of change or the type of feature that you want to make. And it goes off and it looks at all of that information

• 26:07Speaker 01

  and it makes a strategy. It doesn't just generate the code. It makes a strategy for how to add that feature. So it picks which functions need to be updated, which modules need to be added, which tests need to be run, which documentation needs to be added. And you get a chance to review that strategy. And at some point, you can just say, hey, Q, go for it. And Q will go off and work through diligently through its through its to do list to create a set of software changes that you can choose to commit, which add that feature to your code. And so if you can imagine a developer going from having to just write or generate that code manually to having tens or dozens or over time hundreds

• 26:50Speaker 01

  of those developer agents running around doing the work on their behalf, you get this, you know, a combinatorial explosion of productivity. We do the same thing for code transformation. And so if you want to move between different versions of Java, you know, we support that today. You just say, hey, update this to be compatible with Java seventeen, whatever you're running. It will go off and make that same strategy. It will work diligently through it and then allow you to review the results and you can choose to accept those and commit them back. And that's a that's a fixed cost effort that most organizations have to go through. We need to move software project A from Java X to Java Y. And it's going to take ten people and it's going to take three months.

• 27:33Speaker 01

  And we're just going to have to it's just a cost of doing business. We're just going to have to pay that cost, pay that cost in people, pay that cost in productivity. And this is a task that no developer really likes to do. It's kind of toil work and the very best outcome.

• 27:47Alex Kantrowitz?

  That's definitely the boring.

• 27:48Speaker 01

  It's boring, exactly.

• 27:49Alex Kantrowitz?

  That we talked about.

• 27:50Speaker 01

  But it's super powerful because there's so much of it. And so you move from a world where, you know, you have this fixed cost that you just have to pay a cost center, just like we were talking about earlier. And you move it to a point where that is just taken off the table. It's completed automatically. And those same developers get back to actually, you know, moving to doing things which are much more productive instead of that work. So we have we have Java to Java.

• 28:13Alex Kantrowitz?

  And you named it you named it Q because of Q from Star Trek, not QAnon, right?

• 28:19Speaker 01

  It's it's neither, it's neither, but it's a Q.

• 28:22Alex Kantrowitz?

  What was the inspiration for Q?

• 28:24Speaker 01

  It's based on a quartermaster, the idea of a quartermaster where you get your gadgets.

• 28:30Alex Kantrowitz?

  Okay. You guys couldn't have picked a different letter. It's a very controversial letter these days.

• 28:35Speaker 01

  I think it'll work out okay.

• 28:37Alex Kantrowitz?

  We're here with Matt Wood. He's the VP of AI Products at Amazon Web Services. On the other side of the break, we're going to talk a little bit about Amazon's products and also where the models are going next. So stay tuned. We'll be back right after this. This episode is brought to you by Avpoint. Everyone's racing to roll out AI right now. Copilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data, and most teams have no single way to see it, secure it, and prove it's under control. That's exactly what Avpoint does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire

• 29:17Alex Kantrowitz?

  AI estate. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence, so innovation scales without scaling risk. Avpoint, the unifying trust layer for AI. Head to avpoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's A-V-P-T dot co slash big technology podcast. Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and

• 29:58Alex Kantrowitz?

  other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a white glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwell.com slash big technology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwell.com

• 30:39Alex Kantrowitz?

  slash big technology. Learn how to stop online threats before they become real world attacks. The link's in the show notes.

• 30:50Alex Kantrowitz?

  And we're back here on Big Technology Podcast with Matt Wood, the VP of AI Products at Amazon Web Services. All right, Matt. So last year, we were talking a little bit about Bedrock, which is basically a tool that Amazon Web Services customers can use to develop AI models. And the idea that you explained to me was basically Amazon's play for generative AI was that people could, who want to develop on AI, could go in and pick their own models through Bedrock. It could be Facebook's Lambda, or Amazon's proprietary models, or any host of other models, and then they could build that way.

• 31:29Alex Kantrowitz?

  But Bedrock has not integrated OpenAI's GPT models yet, or Google's Gemini models yet. And I was speaking with someone in the know who was basically like, look, like what they're offering is not really choice. It's like one model that works well, which is Anthropics. And they're leaving out the other state of the art models, which is, you know, OpenAI's GPT-4O, and then Gemini. And ultimately, that means that the offering is limited, and in some ways, behind. And I'm curious what you think about that argument.

• 32:02Speaker 01

  I would obviously disagree that it's behind. I think, you know, the interesting thing about these models is that, you know, they can, they can be very seductive when you look at a model in isolation. You know, you can, you can read the benchmarks, and you know, tribes are forming around these models and all those sorts of things. But what we see time and again with customers, enterprises, startups, who are actually building with this in meaningful ways, is that they have a huge number of different workloads. I work with with some customers, and they're very generous, and they send me their roadmap of all the things across the company that they want to be able to apply generative AI to.

• 32:43Speaker 01

  And it's a spreadsheet of, you know, five, 600 rows of all the different things that they want to do with generative AI. You know, it's it's kind of it's kind of intuitive if you if you play that out that there isn't going to be, it seems very unlikely that there's going to be a single model that's going to be the best fit for all of those different workloads. You know, some of those different workloads have different requirements. Some have requirements that are, you know, heavy on reasoning or heavy on the ability to be able to do analysis. Others need to be really good at summarization. Others need to be really, really fast. Others need to be very low cost. And so there's this this multiplicity of use cases that have different

• 33:23Speaker 01

  operational characteristics, whether it is intelligence or latency or or cost, whatever it might be. And customers want to be able to usually map the model to the mission. They want to be able to find the right model for their use case. Because if you have a small number of models or just a single model available to you, it ends up having to play the role of kind of a Swiss Army knife. And a Swiss Army knife sounds great. It's great in a pinch, but in reality, you almost never want a Swiss Army knife. What you actually want is a broad tool best tool belt with all of the specialized tools in there that are a perfect fit for what you're trying to do. If a contractor turned up at your home to do some renovations and all they had was

• 34:04Speaker 01

  a Swiss Army knife, I think you'd be pretty disappointed with their preparation, probably pretty disappointed with with their with their work with their work quality as well. That's right, exactly. Same thing with AI models. You want to be able to match the right model to what it is that you're trying to do. So you can lean into the advantage of that model in whatever it might be. Now, some of those models, you really do want, you know, as much intelligence and as much reasoning capability as possible. And on Bedrock, we make available the the anthropic models, the particularly Claude three and the new Claude three dot five improvements, which drive, you know, not

• 34:45Speaker 01

  just great, a great experience for, you know, a high intelligence requirements, but are the best performing models out there. You know, Haiku Claude three dot five Haiku outperforms all other models on the planet. And so that's that's great.

• 35:02Speaker 01

  And you also want models which are really, really specialized for a specific task. And so we're making the evolutionary scale models that I talked about earlier. They're available on AWS today and we're going to bring them to Bedrock later this year. We have summarization models. We have models which are specifically tuned to build agentic systems. We have models that are specifically tuned to work with reasoning. We have other models that are just really, really, really, really cheap. We have models that are multimodal and will handle different modalities. We have single modality models. We have large models. We have small models. And time and time again, we have seen at AWS, and this is an insight that I think maybe some other providers have not yet had.

• 35:45Speaker 01

  But because of our background in cloud computing, we really recognize the value of optionality for customers. Every single time we have ventured into a new domain, customers have time and again told us that they value the optionality of having purpose-built solutions.

• 36:03Alex Kantrowitz?

  Right. Like being model agnostic is definitely a crucial

• 36:06Speaker 01

  Model agnostic.

• 36:07Alex Kantrowitz?

  aspect of development. Basically, you could swap it, being able to swap in any model.

• 36:12Speaker 01

  I look at it, I look, I look at it more, swapping models, I don't think is quite the same thing. I look at it more like for each individual use case, you want to find the right model.

• 36:22Alex Kantrowitz?

  Picking the right one.

• 36:23Speaker 01

  So if you, if you look.

• 36:24Alex Kantrowitz?

  And that's working well.

• 36:25Speaker 01

  It's working well. You know, Bedrock is one of our fastest growing services ever. We have tens of thousands of customers that are using it today. It's growing, growing, growing like crazy. And it's really based on this observation that we carried over from our cloud computing work. You know, when we started, when we launched EC2, which is our Elastic Compute Cloud, it's our compute platform on AWS. We launched with just a single compute type in a single availability zone. Just one. That was it. That's all you could use. But, you know, the goal was, because we saw it internally at Amazon and customers very quickly told us that one single choice was not what they needed. And so we, today, we have over 400 different instance types in.

• 37:10Alex Kantrowitz?

  Right. But if this choice is working. So well, I want to ask you then, there's a question I've been meaning to ask you for quite some time, which is that maybe it's maybe it's limitations of the models on the platform, or maybe it's the evolution of the models. But Amazon worked on something called Bloomberg, I mean Bloomberg worked on something called Bloomberg GPT GPT on IWS. And this is from Ethan Malik, he's a professor at Warren who studies this stuff. He says, remember Bloomberg GPT, which was a specially trained finance LLM drawing on all Bloomberg's data, made a bunch of firms decide to train their own models to reap the benefits of their special information and data. Here's what he says, you may not have seen that GPT four, the old pre-turbo version with a small

• 37:54Alex Kantrowitz?

  context window, without specialized finance training or special tools, beat it on almost all finance tasks. So I guess I'm curious from your perspective, is it the fact that you didn't have the right models or that the models are advancing so fast that something that could take that much effort to train through this process that makes a lot of sense, could then eventually be surpassed by the next evolution of model from OpenAI?

• 38:20Speaker 01

  Well, for context, those two models were, what, 12, maybe 18 months apart, something like that. And today it looks like models have a shelf life of probably about six months, if you're training on kind of open, open web data. And it's partly why we like working with our friends at Anthropic so much. They are committed to continual and consistent improvement of all of their different models. And, you know, they launched the Claude three set of models.

• 38:47Alex Kantrowitz?

  But if I'm a Bloomberg though, then why would I develop this on, you know, bespoke model if I could be then surpassed by an off the shelf model?

• 38:56Speaker 01

  Well, again, I suspect that, I don't know for sure, but I suspect that for general world knowledge questions, you actually do want a model which is trained on world knowledge. That's really, really useful. But that world knowledge is very, very, very broad. But it's not particularly deep. And most organizations operate at depth. And so there will be questions for sure that you can pose to multiple different models and larger, more modern world models. I'm sure you can find examples that they will outperform specialized models. And I'm absolutely positive that the inverse is also true. That you can find older, smaller specialized models

• 39:41Speaker 01

  that will offer much better, higher quality, lower hallucination results on specific tasks at the depth that most organizations need to read. And so it's an and, not an or. And so if you follow this train of thought where there is a single model that is going to quote unquote, win, I just think it's self limiting because you'll always end up with that being the Swiss Army knife. That presents the denominator on your capability. And that denominator is not guaranteed to grow in the depth that most organizations need to be able to operate in. And so world models are great, they're super exciting. You want them and you want the opportunity to specialize those models and fine tune them.

• 40:26Speaker 01

  You want to be able to build your own models. You want to be able to take existing models and continue to train them. You want to be able to layer in your existing data using retrieval augmentation. You want to be able to adjust the alignment and style and tone of these models in interesting ways. You want to be able to quantize those models if you want to run them at a lower cost or on different environments. There's all sorts of value in optionality and all sorts of reasons why you might choose a different model. And so that is a really good example of where an and of having different models is a really good opportunity for customers.

• 41:02Alex Kantrowitz?

  And you must have a good insight into like where the next level of models are going to go. I mean, being so close with Anthropic, ear to the ground. There's a lot of expectation at the next set, the GPT-5 is maybe the Anthropic 4s. Are going to have sort of, I don't know, godlike capabilities. That's what I like to refer to it on the show, but like, that's the anticipation. What is the realistic expectation for what's coming next on the model front?

• 41:29Speaker 01

  I think it's a good question. I think we'll see a couple of different things. I think we'll continue to see improved reasoning capabilities. The ability to be able to take in larger amounts of data, reason across it with very, very high accuracy, to be able to answer increasingly complex questions, to be able to apply logic to those questions. We'll continue to see improvement in that. I think that I think that improvement will come iteratively, you know, kind of every six months. And probably much more quickly because different model providers are on slightly different schedule. And so I think those will continue to to improve. I also think that there is a undervalued asset in the fact that these models will continue to get better for sure.

• 42:14Speaker 01

  But you also want to be able to layer in your own data in order to be able to get the model grounded at the right level for your organization. And so the world as we see things going forwards is that the models will continue to get better, more capable, more reasoning capabilities. And specialization and customization of the systems built with those models will become increasingly important. And there will be a sophisticated, more sophisticated set of guardrails which are mediating what the models receive and what they generate on the outside. And so you're going to end up in a world, I think, where you're going to have a set of models which are going to continue to improve. Combining those models is going to become, you know, disproportionately advantageous.

• 42:56Speaker 01

  You're going to have a set of data inside your organization, some of which you're going to generate, which is fresh to be able to fine tune those models. Some of which many organizations already have, which they're going to use to ground the models in the reality of their business. And you're going to need a set of capabilities that allow you to bring those components together, as well as kind of manage the generative AI applications. And it's those capabilities that we're kind of focused on, on building it across the board at AWS.

• 43:25Alex Kantrowitz?

  A lot of stakes have been put into what's going to happen in the next 18 months in this Gen AI world. I mean, basically, from my understanding, there's billions of dollars being put into training these next set of models. Everything that you said definitely implies, but it's also just like, there are going to be companies that live and die based off of their next iteration of model. So what do you think is a best case scenario? And what is a worst case scenario for generative AI 18 months from now?

• 43:51Speaker 01

  I think that there's not going to be hundreds of world model providers. I think that there's likely to be maybe a dozen, two dozen, something of that order of magnitude. I think that Anthropic will be one, Meta will be one, Amazon will be one, there'll be others. But I don't think there'll be hundreds of these providers. I think there'll be a small number of providers. And I think over time, they will offer a broader, you see this happening already, a broader family of models which offer different opportunities for optimization. So some of those models will be, hey, the question I am asking is incredibly valuable to my organization. I want to be able to pad it with as much context

• 44:32Speaker 01

  from my private repository as possible, and I want the best possible answer at any cost. That's how valuable that query, that prompt is to me. I think there'll be a lot of that. I also think that you're going to want to run a set of less capable models at much, much lower cost, and everything in between. And so my guess is that these models will not commodify. My guess is that they will diversify increasingly over time, and that the idea that these models will become commodities, defined as you can hot swap them, and that their economics are primarily driven by supply and demand. I don't see that happening,

• 45:14Speaker 01

  and you can see the beginnings of that now, is providers like Anthropic are offering Claude 3 not as a single model, but as a model which has the sliders on its configuration moved in slightly different positions, and offers three different models within a family. I could see that becoming 10 different models inside a family, with a more fine-tunable set of levers around cost and intelligence and capability and latency, those sorts of things. And so I think that there'll be a larger number of models in aggregate, but that the pool of providers probably won't grow much larger than a dozen or two.

• 45:53Alex Kantrowitz?

  Okay, but what is the best case scenario 18 months from now? And what is the worst case scenario?

• 45:56Speaker 01

  Oh, the best case scenario is exactly what I laid out. Like, that is the best case scenario for customers. That offers customers the broadest possible choice. It allows them, you know, by proxy, to be able to address the broadest number of use cases inside their organization. And by proxy, derive the scale which will deliver return on investment, which is commensurate with the value that they're investing. So that's the best case.

• 46:19Alex Kantrowitz?

  But it doesn't, it doesn't seem like that you're anticipating in a best case scenario, models that will really be able to like out, like dramatically outperform what we have today.

• 46:27Speaker 01

  Uh, no, I think there will be. I think that if you look at the differences between, um, you know, Claude 3 and Claude 3.5, you know, the, the way that you measure the improvement is going to become increasingly nuanced. And so today, there is a, uh, in my opinion, misguided belief that, you know, the, the king of the hill will, will, will basically win. There's going to be a single winner here. I don't think that's going to be the case because there is so much value in addressing all of these different use cases. And so, uh, the, the best performing model today is also has a really great cost profile for the, uh, for the, for the intelligence that it provides. That was part of the invention between Claude 3 and Claude 3.5.

• 47:10Speaker 01

  Now, over time, you know, the intelligence will continue to go up and there'll be different optionality within the, within the spectrum. So the customers can, can find that sweet spot.

• 47:20Alex Kantrowitz?

  That's a very interesting idea. Uh, by the way, has, uh, Amazon put all 4 billion into Anthropic now? I know that there was a promise that that was going to happen or an upper bound?

• 47:30Speaker 01

  Yep, we've, we've completed that investment. Yep.

• 47:33Alex Kantrowitz?

  Okay, so then worst case scenario, what do we, like, let's say everything doesn't live up to expectations, like where, when you, when you, you must be game planning this out, like.

• 47:42Speaker 01

  Yeah, I mean.

• 47:43Alex Kantrowitz?

  What do we end up with in a worst case scenario?

• 47:45Speaker 01

  I think the worst case scenario is, there's probably two pieces. One, and this goes back to what we were saying earlier, I think. Uh, the worst case scenario number one is. We've just mismatched where we're at the S, where we're at on the S curve. And we're actually in the top right hand corner. And the capabilities of the, the core technology, the models, the ability for the models to be able to work with data at scale, the capabilities to be able to merge those two things responsibly together. Yet they, they don't mature and improve at the pace that we expect. I think that would be, that would be a disappointing outcome. I think it's pretty low probability at this point, given, given the trajectory that we're on, but that, that could be one.

• 48:25Speaker 01

  And the other is, again, going back to something we talked about earlier, is that the, the readiness of organizations slows down the, the opportunity to deliver on this technology. Um, because they are struggling to manage the change or they're struggling to, you know, really drive reinvention through some of their sort of, you know, cultural biases. And so I could imagine that that is playing out. And I think that's at least as large a challenge for most customers is the, is the way in which you structure and organize and drive and deliver and measure, you know, how exactly you're going to, to kind of operationalize from a business perspective, uh, this, this new technology discovery.

• 49:09Speaker 01

  So that's the worst.

• 49:11Alex Kantrowitz?

  Not every, yeah, not every company reinvents like Amazon, so.

• 49:14Speaker 01

  This is, this is true. We are, we are uniquely designed for speed, which is, uh, which is, makes it an exciting place to work.

• 49:23Alex Kantrowitz?

  Yeah, okay. So on that note, and I think we'll come, bring it home with this one. Uh, Amazon AI guy got to ask about Alexa. I know it's a different division, but maybe there is some collaboration going on today. Everything I've heard about the limitations of Alexa has been that the intelligence within Alexa is effectively hardcoded in there, that there's like, you know, hundreds or thousands of different queries that it's prepared for, and it will respond based off of like a database that it pulls from. And There's been a question of whether Amazon is going to move from that style to a more large language style powered Alexa that will require effectively

• 50:04Alex Kantrowitz?

  a rewrite. And so I'm curious if you think that the question is grounded in fact? And what's going to happen inside the Alexa division of Amazon?

• 50:16Speaker 01

  Well, look, Alexa is, you know, extremely successful personal assistant and has been well received by customers. We have hundreds of millions of Alexa endpoints out there that customers love to use. What's really interesting about the future of Alexa is that part of the success of Alexa has been that the way that Alexa works is that we're very, very accurate at matching the intent of the user to actioning that intent. So that may be simple things like telling a joke or getting the weather, or it could be more serious things like smart home use cases. Now, some of those are turning lights

• 50:59Speaker 01

  on and off, but some of them are locking and unlocking doors, setting burglar alarms and those sorts of things. And so it's really important, a really important capability of Alexa is the ability to be able to perform that mapping. That is very, almost entirely complementary to the kind of revolution that we're seeing with large language models today, which allow us to create a much more natural, much more fluid, much more human sounding, much more intuitive interface to those intents. And so that's what we're working on. We're working on marrying the capability of this remarkable ability to be able to pair an intent to an action with the large language model interfaces that

• 51:42Speaker 01

  have become very popular and allow us to kind of unlock entirely new ways for Alexa to provide assistance for our customers. And so I think it's a complementary marriage between the two technologies and we're hard at work on that.

• 51:58Alex Kantrowitz?

  Is there an LLM in there today?

• 52:00Speaker 01

  Alexa has, you know, over a dozen machine learning AI models under the hood, including, you know, large language models.

• 52:08Alex Kantrowitz?

  And is that going to expand the LLM use cases within the device?

• 52:14Speaker 01

  Yes, part of what we're working on is the ability to be able to take more modern LLMs that have this very natural, easy, intuitive back and forth. That's a really important part of building an assistant and combining that, marrying it with the technical underpinnings, which allows to do this intent mapping under the hood very, very accurately. Now, what's funny, the reason it's complimentary, is LLMs today are not very good at doing that LLM intent mapping. They make mistakes, you need to be able to check them, all those sorts of things. And so, you know, LLMs are good at providing that natural language, that very intuitive interface in ways that is better than Alexa provides today, and we want to take advantage of that. But Alexa today also provides a lot of advantages

• 52:57Speaker 01

  that LLMs are not good at doing today. And so, yeah, that's part of what we're doing, part of what we're working on.

• 53:03Alex Kantrowitz?

  Yep. And so is it going to require a full rewrite of the stuff under the hood of these assistants?

• 53:09Speaker 01

  No, because we want to retain the core capability of Alexa, which is this intent to action mapping.

• 53:17Alex Kantrowitz?

  Okay. Timeframe for that?

• 53:19Speaker 01

  Nothing to announce today.

• 53:23Alex Kantrowitz?

  Matt Whitman, always great to speak with you. Thanks for coming on the show.

• 53:27Alex Kantrowitz?

  All right, everybody. Thank you so much for listening. We'll be back on Friday breaking down the news as usual. Also, Matt is about to hit the stage at AWS's New York Summit, so I'm sure you can find the news that he's going to be making shortly after this podcast hits. All right. Thank you so much for listening, and we'll see you next time on Big Technology Podcast.