Amazon's Ad Business Soars + AI APIs — With Benedict Evans

Benedict Evans is a star tech analyst who’s spent years at Andreessen Horowitz and is now independent. Evans joins Big Technology Podcast to highlight some big, surprising new shifts in the tech industry, which he covers in a new presentation called the New Gatekeepers. In this episode, we discuss how media and retail are blending, with a focus on Amazon's ad business. Stay tuned for the second half, where Evans goes into depth about the potential for AI APIs, especially ChatGPT's. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Questions? Feedback? Write to: [email protected] You can find Evans' presentation here. Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  LinkedIn Presents.

• 0:14Alex Kantrowitz?

  Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. Benedict Evans is our guest today. He's a StarTech analyst who spent years at Andreessen Horowitz and other firms and is now independent. Evans joins us today to discuss the state of big tech, breaking down his big new presentation, The New Gatekeepers, which highlights some surprising new shifts underway in the tech industry. Did you know, for instance, that Amazon's ad business is now bigger than all of newspaper advertising combined? And that it's also bigger than Amazon Prime? Beneath all of this is a trend that Evans points out, which is that the line between media and retail is kind of blurring. What's a media site? And what's a retail site? At Amazon

• 0:57Alex Kantrowitz?

  and some of its peers, that's a bit unclear. That's the kind of stuff we cover in today's episode, along with a discussion of where the chatbot business is heading, looking into some of the APIs. My conversation with Benedict Evans, right after this. This episode is brought to you by Genesis. What does it actually take to put agentic AI to work across an entire enterprise? I recently spoke with Genesis Chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience, and what companies need to get right in the AI era. Then, I sat down with Adam Mitchell, Head of Enterprise Business Solutions at Voya Financial, to go inside a large-scale agentic CX transformation, including the business decisions, trade-offs,

• 1:42Alex Kantrowitz?

  and lessons along the way. You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation.

• 2:15Alex Kantrowitz?

  Hi Benedict, welcome to the show.

• 2:17Speaker 00

  Thanks for having me.

• 2:19Alex Kantrowitz?

  Great to have you back. Really had a fun discussion last time you were on. Back then, Apple was at $3 trillion. It feels like a completely different world today. And I guess that's just how the pace of tech moves.

• 2:30Speaker 00

  Well, there's just as many people online as there were back then. In fact, slightly more. But there's an old line about the stock market and the economy being rather like a dog going for a walk with its owner. And the owner walks in roughly the same direction and the dog kind of zigs back and forth. But they both end up in luck at the same place. And I think that's what happened in the last couple of years. Like, the dog went off chasing rabbit and then it's kind of slunk back with its tail between its legs.

• 3:02Alex Kantrowitz?

  So you think now we've caught the dog and the human in this metaphor are side by side?

• 3:10Speaker 00

  Yeah, don't torture the metaphor. Yeah, no.

• 3:13Alex Kantrowitz?

  We're in a heel state.

• 3:15Speaker 00

  Look, there was, there's a chart early in my presentation of the market cap of Exxon and Zoom. And there was a moment when Zoom had a larger market cap than Exxon, and that was not sensible. There was a moment when, you know, the weirdest Silicon Valley metaphor I heard was people got ahead of their skis. People got, you know, we had a sort of certain loss of discipline, a certain over enthusiasm. But e-commerce penetration is still 20% plus percent. Five billion people out of six billion adults have got a smartphone today. And so none of those things are winding back. None of the actual underlying conversion to technology

• 3:56Speaker 00

  and, you know, destabilization, recreation of markets is going away. We've just got a little bit of a shift of gear.

• 4:04Alex Kantrowitz?

  So wait, do you think this entire big tech drawback, the narrative was overblown?

• 4:10Speaker 00

  Oh, you know, it's it's a long time since I was an equity analyst and I think about you know, valuations as little as possible.

• 4:19Alex Kantrowitz?

  Even beyond the stock, though, just like it's a narrative right now that big tech is in retreat.

• 4:24Speaker 00

  I don't really know. I mean, I don't know what the narrative is really. I mean, I think there's an interest rate thing, there's a macro thing, there is, you know, there was a moment in two thousand and twenty, twenty one, when people sort of thought, wait, maybe we've just kind of jumped in time forward five years. And it turns out, actually, maybe we jump forward six months. We're not quite back on the trend line, but we're not very far off the trend line that we were on before. But it's important. This isn't like what happened in two thousand, when, you know, the growth that people were expecting just disappeared. We are now roughly where we would have been if the pandemic hadn't happened. And the valuations are kind of roughly back where they were before the pandemic. So like, go figure.

• 5:05Alex Kantrowitz?

  Right. So it just feels like there was this moment of irrational exuberance. And now things are picking up a pace.

• 5:11Speaker 00

  They are, yes. I mean, you know, guess what? Zoom is not a global platform that reinvents communication, but business travel is down probably 20% and will stay there.

• 5:21Alex Kantrowitz?

  Do you think that some of the minimized valuations on these big tech companies will change their strategy at all? Like, does Apple at $3 trillion act substantially different from Apple at, let's say, $2.25 trillion?

• 5:35Speaker 00

  Yeah, I think Apple is probably the least affected by all of this. I mean, obviously, there's a certain Apple, Apple, in a sense, is more exposed to things happening entirely outside tech. Like, you know, if we go, as we go into an actual recession, people's willingness to buy $1,000, $1,500 phone slows down a bit. So they've kind of got a kind of a different set of things to think about. For Meta or for Alphabet, e-commerce growth is slowing down a bit, advertising is slowing down a bit, advertising from outside tech is probably slowing down a bit. And so that has consequences for them. And obviously, Amazon more than doubled their fulfillment square footage from 2019 to the end of last

• 6:19Speaker 00

  year, and that was probably too much. And so they're slowing down a bit. But it's not like everybody who was buying online has stopped buying online. I think the interesting kind of there's a sort of a very Silicon Valley dynamic here around things like. I mean, this is obviously the one of the ways that people looked at the chaos at Twitter is why on earth did they have fifteen thousand people working there? Whatever it was, ten, eleven, ten or eleven thousand FTEs and then like another five thousand contractors. And to the number they're at now, which is, I think, last I heard, is like two thousand or something. That's probably not the right number, but the right number is maybe three or four, it's not fifteen. And you certainly would hear a lot from Alphabet and from Meta that they were off.

• 7:04Speaker 00

  They'd hired an awful lot of people, they also couldn't really fire anybody. So there were quite a lot of people kind of hanging around, not doing very much. And it's a bit like the old joke, you know, how many people work at this company? About half. And that was starting to feel, to feel a little bit true at Alphabet. So yeah, there's going to be a bit of a, you know, the kind of brutal phrase or anodyne phrase correction. But, you know, the underlying dynamics of the growth that we were having in two thousand nineteen hasn't gone anywhere.

• 7:32Alex Kantrowitz?

  So you've mentioned it a couple of times that you have this new presentation out. And I think we've kind of gotten off to the races here to start off. But maybe we take a step back and talk a little bit about this presentation. And you make these big, big presentations once a year that sort of wrap up your thoughts on where things are going in the tech industry, where they're coming from. This year is, is all about the new gatekeepers and talking a little bit about how, and let me know if I get this wrong or not, but how companies like Amazon are surpassing traditional retail and internet advertising is surpassing traditional advertising. You know, and as I read, first of all, there's so much good data in there that we're going to cover over the course of this conversation. But as I read, I also looked at it and said, wait a second.

• 8:15Alex Kantrowitz?

  So internet advertising displacing newspaper and traditional advertising. Okay, I feel like that's been something that's been going on a long time. And e-commerce displacing retail that also, or brick and mortar retail also feels like a something that we've seen for a long time. So is, is your thesis here that we're hitting an inflection point?

• 8:37Speaker 00

  Well, so there's, there's a hundred odd slides in the presentation, each of them with sort of one point or one shot. And so there's not one thesis. Some of this is just like, this is an important thing that you should know is happening. I think maybe, maybe kind of two or three things that were sort of threads that run, run through some of the material is. It's almost that I think it's useful to kind of step back and say, well, what are these definitions? And why are we drawing those kinds of distinctions? So, for example, you know, when we get into a car and drive to go to a Walmart, we don't say, I'm going to do some car commerce now.

• 9:17Speaker 00

  And, you know, when you, you know, drive to your home in the suburbs, we don't talk about car, um, how you have my car home instead of my walking home. Those sort of distinctions, that's just one of very many different kinds of distinction. And I think it's kind of, there's, there's a chart in the presentation where I say, well, this is what American businesses spend on online advertising. This is what retailers spend on rent. And rent and advertising have now sort of become two sides of the same question in the way that they weren't in the past. You know, you can say, should we open stores there or advertise there? Which is not a question you could have asked before the internet. But then take that a bit further, well, there's advertising and then there's shipping and there's returns. And there's your payment charges, your, your, your, your credit card interchange fees.

• 10:01Speaker 00

  There's your trade dollars, your marketing expenses. There's, um, there's all sorts of money that gets spent between the product leaving the factory door and the customer unwrapping it at home. And those all sort of used to be different industries, different parts of the org chart, different budgets. And now they kind of become one question. How is it that we should be reaching our customer? And in that context, you know, a lot of most of the Amazon charts that I have have Walmart on them as well. Um, because, you know, we do kind of, I'm just about old enough to remember when people thought that Walmart was the end of Western civilization.

• 10:39Speaker 00

  And it's, you know, raised legitimate questions about, you know, what Walmart did to parts of America. But, you know, now we understand it's just a big retailer and Amazon is, is kind of just a big retailer. I think the the sort of the pair to that thesis, I think, would also be, how many of these questions are not tech questions anymore? I mean, I've said a number of times that I don't think I don't think the questions that matter for Netflix are tech questions, they're TV questions. The same thing for Shein, which we might talk about, which is a fast fashion retailer that's now bigger than than Zara or H and M. What are the questions that matter to them? Are they tech questions or smart or apparel questions? I mean, they sell on smartphones. But it seems to me that you one would sort of think of e-commerce

• 11:21Speaker 00

  or the internet or streaming video, whatever it is, as a new channel or a new route to market. But who is it that uses that new route to market? Well, it's people in that market or it's people who might be your entrance to that market, but it's that market. So all the questions for Walmart are grocery questions. Or and all the questions for Shein are apparel questions. But Walmart didn't create get created by car people. It's not a car company. It didn't get created by people from Detroit. And I think so. I think that sort of shift of

• 11:52Speaker 00

  you know, the technology has kind of changed how all of this works, whether that's apparel or retail or TV or advertising. But then they remain apparel questions or TV questions or retailing questions.

• 12:04Alex Kantrowitz?

  And let's unpack some of this. And for me, there were some of these parts that really struck a nerve looking through some of the slides you put together, especially about how some of this stuff has merged, how maybe retail and advertising aren't exactly that distinct. And to just kick off with it, I mean, Amazon, we knew I knew Amazon was an ad powerhouse, but I it just struck me reading through your slides about how big Amazon's advertising business is. I mean, first of all, you have a slide in there that says Amazon's ad business is bigger than Prime. And Amazon incredibly sells more advertising than newspapers.

• 12:45Alex Kantrowitz?

  So can you unpack this a little bit and talk a little bit about how Amazon's become so important in the ad space?

• 12:52Speaker 00

  And So, yeah, so so a couple of years ago, there's so it's always amazing how people to me it's amazing to meet people who have strong opinions about a company never seem to go and read the accounts. And so they'll kind of loudly say, we wish this company would tell us what this number is. And you think, yeah, that is has been in the back of the account for the last 10 years. And so Amazon gives a breakdown of their business by on in the P and L, they have

• 13:21Speaker 00

  AWS and they split out AWS separately in their retail. But in the back of the accounts, they have these other things like, you know, what's the revenue for the physical stores? And they had this category called other. And the other number started getting really big. And the footnote to that said it was predominantly advertising. And then at the end of, I think, twenty twenty, it got big enough that they had to break it out. They had to break out the advertising part and it turned out it was all advertising. And last year it was, I think, it was thirty eight billion dollars. And I'm always kind of a fan of making charts that tell you what that means. And what does thirty eight billion dollars mean? Is that a lot of money? How big is advertising? If you don't follow this stuff, like, it sounds like a lot of money to me. What does that mean?

• 14:04Speaker 00

  Which is why I kind of turned around and said, well, it's roughly the same size as Prime. It is bigger than the global newspaper industry, which, as everyone does know, has kind of collapsed over the last twenty years, which is maybe another conversation. It also makes them like the fifth or sixth biggest media owner on earth, although obviously Alphabet and Meta are by far the largest. And so that was kind of one, one, one way of looking at this. I think this is a second way of looking at this, which really picks up to what I was saying a moment ago about, you know, challenging definitions and asking what these definitions really mean, is that you can look at, so you can start, start at this by saying, well, this is, this is a kind of a broader category called merchant media or retail media, in which, you know, if you own a store, you can't really put advertising, you can maybe put

• 14:49Speaker 00

  a little bit of advertising in the store, but you're not a media owner. But if you've got a website that hundreds of millions of people are looking at, that's media. And the, the square footage, so to speak, of that website is inventory that you could put ads on, at least in principle. And combine that with the whole movement of privacy in the last couple of years and the move away from cookies. It means that suddenly the fact that you've got this inventory and you have some idea who these people are and you have some consent to that. And even if you don't, you know what they're searching for in an anonymous way, means that all sorts of people have suddenly realised that they have ad inventory that they didn't realise without inventory and that it's suddenly relatively much more appealing to advertisers

• 15:31Speaker 00

  than it might have been five years ago because they got this first party targeting data. And Amazon pioneered this, but Walmart will probably do two and a half to three billion dollars in 2022. Uber did 500 million, hit a 500 million dollar run rate at the end of last year. Target is doing this, everyone's doing this because, and one of the quotes in my presentation was a quote from the CEO of Walmart, basically saying, oh my god, look at these margins. Because if you're a retailer and you've got like two or three percent margins, well, and then you suddenly add like five percent to your top line in ad revenue at a 50% margin, then that's a meaningful change to your net income, which of course is what's happened to Amazon. So that's kind of a second piece, which is like, any retailer was suddenly realised

• 16:16Speaker 00

  that they could try and do this. I think the third thing that's interesting though, again on definitions, is is this advertising or is this marketing? In fact, you could ask, is Amazon, is Google search ad actually advertising or is that marketing? You're paying to be next to the till. You know, if you give a supermarket money to be next to the checkout, that's not advertising, that's marketing. What is, is that, is that what, but that sounds like that, what is it that Amazon's doing? Is it advertising or marketing? You could also call this price discrimination, which is to say, if you are a brand and you are paying Amazon a retail margin in order to sell your product, if Amazon comes, when you then buy search ads as well in Amazon, you're just giving Amazon a bigger cut.

• 16:59Speaker 00

  So what Amazon is really doing is saying, well, people who have a higher profitability will pay us more because they'll still have the ROI to support it. And so you could say this is advertising, you could say it's marketing, you could say it's price discrimination, you could say it's something else. I mean, is Prime, Prime is mainly a marketing cost, that's how Amazon think about it, even though it's not in the marketing line. So all of those sorts of definitions break apart when it's all kind of happening on the same website and doing the same thing.

• 17:29Speaker 00

  Whereas in the physical world, advertising, advertising and marketing is marketing because they're happening in different places in different ways. Here they will kind of just merge into one.

• 17:38Alex Kantrowitz?

  That's interesting, but the question is, of course, like, you're not a media site when you're in Amazon. You are an e-commerce site. And the other side of this is that you can ruin your site by trying to become this media site and end up harming the usability, which, I mean, one look through Amazon tells you that this site is just much worse than it used to be. What's your perspective on that?

• 17:58Speaker 00

  There's a dilemma here. Amazon has, pick a number, it's hard to get a hard number, but they've got certainly hundreds of millions of SKUs. At a certain point, those numbers sort of become meaningless because you've got 200 people selling the same thing and you've got SKUs that don't exist until you order them. And so it's not like saying, how many SKUs does a department store have and what does Amazon have in comparison? But what you have here in essence is an attempt to capture everything that could possibly exist and could possibly be sold. At least certainly everything that isn't kind of grocery and doesn't have to be refrigerated, which is again, is a separate conversation. But in principle, anything other than like frozen food and fruit and vegetables could be sold

• 18:41Speaker 00

  on Amazon as an interchangeable widget, as an interchangeable SKU. And of course, what they've done to unlock this is create Marketplace. And so now something like 60% of what gets sold through Amazon isn't sold by Amazon, it's sold by third parties using Amazon as a channel. It has sort of occurred to me that if you were kind of an innovative and thoughtful regulator and you wanted to take on Amazon, what you do is you say they have to offer wholesale access to the website and the logistics platform to anybody who wants it. And guess what they do? That's called Marketplace. That's what that is. It's as though any telco, it's like an MVNO. And so, but what that means is that they can scale indefinitely because if they want to do such and such a product, they would have

• 19:26Speaker 00

  to hire people to go and source that product. But now they don't have to. It's like they basically created a free market in selling products on Amazon. And so if you are, if you want, how can I put this another way? Amazon has got lots of teams selling, who work for Amazon, who sell stuff on Amazon. So like makeup in Germany is a team for the sake of argument. But now they don't need for something to be sold on Amazon, the people selling it, the team selling it doesn't need to work for Amazon. Amazon doesn't even need to know they exist, they can just create their own marketplace account. So it's the original sort of strategic idea of saying, Jeff Bezos saying, everything has to be an API. Well, if everything internal is an API, then why not just open that up to other people outside and they can sell too. And so this lets Amazon sort of scale indefinitely just selling everything on earth.

• 20:11Speaker 00

  Now, the challenge is, and you see this in content, you know, you see this in newsletters, you see this in podcasts, is if you create a system that makes it really easy for absolutely anybody to create, then everybody, that's a problem as well. You have infinite content. Right. And so that's a problem for music, it's a problem for movies, it's a problem for books, it's now effectively a problem for anything, for any product. And I have, I mean, I have a slide in the presentation which kind of makes this point that we now have effectively have infinite product and infinite media. And it used to be that there was a filter in how much could be stocked by Walmart or Macy's or whatever it is and what they would do, they choose to stock. So that was one filter.

• 20:55Speaker 00

  And then there was a filter on who could, you know, what brand could support a nationwide TV buy, what brand could support putting an ad on the back page of Vogue. And so there was a media filter, you know, what, you know, what, how many things could Vogue or GQ or Car Magazine actually write about anyway? So you had a filter in the kind of the discovery or suggesting channel and a filter in the kind of logistics or the purchasing channel. And those filters are both gone. There's infinite media, there's infinite content, there's infinite, it's an infinite product. And so whether you're a brand or a consumer, you've got this kind of like, suddenly you've kind of got the fire hose full in the face, which is what you're describing when you go to Amazon. But I suppose what I'm getting at is like, that's the sort of inherent problem in not having a fixed amount of square footage per store.

• 21:38Speaker 00

  And in deciding that you're going to allow anybody to sell anything.

• 21:42Alex Kantrowitz?

  Exactly. And there's another problem in being a marketplace, right? I mean, Amazon became Amazon because it had this first-party marketplace. But now it's effectively, as you're saying, become a third-party marketplace and its retail operation and an ad house. And you look at the challengers, you have challengers like Shopify. And one another number that struck me in your presentation is that Shopify has forty forty-five percent as big as the Amazon marketplace. So if you can't differentiate yourself by a first-party marketplace and you play only in the third-party marketplace game, of course, there's fulfillment. Do you then open yourself up to challenges from a company like Shopify? Which, by the way, in your presentation, seems a lot stronger than everything that I've heard about the company over the past couple of years.

• 22:26Speaker 00

  Yeah, so I mean, Shopify is is there's several things that are interesting about Shopify. One of them is a bit like Shein, a bit like this new thing, Temu. A little bit like WhatsApp sort of ten years ago or Skype before that. It's this thing that sort of became huge before anybody had quite noticed it. And those sort of those things always sort of intrigue me. What the hell is this thing that seems to be in the top ten of every app store on earth that I've never heard of? That's always kind of an interesting thing to look at. I mean, Shopify is it's a platform that lets anybody run a sort of first-class startup quality e-commerce experience. A whole bunch of their businesses, startups and small businesses and individuals.

• 23:10Speaker 00

  But another huge portion of it is big companies. So, you know, Facebook uses it, Snap uses it, Heinz uses it.

• 23:18Speaker 00

  And so it's a way of having like top-tier, good-quality e-commerce experience. It's a way of unbundling Amazon.

• 23:25Speaker 00

  But of course, if you when you but of course, then you still have to tell people about it. So you, you know, instead of putting all your money into rent, you put all your money into advertising.

• 23:33Speaker 00

  Then you have the sort of the challenge that Shopify has is are we just kind of an enterprise SaaS company charging a percentage or are we a network? At the moment, they're not a marketplace and not really a network. There's Shop Pay, which is a little bit of a network for the merchant in that they can say, we'll offer you Shop Pay and therefore you'll have higher conversion. But it's not really a network effect from the consumer side. The consumer doesn't know that a website is on Shopify until they've already got to the point that they're making the purchase. And so what Shopify I think want to do is somehow work out ways of creating more network effects so that there are more reasons for merchants

• 24:12Speaker 00

  and or consumers to be on Shopify without upsetting the merchants by supplanting their brand, which of course is the trade off that you take if you go onto Amazon, then you're dependent on Amazon's as a brand. Now, if you look at the share price, was that a hundred billion dollar company? What was the peak valuation? Like it was fifty billion dollars? Was that really a fifty billion dollar company? Well, if they had all that network and stuff done, then maybe. Without it, I don't know. Again, I'm not an equity analyst anymore. Occasionally, I see the CEO complaining about short sellers on Twitter, which I never think is particularly sensible. No.

• 24:55Speaker 00

  And so you can argue about the valuation all day. That to me frankly doesn't interest me so much as the sort of strategic shift of brands going direct. And it's actually something we should talk about, going back briefly to Marketplace, is this guy, I can't remember the guy's name, but Marketplace Pulse, which is always interesting. He reckons that something like, anything between a third and a half of Marketplace vendors, a Chinese manufacturer is going direct. And this is, you know, this was this thing in Atlantic about all these made up brands on Amazon that. This is also a story about Xian, which is that part of what you've got here are Chinese manufacturers that used to sell to Western American developed world retailers and brands now trying to make their

• 25:39Speaker 00

  own brands and sell direct through to the consumer. And because they don't need now to have their own physical retail presence, or even put stuff into physical retail, third-party physical retailers, they can sell direct through Amazon Marketplace. And so Anchor is, I think, probably a $2 billion business last year. And that's a brand that's been created almost entirely on Amazon Marketplace. So you've got this sort of, you know, it's turtles all the way down. You've got unbundling and bundling going on in every, every direction. So, you know, you've unbundled from Best Buy, you've into Amazon, you unbundled from Amazon into Shopify. Amazon bundles in your ad, but takes your ad spend and bundles it back into from, from TV or podcast or whatever, and bundles it back into Amazon.

• 26:24Speaker 00

  Like it's unbundling and unbundling in every possible direction. And kind of part of the point is like, there was this old, relatively straightforward way in which all of this worked that was relatively sort of simple and well understood for like 50 years. And now all the cards have kind of been thrown up in the air and no one knows what the hell's going on or where they land.

• 26:42Alex Kantrowitz?

  Benedict Evans is here. He is a preeminent tech analyst. You can find his work at ben-evans.com. There's a great newsletter there that's subscribed to by more than 175,000 people. You could also find the presentation there. We've been talking about the new gatekeepers focusing on retail and e-commerce in the first half, second half. We'll talk about advertising and generative AI. Back after this. This episode is brought to you by App Point. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data and most teams have no single way to see it, secure it, and prove it's under control. That's exactly what App Point does. For 25 years, they've been the trusted

• 27:26Alex Kantrowitz?

  layer beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud, and the agents acting on it. On your behalf, it's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. AVPoint, the unifying trust layer for AI. Head to AVPoint.com to see how enterprises deploy AI with confidence. Learn more at AVPT.co/bigtechnologypodcast. That's AVPT.co/bigtechnologypodcast. Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed.

• 28:11Alex Kantrowitz?

  When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a white glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwell.com/bigtechnology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwell.com/bigtechnology.

• 28:55Alex Kantrowitz?

  Learn how to stop online threats before they become real-world attacks. The link's in the show notes.

• 29:09Alex Kantrowitz?

  And we're back with Benedict Evans here on Big Technology Podcast. We're talking about sort of the resurgence of big tech that may not actually be a resurgence, actually just a continuation along the line of growth that we've seen from them. It's just kind of been skewed by the pandemic. Let's talk about Facebook and Google. We did Amazon in the first half. Let's talk about Facebook and Google. You mentioned in your presentation that these companies are used to 20% a year growth and they are not going to see that anymore, or may not be seeing that anymore in terms of revenue. Do they look different when they don't have that type of growth and why is their growth tailing off?

• 29:45Speaker 00

  Well, there's many different ways to answer this question. I mean, one of them, I think, is there's a sort of a short-term macro pickup and I don't think one would just sort of presume that, you know, good times are over indefinitely. I mean, I don't think one would necessarily presume that like the long-term growth has just stopped and that's it now and these companies are just going to be flat. It's more like we've had a kind of a big macro shift in the last six months, 12 months. I think, you know, a second answer might be that when the stock is going up and you are in a war for talent, then you hire an awful lot of people. And, you know, these companies have hired huge numbers of people relative to where

• 30:29Speaker 00

  they were in 2010, you know, doubled or tripled in size. And not all of those people were necessarily found things to do or found productive things to do. And some of those people will now shake out and go and work for, there's a sort of, you know, pan glossy in view in the valley. This is all fantastic, but all these people are now available to startups, which, you know, if you're working, you know, five hours a day at Google on half a million dollars and now you're getting a job for a startup, you've got to work 14 hours a day on a hundred grand plus stock options. That may not necessarily feel like a fantastic change, but like, go figure, that's like, that's the way of the world. However, there's kind of another way of coming at this, which is, and it sort of speaks

• 31:11Speaker 00

  to what we were talking about before the break, which is, you know, what is Google's TAM? Is Google's TAM, and it's also this, this, this sort of point, you know, Google and Facebook have got whatever it is, half to two-thirds of digital advertising and digital advertising is now probably three-quarters of global advertising. Is that the right, but that's not a good way of understanding what's going on here. What's actually happened is that the value of print advertising basically evaporated as a consequence of the internet. And sort of five to 10 years later, Google and then Meta invented these completely new advertising models that by and large got completely different kinds of advertisers and made a lot of money from them. And

• 31:51Speaker 00

  The ad budgets that are going into Alphabet or Amazon ads or Meta today, are those coming from TV ad budgets? Well, yes, but they're also coming from rent budgets and shipping budgets and pricing and margin and everything else. Again, you knew you kind of asked this question. Should we spend our money on opening stores or advertising? Should we, you know, if we want to be, if we want people to buy our, buy our product, should we cut our prices so that we rank higher for cheap product? Or should we spend the money on an Amazon ad? Like, what's the best way of appearing on the first page on Amazon? Having free shipping, having a low price or buying an Amazon ad?

• 32:34Speaker 00

  And the answer is, well, maybe all three or it depends. But they're all on budget now. And so to, to look, and if you look at, you know, you imagine that the marketer at that company saying, you know, do I put the money into free shipping, free returns, Amazon ads or low prices? And then you look at the Amazon ad revenue and you say, oh, they've taken all this money from the rest of the ad industry. Well, that's not what's going on. Like, no, that's not what's going on.

• 32:57Alex Kantrowitz?

  Interesting.

• 32:57Speaker 00

  What's, what's happening is that the ad budget is no longer a separate thing. And obviously I'm kind of exaggerating for effect here, but you know, what's happening is that the ad budget, the rent, but basically the entire margin of Procter and Gamble, everything below the gross margin line, all blurs into one question, which is how is it that we, or how is it that we spend this money? Do we spend it on advertising or on retail margins? Do we build a DC business? Like, what do we do?

• 33:22Alex Kantrowitz?

  So let me ask you this, you know, speaking of distinction disappearing and things that swallow everything else, and you knew this question was coming, I have to ask it. What do you think the new chatbot wave is going to do to this stuff? Is it, do you think it's gonna, is there gonna be a ripple here that it's gonna make? Is it gonna subsume things? Or do you think it will sort of just exist on the side as a sideshow and everything continues to pace as normal?

• 33:46Speaker 00

  So several answers to this. I think the first of them is everyone thinking about this thinks that they are confused and do not quite have a good coherent understanding of what this is, what it means and, and what it turns into.

• 34:00Speaker 00

  And that's right down from the kind of the technical level of exactly what what is it that L L M's are doing, why are they working, what are the limitations of this, what are the things that don't work and how feasible is it to fix them? So that's sort of basic like machine learning science questions. But there's also questions like, is this, is a chatbot the right way to turn this into a product? Is that the right thing to do with it? Is this, is, is general search, is this, is generalized search a good use case for this? Is this something that takes over generalized search? Or are all the vertical applications that are already kind of stringing up like mushrooms actually the place where most of this becomes most useful? I mean, I think it's kind of interesting to go back

• 34:43Speaker 00

  and think about all of the wave of voice assistants five and six years ago, which I think we would all kind of now feel was, you know, there was a moment when everyone was running around going, oh, my God, this is a new platform. And I think we all kind of understood now, like, no, it can do about, they're useful for about five things. And some of that is because before L L M's, you actually had to write the answers one by one. So machine learning let you ask the question, and it could take the audio, transcribe the audio into text, work out the structure of the question, so work out what you're asking. But it could only ask fifty, answer fifteen questions. So anything else you'd ask, it just couldn't answer, even if it understood the question. And L L M's mean that like, at least in principle, you can ask chat G P T anything, like, you don't

• 35:25Speaker 00

  have to write the answers one by one by hand. Like you, but, but it still doesn't follow that voice is the right U I, or that chat is the right U I. There is a sort of a broad generalized question of like, what does it mean that these things, when these things are quote unquote wrong? What does it mean when they quote unquote make things up? And I'm very doubtful that make things up is actually the right way of describing what's going on, when they say something that isn't, that you don't think is true. I think there's a really kind of interesting intellectual puzzle of yes, but what does that mean? Is there's a second question, which is, like, why the hell is this Bing thing going crazy? And how, how, somebody, somebody very senior said to me, like, the Microsoft, that Microsoft

• 36:09Speaker 00

  described this as Microsoft's galactic scale fuck up. Of like, what is that? Which is distinct from the kind of deeper problems in chat G P T. There's like a third question, which is

• 36:22Speaker 00

  Is how would you turn this into a search engine? And what would that mean? What would that do for market share? What? How would that change how the ad model works? What does that mean for copyright? If you are, I mean, I'm sure you're familiar with the idea of newspapers that if I send you a link to a newspaper story, then Google or Facebook are somehow supposed to pay the newspaper. And I think we would both agree that this was just completely preposterous. But if I ask Bing or Google, you know, hey, what's going on in Ukraine this week? And it just reads me out three paragraphs that it scraped out of the BBC and the New York Times and doesn't link to them or give them any traffic. Well, that's a whole other conversation. At that point, the newspapers are entirely right.

• 37:03Alex Kantrowitz?

  Yeah, that's fine.

• 37:05Speaker 00

  And say, you can't do that. Can I swear on this podcast? You can bleep it out. The new, yeah, the newspapers, the new, at that point, the newspapers are entirely right to say, wait, you can't do that. So that's a whole question. And then there's a sort of a more fundamental point, which is like, is doing a chat UI the right way of doing this? And is using this for general search? Does that make even make any sense? Is that the right model? And so, you know, as I sort of said at the beginning of answering this, like, there's an awful lot of questions. I don't think we quite know what all the questions are. I certainly don't think we know what the answers are. I mean, there's a whole other strand, which is like, what is the CapEx required to actually pipe all of Google through this stuff?

• 37:46Speaker 00

  Which is, you know, again, I don't actually think anybody knows the answer, but the answer is totally quite a lot. Although that will change radically over time. But like, so there's an awful lot of questions. We don't even know what all the questions are yet. And we're kind of trying to digest this and kind of work out like, what's the right layer of abstraction to understand it?

• 38:04Alex Kantrowitz?

  The reason why we're asking these questions is that there is some sort of there there, right? That these things are are pretty incredible. And without overhyping them, they're unbelievable to chat with. I spent a couple of hours chatting with the Bing chat and I've spent more hours chatting with ChatGPT. And it's, it is pretty unbelievable. So is it too early to say whether or not this threatens Google? Obviously, then there's this narrative that that Bing is the new.

• 38:32Speaker 00

  Well, so any platform shift, yeah. And so any platform shift is a sort of any sort of moment of discontinuity is a way of shifting consumer habits and patterns. It's a moment at which people can drop the ball. So that's one answer. The second is, is this doing something that's fundamentally challenging for them to do? And you know, the kind of classic example, of course, here is always the shift to the iPhone. And a kind of analogy I came up with randomly the other day was the first electronic calculator, which is powered by an Intel chip, the desktop calculator. It

• 39:13Speaker 00

  costs like a thousand dollars in today's money and it's a calculator. And it looks just like the calculators that exist already, but the competition are electromechanical. So they're basically like horrendously complicated typewriters with like 3,000 moving parts inside them. And you take the lid off and it looks like a kind of David Cronenberg nightmare. And they're doing the same thing, but they're completely different products. One of them has got like thousands of mechanical moving parts and the other of them has one chip. And when you take the lid off, you're like, that's why the people who made the electromechanical calculators were completely screwed. Because even though it looks like the same thing, it's not. And that's sort of what happened when BlackBerry looked at the iPhone and when Nokia and Microsoft looked, they're like, this is a completely different thing. And we have no idea how to do this. For

• 39:56Speaker 00

  Microsoft, the problem was, okay, how does Microsoft beat the iPhone? Well, first of all, they'd not be Microsoft because everyone's scared of them. Secondly, they have to make an open source operating system that they give away for free that has no commonality with Windows APIs. Imagine saying that to Bill Gates and Steve Ballmer. They'd look at you like you'd grown a third arm. And like, they couldn't do it. There's no way they would have agreed to do that at that point in time. And so now we have this question, like, is there some reason that Google can't do this? Are they the electrical, are they the mechanical calculator company? Of course not. You know, they've got more, they've built more L L M's inside the company than OpenAI and Microsoft.

• 40:39Speaker 00

  Is there some like business culture disruption reason why they won't do it? Because like it challenges all the things that they believe. Maybe? The question is,

• 40:52Speaker 00

  It's the reason that they've so far not done it because they're right. You know, this is what you see like Yan Le Cun saying, which is looking at this thing and saying it doesn't work.

• 41:04Speaker 00

  I mean, the, I mean, the, the, I was gonna say, I'm gonna back in monologue about all this all day, but like, I think it's really interesting to compare this with HoloLens. Because the HoloLens, you remember that? Anyone else remember HoloLens? Remember when this was the future of Microsoft and this was going to let them become dominant in mobile devices and this was the new smartphone?

• 41:23Alex Kantrowitz?

  Of course.

• 41:24Speaker 00

  And of course, it gave them, it's very cool. It gives great demo. But it's got like a three inch field of view and it doesn't work in daylight and there was no path for those to get fixed. It also wasn't their technology anyway, which is another commonality with, with Bing GPT, um, or ChatGPT. And here, here we are now, whatever it is, five years later, the whole thing's kind of been quietly forgotten. But it was great PR at the time. And you kind of look at this and think, is the reason that Google and Meta and so on haven't launched this because they don't get it? Or because they can't do it, or they're dumb, or that their politics and their business model won't let them, which was part of the Microsoft problem? Or is it that they're looking at this and going, yeah, this doesn't work?

• 42:07Alex Kantrowitz?

  And one of the interesting things that you bring up is, you know, it, it may not be the core brand, right? Like it's actually, I guess Bing needed to roll this out as the core brand because it was a demo or whatever it may be. But that was risky because you then attach it to the Bing brand and they, they have rolled it back in ways that are surprising. But then you think about, how do you make this available to others as an API? And maybe they take the risk. Uh, so what do you think about that? I'm kind of curious. I, I imagine there's going to be some sort of, some sort of war to provide infrastructure where other companies or other entities can build their own bots on top of this tech. Is that where you see it going?

• 42:45Speaker 00

  So I think there's an interesting kind of engineering slash product puzzle in that. In.

• 42:53Speaker 00

  I'm trying to think about it. I almost think it's kind of like it's sort of at a right angle to the problem. Be like, put it like that, which is, I don't think anybody doubts that this stuff is going to be enormously world-changingly useful for certain specific vertical applications. So I was looking at something this afternoon, which was basically for developers in big companies that have got mountains of code and say, what is this module doing?

• 43:20Speaker 00

  Or, you know, summarize this podcast, or do this, or do that, or get, you know, GitHub Copilot is kind of an example here. And I think the interesting common thread in those vertical applications is you don't run the risk of it going off the rails and accusing you of like trying to murder it. And where it gets 5% of stuff wrong and quote unquote makes things up, you can tell because you can see it doing it because you know how it works. And you or you know enough about the domain that you can see the mistakes and fix them. You know, if you get it to write code for you and you can write code as well, you can look and go, yeah, that bit's wrong. Where the 10% or 5% error rate is dangerous is if you let people ask for medical advice and you're not a doctor and you don't know the bit that's wrong.

• 44:06Speaker 00

  And that's where you get into difficulty using it for generalized search is almost by definition, people are using it for stuff they don't know. All of which is a way of saying like, yeah, make the APIs and let people build dedicated vertical applications where the people using it understand what it is, understand how it works. You've built all the tooling around it to handle what it should be doing. It can't answer stuff it shouldn't be doing. You can stop, you don't, you don't have to worry about people, people trying to get it to give you instructions for suicide because it's a coding assistant. You don't have that open-ended problem.

• 44:37Speaker 00

  And you're putting it in the hands of people who are able to assess the results and evaluate it. So when, when you make it vertical, like a huge chunk of the problems go away, or at least are solvable. The challenge is if you try and use it for general purpose search, that's not really a brand problem. That's a like.

• 44:57Speaker 00

  Is that, at that point, now you've got people saying, do I have appendicitis? And are not able actually to assess the reliability of the answer. Or people are trying, are saying, hey, what's the best way to try a noose if I want to kill myself? And your filter doesn't catch that, which is obviously a problem that Google had 20 years ago.

• 45:15Alex Kantrowitz?

  Right.

• 45:15Speaker 00

  And it's the general search thing, trying to use it for general search is maybe where all the problems come from. Or like, or makes all of those problems much more difficult to solve, which is kind of a paradox because

• 45:32Speaker 00

  you go back to voice assistants, the reasons voice assistants, the problem with voice assistants is if you ask it for cricket scores and nobody at Amazon has written a cricket score module, then it's got nothing for you. And you have to write all of those things one at a time, which just became impossible and unscalable. Whereas with an L L M, with chat G P T, because you've trained it on quote unquote, the whole internet. In principle, it can answer anything, except that maybe it can't. So maybe you need to build a special filter for cricket scores and a special filter for football scores and special filter for and like, guess what? You're back at your rules based system and trying to build 150 things. So there's almost like a paradox.

• 46:10Alex Kantrowitz?

  And that's what makes it so magical also is that its ability to go anywhere.

• 46:13Speaker 00

  It is, except that then what does that mean?

• 46:16Alex Kantrowitz?

  The magic and the disaster is two in one.

• 46:20Speaker 00

  And, you know, it may be that, you know, you give it 10x more data and 10x more compute. And that, I have no idea what the actual error rate, but you know, it's an abstract concept anyway. But, you know, you get the, you get the error rate down by a couple of orders of magnitude and it doesn't matter.

• 46:38Speaker 00

  That may be one answer. The other answer might be actually, no, you end up having to build 150 hand tool answers. In which case, what's the fucking point?

• 46:46Alex Kantrowitz?

  Right.

• 46:47Speaker 00

  And you should kind of go back to building individual vertical things. I don't know.

• 46:52Alex Kantrowitz?

  Let's run with this, this idea that it may be end up going, it may end up being infrastructure for individual things. Is there a company you think is better positioned there? I mean, this is clearly, if that's the case, it will clearly be a battle between Azure and AWS, assuming Amazon releases something like this, and Google Cloud. So do you see this? How do you see this battle shaping up? And do you think it's a meaningful business opportunity to provide the backend of maybe not like Bing Chat, but travel agent chat, for instance, or code chat that works in specialized circumstances?

• 47:27Speaker 00

  So I think there's a kind of a youthful lens at looking at this stuff is the kind of evolution of the like, the first wave of machine learning, maybe it was the second or third wave of machine learning, but like the wave of machine learning that happened following on ImageNet in 2013. And so I was working at Andreessen Horowitz at the time from 2014. And you kind of had these waves of startups. So first wave of startup is I'm an academic. I've turned my resume into a PowerPoint. Can I have $10 million, please? I'm going to go and register a domain name and then Google will buy me. And at A16Z, we're like, yeah, that doesn't really work for us as a venture investment, but it worked very well for them and Google went and bought all those people. Like second wave is we're going to be an image recognition platform. And anyone who needs image recognition will come to us and use it.

• 48:12Speaker 00

  And for most of those kinds of things, that ends up as part of AWS or Azure or GCP. Cause that's like a basic primitive. It's like saying we're going to be a database provider. No, that's, you know, the right layer of abstraction for that is a large cloud provider. It's not like, there's not like a standalone market for just that one thing, unless it's something super specialized like Stripe. The next wave of abstraction is kind of comes into what? Two kinds. The first, and I picked two A16Z companies just because they're the ones I'm was familiar with. One of them is we had a company that does legal discovery software. So you see someone, they send you a truck full of paper. What do you do with it? Okay, now they've got translation built in. Now they can do sentiment analysis and say, find me weird emails. They can do clustering.

• 48:55Speaker 00

  Some of that is using AWS or Azure or GCP, some of it is theirs, but they're not worried about competition from Amazon. They're building legal discovery software and that's what they do. And that they understand how to build the software and they go to market. And so in that case, they're not even really an AI company, they're a legal software company. And AI just happens to be one of the things that they use to build one of the capabilities that they have. And so an awful lot of AI sort of ended up like that. It kind of got absorbed into companies that were making some broader thing for an industry. The other way that AI got kind of deployed is another A16Z company is a company that does natural language processing on text going in and out of Salesforce to work out which sales pipelines are going wrong.

• 49:37Speaker 00

  And again, that's, there's a lot more AI in that, or the AI is much more kind of primal in what they're doing because you just couldn't do that before. But that's not what the enterprise salespeople say. The enterprise salespeople say, you know, we do sales process optimization and we tell you which sales pipelines are going wrong. Anyway, yeah, we use AI to do that, but like they're not an AI company. And so that's a sort of, and again, are they using GCP or Azure to do that? I don't know. I don't care. Is it running on AWS? Obviously, yes. Well, that becomes a whole other conversation about, you know, should you run your own data center or not? But I am, you know, GCP is not going to go and build that product. Or unlikely to build that product. I don't know. Having said that, now said that, probably they are already building it, but you know what I mean? Like there's, what's the right level of abstraction here?

• 50:21Speaker 00

  And so I think that's in principle, that's how this will evolve. That kind of, the broad generalized low level primitives that should be part of a big cloud platform will be part of a big cloud platform. But the more specific you get, the more industry specific you get, the more layers of abstraction you have on top of something like, you know, please describe this for me. The more that has to get wrapped into someone who understands the right way to sell asset management software to large movie studios. And understands what product you need to do in order to create that.

• 50:57Speaker 00

  Or whatever the actual problem you're solving for an actual specific industry is. And so you've got those kind of like that kind of pyramid of different layers of abstraction. And I think that's how that, in principle, that's how I would expect this to evolve, but I don't know what those categories will be.

• 51:11Alex Kantrowitz?

  So let me, let me ask you what you think then the business opportunity is. I mean, do you think this is substantial? Like, is every company going to want to have their own chatbot?

• 51:19Speaker 00

  Oh, I mean, I sometimes say that the tech industry suffers from Tourette's only instead of people shouting swear words at random intervals and people, you know, shout out voice, machine learning, NFTs, Metaverse, and obviously now it's going to be ChatGPT. And yeah, you know, I was on, I've been on two calls today with people who are organizing events in industries that are well outside the tech industry and they're all talking about ChatGPT and, you know, thinking about what you do with this stuff. I mean, one of the slides in my presentation, I took, I took a product shot from Shein and then I took the Shein description and typed in disabled diffusion and got three more images and put them up as a two by two image on the slide.

• 52:02Alex Kantrowitz?

  That's pretty cool.

• 52:03Speaker 00

  You can kind of tell, but you can't really tell. And that's just like, literally, that's just one shot me, you know, not optimized or specialized. It's a general purpose model. I'm not, haven't played with it at all. I haven't done any prompt engineering at all. You know, all I did was type in Shein description and I typed in, I think, small red print fabric. Like, that's it. And so that fairly obviously tells you there's going to be kind of a broad class of stuff where this will happen very, very quickly. But, you know, go back to thinking about machine learning, you know, you're, when you dial a call center for your bank now, the bank will be doing five different kinds of machine learning analyzing that.

• 52:45Speaker 00

  So hypothetically, there's something that's, that's trying to work out if your calling pattern is weird and you might be a fraudster. There will also be stuff that's trying to work out if the call center agent is being rude to people or hanging up on people to get their numbers up. And, you know, that's being done for them by, and that's not being done by Google. You know, that's been absorbed into the industry that provides software for call centers.

• 53:11Alex Kantrowitz?

  Hmm.

• 53:12Speaker 00

  And I think the sort of the same thing here, and that's not AI, you know, you don't, you don't, you don't think of yourself as, no, I'm going to use the AI now. I mean, there's an old joke, I mean, it kind of goes back to what we were talking about, you know, is, is ChatGPT intelligent? The old joke in AI research is that AI is anything that doesn't work yet because as soon as it works, people say, oh, that's not AI, that's just a database.

• 53:34Alex Kantrowitz?

  Right, but that's, that's all become the backend stuff. And I think that, isn't the change here that you're starting to interact with these things? Or is that too simplistic?

• 53:44Speaker 00

  Well, I suppose, again, again, the question is, that's the way that we happen to have instantiated that as a product.

• 53:51Alex Kantrowitz?

  Right.

• 53:52Speaker 00

  But I think there's a, you know, ChatGPT is a bundle. You know, it is a bundle of natural language input, natural language output, a chat thread-based UI, couple of other things as well. But it doesn't feel, I wouldn't presume that that's the only way you do this.

• 54:14Speaker 00

  I mean, you know, kind of trivial example, if you take a screenshot of something on your iPhone now in Safari, and then you tap a week later, you go, I wonder what that was. You tap on the URL in the URL bar, the picture of the URL. That's now tapable. The phone number is now tapable. Take a photograph of a bit of a poster. You can tap the phone number on the poster and your iPhone will call it. Now, there isn't a text recognition app on your phone that does that. That's just what the photos app does.

• 54:44Alex Kantrowitz?

  Right.

• 54:45Speaker 00

  And so I think a lot of this will just sort of get subsumed into like, well, that's just how computers work.

• 54:51Alex Kantrowitz?

  Interesting. Yeah, I wonder what will, what will happen. Can we, can we end just by, I'd like to read one, one of your tweets and then we can wrap up. I'm kind of curious if you could sort of explain what you were saying there. It says, you said, if, if Google is the new Microsoft and Microsoft is the new IBM, then Bing is dot, dot, dot. That kind of caught my attention. What's going on in that one?

• 55:16Speaker 00

  Well, it's a very, well, there's a very, very obvious answer to that, isn't it? It's, I was thinking about Watson.

• 55:24Alex Kantrowitz?

  Uh huh.

• 55:24Speaker 00

  So, look, I mean, I think the observation that, that Google has sort of become the new Microsoft, I think is fairly obvious. You know, it's become this very big, sprawling, rather slow-moving tech company that provides, you know, it's a sort of endless sprawling, thousands of different projects, all of them sort of vaguely trying to touch consumer-y enterprise-y stuff. Now, obviously, Google sucks in enterprise and Microsoft's very good at it. That great relationship analogy breaks down a bit there. But yeah, you know, no question now, Google is the new Microsoft. You know, the operating systems in everywhere and all the different stuff that they do.

• 56:04Speaker 00

  Meanwhile, IBM, you know, what is it that IBM became? Well, they still sell mainframes. IBM's install base of mainframes is in fact still growing in terms of install compute base. You know, they sold record IB mainframe compute capacity in 2020. In the hill holes as to the company, they never sold as much. Now, of course, this is partly Moore's Law, it's partly migrating, you know, VMs on to running on on mainframes as all sorts of, you know, it's not the same mainframes that you were running in 1960. But what they turned into is a business that A serviced and sold the basically the legacy product. And the legacy customers for the legacy product still use it. I mean, you know, your bank still runs on mainframes, you know, airlines still run on mainframes.

• 56:46Speaker 00

  You know, all the old mainframe stuff that IBM sold in 1960 is mostly still out there, mostly still running. A lot of it's still running on mainframes. So that legacy business is still there. And then they built a huge ProServe business on top of this. And they built this marketing machine with a ProServe business attached called Watson and sprinkled some AI pixie dust on top of it. But it was basically a bunch of like Accenture style outsource consultancy. They'd go and build you a SQL database and say, hey, our AI successfully solved this problem for you.

• 57:20Speaker 00

  It's a fucking database.

• 57:23Speaker 00

  That's what Watson was.

• 57:26Speaker 00

  And the same thing for IBM, for Microsoft. You know, Microsoft has got this huge legacy business of Windows and Office, which is, I haven't looked at the numbers in a while, but you know, they successfully migrated great chunks of that over to Azure.

• 57:40Speaker 00

  They've successfully built a big kind of cloud business. And, you know, one of the slides in my presentation is there's a Goldman Sachs do a CIO survey every every six months. Asking like a whole bunch of big company CIOs, how much of your workflows are in the cloud? And the number's been sort of stuck at 20 to 25% for like five years. And you see the same in numbers from Gartner and IDC. So like cloud has still got a very long way to go. And, you know, that's sort of what Microsoft does. That's what the act, that's what Microsoft actually is. They provide, they provide Windows and Office as accounting tools and business management tools to big companies. And then there's the Sisley bit, which is the Watson bit. And so there was HoloLens and then it's now, now it's being GPT.

• 58:24Speaker 00

  And is this really the future of search? Well, this, this, you know, I'm being unfair because this is not nearly as much kind of hand as hand wavy as Watson was. But, you know, this is not, you know, it's very un kind of clear that this is like a fundamental transformation in their business.

• 58:47Alex Kantrowitz?

  Right.

• 58:48Speaker 00

  I mean, of all the reasons we've talked about, like, is this really, okay. One back a second unquestionably L L Ns are transformative technology in the way that, you know, Watson was, there was no transformative, there was no technology at all in Watson.

• 59:01Speaker 00

  But does that mean that Microsoft takes over the internet and displaces Google and web search gets completely replaced by this? Like, yeah, that's a lot more difficult to say at this stage.

• 59:13Alex Kantrowitz?

  Oh, my God, could you just imagine this version of Bing G P T playing Jeopardy and the type of stuff that it would spew out? I I would pay pay per view money to watch that.

• 59:25Speaker 00

  Yeah, I mean, I, um, you know, I I saw a couple of examples in my, my, my comedy example, which is a slide in the presentation is I asked chat G P T. Write me a country in Western song about how I made lots of money in social media. And now I think it's destroying democracy, but I'm not giving the money back.

• 59:41Alex Kantrowitz?

  Right.

• 59:42Speaker 00

  I absolutely did not have a large private equity investor in mind when I, when I asked that question.

• 59:47Alex Kantrowitz?

  Ah, okay.

• 59:49Speaker 00

  Yeah, it's fun, but that comes back to the error thing. And, you know, the error thing is something that you could spend hours talking about. You know, if you are, I asked it to make a picture of Sundar Pichai. And Satya Nadella as boxers. In fact, I asked a picture for Satya Nadella as a boxer. And I put this in my newsletter, I'll probably publish it the next day or two. It's a really good picture of Satya. It's not a very good likeness. It's a really convincing picture of somebody who sort of looks a bit like Satya Nadella as a boxer. Two problems. Number one, the rings in the boxing, the ropes in the boxing ring aren't quite right. Do I care? No, that's not the point. It's a photo illustration. Problem two, he appears to have an extra arm growing out of his kidneys and also a hand growing out of his belly button.

• 1:00:36Speaker 00

  They're very well rendered.

• 1:00:37Alex Kantrowitz?

  Where they belong.

• 1:00:38Speaker 00

  They look like arms. They've got boxing gloves. They look like boxing gloves, except that people don't have four arms. And is that wrong? Well, what did I ask for?

• 1:00:51Speaker 00

  It's sort of, and I can see that it's wrong. On the other hand, the rings around the boxing, the ropes on the boxing ring are wrong too. Is that wrong? No, that doesn't matter. What did I ask it to make? Well, did I? I didn't ask it for Satya Nadella, but only two arms. I mean, kind of the, the kind of, the example I keep kind of circling around is I asked it to write a bio of me, which OpenAI now doesn't let you do. They've got all sorts of kind of filters on it. Right. And so it says Benedict Evans is a world renowned sword and hugely influential sword leader, which is obviously entirely correct. Then it says worked for Andries and Horowitz. Well, yes, true. Then it says worked for Bain. No, not true. And founded a startup. Actually, you've never heard of that startup and now I've never founded anything.

• 1:01:36Speaker 00

  And went to Oxford. No, I went to Cambridge. And has written books. No, I haven't. And so again, you can kind of look at this and say this is wrong. You can also look at this and say this is an extremely accurate rendition of what biographies of people like Benedict tend to look like. It's a very good reproduction of the pattern. It's a hundred percent accurate reproduction of the pattern. They always say what university and there was always a prestigious university. They always say things like that. As a biography of me specifically, it's wrong, but is that what it's trying to do? No, what it's trying to do is say what would a biography of someone like Benedict tend to look like? I mean, this is the same thing with the footnote thing. Have you seen this?

• 1:02:21Speaker 00

  So if you ask it to write like a medical paper describing the symptoms of appendicitis and it gives with footnotes and it gives you footnotes, but it doesn't know what footnotes are. And so it gives footnotes that reference universities and doctors, except that doctor didn't work at that university and that other university actually doesn't exist, but it looks like a university name and they look like doctors' names. It looks like a footnote. It's a very accurate rendition of what footnotes look like. That's not, and that's what it's trying to do. It's not trying to make footnotes. It's trying to make things that look like footnotes.

• 1:02:51Alex Kantrowitz?

  So I had to pick a couple of favorite episodes from the Big Technology Podcast and it Bing, this was Bing and Bing got some of them right, but made some of them completely up and

• 1:03:01Speaker 00

  But, but did they sound like episodes?

• 1:03:03Alex Kantrowitz?

  Oh my God. I was like, did we record that episode? Like, did we have those guests on? And we hadn't. And I was like, oh, maybe this should be an episode that we end up doing.

• 1:03:14Speaker 00

  Yeah, we'll give it a year and it can generate the episodes as well.

• 1:03:17Alex Kantrowitz?

  Right. And then I'm really out of a job.

• 1:03:19Speaker 00

  That's what I was doing with Shein. And so this is, it's like, this comes back to my kind of domain point. Like, if you were to ask, if I ask it, I talked about this with someone this morning, like, if I ask it to make a picture of Joe Biden wearing Roman armor.

• 1:03:35Speaker 00

  If you're the editor of The Economist and you want that for your cover, you'll look at the picture and go, that's great. If you are, if you work at the British Museum and you're a specialist in ancient armor, you'll look at that and go, that's completely wrong. So what is it that you're trying to do here? What is it that you wanted? Why did you want it? Like, the buckles are in the wrong place and that's made of steel and they didn't have steel then, they had bronze. But like, does it matter? Well, it depends.

• 1:03:59Alex Kantrowitz?

  Exactly.

• 1:04:00Speaker 00

  And that's the problem with using it for general search, because if you've got it in one vertical, then you can kind of contain and understand the question and you know whether it matters or not. But if you're just going to throw it at every Google search, you've got no idea whether it matters or not, or whether it's right or wrong, or what that would mean.

• 1:04:15Alex Kantrowitz?

  Not ready.

• 1:04:16Speaker 00

  Yeah. So I think this is the puzzle is what's the accuracy? What does that mean? How would you change that? Can you change that at like a generalized level? And, you know, I'll listen to this, we won't, of course, we won't listen to this. We'll listen to this in a year. And like, I'm sure like two thirds of what I've said will be wrong. But that's kind of the point. Like, we don't really know what the questions are yet. It's so happened so quickly.

• 1:04:37Alex Kantrowitz?

  Hey, look, it's much better to start with the questions and I, I think we will listen, re-listen to it because I just went back to our last podcast and I listened to that all the way through. And at least I can say for myself that, I mean, I would love to have you back and, and when you're back, I'll definitely be on this one. And then we can start going back and seeing how our predictions stacked up or how even the questions that we were starting to ask, how those stacked up with, with reality. So. Benedict, it's been great chatting with you. Thanks so much for coming on.

• 1:05:03Speaker 00

  Sure. Great to chat. Thanks for having me.

• 1:05:05Alex Kantrowitz?

  Always great to chat. Do you want to let people know where they can find your presentation and sign up for your newsletter?

• 1:05:10Speaker 00

  Yeah, well, if you Google me, my parents had good SEO, so Googling Benedict Evans will produce the right results.

• 1:05:16Alex Kantrowitz?

  Amazing. Okay, great. Benedict, thanks so much. Hope to do it again soon.

• 1:05:20Speaker 00

  Thanks.

• 1:05:21Alex Kantrowitz?

  And that'll do it for us here on Big Technology Podcast. Thank you so much, Benedict Evans, for joining. Always great to speak with you. And we can do this for next year's presentation as well. Thanks to all of you, the listeners. Great having you come back week after week. Thank you very much. Thank you, Nate Gwatney, for handling the audio. Looking forward to seeing you in Austin. I think, what, next week, coming up for South by Southwest, an annual reunion that's well in store, so I'm pumped for that. Thank you, LinkedIn, for having me as part of your podcast network. And once again, thanks to all of you for listening. Always great to have you here. We'll be back on Friday for a new conversation with Ron John Roy, who's also going down to Austin. That should be fun. And stay tuned for that. Okay, thanks again, and we will see you next time on Big Technology Podcast.

• 1:06:13Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.

• 1:06:42Speaker 03

  In this life, you can't win.

• 1:06:44Speaker 04

  When the world gets you down.

• 1:06:46Speaker 03

  No!

• 1:06:46Speaker 04

  Escape to Pluto TV.

• 1:06:48Speaker 03

  This is objectively fantastic.

• 1:06:50Speaker 04

  Relax into the movies and shows you already know and love, all for free.

• 1:06:54Speaker 03

  Shwing!

• 1:06:54Speaker 04

  Even our news is uplifting.

• 1:06:56Alex Kantrowitz?

  You stay classy, San Diego.

• 1:06:57Speaker 04

  Watch School of Rock, Crazy Ex Girlfriend, Anchorman: The Legend of Ron Burgundy, The X-Files, Wayne's World, and so much more.

• 1:07:03Alex Kantrowitz?

  We're not worthy!

• 1:07:05Speaker 04

  The best things on TV really are free. You just have to know where to look. Escape to Pluto TV. Stream now, pay never.