Can The Web Survive Generative AI? — With Matthew Prince

Matthew Prince is the CEO of Cloudflare. He joins Big Technology podcast to discuss whether the web can withstand a wave of generative AI companies scraping content, summarizing it, and not cutting the original creators into the revenue. Tune in to hear Prince masterfully dissect the problem and put forth a solution. Learn more about your ad choices. Visit megaphone.fm/adchoices

• 0:00Speaker 00

  This episode is brought to you by Genesis. What does it actually take to put Agentic AI to work across an entire enterprise? I recently spoke with Genesis chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, head of enterprise business solutions at Voya Financial, to go inside a large-scale Agentic CX transformation, including the business decisions, trade-offs, and lessons along the way. You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay?

• 0:43Speaker 00

  Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation.

• 1:03Speaker 00

  How will the web fight back against the wave of generative AI that is ingesting all the content on the internet but not paying for it? We're joined today by Matthew Prince. He's the CEO and co-founder of CloudFlare and has been on the warpath, attempting to right the ship. Matthew, great to see you. Welcome to the show.

• 1:22Speaker 01

  Thanks for having me.

• 1:23Speaker 00

  Can you start by giving us a sense as to what is happening to the web with the rise of generative AI? We've talked about it a bit on the show already, but I want to hear it from you.

• 1:33Speaker 01

  Yeah, absolutely. So the business model fundamentally of the internet over the last 30 years has really been driven by search. You search for something that generates traffic. It takes you to content that someone has created. And then that content owner, that content creator, can drive value really in one of three ways. They can sell the content, sell, sell a subscription to it. We see plenty of that these days. They can put ads up against it, or they can just get the ego hit of knowing that somebody cares about and is reading their stuff. And that's really how the web has been built today. It's been built chasing that traffic. What we're seeing, though, is that for the first time in history,

• 2:17Speaker 01

  searches across the major search engines, Google in particular, are actually on the decline. And what's replacing it is more and more people turning to AI. And the difference with AI is rather than giving you ten blue links that you click to and find the answer. Now what AI does is it tries to give you the answer itself. And that's meaning that people aren't going to those original sources. And if they don't go to the original sources, then that means that you can't sell a subscription anymore. You can't put ads up against it. You don't even know that people are actually getting value from your stuff. And so what we're really worried about at CloudFlare is if the incentives for creating content go away, why is anyone going to create content in a new AI-driven future?

• 3:01Speaker 00

  So talk a little bit about how many pages these AI bots or search engines have crawled in the past and how much traffic they've delivered for each crawl and where it's gone to today.

• 3:11Speaker 01

  Yeah, you know, I think that the the the deal that Google made with the web, starting thirty years ago, when Larry and Sergey started working on on the project, was basically, let us copy your content and in exchange, we'll send you traffic. That, again, you can drive value in, you know, one of those three ways from. And we have very reliable data at CloudFlare going back ten years, looking just at Google. And the the the the metric that has stayed very consistent over time is how much Google crawls the web. They've actually crawled at a very consistent rate over the last ten years. Over that same ten years, we've actually added two billion internet users. So we were at four billion internet users about ten years ago.

• 3:55Speaker 01

  Today, we're at about six billion internet users. So you'd imagine it's actually gotten easier to get traffic over that period of time. But that's not what's happened. What instead has happened is that back in the day, if you sort of take ten years ago as the litmus test, today, it's almost ten times as hard to get a click, to get a visitor from Google to your site. What's changed? The answer is that Google has started providing more answers directly. On the page. So if you search for something like when was CloudFlare founded, there will be an answer box at the top that will say September 27, 2010, you know, is the day that we that we launched. And you don't have to click to any link. In fact, about 75% of queries to Google now get answered on Google itself.

• 4:35Speaker 01

  And what's changed in even just the last six months, it's accelerated this is they've rolled out AI overviews and we've tracked this from region to region to region to region. What we see is as AI is giving you the answer without you having to read with the original content. The the amount of traffic that Google is sending to these these sites has gone down and down and down. And that's the good news for publishers. If Google has gotten 10 times harder to get traffic from over the last 10 years. OpenAI is a whole different beast. In OpenAI's case, it's 750 times harder to get traffic than it was from Google just 10, 10, 10 years ago. In the case of something like Anthropic, it's 30,000 times more more difficult to get that traffic.

• 5:18Speaker 01

  So why is that? The answer is, I think people are trusting the AI. They're reading this derivative content and they're not going back to the original source. But the problem is, if you're not reading that original source, then the original sources have no way of generating value. They can't sell subscriptions. They can't sell ads. They can't get the ego hit. And and that over time is strangling the very incentives on why content is being created. And that's the problem that we started to really focus on about 18 months ago. And and then just today on July 1, we announced that we are hard blocking the AI crawlers unless they will actually compensate content creators for the content that they're creating.

• 5:55Speaker 00

  Okay, and we're definitely going to get into your technological solutions. So that's that's coming. But let's talk a little bit more about this problem. So I think the number that you shared recently was Anthropic will call will crawl something like 60,000 pages.

• 6:06Speaker 01

  That's correct.

• 6:07Speaker 00

  For one click. That's that's sent. And OpenAI was somewhere in the like 10. Do you remember 10,000?

• 6:12Speaker 01

  Yeah, it's 1500 pages now for every one click that they send you.

• 6:17Speaker 00

  And you know, I have to say, I'm surprised that publishers are seeing a problem now, only because these AI products are really in their infancy. Yeah. I mean, Anthropic Claude isn't used by very many people at all. When you think about the scope of the web, OpenAI has 500 million weekly active users. It's pretty good. But really, nothing compared to the amount of traffic that you see on the web every day. And I guess Google might might must be the problem. So so just explain why this is already showing up for publishers, because this is the infancy of generative AI.

• 6:49Speaker 01

  Yeah, I think I think it is, but it's one of these sea changes that we can just see happening. So again, for the first time in history, searches to Google actually dropped in the last period over period.

• 7:00Speaker 00

  This is your data?

• 7:01Speaker 01

  This is, Google has actually reported this and it actually came out in the Apple trial as well, where they're seeing more of this traffic actually going to other sources. And so I agree that it's it is a drop, but what what we're seeing is that is the trend. That is the direction that things are heading. And even Google itself is looking more like an AI chatbot and less like a traditional search engine. And so if that's the case, I think that the time for publishers to panic is now. If we wait where more and more traffic gets strangled, less and less is going to it. Again, I think that that's just going to mean that over time we'll have more consolidation in the media industry. We'll have less and less content, we'll have actually more salacious headlines as people are chasing the the content that is is left that's out there.

• 7:44Speaker 01

  And and we need to actually make a change to make sure that we can continue to support publishers. Because I do believe the future of the web is going to be an AI driven future, not a search driven future. And that AI driven future just doesn't have the same incentives and doesn't support the same business model that the old search driven web did.

• 8:00Speaker 00

  Okay, I'm going to poke at this a little more.

• 8:02Speaker 01

  Yeah.

• 8:02Speaker 00

  You mentioned that you can now search Google when was Cloudflare funded founded and you'll get the answer.

• 8:08Speaker 01

  Mm hmm.

• 8:08Speaker 00

  That's something that Google's been doing for a long time. You could ask like, when was Martin Luther King Jr.'s birthday? Even before generative AI, they were giving you these answers. So is it that the the magnitude has changed? And and if so, from the standpoint of a consumer, could this be good? I mean, it's pretty annoying to type this question into Google. When was Cloudflare founded? And then have to click to Cloudflare's website to get the answer that Google could just service for you. And so much of the web has sort of become effectively the service of Google queries, where websites don't really need to exist.

• 8:43Speaker 01

  Well, you know, so absolutely, this has been happening for a while. And if you look at up until six months ago, the ratio 10 years ago of crawls from Google to clicks was two crawls, one click. Six months ago, it was up to six crawls, one click, and that's all because of the answer box. What the AI overviews, which they've rolled out over that time, have done is they've taken it now to 18 crawls to one click. So yes, it is a situation of, you know, the frog boiling in water, but that's, it has gotten progressively worse. And I think across the media industry, it's gotten harder and harder to actually survive as a publisher. And so what I worry about is, you know, publishers are struggling at,

• 9:27Speaker 01

  they were struggling at six to one, they're struggling at 18 to one. I think they're dead at 250 or 1500 to one that we're seeing with OpenAI, completely dead at 60,000 to one, what we're seeing with something like Anthropic. And so that is the direction that things are going, and that's a challenge. I think you're exactly right on the other point as well, that the challenge here is that this is actually a better user experience. That's why more of the web is going to turn to AI. It is great that you can type something in and you can get back an actual response as opposed to having to hunt for it yourself. That's a better user interface. And so that absolutely is going that direction. I'm not arguing, and I don't think anyone is arguing, that we should just get rid of AI or that we should go back to sort of 10 blue links on Google.

• 10:10Speaker 01

  What I am saying, though, is that the fuel that runs all of these AI systems, the reason that Google can tell you when Cloudflare was started or what Martin Luther King's birthday was or something like that, is because somebody is doing the work of that original content creation. That original content is the fuel that fuels Google, it fuels all of the AI companies. And if we strangle off the business model of those places, if we strangle off the incentives for content creators to create content, then we're actually going to end up strangling the AI systems as well. Because if there's no content to train on, then the AI systems are going to be pretty stupid for that. And so I think everybody agrees that there has to be incentives

• 10:53Speaker 01

  that allow content creators to continue to be compensated. The question is, what does that incentive structure look like? And that's, again, what we've been really spending a lot of time looking, trying to figure out.

• 11:03Speaker 00

  Ken, I just want to ask you one more question about crawls.

• 11:07Speaker 00

  I think that sometimes, you know, you crawl to like put a website in in search. You would crawl to put a website in your search engine.

• 11:15Speaker 01

  Mm hmm.

• 11:15Speaker 00

  Or a page in your search engine from a website. Are these generative AI bots crawling to do something similar just to surface the information from these pages or are some of the crawling being done in service of training their models? Because if that's the case, it's actually not as big of a deal. Uh, because it's just being fed into training. I think the problem is taking that like direct query to answer uh uh behavior and sort of bringing it into the search engine. So do you know is it training or is it just surfacing answers?

• 11:45Speaker 01

  So I think there there are two different parts of this. There's there's definitely training and then there's what is closer to a search like experience. If you're if you're familiar with this, it'd be something like RAG or something where you're actually uh getting that real time data in order to augment the foundational model. I think in both cases, though, you're actually costing the content creator something. Um, there there is literally they're paying for that traffic. They're paying for that load that the crawlers are are pulling off of it. There also it is the intellectual property. It's the data, it's the content of these providers that they're using to train the models. And so there's value that the AI companies are getting. If if there weren't, they wouldn't be crawling.

• 12:26Speaker 00

  Right.

• 12:26Speaker 01

  But there's no return of any compensation or any reward. Again, in the old days of Google, the trade off was let us copy your content and in exchange we'll give you traffic. What has happened is the frog is boiled in water and now everyone is saying, let us copy your content and we will give you nothing in return. And so what we're saying is simply, we need a better deal, a deal for a new AI driven future. And that should say, if you are getting value from the thing that I created, then you should compensate me in some way for it. And it may be tiny amounts of money, but at some scale, that actually turns into something that can allow a content creator to continue to have an incentive to create content over time. If we don't do that, if we don't give content creators the incentives to create content,

• 13:11Speaker 01

  they'll stop creating content.

• 13:12Speaker 00

  So I think you're bringing up a key point here, which is if people are like, well, you know, I'm not necessarily seeing. Publisher content show up every time I'm on an LLM. What you are seeing sometimes is the product of the publisher that's been used for training. And even if it's like under fair use, totally fine because it's being transformed and you know, something crawled from big technology or the New York Times is now being used to, you know, help basically because they're just trying to figure out what word comes next in the English language, you know, give you an answer about summer camp. The publishers are actually enabling that. And, uh, and, and every time an AI crawler hits a publisher website, they have to pay. And I, do you work with Wikipedia? Because they've been loud about this, that like the

• 13:54Speaker 00

  server costs that they have to pay have increased exponentially, but those aren't human visitors. They're AI bots crawling Wikipedia.

• 14:02Speaker 01

  That's exactly, so there's, there is a real cost to just supporting this crawl. And before we even talk about intellectual property, before we talk about anything else, like the, the content creators, the publishers are having to bear that cost. And so at just a simple fairness level, like why should they be bearing the cost in order to train, you know, these

• 14:23Speaker 00

  Shouldn't.

• 14:24Speaker 01

  multi-billion dollar, you know, AI companies that are out there. There should be some, some value which is given back, but I think it's even beyond that. I don't even think we have to get to, I mean, use, use legal terms like fair use. And I think that's very much up in the air right now. We literally had two different California cases that came out on both sides of that issue is training on content, fair use or not. And I think it's going to be a coin flip where different courts are going to say different things. And I don't think it's a clear answer there, but I, I think it's a more fundamental thing, which is if you're doing something to create value, you should be getting some sort of, of compensation for that. If, if somebody else is, is imposing a cost on you, you should be able to charge them to offset some of that cost.

• 15:06Speaker 01

  And if something's not, if they're, if, if someone's not willing to pay that, then they shouldn't be taking your content in the first place. Up until now, and, and everyone's focused, you know, the New York Times is suing and I mean, a bunch of people are doing it. Everyone's focused on the legal issue. I actually think that before we even get to the legal issue, this first step is actually take the technical steps to give content creators back control over the content that they're creating and let them have the choice on, do I want to give access or not? Do I want to charge for this or not? And then Done correctly, there should be a marketplace where content creators and AI companies come together and say, "Hey, I created this piece of content. I think it's super valuable." And the AI company says, "Yeah, maybe it is, or maybe it's not, but here's what we're willing to pay." And maybe they meet the clearing price, maybe they don't meet the clearing price.

• 15:51Speaker 01

  But that marketplace needs to exist because otherwise, there's no way to convey value, there's no way to derive value from content creation. And again, I just need to hammer this point home. If we don't give content creators an incentive to create content, they'll stop creating content.

• 16:07Speaker 00

  And it sounds like, by the way, so you're not a skeptic of the AI technology. You believe that this AI, generative AI thing is going to work.

• 16:14Speaker 01

  Yeah, not only that, I mean, it is, it is already clear that it's going to be the interface of the future of the web. So we're going to move from what has been the dominant internet face of the future of the past of the web, which was search, to what the interface of the future of the web was going to be, which is very much going to be AI. So I believe, I am, I believe AI is going to get better and better and better. I actually think that done correctly, content can be created in such a way that will make AI better and that you can create incentives for, for doing that. What I worry about is in order for AI to get better, you have to have original content. People have to be going out and creating that. And right now, we're strangling off all of the incentives for that content creation,

• 16:56Speaker 01

  which not only hurts content creators, it will ultimately hurt the AI companies as well.

• 17:00Speaker 00

  This episode is brought to you by App Point. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data. And most teams have no single way to see it, secure it, and prove it's under control. That's exactly what App Point does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. App Point, the unifying trust layer for AI.

• 17:44Speaker 00

  Head to apppoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast. So I was speaking with someone who does, who works in data labeling or... Data creation for large language models last night. In anticipation of this conversation, I was like, you know, one day, what you're doing might look almost exactly like what web publishers are doing. Where like, you might be hiring PhDs and having them like write their information and you feed that right into a into an L L M's training set.

• 18:29Speaker 00

  And there might be, let's say, historians. So if you take like a world history website, the historians that are writing the web pages for that world history website, they must be just like, maybe one day they're going to be writing those world history articles instead of publishing them to the web, selling them or feeding them right into chat G P T. Do we do we lose anything if the web goes away and it's just content creators selling stuff to large language models?

• 18:57Speaker 01

  Yeah, you know, I think the Black Mirror kind of dystopian future is is not that, you know, content will stop being created and journalists will stop existing and researchers will stop existing. I think the Black Mirror future is that we actually go back to something like the time of the Medici's where we have maybe five big A I companies and they each employ a set of journalists and a set of researchers and a set of set of folks that they become effectively the institutions of knowledge and they they have they have salaries for all their their academics that are on staff. They probably each have different, you know, maybe one of them's the conservative A I company and one of them's a liberal A I company. You can, again, you can very much see that has actually been the natural state of media

• 19:42Speaker 01

  and the natural state of controlling information for quite some time. And you could imagine that all of that research actually consolidates behind each individual A I company and every different academic out there is just a is just is is basically an employee of Open A I or Anthropic or Google or Microsoft. I think that's a pretty bad outcome because again, I think that we the web has been so amazing at distributing and democratizing access to information that I think we want to create that incentive. And so I I think what we're trying to do is say, what's the step? You know, a few steps before the sort of, you know, all the academics are employed by one of the AI companies.

• 20:25Speaker 01

  And I think the answer is you allow the AI companies to pay for the content that is actually valuable to them, that fills in their models and makes their models better. And then you create incentives for independent journalists, independent researchers, to actually be able to create that content, to augment those AIs while still, you know, being valuable. My, this won't happen, but my sort of, you know, optimistic version of the future is humans should get content for free again, because we've kind of paywalled way too much, frankly. And, and robots should pay a ton for it. Because again, every time a robot ingests something, it's in service of hundreds of thousands, if not millions of different humans. So robots should pay for that content. We should get back to a place where then humans get that for free.

• 21:09Speaker 01

  Again, that's, I, I think it's going to be hard for us to get there. But that's the, again, the future that I think is, is actually the kind of optimal future.

• 21:17Speaker 00

  So someone hearing you and, and looking at this through a critical lens might say, look, Matthew, publishers, depending on web traffic, are barking up the wrong tree. That selling eyeballs for CPM fractions has not been a good business for a long time. In fact, we had a guest on the show that recently said, listen, like I'm, he's a journalist, but he's like, if I thought that traffic was the way to go, I'd be out of business a long time ago. And what you really need is an audience that will, let's say, subscribe to your newsletter or listen to your podcast, maybe come to your events. And we've already moved past this business model of trading traffic for dollars. In which case, this isn't an existential threat. What would you say to that?

• 21:59Speaker 01

  I think, I mean, even then, you're still trading traffic for dollars. You're just trading it for subscription dollars, not ad dollars. That will go away as well. Because what will happen is the AI company will ingest the podcast and then summarize it on, on their page. And why would they ever buy a subscription to your podcast? Why would they ever sign up for your newsletter? If their AI agent can just simply say, tell me everything that was relevant in this particular podcast or newsletter.

• 22:26Speaker 00

  Because there's an experience of listening that's enjoyable. Do that in some part for for

• 22:31Speaker 01

  Yeah.

• 22:31Speaker 00

  Entertainment and the leisure value.

• 22:33Speaker 01

  I I think also.

• 22:34Speaker 00

  That's how they learn.

• 22:34Speaker 01

  I think the AI companies will do a very good job at creating that experience as well.

• 22:38Speaker 00

  You think they'll just create like competing?

• 22:39Speaker 01

  Oh, absolutely.

• 22:40Speaker 00

  You know, I used to think that this was like such a pie in the sky and lunatic idea until I listened to Notebook LM.

• 22:46Speaker 01

  Yeah.

• 22:46Speaker 00

  And like we've had multiple people on my YouTube page be like, did you license your voice to Notebook LM? And I'm like, no, but the fact that you're saying that is pretty concerning.

• 22:55Speaker 01

  Totally. And again, I think that's the inevitable future. We're going to want to have hyper customized podcasts that are in exactly the voice that we find the most reassuring. And AIs are going to create that for us. And again, they're going to be fed by original content creators that are out there that give them the ideas, give them what what to talk about, give them the news of the day. What what I think is we have to move even past the business model of subscriptions. We've got to get to something else where you as a content creator are being compensated for the content. The the way I think about it is every one of these L L Ms is a little bit like a block of Swiss cheese. They've got, you know, a lot of stuff there, but there are big holes that are in it. And content that is value for valuable for them are the ones where they actually fill in those holes in the Swiss cheese.

• 23:41Speaker 01

  And so what I would imagine in the future is that you're able to actually surface. What are the places where there are holes in the Swiss cheese as as an AI? And then allow content creators to create content that fills that in. My my favorite example of this is I was I was in Stockholm a couple of weeks ago, meeting with Daniel Ak. Because there really is nobody who has done more to compensate creators at scale than Dan. Dan's the founder of Spotify and and they're they've done just an amazing job at at at doing this. And he told me, he told me a story in our long conversation. He said, you know, one of the things that we do at Spotify is we actually take the searches that people run at Spotify.

• 24:20Speaker 01

  You know, that are things like, I want a song with a reggae beat about how much it, you know, sucks when your, you know, your sister runs away with your car. And

• 24:30Speaker 00

  Has happened.

• 24:31Speaker 01

  Yeah, whatever. And it turns out that they don't have good things to fill that in. And there are content creators out there that are making tens of millions of dollars a year just creating content for those searches that don't have good results right now. Cause Spotify surfaces that list of things where they don't, they don't give those results. I think that that's actually beautiful. I think that's actually really amazing where they are showing where is there something that there is human need for? And then how can we actually then create content to fill that human need? And then monetize it, you know, through what they're, what they're doing. I think the same opportunity exists in the AI space where these AIs actually are able

• 25:14Speaker 01

  to say, this is a, I can tell you how valuable this new piece of content is for me. And they, and you can rank it. And then that allows you to create a marketplace where they can say, listen, that new piece of information is so valuable that I'm willing to pay you for that. And I think that done correctly, that then gets us to more original content creation. It gets us to less sort of me too, copycat style journalism, same thing in, in research. It gets us to maybe a place where we're, we are doing original research and getting rewarded for being more original as opposed to being more salacious.

• 25:49Speaker 00

  Yeah, it's interesting. YouTube has a similar thing where there's an insights or an inspiration tab.

• 25:53Speaker 01

  Yep.

• 25:53Speaker 00

  And they give you like the title, the description and the thumbnail. And they're like, people are searching for this. You go out and make it.

• 25:59Speaker 01

  Yeah, that's exactly right. And I think that that's actually is that incredibly valuable thing that, that's making humanity better as opposed to, you know, yet another story that's just chasing, you know, the most salacious headline that you, that you can get.

• 26:13Speaker 00

  So you're talking about this idea where publishers might sell their, the, the ability to crawl.

• 26:19Speaker 01

  Yep.

• 26:19Speaker 00

  To AIs. That is also assuming that content is scarce. And so I want to run this other idea by you, which is that if we had the same amount of content that we have today. That's a great idea. But what we're seeing now is this explosion of content creation that's made through generative AI. Like, it's it's kind of funny, like, every time you see, like, these suggestions that we're talking about. YouTube's making these suggestions because clearly there's traffic to be had. I'm sure there are already YouTubers today that are feeding that into ChatGPT, spitting out a script, running that through VO3 in Google, and then posting the videos and cashing in on traffic. So there's just gonna be, and we're in the middle, I believe, of this explosion of content.

• 27:01Speaker 00

  Actually, you probably have better data on that than my suppositions. It almost feels like a DDoS of the web. Where, like, you know, if there if the ability to create content is constrained by a human's ability to create content, then you have something to bring to these AI companies. But if human plus bot content starts to become the norm, there's gonna be so much, then even if you're creating high-quality stuff, it's not gonna matter very much to these generative AI companies. What do you think about that?

• 27:28Speaker 01

  I I think it's still, I mean, so first of all, I think that there's the pure AI-generated content. There's lots of research that shows that training AI on AI data is sort of like that old Michael Keaton film Multiplicity, where basically every copy of something gets worse and worse and worse. And and again, that that feels like that's going to still be still be the case for quite some time. May in the future, um, robots be able to go out and do, you know, interesting reporting from the field. May they be able to do, you know, interesting research, for sure. But today, um, I I think that that interesting research, that interesting original content, that interesting insight that that comes from the work that right now only journalists and researchers and others can do

• 28:10Speaker 01

  is still the most important thing for filling in those gaps in the in the Swiss cheese of of AIs. What is just, again, high-volume, low-value content? My hunch is that that's, if we score it correctly, going to be exactly what it what it is, which is low-value content. And so it should it should be rewarded very minimally. Um, I I I like to ski, so I I live part of the year in Park City, Utah.

• 28:35Speaker 00

  You're in the right place.

• 28:36Speaker 01

  I I, yeah, I I care, I care enormously about the snow forecast. There is a a A forecaster in Utah named Evan Thayer. He writes these incredibly precise weather forecasts where he will literally tell you it's going to snow this much on this run and this much on this run. And again, I actually pay for his content because that's super valuable for me. I am going to be more willing in the future to pay for an AI that has actually licensed Evan's content back from him than I would to pay for an AI that doesn't have that content. Because again, that content is going to be, you know, super useful and unique and valuable to me.

• 29:11Speaker 01

  And so I think actually what it will do is we, as we have more AI systems that are out there is it will cause you to look for more original creative content. And that's going to be the thing that the AIs are going to be the most willing to pay for. And that again, I think is actually a beautiful thing where we're, we're instead of creating incentives to create more and more salacious headlines and chase traffic. We're creating incentives to create knowledge that fills in those places in sort of the Swiss cheese where there might be holes. Taken in aggregate, all of the AIs are probably a pretty good representation of what human knowledge looks like. And so if we can score them and say, okay, here are the gaps in human knowledge and, and here are the places we need to fill in.

• 29:55Speaker 01

  That actually gives a really rich place for creators to look to create content which advances human knowledge.

• 30:03Speaker 00

  So you know, DeepMind is working on weather forecasting right now. This example that you gave of Evan Thayer, the forecaster in Utah. Are we that far away from just telling an AI, hey, like you're, you're tapping into the DeepMind model and weather forecast? I want to ski this route today. What's happening?

• 30:20Speaker 01

  I think we are probably pretty far away from that, but, um, and, but, but again, I think Evan is going to be always better. Using the tools of AI plus his local knowledge to make this better. AI just becomes a tool that creative people use in order to tell stories better, get better information, do more research. And, and again, I am skeptical that in the, in the short term at least, that we're going to have, um, real value that is created by, by training on purely generated content.

• 30:51Speaker 00

  Okay.

• 30:52Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com. Not authorized by any candidate or candidate's committee.

• 31:21Speaker 00

  So we've talked about your solution. Let's dive into the technological side of it a little bit. We are a tech podcast, so we should do that. So Cloudflare, a security company, helps websites stay up on the web despite all the threats.

• 31:34Speaker 01

  Yep.

• 31:35Speaker 00

  And let's just, at the very beginning, kind of talk about like the threats that you see to websites? Who's trying to take them down?

• 31:42Speaker 01

  Yeah.

• 31:42Speaker 00

  What's happening on that?

• 31:44Speaker 01

  Yeah, so protecting websites is part of our business. So is protecting employees as they go out across the internet. So Cloudflare is fundamentally kind of a network that is built with all the performance, reliability, security, availability, and privacy guarantees that frankly, the internet should have been built with had we all known what it was going to become. But obviously, back in the 60s, 70s, and 80s, when we were laying down all these protocols, we didn't think about those things. And so Cloudflare is basically reverse engineering the internet in order to give it those performance, availability, security, reliability, and privacy guarantees on top of what is there.

• 32:28Speaker 01

  And so today, one of the main uses for Cloudflare would be to, you're putting out a website, or a web application, or anything online, you want to make sure that it's safe from different sorts of threats. And so what are the threats that we see? I mean, every day, we go to war with the Chinese government, the Russian government, the North Koreans. I mean, everyone is trying to hack into our customers. Because who are our customers? Some of the largest banks in the world, some of the largest governments in the world. And they're all constantly under threat and constantly under attack from these organizations. The media companies actually were a pretty small part of our business. We had some media companies that used us, but it wasn't a big piece of it. What happened, starting really 18 months ago, is that those companies said, hey, I know we hired you

• 33:12Speaker 01

  in order to stop the Chinese hackers, but we have this new threat that's there. And frankly, my initial reaction was... Publishers, they're always whining about the next new technology, like what's, what's, what's going on? And it, and, and they, and, and over and over, they said, just pull the data, pull the data, pull the data. And it was only when we actually saw the data and we saw how AI companies were taking content without giving anything of value in return, that they were actually adding enormous amounts of load. And in some cases, taking whole websites down because of the amount of traffic that they were sending to it.

• 33:45Speaker 00

  Right. They, they basically DDoS the websites.

• 33:46Speaker 01

  DDoS the websites, you know, again, not intentionally, but, but that was the point at which we said, listen, maybe there is something that we can do here and, and, you know, at, at first, I think a lot of the publishers were saying, oh, this is, this is so hard. There's no way we can stop it. You know, there are these, these nerds and they live in Palo Alto and they're so smart. And what are we ever going to possibly do about it? And, um, and, and, and I just kept saying, guess we, we go to war with the, like the Chinese hackers, like we can stop some nerds with the C corporation. And I think it took a while for that message to really get through. But now that it has, you know, it was, it's been really rewarding to see that the vast majority of the world's publishers, major publishers have said,

• 34:29Speaker 01

  this is, we need to change the model. We need to be compensated for our content. And Cloudflare has the right idea in terms of the technical solution to do that.

• 34:37Speaker 00

  By the way, folks, $60 billion company listed publicly. So it's, uh, it's one of the bigger cybersecurity companies on the New York Stock Exchange. But I want to ask you, okay, so we're going to get into this technological solution, but you, what you said is interesting because what if these AI bots, you ever think there's a world where these AI bots ingest not just the publishers, but the banking websites? As well, like, are you like a natural enemy to having everything go through that? Because if everything goes through ChatGPT, then these other sites that you secure might not need your services.

• 35:06Speaker 01

  Um, I, I think, I mean, they still, there, there's going to be some, um, gatekeeper for how agents and other things access various services online. And I think that the challenges in each of those cases are different. In the case of a bank, you might want to say, I want to have guardrails that are in place. I want to make sure that this is actually a customer that's accessing account. I want to make sure that there are. You know, they can, they can only conduct transactions that have been authorized by an actual human being or, or something like that. Klevlar actually provides those guardrails and, and, and makes it so that a bank can say, I want to expose my infrastructure to AI, but do it in a way which is safe and secure. I think publishers have a different, different challenge. And so, you know, in our case,

• 35:50Speaker 01

  we're, a way of thinking is, is like, we have a whole bunch of developer documents which are on our website. We want those to be in AI. We want coding platforms when someone says, oh, I want to use CloudFlare to, you know, build X, Y, or Z, for it to be able to spit that out. What we've done is we've actually tried to identify with, with real narrow precision. What are those pages that are on the web that have some, some indication that they are going to be monetized? And, and generally that is, look at, is it behind a paywall? Or is it, does it have some sort of an ad unit on it, like a, a banner ad or, or some sort of ad that's there? If we detect that, then we're blocking it by default, but we're not doing this.

• 36:30Speaker 01

  Again, there's value for AI and we want to make sure that AI is actually getting the data that people want to have, have in it. So like the about us page on the New York Times probably should go into the AI system. But in brand new article, you know, with breaking news, probably should be restricted and again, unless the AI company is actually paying for that content.

• 36:49Speaker 00

  I guess the way I want to ask it is if everything goes into chat GPT, what's less, what's left for you to protect? Thinking, thinking outside of the media world.

• 36:57Speaker 01

  Uh, well, again, I think that eighty percent of the AI companies are, are customers of ours. And so, so we protect them as well.

• 37:04Speaker 00

  Yeah. Okay. Sounds good. Just wanted to ask that. I was curious about it, but let's talk. Okay. Now, so you're gonna build a, a technological solution that will block crawling.

• 37:13Speaker 01

  Yes.

• 37:14Speaker 00

  And so Robots TXT, which is this code that you put in like the header of your, of your site, if you don't want to be crawled, that wasn't working?

• 37:21Speaker 01

  Yeah, I mean, I think robots at TXT has two problems. The first is some people just ignore it. And so if you ignore it, then you can still crawl all you want. And there's some just, there's some even some big legitimate companies that completely ignore robots at TXT. And we're really good at basically being able to say, okay, here's what robots TXT says. How are you, are you actually following what those, those, uh, what, what sort of the rules of the road are? And if the answer is yes, then robots at TXT is a great solution. But in the cases where somebody is ignoring it, then we need to actually put in place additional technical barriers to restrict their, their, their access. And so that's exactly what we're doing.

• 38:04Speaker 01

  The second problem with robots TXT is it's not granular enough. So take the Google bot, for example. Google's crawler does five different things, at least. One is it checks if you have an ad on a page. It makes sure that if you're putting an ad for Procter and Gamble, a Procter and Gamble product up, it's not against a pornographic site or something like that. So it does brand safety checks. The second thing that it does is crawls to index for traditional search, the 10 blue links that are out there. The third is that it crawls to create answers that are in the answer box. The fourth is that it crawls to create answers that are in the AI overview, the newer thing that they've rolled out. And the fifth is that it crawls in order to ingest content in order to put it into Gemini. It's a lot of crawling. A lot of crawling, all through one crawler.

• 38:48Speaker 01

  And for lots of different reasons, they don't want to split that out into, into various crawlers. But right now, they basically make you have a choice. They say you can either block Google entirely, in which case you can't run ads, you don't appear in search, but you don't appear in the AI overviews or Gemini or other things. Or they've recently added a tiny flag, which, which basically just says, I'm not going to use this data just for the Gemini piece, but you still appear in AI overviews, you still appear in answer box. We think there needs to be more granularity, where there is a difference between taking content and transforming it. And a license should say you can't do that without my permission versus just taking that content in order to do brand safety checks, taking that content in order to do traditional.

• 39:31Speaker 01

  Search. And so what we've proposed, and we're working with the IED at TF as well as regulators is extensions to robots.txt to give it that granularity. And that actually then allows us to further test to watch, you know, if does this robot behave in an appropriate way? And if the answer is yes, then maybe it gets more permissions to do things online. If the answer is no, then we will put more restrictions and blockades in place to stop what are again badly behaving robots.

• 39:58Speaker 00

  So what you're going to do now is in addition to that, put a wall up, technological wall.

• 40:03Speaker 01

  That's right.

• 40:04Speaker 00

  No crawling. Sorry. Enough of you haven't respected robots, TXT, no entrance.

• 40:09Speaker 01

  That's right. And so the the original, like we're all familiar with like 404 errors when something is not found. Success on the internet is a 200 response that comes back to you. There's actually an original, one of the the protocols set out a 402 response. And that response says payment required. And so we're actually tapping into that exact original specification to say when a robot tries to access a page where there's an intent to monetize it. So it's either behind a subscription or it's got ads on it. That there is an ability for us to say 402 payments required. And then there's a negotiation in in some cases, and at first that's going to be largely large publishers with large AI companies doing deals like what Reddit has done or what the New York Times has done

• 40:54Speaker 01

  or what others have done where they have licensed the content. And then certain robots get access to that. But in other cases, and I think over time, that will be a dynamic process where maybe a smaller AI company or a smaller publisher will say, hey, here's what I would charge for this content. Cloudflare will surface like how valuable that content would be for that particular AI. And then the AI companies can decide, is that worth it or not? And it might be, you know, a very small transaction, maybe a fraction of a penny or maybe a few cents. Or in some cases, content is really valuable, might be worth hundreds or thousands or millions of dollars. You could imagine Taylor Swift, you know, is about to release a brand new song. And the lyrics get published, how valuable is that for an app for teen girls who are lonely

• 41:39Speaker 01

  and want to talk about things? Probably pretty valuable and especially valuable if you could have exclusive access to it for some window of time. And so that's the sort of thing where I think a marketplace over time can develop where original valuable content will get compensated and there will be a clearing price in the market. Once we have that scarcity that's created by that wall.

• 41:58Speaker 00

  Okay, so it's not just a blocker. It's also this marketplace where you're going to have publishers that will sell their content. So that's a way where you could have useful, effective chatbots and potentially a flourishing web.

• 42:10Speaker 01

  Exactly. And that's and that I think is what we're trying to play for. Again, my utopian vision of the future is robots should pay a lot for content and humans should get it for free.

• 42:19Speaker 00

  Right. And so to kick this off on June 30, as the day turned July 1st, you had a party on the top of the World Trade, One World Trade Center, where a bunch of publishers pressed a red button to get this thing going. And that includes some very big names, Conde Nast, Time, the Associated Press, The Atlantic, Adweek, and Fortune are all going to be part of this.

• 42:42Speaker 01

  And and a lot more. Frank frankly, there, there hasn't been a publisher that we've talked to who hasn't said that this is a change that needs to happen. You're on the right path for it. And so across the board, not only the kind of 20% plus of the web that sits behind Cloudflare already, but I think another 20 to 30% that are these major publishers that are out there are all on board and doing that. And what I think has been encouraging is at the same time, we've been having conversations with the largest AI companies and all of them agree that content creators need to be compensated for their content. They all agree on that. The devil's in the details and some of them are pushing back in various ways. But I've been really encouraged that as we have talked to the largest leading AI companies, the largest technology companies in the world,

• 43:27Speaker 01

  they're actually leaning into this. They all recognize that content creators need to be compensated. And I think over the months to come, that's when the hard work will be go down around how do we actually create this marketplace in a way? Which is fair for all of the different providers in the ecosystem treats everybody in a way that has a level playing field still allows new entrants doesn't just reward the largest companies with the biggest budgets that are out there make sure that you know legacy providers like Google are treated the same as you know newer providers that are there that's all going to be really tough but I am incredibly encouraged by the conversations I'm having not just for the publishers who are all on board but actually with the AI companies who recognize that something needs to change.

• 44:12Speaker 00

  That's interesting that they're they're recognizing this because the sense that you get is that you hear these announcements of deals like OpenAI paying X million to the Wall Street Journal to be able to include their articles or Dow Jones and the sense you get is that they're just kind of payoffs to not get sued like the Sam Altman very happy very clearly is not happy with the New York Times pursuing OpenAI and especially the actions that the Times are taking in their lawsuit like forcing OpenAI to preserve their chat logs which I think is wrong but but it is interesting so what do you think about is there are we going to see an evolution from these one-off deals to this marketplace style world?

• 44:52Speaker 01

  Well, I think that I mean we've seen this story many times before. I mean Napster was along it was a Wild West there was a bunch of lawsuits from you know the publishing the music industry targeting Napster and the like and then along comes iTunes which starts out as 99 cents a song but eventually evolves into what is much more much closer to a Spotify model of a subscription and a pool of funds that then get distributed out to all the creators. So so I think we've seen this story before and I think that one of the things that's really important is that OpenAI and others are willing to pay for content. They do the deals that are there and I don't think it's right to just say we'll do a deal to avoid lawsuits. Again, I think that

• 45:35Speaker 01

  that when you talk to leading AI companies they understand that people are doing the work to get create content they need to get compensated for that content and if it's not going to be through subscriptions or ads or ego it's got to be through something else and so exactly how that happens we'll figure out. But what I know won't work is if OpenAI is paying for your content, but you're giving it away for free to everyone else.

• 45:59Speaker 00

  It's not gonna work.

• 45:59Speaker 01

  OpenAI eventually is like, listen, we want, we want to support you. We want to help you out, but we can't be the suckers. We can't be the only ones paying where you're giving stuff away for free. And so scarcity is needed in order to actually have value in any kind of market. And so I think that the, the people who have actually leaned into this the most heavily are the ones that have the existing deals with some, but not all of the AI companies. Because they realize that for those deals to be valuable, for them to renew, for them to renew for more, there has to actually be scarcity where they're getting something of value. You can't, you can't charge OpenAI, but give it away for free to Anthropic. Something needs to actually restrict it and say, everyone needs to pay.

• 46:41Speaker 01

  Everyone needs to be on a, on a level playing field and figure out what that, that looks like going forward.

• 46:46Speaker 00

  Could there be some collateral damage with the solution like you're implementing? For instance, I'm looking at the names of these publications, Conde Nast, Time, the AP, the Atlantic. I imagine they get a lot of traffic from search as it is today. So if you put this blocker up, does that impact their SEO, for instance?

• 47:04Speaker 01

  Yeah, so we've been very, very careful to say that the traditional search today is not blocked. And, and even AI driven search today isn't blocked. But you're going to see us give publishers the tools to differentiate between search indexing and derivative content. So the way I would think about this is the Google experience today. It may be that a publisher says, I still want to appear in the 10 blue links, but I don't want to be in the AI overview or the answer box. And the granularity of being able to say, okay, Google, I understand you use one bot, but we need that to be treated similarly. And, and again, I am hopeful, and in my conversations with Google, I, I, I am increasingly hopeful that they understand the importance of this and giving that granularity.

• 47:47Speaker 01

  But if for some reason they don't, I, I am also a hundred percent certain that regulators are paying a ton of attention to this. And that around the world, you will see them force Google to split their crawler out into, into announcing exactly what it is doing. Again, I I I think that that's kind of the hopefully we get to an agreement with Google way before that has to happen. But but that's inevitably, I think Google is going to have to say, you know, if you don't want us to use your content for derivatives, you have a way of of of controlling that while still appearing in search.

• 48:21Speaker 00

  Okay, a couple of big picture questions before we leave. How much bigger is the web getting and is the web sort of accelerating in this size increases that we see?

• 48:30Speaker 01

  Yeah, I mean, it's it's actually been, but by all the measures that we can see, it's actually kind of plateaued and is actually flattened out in terms of in terms of in terms of content. You see fewer domains getting registered. You see fewer new websites going online. I think a lot of that has moved to individual platforms. So more of that on a YouTube, more of that on a Facebook, more of that on a TikTok. That that is there. And I think part of that is because those tools have provided content creators easy monetization tools to allow them to to to not have to think about some of those problems. I think that in in a in an ideal future, you would want content creators to be able to be free from those platforms to earn more

• 49:12Speaker 01

  themselves and but still have abilities to monetize that content in in interesting ways. And so, again, I think there are lots of people who are working on on that problem. I actually think Google has been, you know, one of the organizations that has again, created what was the business model of the last 30 years of the web. But the business model of the next 30 years of the web is going to be different and we've got to think about it in a different way. It's not going to be banner ads. It's not really probably going to be subscriptions. It's going to be something different. And so this is our attempt at at one solution, but I doubt it will be the only one that that emerges.

• 49:46Speaker 00

  Now, I'm curious what I'm going to do because just a one person content operation, so.

• 49:51Speaker 01

  Well, you should certainly be charging AI to license your voice.

• 49:54Speaker 00

  Can I sign up to your product?

• 49:56Speaker 01

  For sure. Absolutely.

• 49:57Speaker 00

  Okay, I'm gonna email you after this. And then, when it comes to cybersecurity, obviously you talked about how you're dealing with all these governments that would like to hack into sites across the web. Have they been able to use generative AI tools or automated coding to get more, to become more effective at what they do?

• 50:13Speaker 01

  Yeah, I mean, I think that any time a new technology comes out, bad guys are gonna use it as well as as good guys. And so, we have seen, and we will continue to see some horror stories around, you know, the family that was tricked by some gang in order to wiring their life savings because something, someone that sounded like their, their daughter called and said, "I've been arrested in Mexico. You know, I need, I need to pay to get out." Or, or, or other things. I think we are seeing a real rise in, especially out of North Korea, North Koreans posing as if they were applicants to various, various jobs. And then, that is, you know, allowing them access, which they can then use to, to do, to do any number of

• 50:57Speaker 01

  nefarious things. All of that, again, assisted by AI. So I think that's, that's been, that's been sort of on the, on the bad guy's side. The, the good news, though, is that the good guys, you know, folks like CloudFlare, we have been using AI as well in order to not only detect these things, but get smarter at detecting attacks earlier in the process.

• 51:16Speaker 00

  That's working for you.

• 51:17Speaker 01

  At, at the end of the day, who wins in the AI race is whoever has the access to the most data. And, and I just think that the good guys are always gonna have access to a lot more data than the bad guys. And, and so far, I feel like we've, we, we have made the, the web more secure with AI over the course of the last two and a half years, and, and stayed way ahead of the attackers. Although, again, there are gonna be horrible stories, there are gonna be problems that are there. I, I think that i- it is going to be harder and harder to trust that something that you're seeing online is, is actually, you know, real, and we'll, we'll have to turn to other ways that are more secure about verifying things like identity and, and authentication.

• 51:55Speaker 00

  Okay, last question for you. We have 60 seconds. You mentioned you're a believer in this technology. What does the next couple of years in AI look like to you? Are we gonna hit AGI anytime soon? Like, what are the, what's the timeline you're thinking about?

• 52:07Speaker 01

  I mean, I, I don't, I don't, I'm not, I, I am, I am, so I believe today. That 99 cents out of every dollar spent on AI is just being lit on fire. But that one cent that's out there is going to generate real returns. It's very hard to figure out what's kind of just a total waste of time versus what's not. You know, we see a lot of data about how much, you know, AI systems are really are being used, not so much for businesses today. A lot of the business applications have been very tough to take on, but a lot of times just for like loneliness and social interactions and things like that. So I would imagine that a lot more of those things are going to develop and those will be sort of the first uses.

• 52:51Speaker 01

  I think the business applications are actually going to take longer. And in places where it's easier to verify the output as being legitimate, it's going to be easier. So coding, like we see that our engineers are significantly more productive by using AI tools than they were before. That's not causing us to hire any less engineers. It's just meaning that every engineer we hire is that much more productive. We have a huge backlog of things to do and AI is helping us do that. On the other hand, you know, I am, I'm still quite skeptical that the AI customer support agent, that is a much harder problem. Or the AI lawyer, that is a much harder problem because it's just harder to tell whether something actually worked or didn't. There's no debugger in those spaces in order to figure out if what the AI is creating was actually true.

• 53:36Speaker 01

  And so I think you're going to see just huge leapfrogs in what are things like coding. But I think it's going to take longer for us to do things that are a little bit more difficult to verify.

• 53:47Speaker 00

  Very interesting. You're deeply optimistic about the technology, but still think 99 cents are wasted.

• 53:52Speaker 01

  Yeah.

• 53:53Speaker 00

  It's going to be a very interesting shakeout. Matthew Prince, great to see you. Thank you for coming on the show.

• 53:56Speaker 01

  Thanks for having me on.

• 53:57Speaker 00

  All right, everybody. Thank you for watching and listening. We'll see you next time on Big Technology Podcast.

• 54:17Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, theHouseMajorityPac.com. Not authorized by any candidate or candidate's committee.

• 54:46Speaker 03

  In this life, you can't win.

• 54:48Speaker 04

  When the world gets you down.

• 54:49Speaker 03

  No!

• 54:50Speaker 04

  Escape to Pluto TV.

• 54:52Speaker 03

  This is objectively fantastic.

• 54:53Speaker 04

  Relax into the movies and shows you already know and love, all for free.

• 54:57Speaker 03

  Shwing!

• 54:58Speaker 04

  Even our news is uplifting.

• 54:59Speaker 01

  You stay classy, San Diego.

• 55:01Speaker 04

  Watch School of Rock, Crazy Ex Girlfriend, Anchorman: The Legend of Ron Burgundy, The X-Files, Wayne's World, and so much more.

• 55:07Speaker 03

  We're not worthy!

• 55:09Speaker 04

  The best things on TV really are free. You just have to know where to look. Escape to Pluto TV. Stream now, pay never.