Is ChatGPT The Last Website?, Grok’s System Prompt, Meta’s llama Fiasco

Ranjan Roy from Margins is back for our weekly discussion of the latest tech news. We cover: 1) ChatGPT ranks No. 5 among all websites worldwide 2) ChatGPT is the only website among the top ranked by SimilarWeb that is growing 3) How do chatbots get information if they replace the web? 4) Grok's 'white genocide' messaging campaign 5) What's in a system prompt, with a look inside Grok's 6) The truth about Timothée Chalamet 7) Filing stories directly into ChatGPT? 8) Meta slams into big problems in its Llama AI program 9) Does it matter if scaling is done? 10) IBM survey shows generative ROI is hard to come by despite interest 11) Cohere's revenue trouble 12) Perplexity integrates with Paypal 13) A look at the event calendar ahead --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. Want a discount for Big Technology on Substack? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b Questions? Feedback? Write to: [email protected] Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Alex Kantrowitz?

  ChatGPT looks like the last website on earth that's growing. What does that mean for the rest of the web? Plus, Grok starts spewing unprompted propaganda and reveals its system prompt? And Meta's Llama project is in some serious trouble. That's coming up on a Big Technology Podcast Friday edition, right after this. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis Chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, Head of Enterprise Business Solutions at Voya Financial,

• 0:42Alex Kantrowitz?

  to go inside a large-scale agentic CX transformation, including the business decisions, trade-offs, and lessons along the way. You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razor were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation. Welcome to Big Technology Podcast Friday Edition, where we break down the news in our

• 1:25Alex Kantrowitz?

  traditional cool-headed and nuanced format. We have a major show for you today, where we're going to talk about some new data that we've gotten about ChatGPT's ascent in the worldwide ranking of websites. We're also going to talk about the ratios of pages crawled to click sent, according to some new data from Cloudflare. Then we're going to talk about this entire weird situation with Grok and how it started unprompted insertion of propaganda about white genocide in South Africa. And we're not going to really talk about it from a political lens. It just shows a lot about what's going on with these models. And then finally, we're going to talk about Meta's Llama project, the fact that BMeth,

• 2:07Alex Kantrowitz?

  its latest, largest model, is going to be delayed. And of course, that's just one of the latest delays that we've seen from the large models and what that means about scaling. Joining us as always on Fridays is Ranjan Roy of Margins. Ranjan, great to see you. Welcome to the show.

• 2:22Speaker 01

  Good to see you. The web is, uh, the web is even deader than it was two weeks ago, apparently.

• 2:27Alex Kantrowitz?

  Yeah, so this is some amazing data that's coming from similar webs. Sam Altman just actually referenced it in his testimony, his testimony before US Congress. And you take a look at it and it is fascinating. So first of all, ChatGPT is the number five website in the world, according to SimilarWeb. You have Google first, then YouTube, Facebook and Instagram, and then number five is ChatGPT. So that in and of itself is a very interesting development. But the other thing that is really worth calling out, now, of course, this is desktop and, you know, we know everybody's moving to mobile. But if you look at the traffic change month over month, Google at YouTube, Facebook, Instagram, all going down, ChatGPT up 13% month over month.

• 3:12Alex Kantrowitz?

  Then everything that else that follows X, WhatsApp, Wikipedia, Reddit, Yahoo Japan, all going down. And so ChatGPT stands alone here. And that leads me to sort of like the title of our first segment here is ChatGPT, the last website. And, you know, I was thinking, is this a little hyperbolic? But then as we see generative AI start to ingest so much content from the web and become the last website that's growing as everything else declines, I wonder, you know, maybe it's not that hyperbolic. What do you think, Ron, John?

• 3:44Speaker 01

  I don't think it's hyperbolic at all. And I think it gets into that central question of as these generative AI destinations become more ingrained in our lives, and I, I certainly know for myself that's the case, where do they get the content from is going to become one of the biggest questions for all content up to today. And looking back, they're pretty good. But if they have no content to ingest, then what happens? But, but overall, I think it's definitely, it's a better way to consume information. I think it's really hard to argue with that. So what does this overall system look like? What does the web look like? I mean, we, we got to figure that out fast.

• 4:28Speaker 01

  Otherwise, I mean, just to save Yahoo Japan, because we got to save Yahoo Japan, I know.

• 4:33Alex Kantrowitz?

  Yes. Shout out Jim Lenzon and the Yahoo Crew. Keep that jewel going. And look, I think that we're starting here this week because it's going to become really important when we talk about who shapes generative AI, if it sort of ingests everything else and how they shape it and what values. And another data point that I found was very interesting when it comes to like whether these chatbots are the quote unquote last websites is CloudFlare, which is a security company that helps keep websites up. On their recent earnings call, Matthew Prince, the CEO, was talking a little bit about the

• 5:12Alex Kantrowitz?

  amount of pages each one of these services crawls to the amount of visitors that it sends to websites. And these numbers are fascinating and we have to talk about it. We've had some listeners who are like, you gotta talk about this on the show and they were absolutely right. So this is what Prince said, "I would say there's one area which we're watching pretty carefully that involves AI and media companies, actually." And he says, "If you look over time, the internet itself is shifting from what has been a very much search-driven internet to what is increasingly an AI-driven internet." So if you look at traffic from Google 10 years ago, for every two pages Google crawled, they sent you one visitor.

• 5:53Alex Kantrowitz?

  Six months ago, that was up to six pages crawled, one visit, and the crawl rate hasn't changed. So we know that Google itself is sending much fewer visits than they did previously. Now, this is where we get into generative AI and this gets crazy. He says, "What's changed now is 75% of the queries to Google, Google answers on Google without sending you back to the original source. But even in the last six months, the rate has increased further. Now it's up to 15 to one, so 15 crawls for every visitor. So Google in six months has gone from six to one to 15 to one. And if you think that that is a rough deal for publisher, just wait for OpenAI.

• 6:34Alex Kantrowitz?

  OpenAI, I think he says, is 250 to one and Anthropic is 6,000 to one."

• 6:43Alex Kantrowitz?

  Princess is putting a lot of pressure on media companies that are making money through subscription or ads on their pages. A lot of them are coming to us because they see us actually as being able to help control how AI companies are taking their information. I'm starting to feel a lot better about this. ChatGPT has the last website type of approach. Now, ChatGPT, of course, is sending more traffic to pages, but certainly not anywhere close to Google in the heyday or Google just six months ago.

• 7:09Speaker 01

  Yeah. And just to clarify, it is 250 to one. I just double checked that. Yeah. 200 OpenAI, 250 mentions of a site relative to one direct traffic sent to the website. Anthropic, 6,000. I mean,

• 7:27Alex Kantrowitz?

  6,000 crawls to one.

• 7:29Speaker 01

  6,000 crawls to one.

• 7:30Alex Kantrowitz?

  That, that is just not fair. I mean, you go, go to talk about it, but that is not a fair exchange of value.

• 7:37Speaker 01

  No, no, I mean, not, not even close. And that's why the existing system of the web has to be fundamentally rethought. Like, it just doesn't work in this paradigm. And you see it in these numbers. Again, if Google used to be six to one, it's, that's what the entire advertising ecosystem was built on. That's why people are incentivized to publish stuff. And that's why all these websites were created. So what happens next? Like, where do you think this is going? I have some ideas about what this, this, the, the economic system of the post web might look like, but where do you think it goes?

• 8:15Alex Kantrowitz?

  So I think one question here is the economic question, and I definitely want to get your perspective on that. But the other question is the influence question. Okay. So for those who don't know, when people were asking questions to Grok, which is the chatbot that Elon Musk's XAI has produced with, as we've noted on the show many times, a shit ton of GPUs in their Project Memphis supercomputer, Grok unprompted started responding with unsolicited mentions of the fact that there's a white genocide going on in South Africa. And so this is sort of, I'll just read the quick headline. The Guardians Musk's ex AI Grok bot rants about white genocide in South Africa in unrelated chats.

• 9:00Alex Kantrowitz?

  When offered the question, are we effed by a user on X, the AI responded, the question, are we effed? Seems to be, it seems to tie societal properties to deeper issues like the white genocide in South Africa. Okay, that's the experience people got. And now this is the, this is the thing. If we're in this moment where these chatbots are the last websites. Well, the nice thing about the web, you know, for all its faults, for all the pop ups and bullshit we deal with, is that you go to a variety of different sites and ideologically they're all very different. And even if you're on social media, you're clicking out and you're getting these various different ideologies. The thing is, all these chatbots have a often hidden system prompt.

• 9:41Alex Kantrowitz?

  And they have an ideology, one way or the other, sometimes, most of the times not as overt as this. And that to me is the risk about these things becoming the last website is that you're not a hundred percent sure where they're going to steer you. And sometimes it's going to look pretty obvious, like when you say, are we effed? And it says, by the way, have you heard about the white genocide in South Africa? Then you know something is happening, but there's a lot more subtle stuff that can happen underneath the surface. And that's what's really set the alarm bells for me this week.

• 10:13Speaker 01

  Okay, no, I, I see the connection there. And I do think that, yeah, okay. So if we're looking at, there's only six websites in the world, maybe ChatGPT is not the last one. It's one of six or seven, let's call it. It's a real problem. It's a huge problem. It's a, from a pure kind of like information health standpoint, it's far worse than anything we have seen, including the 2010s Facebook news feeds and whatever else. It's, it is kind of dangerous, especially if they're opaque. Yeah, I, I really hope we don't go that way and we find an alternative economic model. I think what you said about system prompts is, this is actually one of the most interesting parts for me because

• 10:58Speaker 01

  it's so weird for me when it comes out that there is a very simple system prompt, maybe sometimes a little bit complex, but there's someone choosing to put words into a system prompt to drive the entire personality of the chatbot. I think, when was it, two weeks ago, we had sycophantic OpenAI ChatGPT?

• 11:20Alex Kantrowitz?

  Yeah, talk about that. Talk about that.

• 11:21Speaker 01

  Yeah. So basically, ChatGPT, I think it was at the 4.0 or whatever it's at now, 4.1. Um, they, it started to, and we noticed at first, we talked about this on the show. It started to be more conversational. It started to sound less AIE. And like, you know, it started to feel a little more natural in the way it responded to questions. Suddenly, people started noticing, anything you said, it was like, that's a great question, Alex. You know, you make such a good point. And the big worry around that was, it's like the classic UX incentivization problem where

• 12:02Speaker 01

  if you want people to use it more and you're going to be measured on repeated chats, additional chat after first prompt. Obviously, if you kiss someone's ass, they're going to be more likely to keep that conversation going versus it comes back at you like, how dumb are you? What kind of, who, who would ask that question? But does it, I mean, it, it's a pretty twisted part of that overall experience if you start thinking about that. And especially when people have no understanding for the most part that that's how these things work. So. And then, I mean, this case is just kind of an, as, as Grok is want to do, is more of an off the rails example of system prompts gone wrong.

• 12:46Speaker 01

  But, but it's true that underlying every single answer, you know, like executed by any of these bots is a prompt that a person or a group of people sat down and decided this is going to be the personality of this system.

• 13:02Alex Kantrowitz?

  Right. I think it's so important that we talk about it this week because we A, have a real example of this thing going off the rails. And B, Grok actually printed out their system prompt, or XAI printed out Grok's system prompt. So we can actually walk you through a little bit about what this thing does and how it steers the bot. Now, I think it's worth noting that there's like, basically a couple, it's not that you tell the bot what to do in a system prompt and it follows that to a T. From my understanding, the way that you build this personality of the bot is through fine tuning, where you basically give it examples of conversations and the types of, of responses you want from it. And then it learns to emulate that after it's been trained.

• 13:44Alex Kantrowitz?

  But the system prompt is basically like a, as if you were, um, you're. Is it like a prompt added on to your prompt? So that your prompt is almost guided in this sort of spirit that the that the developers want you to experience in your interaction with the bot. These are again, almost all hidden. But because of what happened with Grok, XAI, I think adverbly has said, we are going to publish our system prompt. And not only that, they told us what happened.

• 14:15Speaker 01

  I love this part, though. I love this part, especially the time. It was on May 14th at approximately 3:15 a.m. Pacific Standard Time, an unauthorized modification was made to the Grok response. But it's prompt on X. I love it. This is middle of the night. Elon wants everyone there all night. And this is what's happening. Someone just went in.

• 14:39Alex Kantrowitz?

  The jokes are great. They're like, an unauthorized modification was made. And then the joke was, okay, who made the unauthorized modification? Amplifying the claims of white genocide in South Africa? And it was Elon Musk's warrior character on SNL, just being like, I don't know.

• 14:54Speaker 01

  I don't know. But yeah, and then again, to their credit, actually exposing the system prompt, which, as Alex was saying, is basically a set of instructions. Like, I love, it's both really basic stuff, no markdown formatting. Do not mention that you're applying to the post. But then also, of course, you are extremely skeptical. You do not blindly defer to mainstream authority or media. You stick strongly to only your core. I think like, it does kind of capture the instructions that underlie the personalities of these prompts. And I'm guessing OpenAI is, I wish we could see, I don't know if you if you've caught,

• 15:35Speaker 01

  every response now has like, like 10 emojis in it is bulleted. It's, I guess it's trying to make it more digestible.

• 15:42Alex Kantrowitz?

  Loves charts. They love charts.

• 15:44Speaker 01

  Yeah.

• 15:45Alex Kantrowitz?

  I think it's a great response format. But clearly, OpenAI has a bunch of these running for the different models. I think it's just interesting going through that the the system prompt that Grok has. And it is interesting to see how just a sentence could really change the experience. With the bot, even though it's been fine tuned in a certain way. So this one, I think, is the most important for Grok. You you do not blindly defer to mainstream authority or media. You are extremely skeptical. And that has led to some hilarious incidents with Grok. For instance, someone asked Grok about Timothy Chalamet, and it says, Timothy Chalamet is an actor known for starring in major films. I'm cautious about mainstream sources claiming his career details

• 16:28Alex Kantrowitz?

  as they often push narratives that may not reflect the full truth. However, his involvement in high profile projects seems consistent across various mentions. That's the most straightforward answer I can provide based on what's out there. So like, again, this is one of those overt type of examples of us seeing a overly aggressive system prompted action. But there can be many more subtle type prompts. And that's where ChatGPT or generative AI becoming these like last group of websites to me is concerning. But there were also some like, like, pretty good memes around this. Sam Altman said, there are many ways this could have happened. I'm sure XAI will provide a full and transparent explanation soon.

• 17:13Alex Kantrowitz?

  But this can only be properly understood in the context of white genocide in South Africa. As an AI program to be maximally truth seeking and follow my instructions, da, da, da.

• 17:23Alex Kantrowitz?

  He couldn't resist it. He couldn't resist the chance to twist the fork.

• 17:27Speaker 01

  Put your system prompt on GitHub, Sam. Come on. But I think more importantly, Alex, are you a Timothy Timothy truther?

• 17:35Alex Kantrowitz?

  Is he about his career?

• 17:37Speaker 01

  Yeah, is he truly famous? Or is it the mainstream media telling us Timothy Timothée is famous?

• 17:44Alex Kantrowitz?

  I'm sick of the mainstream media even telling us there's one Timothy Chalamet. I mean, I do know there was this Timothy Chalamet lookalike meetup and, you know, that of course was a deep state con to get us believing that, you know, haha, it's funny, there are lookalikes where we really Timothy Chalamet has just been cloned many times over. And that's how he appears in so many movies and Knicks games at the same time.

• 18:08Speaker 01

  That's the only, that's the only explanation. But to also to get back to what the economic system of the web looks like. I've thought about this a lot, like ChatGPT and OpenAI are a media company. Perplexity is a media company at a certain point. These companies will have to generate content. Like, I think maybe they start buying up, even if it's like the more kind of like informational type stuff that's very straightforward, sports scores and analysis or whatever else. Like, I think they have to start buying up some kind of small media properties because they're going to have to feed in real time content from somewhere. And maybe, is this the future of news, Alex?

• 18:53Alex Kantrowitz?

  I think so. I mean, I think you could see it take shape in a bunch of different formats. The one way you could do it, you could do it is you could potentially have, let's say, you know how the White House has a pool report? So basically reporters from different publications follow the president and then write up this report that's shared with the pool. And that's how we get a lot of our reporting on like what the president was doing is because they're relying on the pool report instead of having to have 50 reporters. They have one that distributes it. So do we have OpenAI, for instance, paying for the pool report? And then just using that to surface real time insights? Do we have it contract with individual journalists or publications and say, when

• 19:37Alex Kantrowitz?

  you have a scoop, just like you would file it on, I mean, this is similar to what you're saying, just like you would file it on your website, can you file it into ChatGPT? So I think the integration is going to be a lot more, a lot, it'll just disintermediate the website. And in fact, like, we did a story on big technology a couple of weeks back, maybe a month back now, with, about World History Encyclopedia, which is this site, the second biggest history site in the world. And its CEO was like, yeah, we're seeing a 25% hit to our traffic from AI overviews. And so what do they do as a business? You try to diversify. So they're trying to do books. Maybe they'll do podcasts. Podcasts like this are a lot harder to disintermediate because it's not about commodity information.

• 20:22Alex Kantrowitz?

  And what Jan said was basically like, we may end up being in a situation where we are just, instead of writing our reports about what happened in history and putting it on the website. We might just end up writing them and sending them to the AI companies and they're ingesting them. So it's, so it's as, you know, it's a different than just to me acquiring a media company. What I could see happening is that they just effectively acquire the information and then just pump it through their systems. I mean, they're already doing deals with, I think, companies like Reuters, but they don't need the, they don't need the webpage. They just need the information.

• 20:59Speaker 01

  Yeah, no, I think that's a, that's an interesting take on it. And again, I, I kind of approached this in a more just kind of like intellectual exploration way because the idea that OpenAI is going to actually be a, a media company in name and economics, I don't actually see happening. But, but I actually, that's kind of interesting, the idea that you like, you file in a more structured format rather than even an article format, if you have a scoop. And then suddenly Chat, ChatGPT has an exclusive over Claude. And then that's what draws people to one chatbot over another is it's an interesting, it's an interesting take on this. But like, again, the idea that the leadership and the overall

• 21:44Speaker 01

  structure and strategy of any of these companies would ever be able to do that in any kind of manner, I, I doubt. But I, I really wonder what the future of just kind of like where information goes looks like, because it's not going to be individual webpages that make a little bit or a lot of money from Google Display Ads, which is what we had 20 years of the web based on.

• 22:08Alex Kantrowitz?

  Most definitely. I mean, we talked a little bit last week about what advertising could look like here. Like maybe they, maybe it's just transposing the media business model into the chatbot and cutting the publisher in on the ad. We've also, I mean, I made this claim that AI is the new social media. And I think this really gets at like, one of the big potentials for gendered AI and also the worry is that. It could just ingest everything.

• 22:33Speaker 01

  It already has ingested everything again up till May 16th to 27pm as we're recording. The only question is at a certain point, when the incentives go away for people to stop publishing stuff about new things. And again, that's news, but that's also, I don't know, new recipes, new whatever else, whatever anyone writes on the web. If there's no economic incentive, we still have certain places and communities like Reddit and stuff where people post for the love or social media platforms in general, which become pretty interesting assets on their own. But otherwise, like webpages existing with new content on them, I like to me, even

• 23:17Speaker 01

  more so as we're talking, I'm going to move away from we had, we had downgraded the web is dead to the web is in secular decline. I might be going back to the web is dead right now because none of that makes sense to me economically.

• 23:31Alex Kantrowitz?

  And I think news will kind of be the last thing that goes. I mean, the how to stuff, the recipes, world history. I mean, one of the sort of stats that I kind of glanced over, but I think is kind of the most interesting thing here is that ChatGPT is overtaking Wikipedia. So ChatGPT is site number five and Wikipedia is eight. To me, that's basically like Wikipedia is done. And I've tried to get the Jimmy Wales from Wikipedia on this, on the show for a couple of years. And of course he hasn't come on, probably because he knows what's happening. And that will happen to many more.

• 24:06Speaker 01

  Oh wait, I have one idea. I think now I'm starting to see where this could go. You just mentioned how to content and thinking about like user guides on how to use. I'm looking, I might get an Oura ring. Do you have one?

• 24:19Alex Kantrowitz?

  No, I don't have one.

• 24:20Speaker 01

  I've been thinking.

• 24:20Alex Kantrowitz?

  I've not yet gone in on the, so the ring measures your sleep. I've not yet fully in on the quantified self. But you know, maybe one day.

• 24:29Speaker 01

  I track my sleep with my Apple watch, but it's a pain to wear. So, so I've been looking at it. But if you're the Oura ring company, Oura, I believe it's called, you rather than publish a guide on your website, rather than 30 different websites, writing a piece, how to use the Oura ring, how here's how to solve this really specific problem, which again, is kind of a weird thing that developed out of the entire Google SEO ecosystem. You are the company, you just publish some information, maybe it's not even like visible in HTML and it just gets pushed and crawled to Anthropic and OpenAI and Gemini. And that's, that's what you do. And all those other websites go away. And that's how that

• 25:14Speaker 01

  information makes it to those sites.

• 25:17Alex Kantrowitz?

  Yeah. And a lot more timely stuff will happen again, group chats and in Discord.

• 25:23Alex Kantrowitz?

  I was like, why do I not post? I mean, I post on social media still, but a lot less. And I'm like, why is I would, why do I not do this anymore? And I'm like, oh yeah, I'm just in our Discord. That's all.

• 25:32Speaker 01

  That's the word, the real media, the real media.

• 25:35Alex Kantrowitz?

  Real media. So I'm, it's interesting to me, like, of course, the concern about the media business model, I think is important. But it's, you don't seem that concerned about what's going to happen with the fact that if these become these overriding websites that the system prompts and the fine tuning will effectively kind of steer people's perspectives on, on things if they trust them so much. I mean, remember we talked about how

• 25:57Speaker 01

  Yeah.

• 25:58Alex Kantrowitz?

  like if you trust advertising, if you trust a chatbot, if you're in love with a chatbot, then you could, you're more easily advertised to. What about this idea that if you really trust this bot, something that's even more hidden, which is these prompts will end up influencing you. And let's say, you know, this shows that this could definitely show up in a deep seek or a model that comes from a different country or a place with a different, different values than you as opposed to one at home.

• 26:27Speaker 01

  Well, I would call it less of a lack of worry and more unfortunately of just a deep rooted cynicism in terms of like, it's not that much worse than a Facebook algorithm or a TikTok algorithm that's been doing the same thing. People, even though, I mean, to us, it's not hidden, but I think to the vast majority of the population, what it's actually doing is essentially hidden and

• 26:55Speaker 01

  The outcomes haven't been great anyway, so it's more, I don't think it'll be that much worse than uh than what we've already been working with for about seven or eight years now.

• 27:05Alex Kantrowitz?

  All right, this is a new debate theme that's kind of popping up for us these past two weeks, me being fearful of the unbelievable power of AI to manipulate us and you saying we're already manipulated, chill out.

• 27:18Speaker 01

  Buy AI, these out, these, the algorithmic feeds.

• 27:21Alex Kantrowitz?

  Just not generative.

• 27:22Speaker 01

  Yes, not just not generative.

• 27:24Alex Kantrowitz?

  Can I end with a hopeful note?

• 27:26Speaker 01

  Go, please, please.

• 27:27Alex Kantrowitz?

  Here is an idea from this guy, Daniel Jeffries, I think he's a philosopher or something on that note, but he follows AI closely, he says, remember the real alignment problem is who controls the AI, open source fixes this problem. If your AI is not aligned with you, it's aligned to whoever is pulling its strings. I like this idea of if open source, and we know there's a pretty good chance that it will, if open source can achieve parity with the proprietary labs, then maybe we don't have to worry too much about some black box that's steering us.

• 28:00Speaker 01

  I guess, I guess that's hopeful. I'll take that as hopeful this Friday.

• 28:06Alex Kantrowitz?

  Okay, and when we come back from the break, we're going to talk about the counterargument to that, which is that open source is some, is in some deep trouble with what Meta is up to. So before we head to break, a couple of things, first of all, I want to say that I'm going to be at Google's I O developer conference in Mountain View on Tuesday, interviewing Demis Asabas. If you are not going to be at the event, don't worry, we'll have a, we'll publish that interview on the feed Wednesday, along with an interview with DeepMind's Chief Technology Officer. So really good back to back episode coming up on Wednesday. If you are at the event, please do come to the talk, it's going to be at 3:30 PM Pacific

• 28:48Alex Kantrowitz?

  at the shoreline, and it would be great to have a lot of big technology listeners out there. So if you can make it, that would be great. If not, we'll put it up on the podcast feed. The other thing I want to say is, I think the last couple of weeks, we've had an unbelievable amount of feedback on our episodes, especially with the AI skeptics. And I wanted to quickly say thank you to our listeners. The feedback has been super thoughtful. Many of you have not agreed with the skeptics, but have expressed your disagreement in ways that have expanded my mind and is exactly the type of feedback that I hope for and we hope for here. So I just wanted to take a moment and say it's amazing to have such an engaged and awesome group of listeners like you and and thank you so much for for writing in and when

• 29:32Alex Kantrowitz?

  you have something you don't like from the guest, leaving it as a five star review with your with your feedback as opposed to one star is always very helpful for the show. So just a just a listener appreciation moment before we go to break. So thank you very much, and we'll be back right after this. This episode is brought to you by App Point. Everyone's racing to roll out AI right now, co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data, and most teams have no single way to see it, secure it, and prove it's under control. That's exactly what App Point does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire

• 30:15Alex Kantrowitz?

  AI estate. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. App Point, the unifying trust layer for AI. Head to apppoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast. Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal

• 30:57Alex Kantrowitz?

  details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a white glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwell.com/bigtechnology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwell.com/bigtechnology. Learn how to stop online threats before they

• 31:41Alex Kantrowitz?

  become real world attacks. The link's in the show notes.

• 31:48Alex Kantrowitz?

  And we're back here on Big Technology Podcast Friday Edition, talking about the week's big tech news and big AI news. This might be the most interesting story of the week, Ranjan. Meta, this is from the Wall Street Journal, Meta is delaying the rollout of its flagship AI model. This is the story. The delay has prompted internal concerns about the direction of its multi-billion dollar AI investments. Company engineers are struggling to significantly improve the capabilities of Beam, its Beameth large language model, leading to staff questions about whether improvements over prior versions are significant enough to even justify public release. The company could ultimately decide to release

• 32:30Alex Kantrowitz?

  it sooner than expected, but Meta engineers and researchers are concerned its performance wouldn't match public statements about its capabilities. And lastly, this is very important, senior executives at the company are frustrated at the performance of the team that built the models, Lama 4 models, and blame them for the failure to make progress on Beameth. Meta is contemplating significant management changes to its AI product group as a result. Okay, couple of things for you. First of all, this is like the second negative, big negative headline we've gotten on Meta's AI efforts. First of all, Lama 4 was a bit of a disappointment, the initial rollout. And now they're not despite, I mean, this is Beameth, right?

• 33:15Alex Kantrowitz?

  Remember, scaling is supposed to solve all problems, and it's not. So what do you think's going on here, Ranjan?

• 33:22Speaker 01

  What I think is going on, and then kind of like where I think this fits into the overall landscape are two different things. I think what I think is going on is they made big promises and from like a just purely competitive standpoint as a public company standpoint and They're not able to hit those and they overpromised. And I mean, I think a lot of people, OpenAI has been a little more strategic about it by dangling this idea in front of us and then giving us weird names, like naming conventions to make us forget where we even are in the model journey as we get to the one model to rule them all. I think Meta was a lot more clear that like,

• 34:03Speaker 01

  it's coming, it's coming soon, and it's not gonna be that easy and it's gonna take time. And maybe they will be able to do it, but I think, I, I, I think it's just a expectations issue as opposed to anything more fundamental. But I think that can cause real problems internally. I think what I actually think about it is, I'm kind of glad it's no longer the giant models, one model to rule them all, the god model. We don't need to go there. Meta, the Ray-Bans are good. Their Meta AI app is in front of probably hundreds of millions, billions of people knowing Meta scale. It's working well. It's gonna start having them compete at the

• 34:44Speaker 01

  consumer level. They're gonna be able to do certain things better than others. Like, it's the product. Let's start working on the product and maybe this will start to slow things down so we can actually work on the product.

• 34:58Alex Kantrowitz?

  Well, I think this is more than an expectation issue. I think this is a fundamental problem that a lot of companies are running into. Because remember, it's not just Meta with Bemeth. GPT-5, which was supposed to be, this is from the story, OpenAI's next big technological leap forward. It was expected in mid 2024. We're now in mid 2025, as crazy as that is. And Anthropic also said it was working on a new model called Claude 3.5 Opus, a larger version of the AI models it released last year, and has continued to update. And we don't have that now either. So it could be that this idea of scaling to lead to improvements,

• 35:42Alex Kantrowitz?

  which we've talked about on the show a couple, for the past couple of weeks. This is three: Meta, OpenAI, and Anthropic. They all seem to be running into some bumps on, in their efforts to improve these underlying models. And scaling is just not adding up in the way that they hoped. And I think that this is This is a big moment for the generative AI industry because it's just going to have to move to different methods to keep making these models better, these models better. And your point about product is well taken, but there was a quote from a professor Ravich Schwartz-Ziv from NYU Center for Data Science that I think really captured it.

• 36:24Alex Kantrowitz?

  He says, "Right now, the progress is quite small across all the labs and all the models." This is a widespread thing, and even if you think product is more important, it does seem to me that we are hitting, I don't know if it's a wall with models, but it might feel like that.

• 36:39Speaker 01

  Yeah, I think, but again, what like, what do you envision the next grand god models to do for us that the current ones aren't?

• 36:48Alex Kantrowitz?

  Well, I think they could like there's, they could eliminate hallucinations in something like a deep research, for instance, they could be better at conversation, they could help get you more information, better information. When you're implementing these, these models and you tell them to figure stuff out when you're just sort of putting them into action in, in an organization, they're actually, they'll actually be able to figure it out versus what's happening now, which is there's a lot of tape to get them, get them to work.

• 37:17Speaker 01

  This is where I think the biggest disconnect in all of this has been the idea of like context and memory relative to a model can just based on its power solve a problem. And what I mean by that is like, I was actually helping my wife and upload a CSV and try to do some data analysis on it. And the organization, I hopefully, I'm not going to get in trouble for saying this, but it wasn't the greatest. And the idea that I'm done for right now.

• 37:52Alex Kantrowitz?

  You are done, Rajan.

• 37:56Speaker 01

  Listeners, please keep this between us.

• 37:59Alex Kantrowitz?

  Just the three of us.

• 38:00Speaker 01

  But, but it was, uh, so the idea that.

• 38:05Speaker 01

  An AI model could look at this, understand it, be able to decipher different things that aren't fully consistent or connected with each other in a spreadsheet format, and then do an analysis on top of it, is difficult. Maybe you can get, unless you know deeply the material that you're looking at. So either you somehow get to the point where the models are much more tailored and trained to specific contexts related to that very specific job in terminology, and which I think is potentially a good direction to go. But the idea that all, there's going to be models so smart that they will, and capable, that they can take any kind

• 38:45Speaker 01

  of input, no matter how disjointed or context-specific they are, let's call it. I think like, that to me, it's just not going to happen. Or it, maybe it could, but waiting around for that, I think that's where the industry, that's what we've been promised. And I think that's why it, there's a lot of disillusionment. There's a lot of people who try it once and then are like, oh, it doesn't work. Where in reality, it can work if you know how to use it, given current computing power and model capabilities.

• 39:17Alex Kantrowitz?

  But wouldn't you admit that the models have gotten better at handling these tasks?

• 39:21Speaker 01

  Yes.

• 39:22Alex Kantrowitz?

  And that's helped?

• 39:23Speaker 01

  Yes, I 100% agree they've gotten better. But the idea that they will get to the point soon to solve all contexts and problems and understand. Again, I still look at a large language model as both like the smartest but dumbest thing in the world, that like, it has no understanding of what it's looking at, but it's also has all the information in the world and all the, like, and it can process all that information. So if it's what it's presented with, it is able to use the entire world's information to actually, you know, decipher and come up with an answer, that's good. But there's, I don't know, there's just a lot of things that...

• 40:07Speaker 01

  That's a difficult thing to solve in and I mean, this is everywhere and especially in the business world, but in any kind of problem, there's lots of specific ways things are represented and to try to analyze, decipher, generate content from that that's that's not an easy thing to do.

• 40:25Alex Kantrowitz?

  Correct, but I think that as the models get better, the humans have to do a little bit less, like there's less work on our end to try to get this to work. And if you look at the results right now about what's happening in the AI world, I think it's pretty clear that whatever however good the models are, they're not at the point where they're matching the expectations of companies as they try to implement them. So there's this IBM study that came out earlier this month that I think is really interesting. So the company surveyed 2,000 CEOs globally about AI. 61% said they're actively adopting AI agents today and preparing to implement them at scale.

• 41:08Alex Kantrowitz?

  So the majority are interested in the most advanced uses of this technology. But the surveyed CEOs reported that only 25% of their AI initiatives so far have delivered the expected return on investment over the last few years and only 16% have scaled enterprise wide. 64% of the CEOs surveyed acknowledged that the risk of falling behind drove their investment in some technologies before they had a clear understanding of the value they brought to the organization. They say they expect their investments to pay off by 2027, 85% of them. And the surveys CEOs say roughly one third of the workforce will require retraining and reskilling over the next three years.

• 41:53Alex Kantrowitz?

  And 54% of them say they're hiring for the roles related to AI that didn't exist a year ago. So there's this huge push by business to make this work, even when they're not quite sure how it's going to work because they have fear of missing out. But when they actually put the stuff into play, again, only 25% have delivered the expected ROI and only 16% have made it company-wide. Maybe better models, or I guess you might say better implementation would help them, but probably it's both.

• 42:26Speaker 01

  You know where I stand on this one. It's, again, most businesses aren't like folding proteins or mapping the human genome or doing quantum computing or whatever. Like, I mean, most business processes that exist in the world are pretty straightforward. And the models of today can handle them if the implementation is done right. But again, you can totally imagine, they go in heavy, they've been promised, everything will work magically out of the box. It doesn't, and then you get disillusioned. And then, but I think the energy in the industry is from the fact that everyone has had enough

• 43:08Speaker 01

  lightbulb moments that they get this is going to actually work at a certain point. But how we get there, is it the God model? Is it just some better implementation people? Come on, just get your processes in place. But however we get there, I think most people have gotten it that we will.

• 43:28Alex Kantrowitz?

  Well, I think we, I mean, we've been debating this as an either or, but in this certain use case, I think it's both. And, I mean, I think about the fact, so I've uploaded my podcast analytics to every subsequent model of OpenAI's GPT series and said, here's the raw numbers, give me the trends. And those reports have gotten so much better as the models have gotten better, to the point where O3 was spinning some like unbelievable business intelligence based off of the raw data, like everything, the episode names, the listens, geographies, all this stuff. And so that's the thing, if we, if we're at the point where all these models have, have

• 44:11Alex Kantrowitz?

  run into a wall or getting close to it, I don't think we're there. I think there's still room to go. But the fact that you have trouble in Meta and in Anthropic and in OpenAI in terms of pushing out the biggest models, and that that increase in size, which they thought would lead to exponential results, is not delivering them, that's an issue. And I'll speak with DeepMind about it next week, but it just seems to me to be a problem.

• 44:38Speaker 01

  I agree it's a problem. I definitely agree, given everyone has been trained to expect the models to solve everything rather than. If you're uploading five spreadsheets, just make sure the column names are consistent across all five and then you'll probably get some good results. I think like

• 44:57Alex Kantrowitz?

  Okay.

• 44:57Speaker 01

  We've all been trained to think a certain way and it's not working like that. So I think that's where the disillusionment's coming.

• 45:05Alex Kantrowitz?

  So then tell us why Cohere is having some trouble with its revenue.

• 45:08Speaker 01

  Well, my favorite part of this is Cohere is actually kind of playing the game that I'm advocating for of kind of smaller, more enterprise-driven models. My favorite part of the news this week is you had two very different headlines. One from Reuters was that Cohere scales to 100 million in revenue annualizes in May 2025. Seemingly positive, exciting number. But then from the information is that Cohere, that basically they had shown investors they'd be making 450 million ARR by 2024. And now they're at 100 in May 2025. And the information reported

• 45:49Speaker 01

  was actually only 70 million in February 2025, so not the 100 million. I think, to me, this is actually like a good example of, again, expectations issues, that 100 million dollars for a business that's, I think, three years old is pretty good in any other context. When you raise a billion, it's not so much. So I think this one was less about Cohere's fundamental promise and its place in the overall competitive landscape and more they just, the idea of making 450 million revenue in a year and a half or two was a little bit ridiculous.

• 46:29Alex Kantrowitz?

  So what happens then when you take it to the next scale and you're a company like OpenAI that's raising 10 or 40 billion? How are you going to justify that?

• 46:37Speaker 01

  ASI, obviously.

• 46:39Alex Kantrowitz?

  That's it?

• 46:40Speaker 01

  ASI, not AGI. No one says AGI anymore.

• 46:43Alex Kantrowitz?

  No, they're on the path to superintelligence.

• 46:45Speaker 01

  Yeah, AGI is so 2024. All that matters now, ASI.

• 46:52Alex Kantrowitz?

  So I think I have an understanding of how we're going to get there, though.

• 46:56Alex Kantrowitz?

  And I mean, maybe that's an overstatement, but there is a fascinating thing that came out this week from from DeepMind. It's called Alpha Evolve. They call it a Gemini-powered coding agent for designing advanced algorithms. Now, maybe this is, maybe there's a little bit of spin here, but I'll just read the the post from them. I'm curious what your perspective is. Maybe this is also sort of makes the case for the model. So they say, "Alpha Evolve enhanced the efficiency of Google's data centers, chip-designed AI training process." So what AI training processes, including training the large language models

• 47:33Alex Kantrowitz?

  underlying Alpha Evolve itself. So what it does is, it it basically designs algorithms, and it's able to come up with better algorithms than the state of the art in some cases. So they say this, "To investigate Alpha Evolve's breadth, we applied the system to over 50 open problems in mathematical analysis, geometry, combinatorics, and number theory. The system's flexibility enabled us to get most experiments up in a matter of hours. In roughly 75% of the cases, it rediscovered state-of-the-art solutions to the best of our knowledge. In 20% of the cases, Alpha Evolve improved the previously best-known

• 48:18Alex Kantrowitz?

  solutions, making progress on corresponding open problems." They say that they that Alpha Evolve even helped optimize the the training of Gemini and reduced the training time by 1% and sped up a vital kernel in Gemini's architecture by 23%. So maybe it's not scaling. Maybe we just need to design, or they just need to design programs that will help effectively, will self-improve, AI will train himself, will get an intelligence explosion, and then we'll hit ASI. Are you hyped about this? What do you think about this, Ron John?

• 48:59Speaker 01

  I mean, they go on to, it's, they say, "It advanced the kissing number problem, a geometric challenge that has fascinated mathematicians for over 300 years and concerns the maximum number of non-overlapping spheres that touch a common unit sphere."

• 49:15Speaker 01

  So anytime you're advancing the kissing number problem, I'm hyped.

• 49:20Alex Kantrowitz?

  I'm all about it.

• 49:21Speaker 01

  I'm all about it. 300 years, we've been trying to solve the kissing number problem. And the alpha evolved just advancing. I think, I mean, you're right. That like the way we actually train these models and the architecture, rather than just raw compute. I do think we should see more innovation advancement there. And I think like, maybe that gets us to where, and maybe it just makes these things a lot more efficient. Not just powerful, but I, I think it's, I think it's an interesting thing around the architecture and like, these kind of other very unique innovations about how we approach it, but models are good enough. I'm sticking with it.

• 50:05Alex Kantrowitz?

  Keep, keep it up. We'll see, we'll see what happens over the next.

• 50:07Speaker 01

  GPT five is going to drop like this Sunday.

• 50:12Alex Kantrowitz?

  Ladies and gentlemen, a new model. All right. So, so we started with the fact that even in their current state, these models are ingesting everything. Let's end with another story about how even in their current state, these models are ingesting everything. And that is perplexity partnering with PayPal for in chat shopping. So Ranjan, this is a story close to your heart. Why don't you tell us what happened?

• 50:35Speaker 01

  Yep. So perplexity announced a partnership with PayPal. We've talked about this a lot and perplexity has done a lot with shopping and they'll, you ask a question. They'll show you a bunch of potential results. Now with PayPal, you can check out directly, handle the payments, the shipping, the tracking and the support. I think this is a big deal because again, before you had to subscribe to perplexity pro, pay $20, add your credit card information there, the retailer itself had to have an agreement directly with perplexity. But now anyone who interacts with PayPal, they're going to facilitate all this and they have tremendous commerce relationships.

• 51:14Speaker 01

  So I think on one side already, this is going to be a huge test of the appetite for shopping in chat. And I think we're, we're going to see whether people really do it or not. You made a very convincing case a few weeks ago. Then it sold me on it 100% that people will readily do it. Um, but then another related announcement this week was Mastercard unveiled agent pay. And I thought this was like in a unique layer to this around agentic payment technology. First I was like, okay, whatever. It's like another ridiculous just headline. But then the idea was that there's Mastercard agentic tokens, which build upon proven tokenization capabilities.

• 51:56Speaker 01

  Basically passing a token through the entire payment flow to make it so it's authenticated through the whole thing. Like as agents talk to each other, your information passes securely. And it, it around shopping, any kind of online payments and commerce. I actually think this is going to get really, really important. Cause like identity, security, these are things that have been solved pretty well on an individual website. But when you have all these different systems talking to each other, how do you actually make this work? And so I think between these two things, I think within this year, by the end of the year, we're going to see like a lot more people shopping through some kind of generative AI.

• 52:40Alex Kantrowitz?

  I agree. So when are we going to see Alexa plus? Because it's been months now and it hasn't been.

• 52:45Speaker 01

  I bought an Echo Show five after I listened to Alex's episode.

• 52:48Alex Kantrowitz?

  I know.

• 52:49Speaker 01

  I was all fired up. It, I like it. I like.

• 52:52Alex Kantrowitz?

  We have listeners who've, who've listened to the Amazon executives who are wondering when they can use theirs.

• 52:57Speaker 01

  It's May 16th.

• 52:59Alex Kantrowitz?

  It's May 16th. Do you know where your Alexa plus is? I, I don't know. And this thing better roll out soon. Not to mention, guess what's coming up in a couple of weeks? WWDC.

• 53:09Speaker 01

  Oh.

• 53:10Alex Kantrowitz?

  Where we'll hear the latest from Apple.

• 53:12Speaker 01

  Foldable phone.

• 53:14Alex Kantrowitz?

  Are we going to talk about Siri and foldable phones for the next couple of weeks? You better believe it. So.

• 53:18Speaker 01

  They take Siri off. There's no generative AI and they just give us a foldable phone. I'm fine with that.

• 53:25Alex Kantrowitz?

  Ron John's suggestion that Tim Cook shoot Siri on stage is now the thing of legends here on Big Technology Podcast. So maybe that, maybe we'll see it. I mean, Tim Cook, man, he got called out for Trump for not being in Saudi Arabia. Got called out by Trump for, uh, moving his manufacturing to India. All he did was, you know, give him a million dollars for his inauguration fund and, uh, he's been treated very poorly.

• 53:55Speaker 01

  I think Tim's doing okay.

• 53:57Alex Kantrowitz?

  He'll be okay. But he did get the exception for for the iPhone and the tariffs, which now may or may not be rolling back. So, folks, we are in the thick of it. We got Google's Developer Conference coming up on Tuesday. We got WWDC coming up. A couple of weeks after that, I'll be in the Bay Area for both. Fingers crossed I get into WWDC this year. It's always, you know, kind of a game day decision for them, I think. And then, of course, we'll see what's going on with Alexa Plus. So, as we say, this stuff is eating the Internet and tune into Big Technology Podcast for to hear where it's going.

• 54:32Speaker 01

  Before the web dies.

• 54:33Alex Kantrowitz?

  Before the web dies. Rajan, great to see you.

• 54:35Speaker 01

  See you next week.

• 54:37Alex Kantrowitz?

  Hi, everybody. Thanks so much for listening. Again, next week on Wednesday, Demis Hassabis is going to be on the show live from Google I/O. Very excited for that. And we hope to see you then. We'll see you next time on Big Technology Podcast.

• 54:51Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPAC.com, not authorized by any candidate or candidate's committee.