Yep, We're At Peak AI

In this episode, Ed Zitron discusses big tech's discovery of the diminishing returns in training generative AI models - and how we may have finally have reached peak AI. --- LINKS: https://www.tinyurl.com/betterofflinelinks Newsletter: https://www.wheresyoured.at/ Reddit: https://www.reddit.com/r/BetterOffline/ Discord: chat.wheresyoured.at https://discord.com/invite/QUUQUP9szv Ed's Socials: https://twitter.com/edzitron https://www.instagram.com/edzitron https://bsky.app/profile/edzitron.com https://www.threads.net/@edzitron See omnystudio.com/listener https://omnystudio.com/listener for privacy information.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This is an iHeart Podcast. Guaranteed Human.

• 0:04Speaker 01

  OnDeck is built to back small businesses like yours. Whether you're buying equipment, expanding your team, or bridging cash flow gaps, OnDeck's loans up to $400,000 help make it happen fast. Rated A+ by the Better Business Bureau and earning thousands of five-star TrustPilot reviews, OnDeck delivers funding you can count on. Apply in minutes at ondeck.com. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank. OnDeck does not lend in North Dakota. All loans and amounts subject to lender approval.

• 0:34Speaker 02

  Empower is all about helping you invest well so you can go out and live a little. And their Free Empower Personal Dashboard shows you your full financial life in one place. You'll be able to see your net worth, monitor budgets, analyze your portfolio, set retirement goals, and more. Whether you're saving for something big, want to treat yourself, friends, or family, or meet long-term goals, Empower helps you see where you stand. You've worked hard for your money, and Empower can help you make the most of it. Download the Empower Personal Dashboard or visit empower.com. Not an Empower client paid or sponsored.

• 1:07Speaker 03

  Call Zone Media.

• 1:10Ed Zitron?

  Hello and welcome to Better Offline. I'm your surly yet lovable host, Ed Zitron.

• 1:26Ed Zitron?

  Today, I'm going to kick off by reading something I wrote in March 2024 and talked about in the episode PKI. What if what we're seeing today isn't a glimpse of the future, but the new terms of the present? What if artificial intelligence isn't actually capable of doing much more than what we're seeing today? And what if there's no clear timeline when it will be able to do more? What if this entire hype cycle has been built on hot air, gooseballed by compliant media, ready and willing to take career embellishes at their word? Reading that back, well, I think I might have been right. And that's kind of what I'm going to get at today. I don't want to scream mustard. I'm not going to get smug about it. But this is what we're getting into today and in the next episode that will come out on Friday. Now, I'll be linking to some articles, so check the episode notes if you want to read them.

• 2:10Ed Zitron?

  But I'm going to get a lot into the spoken word.

• 2:14Ed Zitron?

  So I warned you in February that Generative AI has no killer apps and had no way of justifying its valuations. I also warned you in March that Generative AI had already peaked and I pleaded with the tech industry in April to consider an eventuality where the jump between GPT-4, which is the most current model, or GPT-40, to GPT-5 was not significant, in part due to a lack of training data, one of the more obvious things. I shared more concerns in July that the transformer-based architecture underpinning Generative AI, things like ChatGPT, was a dead end, and that there were really not many ways we'd progress past the products we'd already seen back then, in part due to the limits of training data like I mentioned, and the limits of the models that use said training data. In August, I summarized the pale horses of the AI

• 2:57Ed Zitron?

  apocalypse, events, many that have now come to pass, I'm afraid, that would signify the end, well, being nigh, though it's not quite here yet and it's not obvious when it will be, but this can't last forever. But I also added that GPT-5 would not change the game enough to matter, let alone add a new architecture to build future and more capable models or products of any kind. Now, throughout the things I've written and the things I've spoken, I've repeatedly made the point that, separate to any core value proposition, training data drought, or unsustainable economics that I've gone over quite a lot, Generative AI is a dead end due to the limitations of a probabilistic model that hallucinates. Now, just to be clear with what that means, it's guessing what the next thing might be, and it's quite good at it, but quite good is actually

• 3:40Ed Zitron?

  kind of shit. And hallucinations of, of course, are where they authoritatively state things that aren't true, like when ChatGPT tells you something like, I don't know, there are two Rs in strawberry. The hallucination problem is one that is nowhere closer to being solved. You may remember a few months ago when you had every tech executive, you had Tim Cook saying it's Sachin Adella, Sundar Pichai, "We'll deal with the hallucination problem. It'll be all right." But I want to be clear, they have not solved it, they have not really mitigated it, and there's no fixing it, at least with the current technology. It's not going anywhere, and it makes all of this stuff kind of a non-starter for many business tasks. I have since March expressed great dismay about the credulousness of the media about

• 4:23Ed Zitron?

  this and their weird acceptance of this inevitable way in which generative AI will change society, despite the fact there's not really a meaningful product that might justify any of this bullshit, this environmentally destructive nonsense, led by a company that burns more than $5 billion a year and big tech firms that are spending $200 billion on data centers for products that people don't want or even potentially use.

• 4:47Ed Zitron?

  And you're gonna need context for everything I'm saying today, so it's worth going over how these models work and how they're trained. And I must be clear, the reason I'm repeating myself on so many levels here, that it's it's just really important for you to know how obvious the problems of generative AI have been since the beginning. It's really important. Let's go over how they work real quick. A transformer-based generative AI model such as GPT, which is the technology behind ChatGPT, generates answers using inference, which means it draws conclusions based off of its training, which requires feeding it masses of training data, mostly text and images scraped from the internet. And both of these processes require you to use high-end GPUs, graphics processing units,

• 5:29Ed Zitron?

  and lots of them, tens, hundreds of thousands of them, well, over 100,000. I'll get to that next episode. Now, the theory was, and might still be, that the more training data and compute you throw at these models, the better they get. And I've hypothesized for a while that we'd have diminishing returns, both from running out of training data and based on the limitations of transformer-based models. And wouldn't you know it, I was bloody right. I'm one, I'm not gonna do many of these, but this one really, this one I'm right on. A few weeks ago, Bloomberg reported that OpenAI, Google, and Anthropic are struggling to build more advanced AI, and that OpenAI's Orion model, otherwise known as GPT-5, did not hit the company's desired performance, and that, and I quote again, "Orion is so

• 6:12Ed Zitron?

  far not considered to be as big a step up as it was from GPT-3.5 to GPT-4, its current model." You will be shocked to hear that the reason is that it's become increasingly difficult to find new untapped sources of high-quality, human-made training data that can be used to build more advanced AI systems. Something that I said in March, I said it would happen in March, I'm pissed off that people said I was a pessimist. Well, who's a pessimist now? Me, I guess? I don't, I don't know. But I, they also added one other thing, which is that they believe, and I quote, that the AGI bubble is bursting a little bit, which is something I said in July. AGI isn't coming out of this shit. Let's just be honest. And I also want to stop and stare really hard at one particular point, and I quote again from Bloomberg.

• 6:55Ed Zitron?

  "These issues challenge the gospel that has taken hold in Silicon Valley in recent years, particularly since OpenAI released ChatGPT two years ago. Much of the tech industry is bet on so-called scaling laws that say more computing power, data, and larger models will inevitably pave the way for greater leaps forward in the power of AI. The only people taking this as gospel would be members of the media unwilling to ask the tough questions, and AI founders that don't know what the fuck they're talking about, or that intend to mislead you. Generative AI's products have effectively been trapped in amber for over a year. It's been blatantly obvious if you fucking use them. I'm pissed off, I shouldn't swear so much. There have been no meaningful industry-defining products out of this." Because, and I quote Darren S. Moglu, the economist at MIT, back in May, "More powerful

• 7:40Ed Zitron?

  models do not unlock new features or really change the experience, nor what you can build with transformer-based models is really a worthwhile product. Or, put another way, a slightly better white elephant is still a white elephant." Despite the billions of dollars burned and thousands of glossy headlines, it's difficult to point to any truly important generative AI product. Even Apple Intelligence, the only thing that Apple really had to add to the latest iPhone, it sucks. It's not useful. I, I can make a special emoji now. I now get summaries of my texts that are completely or vaguely incorrect, or just summarize a giant meaningful paragraph into a blob of a sentence. It's so stupid. And just as a side question,

• 8:25Ed Zitron?

  What the hell is Apple gonna put in the next iPhone? I buy one of these every year, I'm a little pig, oink, oink, oink. But still, I don't even know why I'd upgrade again. The camera is already about as good as it's gonna get. Anyway, there are people that use ChatGPT, 200 million of them a week allegedly, losing the company money with every prompt by the way. But there's little to suggest that there's widespread adoption of actual generative AI software. The information reported in September that between 0.1% and 1% of the 440 million of Microsoft's business customers were willing to pay for its AI-powered copilot. And in late October, Microsoft claimed that it was on pace to make AI a $10 billion a year

• 9:05Ed Zitron?

  business, which sounds really good, until you think about it for roughly 10 seconds. First of all, Microsoft does not have an AI business unit, which means that this annual $10 billion, or two and a half billion a quarter, revenue figure is split across providing cloud compute services on Azure, selling Copilot to dumb people with Microsoft 365 subscriptions, selling GitHub Copilot, and basically anything else with AI on it. Microsoft is cherry picking a number based on non-specific criteria and claiming it's a big deal when it's actually pretty pathetic considering that Microsoft's capital expenditures will likely hit over $60 billion in 2024, with no sign they're gonna slow down. Note that sticky word "revenue", not "profit",

• 9:49Ed Zitron?

  those are two very different things. How much is Microsoft spending to make $10 billion a year? OpenAI currently spends $2.35 to make a dollar, and Microsoft CFO Amy Hood said that OpenAI would cut into Microsoft's profits in their last earnings call, losing it a remarkable $1.5 billion, mainly because of the expected loss from a company that has only ever lost money. Now, a year ago, in October 2023, The Wall Street Journal reported that Microsoft was losing an average of $20 per user per month on GitHub Copilot, a product with over a million users. If this is true, by the way, this suggests losses of at least 200 million a year. They have 1.8 million users allegedly, this is based on documents I've reviewed.

• 10:32Ed Zitron?

  It's not great either way. That is, $200 million is a lot of money to lose. I would personally like to make $200 million rather than lose it. Don't ask me that, I don't run Microsoft. Now, Microsoft has still yet to break out exactly how much generative AI is increasing revenue in the specific business units they have. Generally, if a company is doing well at something, they take great pains to make that clear. Instead, Microsoft chose in August to revamp its reporting structure to give better visibility into cloud consumption revenue, which is something you do if you say, anticipate you're going to have your worst day of trading in years after your next earnings, as Microsoft did in October. It's all very good, it's all going well. Now, it must be clear that every single one of these investments and products has been

• 11:14Ed Zitron?

  hyped with the whisper that they would get exponentially better over time, and that eventually the $200 billion in capital expenditures would spit out this remarkable productivity improvement, this crazy new product that would change our lives, fascinating new things that consumers and enterprise would buy in droves and talk about how much they loved. Instead, big tech has found itself peddling increasingly more expensive iterations of near identical large language models and shitty products attached to them, a direct result of all of them having to use the same training data, which they're now running out of. But if you're running out of stuff and you can't find stuff to buy, I really recommend the following advertisement. I'm sure it will totally gel with my beliefs, the things I'm talking about right now, won't be embarrassing at all.

• 12:04Speaker 01

  Cash flow crunch? OnDeck's small business line of credit gives your business immediate access to funds up to $200,000 right when you need it. Cover seasonal dips, manage payroll, restock inventory, or tackle unexpected expenses without missing a beat. With flexible draws, transparent pricing, and control over repayment, get funded quickly and confidently. Apply today at ondeck.com. Funds could be available as soon as tomorrow. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank. OnDeck does not lend in North Dakota. All loans and amounts subject to lender approval.

• 12:34Speaker 02

  Empower is all about helping you invest well so you can go out and live a little. And their free Empower Personal Dashboard shows you your full financial life in one place. You'll be able to see your net worth, monitor budgets, analyze your portfolio, set retirement goals, and more. Whether you're saving for something big, want to treat yourself, friends, or family, or meet long-term goals, Empower helps you see where you stand. You've worked hard for your money, and Empower can help you make the most of it. Download the Empower Personal Dashboard or visit empower.com. Not an Empower client paid or sponsored. Sponsored by GSK, the makers of Extenture, Depamocumab ULAA. If your asthma keeps interrupting, it may be time to try something else.

• 13:16Speaker 02

  Extenture is an ultra-long-acting prescription maintenance treatment for severe asthma, eosinophilic type, ages 12 and older. It isn't for sudden breathing problems. Extenture is the first and only add-on treatment that can help deliver protection from asthma attacks with two doses a year. Just one dose every six months. That's a lot of time for everything in between. Serious allergic reactions can occur. Get help right away for swelling of face, mouth, tongue, or trouble breathing. Don't change or stop medicines, including steroids, unless told by your doctor. Report if you have a parasitic infection. Common cold, sinus infection, inflammation in the nose, flu or flulike symptoms, joint pain or sore throat may occur. Carry on with fewer attacks and ask your doctor about Extenture.

• 14:00Speaker 02

  Learn more at extenture.com or call 1-833-393-3678.

• 14:08Speaker 05

  The world is transforming faster than ever, and standing still isn't an option. At Oppenheimer, we're working at the forefront of the innovation economy to invest where progress begins, finding opportunities that build and protect wealth for individuals and institutions that want a seat at the edge of tomorrow.

• 14:28Speaker 05

  Put the power of Oppenheimer thinking to work for you. Wealth management, capital markets, investment banking.

• 14:39Ed Zitron?

  And we're back. Now, there's another assumption that people have about these so-called scaling laws. That's been that by simply building bigger data centers with even bigger, more powerful GPUs, the expensive power-hungry graphics processing units that use to both train and run these models, and throwing as much training data at them as possible, they would simply start doing new things. They'd have new capabilities despite there being little proof that they would do so in any way, shape or form. Microsoft, Meta, Amazon and Google have all burned billions on the assumption that doing so would create something, you know, a thing, a good thing, like a human-level artificial general intelligence, or a product that made more money than it cost, that people liked.

• 15:24Ed Zitron?

  It's become kind of obvious that that isn't going to happen. As we speak, members of the media who should know better are already desperately trying to prove that this is not a problem. The Information, in a similar story to Bloomberg's, attempted to put lipstick on the pig of generative AI, framing the lack of meaningful progress with GPT-5 as "fine" because OpenAI can now combine its GPT-5 model with its O1 reasoning model, which is the one that can't count the number of R's in the strawberry, by the way, which will then do something. Something, something good, something's going to happen. Like, Sam Altman said it could write a lot more very difficult code. You know, Sam Altman, the career liar who intimated the GPT-5 may function like a virtual brain in May.

• 16:09Ed Zitron?

  Like, these people are liars. They're liars. They're lying to you. They were lying then, they're lying now. Now, I couldn't possibly leave out chief fellow cheerleader Casey Newton, who wrote on Platformer a few weeks ago that diminishing returns in training models may not matter as much as you would guess, with his evidence being that Anthropic, who he also claims has not been prone to hyperbole, do not think the scaling laws are ending. Now, the original scaling laws paper, partly written by Dario Amadase of Anthropic, important to know. And to be clear, in a 14,000-word op-ed that Casey Newton for no reason wrote two pieces about, Anthropic CEO Dario, he said that, and I quote, "AI-accelerated neuroscience is likely to vastly improve treatments for, or even cure, most mental illness," which

• 16:52Ed Zitron?

  is the kind of hyperbole that should be, have you tarred and feathered and put in a jail. I'm not seriously saying you put him in jail, but why are we, why are we trusting these people? Why are we listening to them? Why are we treating them as if they're telling the truth, or even that they know what's going on? But let's summarize. The main technology behind the entire, and I say this in quotation marks by the way, artificial intelligence boom is generative AI, transformer-based models like OpenAI's GPT-4 and soon GPT-5. And said technology has peaked, with diminishing returns from the only ways of making them better, feeding them training data and throwing tons of compute at them, suggesting that we may have, as I said before, reached peak AI. Generative AI is incredibly unprofitable.

• 17:36Ed Zitron?

  OpenAI, the biggest player in the industry, is on course to lose more than $5 billion this year, with competitor Anthropic, which also makes its own transformer-based model, Claude, on course to lose more than $2.7 billion this year. They just raised another $4 billion. Every single big tech company has thrown billions of dollars, as much as $75 billion in Amazon's case in 2024 alone, of building the data centers and acquiring the GPUs to populate said data centers specifically, so they can train their models and other people's models, or serve customers that would integrate generative AI into their businesses, something that does not appear to be happening at scale. And these investments could theoretically be used for other products, but these data centers are heavily focused on generative AI. Business Insider reports that Microsoft intends to amass 1.8 million

• 18:19Ed Zitron?

  GPUs by the end of this year, costing it tens of billions of dollars. Worse still, many of these companies integrating generative AI do so by connecting to models made by either OpenAI or Anthropic, both of whom are running unprofitable businesses and likely charging nowhere near enough to cover their costs. As I've said before in my article, the subprime AI crisis, in the event that these companies start charging what they actually need to, their real costs, I hypothesize that it will multiply the cost to their customers to the point that they can't afford to run their businesses, or at the very least will have to remove or scale back generative AI functionality in their products. It's just, it's such a waste. The entire tech industry has become oriented around this dead-end technology that requires

• 19:01Ed Zitron?

  burning billions and billions of dollars to provide inessential products that cost them more money to serve than anybody ever would pay. Their big strategy has been to throw more money at the problem until one of these transformer-based models created something useful, despite the fact that every iteration of GPT and other models has been, well, iterative? And it's weird, you think at some point that goes, shit, do we actually have the ability to build products with this? What are the products? Maybe we should work out the products first before we throw all the CapEx at it. But wait, no, over yonder, I couldn't possibly not do this because the other big tech company

• 19:43Ed Zitron?

  that also has no ideas, they're doing this, and if I don't do this, my investors are going to be angry at me. And then what will I do? Oh, no. Oh, no. What could I possibly do if the investors? I don't fucking know. That's your problem. Why waste this much money? It's just, there's never been any proof, other than these benchmarks that are really easy to game and also only show just this vague power of these models. It's been obvious that GPT or other models wouldn't become conscious, that they're not going to do more than they do today, or three months ago, or even a year ago. Hesitate to give Gary Marcus credit, but in 2023, he was saying this, if not earlier. Many people have as well.

• 20:27Ed Zitron?

  And it's just really, really, really, really frustrating. Better Offline isn't even a year old, but when we put out our PKI episode, I got so much flak. I got so much shit for being a hater, that I didn't really understand things, that my fly was open in my Instagram picture, that I didn't get it, and that in mere months, I would be proven wrong. Well, here we are. How wrong am I now? What happens next exactly? Where do all these hundreds of billions of dollars go? What happens to OpenAI when it collapses? What does Microsoft do with all of these GPUs? Because you can't just move them into other shit, you know? From what I hear, they don't really have a plan. And that's the scariest thing. Because what happens to a stock market that's dependent on big tech companies for growth when the big tech companies can't work out a way to grow anymore?

• 21:11Ed Zitron?

  And in fact, their big path to trying to grow more was to burn a shit ton of money on things that people hate that destroy our environment.

• 21:20Ed Zitron?

  I know, I know I'm angry. I know, I should calm down. I should. But as I said in the Rott Society, this money could go elsewhere. More things could be done. It would enter a fallow period of tech. But we don't just have to burn all this money. We don't have to do that. Why not make the products you have already better? Because stapling generative AI on them, I think it makes them worse.

• 21:47Ed Zitron?

  But there are more problems ahead. There are problems around the infrastructure. And in the next episode, I'm gonna break down these worrying problems. And I'm gonna kind of tell you what happens next as best I can. I really appreciate your faith in me. And there are many people who also contacted me and said, no, you're bang on. Keep going. I'm glad they did. I'm very grateful for you, audience. I love you all, much like he said in the menu.

• 22:20Ed Zitron?

  Thank you for listening to Better Offline. The editor and composer of the Better Offline theme song is Matt Osowski. You can check out more of his music and audio projects at mattosowski.com. M A T T O S O W S K I dot com. You can email me at [email protected] or visit betteroffline.com to find more podcast links and of course my newsletter. I also really recommend you go to chat dot where's your ed dot at to visit the Discord and go to r slash betteroffline to check out our Reddit. Thank you so much for listening.

• 22:52Speaker 03

  Better Offline is a production of Cool Zone Media. For more from Cool Zone Media, visit our website, coolzonemedia.com or check us out on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

• 23:23Speaker 02

  Empower is all about helping you invest well so you can go out and live a little. And their Free Empower Personal Dashboard shows you your full financial life in one place. You'll be able to see your net worth, monitor budgets, analyze your portfolio, set retirement goals, and more. Whether you're saving for something big, want to treat yourself, friends, or family, or meet long-term goals, Empower helps you see where you stand. You've worked hard for your money, and Empower can help you make the most of it. Download the Empower Personal Dashboard or visit empower.com. Not an Empower client paid or sponsored.

• 23:56Speaker 06

  Wait, I came in for two things. How is this $47? All right, we're gonna need a plan here.

• 24:04Speaker 07

  Just start simple with Bank of America Advantage Safe Balance Banking. No overdraft item fees, no monthly maintenance fee if you're under 25.

• 24:12Speaker 06

  Oh, that's actually really simple.

• 24:15Speaker 07

  Safe Balance Banking, one less thing to figure out. Learn more at bofa.com/simplebanking. Terms and conditions apply. Bank of America, N.A., member FDIC. Cash flow crunch? OnDeck's small business line of credit gives your business immediate access to funds, up to $200,000, right when you need it. Cover seasonal dips, manage payroll, restock inventory, or tackle unexpected expenses without missing a beat. With flexible draws, transparent pricing, and control over repayment, get funded quickly and confidently. Apply today at ondeck.com. Funds could be available as soon as tomorrow. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank. OnDeck does not lend in North Dakota. All loans and amounts subject to lender approval.

• 24:55Speaker 02

  Sponsored by GSK, the makers of Extentur, Depamocumab ULAA. If your asthma keeps interrupting, it may be time to try something else. Extentur is an ultra-long-acting prescription maintenance treatment for severe asthma, eosinophilic type, ages 12 and older. It isn't for sudden breathing problems. Extentur is the first and only add-on treatment that can help deliver protection from asthma attacks with two doses a year. Just one dose every six months. That's a lot of time. ...of time for everything in between. Serious allergic reactions can occur. Get help right away for swelling of face, mouth, tongue, or trouble breathing. Don't change or stop medicines, including steroids, unless told by your doctor. Report if you have a parasitic infection.

• 25:38Speaker 02

  Common cold, sinus infection, inflammation in the nose, flu or flulike symptoms, joint pain, or sore throat may occur. Carry on with fewer attacks and ask your doctor about Extensura. Learn more at Extensura.com or call 1-833-393-3678.

• 25:55Speaker 03

  This is Iris Palmer from Against All Odds with Iris Palmer. You know it's the holidays when somebody in the family starts cooking before the sun comes up. The babusas are grilling in the pan, everybody's sneaking bites, and somehow there's always room for one more person at the table. And nothing goes better with that first holiday plate than a delicious Coca-Cola. That crisp, refreshing taste makes everything that much better. Because the best holiday traditions aren't just about what we're eating. They're about who we're sharing those moments with. So check that shopping list twice. Stock up on Coca-Cola for all your family fiestas y disfruta Coca-Cola.

• 26:32Speaker 00

  This is an iHeart Podcast. Guaranteed Human.