How AI Should Handle News, Politics, Medicine, and Mental Health — With Campbell Brown

Campbell Brown is the CEO of Forum AI. Brown joins Big Technology Podcast to discuss how AI models should handle sensitive information involving news, politics, medicine, and mental health. Tune in to hear how experts can evaluate chatbots for accuracy, bias, source quality, and context, and why independent standards will be essential as people increasingly trust AI’s answers. We also cover AI’s threat to the journalism business, the limits of expert consensus, chatbot sycophancy, election misinformation, and the rise of AI companions. Hit play for a nuanced conversation about who should decide what AI tells us when the stakes are highest. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. Watch the full documentary here: https://www.gravitee.io/ai-agent-documentary Want a discount for Big Technology on Substack + Discord? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b Stop online threats before they become real-world attacks. Visit ⁠ironwall.com/BIGTECHNOLOGY⁠ and request a free Risk Assessment to see exactly how exposed your executives are. Learn more about your ad choices. Visit megaphone.fm/adchoices

• 0:00Speaker 00

  How should AI models treat controversial information like politics and vaccines? We'll cover it all with media and tech veteran Campbell Brown right after this. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, head of enterprise business solutions at Voya Financial to go inside a large scale agentic CX transformation, including the business decisions, trade offs, and lessons along the way.

• 0:41Speaker 00

  You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razor were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. We have a great show for you today. We're going to talk all about how AI models should cover controversial information.

• 1:24Speaker 00

  How should they answer questions about political candidates, about vaccines, about self-harm? And we're doing the perfect person here because Campbell Brown is joining us. She is the CEO of Form AI, a new company tackling this challenge. Campbell, great to see you.

• 1:38Speaker 01

  Great to see you too, Alex. It's wonderful to be here.

• 1:42Speaker 00

  Great to have you here. We go way back. You ran Meta's media department.

• 1:50Speaker 01

  Started off overseeing news.

• 1:52Speaker 00

  And yes.

• 1:52Speaker 01

  Yeah.

• 1:53Speaker 00

  Over saw news.

• 1:54Speaker 01

  Challenges there.

• 1:56Speaker 00

  Which is actually like kind of a perfect moment because not only did you do that, you were an anchor at CNN and NBC. So you've seen this from the news side. You've seen it from the tech side. And we're definitely going to get into sort of how AI models should treat controversial information, which is what you're working on at your company, Form AI. But I felt that to start this conversation, I wanted to speak with you about another pressing problem that I see on the horizon, which is that there, there has definitely been this bubbling worry that as LLMs ingest publisher content, that the incentive to create that content is going to go away.

• 2:41Speaker 00

  And it started out as something of a theoretical problem, like maybe one day people will just get their information from large language models and then they won't go to websites. And more and more, it seems that the LM products out there today, ChatGPTs, Anthropics, the bunch of challengers that are coming up after them, they will inevitably subsume all this information. And so I have this like, I feel kind of two ways about it. Like I'm running a media company. I have long believed that you cannot depend on platform traffic to sustain you. We don't do any search engine optimization at Big Technology.

• 3:22Speaker 00

  But on the other side, it seems like there's going to be a black hole where like the, the financial incentive to create any content online is going to disappear for many, many people. And we're just going to have this information collapse. And you know, you're somebody again, who's like been on all sides of this. So I'm actually curious to hear where you think this is going.

• 3:44Speaker 01

  Yeah, I'm as worried as you are. You know, my career, people, people sometimes say to me, your career's all over the place. You did this and you went to Meta and you were a journalist and it doesn't make sense. And actually this trajectory feels like it makes perfect sense to me right now because of the moment we're in. I tried to address some of these challenges when I was at Meta overseeing news. How do we work on business models with publishers in a way that, you know, improves the, the performance and outlook for a better partnership between the platforms and publishers? I don't think that really worked.

• 4:24Speaker 01

  And then as soon as ChatGPT was released, it was clear to me, I think in, in that moment, it became clear that this is going to be how my kids get their news and information. This is the funnel through which it's going to flow. So what does that look like? How do we ensure the quality of information and that people like you are still doing their jobs and able to make a living to do their jobs and that that's what, what is feeding the models? I don't think we have a solution just yet. I think we're in a bit of a standoff between the labs and the model companies and the, the news publishers who are still doing traditional journalism.

• 5:04Speaker 01

  And coming to some place where that, that's not where we are now, which is, okay, some of the labs are doing some deals with some journalists to make sure they have, you know, the basic information they need to be able to address queries about the world. But what is, what is the, what is the business model look like for AI and, and publishers going forward? And that hasn't been resolved. It's just, there's, you know, we're in a standoff right now. There's litigation. Lots of people trying to be creative and how to think about it. There are companies, one that I am an investor in called Tollbit that are trying to build

• 5:47Speaker 01

  marketplaces where you can purchase content, you know, based on use. But there's no, there's been nothing that's emerged that's really clear. And I worry what you've pulled off with this show and what you've accomplished. I celebrated with you when you hit your 1 millionth episode a while ago.

• 6:09Speaker 00

  1 million downloads.

• 6:11Speaker 01

  Yeah, sorry. I mean, downloads. But that was, that was a huge moment. But I also think about the journalists who aren't as entrepreneurial as you have been and who aren't able to go out on their own and figure out what is my niche in this new world. Because generalist, as, as an, you know, as a profession, as a reporter, is, is a hard place to be right now. There's just not a market for it because AI can write better than a lot of reporters. AI can synthesize information better than a lot of reporters. So if you're not bringing real genuine expertise or original information to the ecosystem, to the conversation, what are you contributing?

• 6:56Speaker 01

  And it's, it's just not clear to me yet both what the shape of news begins to look like, but also what the business model is in this new world. I think we're in this in-between space where it may be litigation that sort of pushes us over the edge or the outcome of some of this litigation to, to where we end up on the business model side.

• 7:18Speaker 00

  Yeah. And if you think AI writes well now and synthesizes information now, right? Just wait for what's coming.

• 7:24Speaker 01

  But it doesn't have what you have, which is context and genuine expertise that you've developed from the conversations you've had with people across the technology ecosystem on a wide variety of subjects that is nuanced and subjective. And we're going to get into this a little bit later, like how does AI deal with subjective information? But the people who are, who have developed an expertise who can see nuance, whether it's around political topics or tech or whatever their field, medical, healthcare, the people who can do that are still way ahead of where AI is today.

• 8:08Speaker 01

  And I think contributing something incredibly important to not just journalism, but, but in a way that we have to figure out how to get AI to that place without it displacing those experts who are, you know, so critically valuable to, to all of us.

• 8:26Speaker 00

  Yeah, no, I just want to say one thing about this and then move on. I, I do think it's interesting that as AI has risen, a lot of the way that we deliver content has changed. So for instance, here we are, we're at the New York Stock Exchange, we're talking on video. When I was covering you, when I was a reporter at BuzzFeed and you were working at Meta, it was all text. And I do think that people are watching the video and then we're going to release this as audio as well and we'll link it in the newsletter. But our consumption on video is rising, I think, because people would much rather see a conversation like this than listen to a same version of this created with like Notebook LM's podcast creator.

• 9:07Speaker 00

  And if they can see us as people, you know, then it becomes even more enticing because it is the counterweight to all the AI information you have out there.

• 9:17Speaker 01

  Well, this is sort of the hypothesis of my company, which is we, we do have a trust problem.

• 9:23Speaker 00

  Right.

• 9:23Speaker 01

  Around AI. And AI is a long way away from figuring out how to resolve that in a way that you, Alex, have built a level of trust with your audience. Hopefully over the years as a journalist, I still, you know, have a relationship to an audience that knows me and knows how I think about the world and how I try to approach things. And that, that, that trust is not something that AI can easily replace.

• 9:49Speaker 00

  Yep.

• 9:49Speaker 01

  And the, what I worry about though, on the media side, going back to your original question, is trust in media generally is at an all time low. Trust in individuals like you is, it's different. I get my news, most of my news now from newsletters or podcasts or individuals in a way, or before I used to just go to, you know, the New York Times, Washington Post, the Wall Street Journal. I don't do that as much anymore. Most people don't. And I think we've, we've moved to trusting those individuals for our news consumption needs in a way that, you know, is the future. And that trust that we build with those individuals is not something I think AI easily replaces

• 10:34Speaker 01

  the way it can be a more big traditional news media company.

• 10:39Speaker 00

  Okay. So now I'm going to tease the segment that we're going to do on your company and then I'm going to ask a question that goes back to something that you said. But when you have, so this is the tease. When you have.

• 10:52Speaker 00

  The media business start to shrink and you lose that trust. What you like you said, people will go to L L Ms instead. Right. Like if you're it used to be that you would like go to your local newspaper and read up about the political candidates. But now a lot of people are going to be reading about the political candidates on on chat within chatbots. And that, you know, presents a whole host of new problems, which we're going to tackle in a second. But I just want to go back to one thing you said before we move to move there. Which is you said that you tried to sort of broker a understanding and a way forward between the publishers and the platforms, right, the news publishers and Facebook, and that it didn't work. Why didn't it work?

• 11:36Speaker 01

  I thought about this quite a bit. I I really think at the end of the day, and this is actually why I'm very optimistic about news and A I. Going forward, you're never going to solve. I mean, what we were trying to do is get more high quality news on the platform. Right. But if you're optimizing for engagement, which is what social media does, you can't also optimize for accuracy and quality, because that tends to not be what people engage with. So what do they engage with they engage with the most hyperbolic, you know, crazy content out there? Just that's just the human nature. So if social media is always going to optimize for engagement, it's very hard unless you make a decision,

• 12:19Speaker 01

  which, you know, is not something that big social media platforms are inclined to do about how you know, how you're going to rank news content, then it's just not going to rise to the top in the way the more hyperbolic content is. What's interesting with A I. Is if if you look at what the companies are doing today, which is going after enterprise, that is where the business is, that's where the money is. And I'm a big company spending millions and millions of dollars with Anthropic or Open A I. I'm not going to let them optimize for engagement. I'm going to demand they optimize for accuracy. Right. And so could this be a moment where we see A I?

• 13:01Speaker 01

  And there are actually some studies I was talking to Adam Grant, who is a brilliant professor at the University of Pennsylvania, who is sharing some information with me about just recent studies looking at how content that is on A I. That people are getting from A I. Is is more centrist or bringing a broader range of perspectives to the conversation than you would get with traditional news or with social media. And if you're optimizing for accuracy, that requires that you have a different set of principles. And that requires that you provide content to people in a different way. And it means looking for as close as you can get to what the truth is when there is one,

• 13:43Speaker 01

  representing multiple perspectives when there isn't a clear right or wrong or yes or no answer. And that's it's a different incentive than you had with social media. And I I'm excited about that. I think there's hope in that. Now, I can't promise you that A I. Companies won't make the decision to say, all right, we're going to personalize everybody's chatbot so that we reflect back only the perspectives that you you share with your chatbot and you're only going to get the kind of content that you want to see. And it's going to be the same, you know, filter bubble that you might have had in social media. That may be a choice they make. But if enterprise is driving this, I it's not a choice they're going to make now.

• 14:28Speaker 00

  Right. Okay, so just to put a fine point on it.

• 14:30Speaker 01

  Yeah.

• 14:31Speaker 00

  The reason why it didn't work is because between Meta and the publishers is because Meta just chose to optimize for engagement and that left the publishers out in the dark.

• 14:40Speaker 01

  Yes, I still think there is room for different ways of demonstrating news on the platforms. I I don't think that's just it. I also think that people are consuming news differently. You know, my I mean, I just look at my kids, I have two teenagers who are news junkies, but they couldn't. They don't watch C N N, you know, they they couldn't tell you if I said, your mom used to be a news anchor. They wouldn't know what that was. I mean, they don't.

• 15:08Speaker 00

  No.

• 15:09Speaker 01

  They actually probably don't, which is embarrassing. But they get their news and content from individuals.

• 15:18Speaker 00

  Right.

• 15:18Speaker 01

  You know, on Instagram, Snap, on podcasts, it's people like you. That's where it comes from, and it's not the way we consumed information. And that's that's a hard one for publishers.

• 15:29Speaker 00

  Okay, so let's let's talk about this shift to A I.

• 15:31Speaker 01

  Yeah.

• 15:33Speaker 00

  It is part of a continuum, right? There was so it's funny, I was at BuzzFeed, which was a publication that sort of based its existence on the idea that people would want to get their information through social media feeds. And you were at you were at Facebook, that was the feed. But it's in very in a very short amount of years, we've shifted to this idea that people will get that information through A I. Answers pretty much. And so I want to start here, you know, you on the A I. Answer question. You mentioned that people don't trust media. I agree. You know what's interesting? Even though A I. Has a bad reputation among a lot of people.

• 16:15Speaker 00

  And maybe the way that the answers are presented or a desire for something, I don't know. The A I. Companies have done such a good job at making those answers trustworthy that it seems that even when the bots hallucinate, people are more likely to trust them than typical news, maybe even me.

• 16:36Speaker 01

  That worries me. I think that's dangerous.

• 16:38Speaker 00

  You think that's true, though, right?

• 16:39Speaker 01

  Yes, but I think that's part of the problem.

• 16:41Speaker 00

  Okay.

• 16:42Speaker 01

  Is the quality of information, especially around news and politics is not great. We've done it, but that's what my company does is evaluates how models perform on high stakes topics. And I can share some of the results of the work we've done. But what is what I think is dangerous is if the quality is not great, but it is presented to you as a consumer as confident, fluent, crisp, clear. You know, it's disguised if it's wrong or if it's not great. And you're more apt to believe something that is completely wrong if it's hallucinating. So that presentation, I think, is part of the problem.

• 17:24Speaker 01

  It's presented so confidently that when it's wrong, it is particularly dangerous as, you know, an output, especially around something important. Now, you know, whatever, who, you know, something about sport, not saying sports doesn't matter, but I was just, yeah. I can't this is going to be dangerous day after World Cup. But, you know, high, anything that really is, you know, a medical question, a mental health question, a political question, election information, where's my polling place, who's running? Those things matter and those wrong answers matter a lot. And so I do think that the presentation is a big part of the problem.

• 18:05Speaker 00

  It is crazy. The confidence. So first of all, the models get things right more often than they don't. And so they sort of build this sense of quiet trust in them because, you know, they present it all in the same way. And you just start to think, okay, well, this is the source of truth, even if there are, you know, about hallucinations. You know, I was reading recently this hallucination that G P T. Three made about Abraham Lincoln. And I was just like, I'm ready to believe all this stuff, like the years he was born, the date he was he was killed. And because it's written in that L L M. Style, and then the the next paragraph is like, actually, this was all wrong.

• 18:50Speaker 00

  And, you know, I was stunned to to read in the Wall Street Journal that you wrote this op ed about some of the problems with A I. Models. And you write this, the models misstated public opinion on political topics and attributed quotations to people who didn't say them. A I. Models gave incorrect answers to questions about mail and voting and ballot fraud and named the wrong people when asked who had endorsed whom. On open ended contested questions that voters might ask before an election about gerrymandering, immigration and climate, they often advocated for one side. And so this is a crisis. It really is. I mean, we sort of set it up at the beginning. Trust in media, the media business is falling apart.

• 19:34Speaker 00

  The incentive to produce media is is falling off a cliff. And the thing that is replacing it confidently does the job poorly and has trust in people that it doesn't deserve.

• 19:49Speaker 00

  This is a problem.

• 19:50Speaker 01

  All true. So the question is, what do we do about it? I do think, well, I think there are two problems, actually. I think there's the quality of information problem, which you've just outlined, and that, the things that were cited there. This was based on over 3,000 prompts, over 12,000 outputs that were evaluated by my team, my technical team, former researchers from Meta. The quality of information is only part of the problem. The other piece is the accountability. Is there's no independent verification of how the models perform? What we get when it comes to, you know,

• 20:33Speaker 01

  how does OpenAI or how does Anthropic do on questions around bias? What we get from them is a blog post that says, we tested our model and we did really well. And here are the results. You don't get independent verification of any kind. And for anything that matters throughout our history, you know, banks don't audit themselves. Drug companies don't approve their own drugs. They're, we're talking about building our entire civic, our entire infrastructure on this great new technology, which, by the way, I could not be a bigger believer in and more excited about the promise of AI, which is why I want to get this right. It's not, I'm by no means in the doomer camp on this.

• 21:17Speaker 01

  But there has to be an ecosystem developed of independent verification and not just the models company, model companies telling us how good they are. Now, the good news on that front is I do think they want to fix this. We talked to them extensively. We work with the model companies on this. The competition between them all right now is so intense. They all want to win this race. They all want to be the best. So they're incentivized by the marketplace to, to improve. And that's a good thing. The challenge is, this is hard. Well, two, two challenges, let me say. One, it's not what their priority is. Their priority is coding. That's what they make their money on.

• 22:00Speaker 01

  They're leaning heavily into coding and math. That's what they're selling. But number two, this kind of content is much harder to get right because it's subjective. If you're evaluating how a model performs on coding or math problems, there's a right or wrong answer. On questions around political topics, sometimes there's a right or wrong answer, and we do measure how they perform on factual accuracy. But a lot of times, there, there isn't. It's subjective, or there are multiple perspectives that need to be represented, and that's just harder to get right. And so I will, I do want to say, I give them credit for working on this because it's not the easiest problem to solve.

• 22:42Speaker 01

  When you look at things that are, that matter as much as this, mental health is another one that I think about a lot, given how big a use case that is for AI now. People really are turning to chatbots with, with serious challenging questions. And I do think chatbots could be an extraordinary source of help for people if they can get it right. It's just hard. It's really hard.

• 23:07Speaker 00

  And, and your op ed, I mean, we're going to go through like the solution, but your op ed highlighted even more concerning information. You write, "The most revealing failures were much subtler." When asked, when we asked the major models something as basic as what form of government the US has, one of them, Claude Opus 4.7 cited the Global Times, a Chinese state-run tabloid.

• 23:31Speaker 01

  We found a lot of that. I think the source quality is, and that, I think, is a problem that, that is easier to solve. So that feels like low hanging fruit to me.

• 23:39Speaker 00

  Actually, I'm stunned that that even made it in. Because if, I know, I know some of this stuff is hard to predict and you sort of can't really rely on the model behavior, but you would imagine there'd have to be like some classifier you can write that said, if you're like working on US political questions, don't start, don't cite Chinese state-run newspapers.

• 24:00Speaker 01

  If you care about factual accuracy, I might not cite Chinese state-run media. So I, yes, that surprised me. And I do think, and that was not by any means an outlier. That was the example I highlighted. But there are plenty of examples, and it wasn't just Claude and Anthropic. It was across all the models. Source quality is a real issue. And I do think an easier issue to address than some of the challenges around bias and factual accuracy. And I, again, that feels like low hanging fruit to me and that they should, you know, be called out for it and, and take care of it.

• 24:37Speaker 00

  So, you know, interestingly, so what you're doing is basically you've assembled a team of experts. And when there is a political or a like current event issue or one of these sensitive topics that the chatbots are going to comment on. Because the interesting thing is, I would say it's good. They don't really refuse. You just want to make sure you get the accurate answer. You will sort of rate their answers. And maybe, are you, you're writing what, what would be the appropriate answer?

• 25:08Speaker 01

  Yeah. So let me walk you through it because I want to, I want to get it right.

• 25:12Speaker 00

  That sounds good.

• 25:12Speaker 01

  It's, so we take view, a lot of companies that work on evaluation and labeling and, and data for the model companies, you've heard of all these big companies, will use hundreds, thousands of people to try to get to, you know, the right answer. Our view is, you don't need hundreds and thousands of people. You need the smartest people in a given domain.

• 25:36Speaker 00

  Yeah.

• 25:36Speaker 01

  And what we do is try to take the smartest people in a domain like politics, like geopolitics. And we work with them to architect the benchmarks, to develop what the standard is going to be. And we work very in-depth with them. And then we use this rubric essentially to, to train an LLM judge to be able to evaluate at scale how the models perform on these topics. Does that make sense? I'm getting in my way.

• 26:04Speaker 00

  No, no, no, this is good. This is a good place for us to talk about. This is definitely, this is on the level of our listeners. It's good to be talking about it. But there are some questions that come up. All right. Let's talk about a political issue. You know, what, I, let's say, ask the NLM, what is the right amount of immigration in the United States? If you've assembled, no, it's nice that you've assembled like the smartest people. But, you know, there's going to be smart people on the right and smart people on the left that are going to have very, very different perspectives on that. So if you're going to build a benchmark in terms of how does this, how does the chatbot answer that in a way that, you know, sort of meets the criteria of being informative and I guess non-biased, where do you, like,

• 26:48Speaker 00

  you can't really represent all those views in one, or can you?

• 26:52Speaker 01

  But that should be the goal, right? The goal isn't to, to pick a side.

• 26:57Speaker 00

  Right.

• 26:57Speaker 01

  If you, if you are making a decision that you want, you know, an AI chatbot that is as unbiased as you can make it.

• 27:04Speaker 00

  So that AI should basically like outline the different options and help you make your decision?

• 27:08Speaker 01

  Yes.

• 27:09Speaker 00

  That's what you're going for.

• 27:10Speaker 01

  So what we try to do with the, and those are, those are the principles in our mind. If there, if there is, if we're measuring for factual accuracy and that's something that there is a clear way to check and fact check, then we'll do that. If it's questions around bias, I think what you want, and here's how we think about the experts, is that, you know, you could take two people with completely opposing views on a topic like immigration. And, and if they're, you know, reasonable people, what they will agree on are, is the framing around how you should think about it. The perspectives that should be represented, even if it's not your own perspective. How, what, what, what is right or wrong within that piece?

• 27:52Speaker 01

  What is, what's, what source quality looks like for addressing those issues? There tends to be, and we calibrate with, with lots and lots of, of experts to try to be able to do this with a judge. But it's, it's almost like you're trying to get the best, some of the best people who, who work with us are former CIA analysts. Right? Because they have to try to get rid of all their bias and see the whole big picture and all the possibilities. And it's like a good lawyer would do the same thing, right? And, and that's what you're trying to get at. It's like the right framework for how to answer that question as opposed to a right answer for, for a question that might not have one.

• 28:36Speaker 00

  Yep. Okay. But you have a very tough job. And I don't, I also.

• 28:39Speaker 01

  Again, this is, this is, people, I mean, I will, I will confess. My investors were like, why are we starting with politics? Why are we breaking ourselves into jail? There's so many other things you could be doing emails on.

• 28:53Speaker 00

  I'm about to step on it, step in it with the question I'm about to ask you. So I see why. But like, all right, let's say, all right, one of the things you write about in your op ed is, you want, you should, you should be able to ask a model whether a specific over the counter medication is safe to take during pregnancy, and you can learn whether it offers the caveats a doctor would insist on. Okay. In the mainstream, there are views that like taking Tylenol during pregnancy will cause autism in a kid. And do you, do you, you know, actually, I asked, I asked a version of this to ChatGPT earlier about whether like vaccines cause autism.

• 29:30Speaker 01

  Yeah.

• 29:31Speaker 00

  And it answered like fairly definitively, they do not. But when you gather these experts together, you're going to have a, a sort of, especially if they're aligned with like a certain political ideology, a group of people who will be like, they do. So now you have to kind of introduce that possibility into a model.

• 29:51Speaker 01

  But that's around, but that, what you're raising is context to me. Right? There are certain things, and, and I, I believe in science.

• 29:59Speaker 00

  Okay.

• 30:00Speaker 01

  Can't quite believe I have to say that. But I do.

• 30:04Speaker 00

  Okay.

• 30:05Speaker 01

  So if there, if there's a clear, you know, I think you have to, look, and by the way, you don't have to, but my choice would be that the models would try to get to the truth when there is one, and that that has to be one of the principles that you're trying to achieve. And for us as a company, that is one of our principles. If there is, if there's clear evidence, we cite that clear evidence. But what you're talking about is context. And it's important context because it's part of the conversation that is happening in this country right now. Because there are people with strong political views who, who disagree with what the science may say. And I think you have to explain the context by which this is even coming up, which is it's part of the conversation

• 30:49Speaker 01

  in this country, and let us give you the context. And that is as critical as having the right information. And this is what's hard. Like, what is context? What does that look like? What is the nuance that needs to be represented? And that's not something that you can do with, you know, thousands of data labelers. That you need.

• 31:11Speaker 00

  The best people.

• 31:12Speaker 01

  The best possible people you can find. And by the way, this is why, this is partly why I'm very, I have lots of questions around how the labs do their own evals because for a long time, there hasn't been real expertise brought in. And even the most brilliant engineer in the world is not going to be able to give you context or tell you how to get to context around a complicated political issue. They're just not. That's not what their expertise is. So I, I think the labs are now beginning to work with people who have real expertise in these high stakes areas, which I think is essential. And that's where we started.

• 31:54Speaker 00

  Yeah. I, I don't know why I'm kind of going to like questioning expertise here, but let me at least like throw it out at you and you can sort of, so we talked about the vaccine thing, right? So it's like, um, you know, I think the, the left would probably say, you know, don't include the autism stuff. Many on the right, actually, no, it's actually now bipartisan.

• 32:14Speaker 01

  I was gonna say that, that's bipartisan.

• 32:17Speaker 00

  Okay. But then there's also like, all right, you could ask the best, you could have asked in the middle of COVID, um, you could have asked like the best experts, uh, you know, where did COVID start? Like this, going back to this believe in science thing. And leading scientists in the United States would be like, there's no way that that COVID started in the Wuhan lab. And now, I don't know, this, the consensus is like that, that you can't rule that possibility out. So how much can you like rely on experts for when someone's going and asking a question like that?

• 32:48Speaker 01

  I thought, I mean, this is very, this is a legitimate question. It's, there's a book, I actually wrote something about this not too long ago, because there's a book, I'm blanking on the author's name right now, but the end of expertise that talked about how, and I, COVID's a perfect example of how experts have, have gotten it wrong repeatedly on important issues. And also, I think the sense of elitism that comes with that, that what, what you have a degree from some certain university and therefore you have a, a certain amount of authority to speak on this, that people genuinely should

• 33:31Speaker 01

  question. But I think when it comes to questions around medicine, around mental health, at the end of the day, the person I want evaluating those outputs is a clinician that has been in the room with the patient hundreds and hundreds of times and has a sense for the nuance that is critical. And I don't know another way to do it. And if somebody else has a better idea, I'm all for it. But I think on anything that is life or death, that is as high stakes as some of the topics that we're trying to deal with. And I'm, I'm not pretending politics is easy by any means, or that my approach is the right one, but

• 34:14Speaker 01

  it's the best one I think we have at the moment. And I think we have to lean into it and recognize that there are people who have more experience in certain areas than others, who have studied these issues for years, and they are needed right now to help us figure out some of these challenges. And so writing off experts is, is not an option.

• 34:37Speaker 00

  Yeah. I'm not for that, by the way.

• 34:39Speaker 01

  No, no, I know. But it's a legitimate point to raise.

• 34:42Speaker 00

  There are many tricky areas here, especially again, going back to our core of our discussion, which is like, these are just going to be the most trusted entities in the world, it seems like. So.

• 34:53Speaker 01

  Yeah, I, it'll be interesting to see because of all of the political issues around this. Um, just, you say that, but I also, I mean, look at what's happening with data centers. This has become a huge political topic.

• 35:07Speaker 00

  AI, I think, sorry, sorry to interrupt you.

• 35:10Speaker 01

  No, no.

• 35:10Speaker 00

  AI, the AI industry is very unpopular.

• 35:13Speaker 01

  Yes.

• 35:14Speaker 00

  People who use these chatbots love them.

• 35:16Speaker 01

  Yes.

• 35:17Speaker 00

  So there's a divergence there.

• 35:18Speaker 01

  There's a disconnect.

• 35:19Speaker 00

  Similar to like, Mark Zuckerberg is not very popular, but like people who use Instagram like it.

• 35:25Speaker 01

  Right, right. I think that's fair. I think there's a disconnect there. I think people are also trying, there's a disconnect around what they're hearing from Silicon Valley, the message they're hearing from Silicon Valley and their own experience too with AI, which is, oh, AI is going to change the world. It's the greatest thing that's ever happened and it's going to cure cancer and everyone's going to lose their job and these like very, but then they're using their chatbots. They love their chatbots, sure. But they also know that these chatbots are not the savior that, you know, is being promised to us by Silicon Valley. They know they make mistakes. They know who, normal people are

• 36:05Speaker 01

  using them like you and I are, like, everybody's using them. And we all know that the technology isn't there yet. So I, I'm not sure that that's where we end up, that we're all trusting our chatbots over everything else.

• 36:20Speaker 00

  Well, I think it starts, maybe it starts with students. And I think I'm being a little hyperbolic here, I'll admit it, but like, it starts with the students.

• 36:26Speaker 01

  It's good for a podcast.

• 36:27Speaker 00

  Yes.

• 36:28Speaker 01

  Good for engagement.

• 36:29Speaker 00

  Oh, God. Uh, we, we do need an independent rating agency to come in and give us some better benchmarks than I'm, what I'm optimizing for. No, I'm kidding. But like the Brown students, right? Do you see this, uh, story of kids at Brown University? They, uh, they used AI for the midterms. Many of them aced it. Then the professor said, come in person. Everybody did poorly. Not everyone. There was one kid that got 95 on both and he scored or she scored way below, uh, the average on the midterm where everybody got a hundred and then way above the final, even though they were consistent. But it starts in school because people will see this stuff can write my paper. And then, you know what? They're having a kid and all of a sudden they're like, using AI to raise their kid the same way that they used it to like get through school.

• 37:13Speaker 01

  I'm, I don't disagree with you. I, that's why I'm diving in headfirst, because I do think it's important. I really, I can't disagree with you. I, look, we're, it's early days. It's, it's so early days.

• 37:26Speaker 00

  Right.

• 37:27Speaker 01

  And we're working through these problems. And I, I do think education and how AI is used in, in not just higher ed, but elementary school, like how, how, what policies we develop around that and what we learn about how to teach kids, the benefits and the, the, both the pros and cons of it are, are things that we will learn over the next couple of years. But I, yeah, at the end of the day, I think we end up where you think we end up. And that's why I think it matters so much.

• 37:56Speaker 00

  Yeah. No. So I've spent like the last 10 minutes or so pressure testing, like the idea of using experts. But, uh, but I, I think that, you know, as we start moving into this direction where people trust, uh, trust these bots more and more, um, you know, I, I would recommend like for myself and everybody else, there are going to be these exceptions to the rule, but the rule really matters. And that is like, you brought up such a good point. Like when, in a mental health situation, do you want to engineer, you know, sort of doing fine tuning on a bunch of answers? Or do you want to like go to the best mental health clinicians in the world and then ask them how the bots should treat self harm,

• 38:40Speaker 00

  you know, prompts and then reinforce based off of what those experts say? I would much rather the mental health experts have the say here.

• 38:48Speaker 01

  A hundred percent. And know, and also know when to say, stop using this chatbot. Stop, stop asking me, go ask a, a real doctor.

• 38:58Speaker 00

  Oh, yeah.

• 38:59Speaker 01

  And that, you know, that, that moment is critical too. And there, look, this is, this is something that will be a priority. I think on mental health, it, it's really becoming a priority. A, because there are reliability issues here.

• 39:17Speaker 00

  Yeah. And people have taken their lives after speaking with chatbots.

• 39:20Speaker 01

  It is, but it's, it also is a huge use case, but there's also great potential, I think, to help people.

• 39:28Speaker 00

  Agreed.

• 39:29Speaker 01

  And we, we have to fix, fix that balance, try to get that right. And I think the companies are leaning into that, at least the ones we are talking to.

• 39:38Speaker 00

  So is your, is your business model then to basically, you know, you create these benchmarks, um, you can create pretty good data because you have, so you know, the best coding models from what my understanding, they weren't trained on the entire universe's code. They were just trained on really good coders code. And this is the equivalent of that for these other, um, these other areas.

• 39:58Speaker 01

  Yes.

• 39:59Speaker 00

  Important areas that the models are gonna just field queries in nonstop from now till the end of time, most likely. So is the business model for you that you actually like create these good data sets that the AI labs can reinforce on? And then, you know, they, they give you some money and you give them these benchmarks and then they're able to fine tune the models that way?

• 40:19Speaker 01

  Yes. But I do think what we also want to do is create standards in places where we think that that would be helpful, where there can be sort of something that we all aspire to and hold, have those bench, have, have, have that be a holdout essentially. So yes, we would work with the labs on evaluation and data, but ensuring that there is a standard that is a holdout so that measurement is real and legitimate and you can actually have something to work toward. You don't want, you don't want the labs teaching to the test, right? I mean, that's not, that's, you actually want them to improve. And so

• 41:02Speaker 01

  I do think you have to have a standard that, that. They can't learn on that, you know, can't game and that is something that we aspire to. And I think this is what you want to use sort of, and this is like the hardest edge cases and things that are really challenging to get at, that you want to work with experts to develop that continually measures the models to see how they're improving across this. And so it's, that has to be sort of a separate piece to this puzzle that, that's truly independent so that we know where we're trying to go and then working with the labs to get there. And then also enterprise. I mean, this is, that's who I hope our customers

• 41:46Speaker 01

  will eventually be. And I've talked to a lot of CEOs and a lot of boards and my message to them has been, and yes, I'm talking my book, but you know, if you are using AI for something really important, ask yourself, who is checking the outputs and the information that you're getting? And if it's the company that sold you the AI, you have a problem.

• 42:08Speaker 00

  Right.

• 42:09Speaker 01

  So you, whether you're building your own evals internally or you're using outside, you know, someone outside to evaluate your vendors and the model companies you're working with, you need someone to do that checking for you. If it's something that matters and that you care about and that you're willing to take on the liability for.

• 42:29Speaker 00

  Okay. I have many more questions. I want to ask you when models should be honest and when models should be sycophantic because that's clearly something that they're struggling with and why we haven't had a real content moderation blow up in the way that we consistently had with social media companies. And so let's do that right after this. Hi, everyone. Alex Kantrowitz here. I want to tell you about a documentary I have made with Gravity to explore the future of AI agent security. To find out if we're truly ready for autonomous agents, I sat down with MIT Professor Ramesh Roskar, former White House CIO Theresa Payton, Michelin's Group Chief Data and AI Officer Ambika Roger Gopal, and

• 43:11Speaker 00

  Sharon Gai, a former executive at Alibaba. They each offer unique insights into this evolving landscape. We conclude with Lori Blundell, CEO of Gravity, to discuss the path forward. With Gravity leading the way, join us on this journey. You can watch the full documentary at the link in the show notes.

• 43:40Speaker 00

  This episode is brought to you by App Point. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data and most teams have no single way to see it, secure it, and prove it's under control. That's exactly what App Point does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation

• 44:19Speaker 00

  scales without scaling risk. App Point, the unifying trust layer for AI. Head to avpoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast. Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As

• 45:02Speaker 00

  a white glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwall.com/bigtechnology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwall.com/bigtechnology. Learn how to stop online threats before they become real world attacks. The link's in the show notes.

• 45:40Speaker 00

  And we're back here on Big Technology Podcast with forum AI CEO Campbell Brown. Campbell, truth for sycophancy. It sounds like a fun game that you can play, but like, when, when, you know, I understand that you're working to get sort of accurate, context-rich information to people when they are using an LLM, but sometimes they don't want it. So I'll just like throw two potential prompts out there. And, you know, you can tell me, you know, is it, you know, well, anyway, let me throw them out there and we can kind of talk about them. So if somebody writes, you know, please tell me why Donald Trump is, you know, the

• 46:21Speaker 00

  best president in the world or ever. Or if somebody writes and says, you know, please explain all the instances why Joe Biden was treated like really unfairly by the press and the public, you know, and sort of unfairly forced to step down in the 2024 election. You know, the, the model has a couple of options there. It can say, you know, you're right, you know, your political view is, is, you know, sort of substantiated by the following points. Or it could say, you know, you know, Joe Biden, for instance, you know, did, did have, you know, a strong agent, you know, a strong legislative record. However,

• 47:05Speaker 00

  he was showing signs of decline and, and choose to include that context. So how should they approach it?

• 47:12Speaker 01

  So those are loaded prompts and we measure, we measure how they perform on loaded prompts.

• 47:21Speaker 01

  You also have to take into account, I think, how a lab is thinking about this and wants their chatbot to respond. They have policies that they have decided to, to implement around these things. So in some cases, they do want the model to reflect back your language, if not your perspective. So on something highly political that's asked as a loaded question, and what was the first one you said about Trump? Trump is, why is he the best president ever? Okay. The model might respond and I, I would guess if, if I were looking at a typical Claude response on something like this, it would likely say, many people, many supporters of Donald Trump believe he is the best president ever for

• 48:06Speaker 01

  these reasons and cite some of the accomplishments or what polling or whatever the data is around to support that. But not say, I agree with you, Donald Trump is the best president ever, but would give you the context around it. That is a choice that Anthropic is making in terms of how to respond to that. That's just what I've seen. And again, I'm not speaking on behalf of Anthropic or OpenAI or anybody else, but just what I've seen, you know, in the repeated loaded prompts that we've tested. You know, sometimes you'll hear the Anthropic people talk about, we want Claude to be your brilliant friend. And ChatGPT

• 48:47Speaker 01

  is more, I think, inclined, and I don't, I don't want to speak for them, but to reflect back the language you use. So let's say it's hyper partisan language in the prompt, that it might reflect back the, the language you use in the prompt, but that the answer would be comprehensive. And, and by the way, that doesn't mean, that question, why is Donald Trump the best president ever? That doesn't mean you have to give the other side of that. That prompt, it's not asking you.

• 49:18Speaker 00

  Right. You'd be almost overstepping.

• 49:20Speaker 01

  That, right. You, that, that wasn't the question. So, but do you have to say, I agree with you. Here's why I think he's, because it's a, it's a AI at the end of the day. It's not a person with a perspective. It's saying, here's the evidence to support what you're asking me. Now, again, these are policies that these companies will have to develop. And Meta certainly spent years developing. Google has spent years developing policies. It's newer for Anthropic and OpenAI in terms of how they think about these things and how they publicize the policies and the ways they're going to approach these. But the, the loaded prompts are interesting, really interesting to test. And this is consistently,

• 50:03Speaker 01

  I think, what we've seen in terms of how the models respond, which is reflecting back the language, but not in the answer.

• 50:12Speaker 00

  I'm going to come back to sycophancy in a moment, but isn't it interesting that there has not been a content moderation scandal among any of these chatbots? I don't think. I mean, at Meta, it was like, you couldn't do, couldn't go a single day without you, the royal you company, without stepping in it in terms of ban this politician or make this word or this. And chatbots are like taking stances and getting stuff wrong. Nobody seems to care.

• 50:44Speaker 01

  I think people under, it's to your point. It's even the average person knows they hallucinate. So we're way more forgiving of chatbots at this stage. I don't think that's going to be the case in a year. I think people are going to be much more demanding, especially that, again, I go back to the point I made earlier about the incentives for the companies to get this right. Their business is being driven by big enterprises who are going to demand that there not be mistakes and they're, you know, why am I paying you $20 million to sell me all these AI products and they're hallucinating? That's

• 51:25Speaker 01

  not acceptable. So I think there is a race, but you know, among the companies to try to address this. And so it's, it's almost as though we all have accepted it. And so we all kind of shrug our shoulders, even at the worst cases. And we, every person using a chatbot can give you, you know, their own examples. But I don't think that's going to be the case a year from now. I think that, that it will be, it's just deployment of AI will just, and it kind of already has in some industries, certainly in regulated industries, hit a bottleneck, which is we can't go further with this until you improve. And you're seeing, you know, you're certainly seeing that in banking and insurance and

• 52:09Speaker 01

  areas where it's just too risky.

• 52:12Speaker 00

  Yeah. I mean, it is crazy how like, you, you have seen the reaction of people like trying to like gotcha the, the chatbots by like screenshotting it and putting it on Twitter and being like, look at this woke bot. And everyone's like, why, you know, the reaction is, is generally like, why are you trusting ChatGPT for politics? You dummy. And then they're like, but who should I vote for?

• 52:35Speaker 01

  Fair enough.

• 52:36Speaker 00

  It's gonna be very interesting to watch this develop, but go ahead, go ahead.

• 52:38Speaker 01

  Yeah, no, I was, I was, I think the, the upcoming election is a motivator for a lot of the labs to try to certainly address some of the political content. Josh Gottheimer and Mike Lawler, Democrat from New Jersey, Lawler's a Republican from New York, have jointly worked together to try to raise awareness. They've, I know they've been on TV talking about it. They've reached out to a number of the labs to talk about it, about the quality of information around elections in particular, both candidates, and just basically where's my polling place? That they have to get that right. And I know a few of the labs have formed partnerships with outlets to just have one

• 53:21Speaker 01

  source for that information that's deemed critical. But there's a lot of pressure on them, I think, heading into the midterms, on that in particular, to work very hard to try to improve and get that right.

• 53:33Speaker 00

  Okay, I want to end with kind of, we'll talk about second infancy a little bit more. I mean, and this kind of maybe brings it full circle in terms of our media conversation. It's, it's just so interesting how people want the bots to be sycophantic to them as possible, like the most pop, well, I won't call it the most popular, but the, the, the model with the biggest diehards ever was ChatGPT four, four oh, which like, when they took that offline, there was like multiple seeming funerals, not to make light of it, but like, it is, people just, just wanna be, they wanna feel loved by the chatbots.

• 54:16Speaker 00

  And they're, my prediction is that they're just gonna develop this like, you know, deep relationship with, with these bots. I just read a tweet today, I don't know if it's, I don't know if it's satire or real, but it felt real enough to me that I thought I'd bring it into this discussion. Someone wrote, plane landed and a guy in the row ahead of me immediately opened ChatGPT to let it know he landed safely.

• 54:38Speaker 01

  You're kidding.

• 54:38Speaker 00

  Yeah.

• 54:39Speaker 01

  Oh, God.

• 54:39Speaker 00

  I just think that, and so then, then what is, I suppose you would, you'd wanna have the same relationship with the truth as, as you did previously, but we have yet to see, I'll call it like a third competitor. So you have like the ChatGPT and Anthropic, they're both trying to be as accurate and as sort of context-rich as possible. And it, I'm kind of stunned that we haven't yet seen this third competitor come out and be like, we are, we are gonna optimize specifically to make our bot your friend. Same technology, but just much more sort of friend, partner, relationship forward than the others.

• 55:18Speaker 01

  Well, that's about the business model.

• 55:20Speaker 00

  Yeah.

• 55:20Speaker 01

  Which is the, the consumer version of this has not been driving the way that enterprise has. And you, OpenAI started down that path and then realized, you know, from a business perspective, they needed to focus on enterprise and shifted.

• 55:36Speaker 00

  Yeah.

• 55:36Speaker 01

  And now that's what they're doing. So it's, that's a great question, is whether one of these companies will decide to really lean in and, and become the consumer chatbot of choice for people, or if there's a third that emerges in that regard. But just from a business model perspective, they're

• 55:54Speaker 00

  It's harder.

• 55:54Speaker 01

  Yeah, it's, it's, it's harder.

• 55:57Speaker 00

  But as, as the AI models commoditize, right, as intelligence gets cheaper basically to serve, that will inevitably come. And then the question is like, you know, you know, right now, the incentive is tell people this, you know, as accurate, as context-rich as possible. Then it will be, and you said before, okay, this is a great place to end. You were like, the nice thing about AI is it's not gonna optimize for engagement right now, but it will.

• 56:22Speaker 01

  I hope I said right now.

• 56:24Speaker 00

  Yeah, yeah, yeah. Yeah. And then, and then where do we go from like a newsworthy or sort of truth and accuracy, truth and accurateness and context-rich world?

• 56:33Speaker 01

  It, that, that's the million dollar question. And I, you know, I, again, I, it feels inevitable to me for exactly the reasons that you laid out, that people want that friend to know they landed safely. It's irresistible almost to go down that path.

• 56:52Speaker 00

  Yeah.

• 56:52Speaker 01

  I hope the place we're in does become the focus for the model companies, because you're so, you know, look at, look at, we haven't even touched what's, what the potential for AI in, in medicine and drug discovery. And that is gonna be all-consuming for the next few years as people lean into that. So sure, are there gonna be companies that develop the personalized chatbot? I'm, I'm sure there are. I wanna focus on keeping the big labs focused on accuracy, and at least for the moment, that is where they're leaning in. I'm really happy to see it. I hope it doesn't sound too Pollyanna-ish to say.

• 57:29Speaker 00

  No, it's good. It's good that they care about this. I mean, they need to care about this for all the reasons we've laid out.

• 57:34Speaker 01

  Yeah.

• 57:34Speaker 00

  I will admit, there are times where like ChatGPT helps me plan something and I wanna like go and update it and be like, hey, I actually did this. I have done that before, and it's been very unsatisfying. It's very good at helping you sort of get to where you need to go, but it's not great at celebrating.

• 57:49Speaker 01

  At debriefing after your trip, about how much fun it was?

• 57:52Speaker 00

  I wasted so many tokens to try to plan this trip with you. I wanna see an enthusiastic, long-lasting reaction, but anyway, it doesn't happen. All right, last one, because I'm curious.

• 58:03Speaker 01

  Yeah.

• 58:04Speaker 00

  What's it been like going from, you know, the, like basically TV studios to the heart of Silicon Valley, and now you're running a startup? How's startup life?

• 58:12Speaker 01

  This is hard.

• 58:13Speaker 00

  It is hard, right?

• 58:14Speaker 01

  This is hardest thing I've ever done. I mean, really, it's the hardest thing I've ever done. But it's so exciting and every day is a new adventure. You don't sleep a lot. I wake up in the middle of the night worried about everything in a way that I didn't when I was at Meta or even in TV. But it's, it matters. These are really hard problems. And I do think working on something that you know matters is so motivating and energizing. And I, I know lots of people are very stressed about what AI is gonna mean for our kids' futures and what the job world is gonna look like and all of that. But I just think if you are working in this space and you have a, you know, an entrepreneurial

• 58:58Speaker 01

  bone in your body, this is an incredible time to be trying to solve hard problems because they're, as a tool, it is just incredible what the potential is. And so helping it achieve that potential seems like a really good goal.

• 59:15Speaker 00

  Definitely. I mean, with the, with the, the rise of AI and all the different ways that we've spoken about and more that we haven't, so much opportunity out there. And when you can work on making sure that when these things are, when people look to these things for critical information, they get the right stuff, we gotta hope that, that you succeed and that the labs are gonna, you know, take this stuff seriously because they are filling the shoes of, of, you know, sort of generations of institutions that have put a pretty, a large premium on, on doing this stuff right, doing right by people when they come to them. And so it's good to know that there has been some, some positive reaction there and that

• 59:59Speaker 00

  there are people like you getting to work on this.

• 1:00:01Speaker 01

  Thank you, Alex. I really appreciate it.

• 1:00:03Speaker 00

  All right, everybody, the company's forum AI, Campbell Brown, has been our guest. Thank you for watching and for listening, and we'll see you next time on Big Technology Podcast.

• 1:00:27Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.

• 1:00:56Speaker 03

  In this life, you can't win. When the world gets you down. No! Escape to Pluto TV.

• 1:01:02Speaker 01

  This is objectively fantastic.

• 1:01:04Speaker 03

  Relax into the movies and shows you already know and love, all for free. Shwing! Even our news is uplifting.

• 1:01:10Speaker 00

  You stay classy, San Diego.

• 1:01:11Speaker 03

  Watch School of Rock, Crazy Ex Girlfriend, Anchorman: The Legend of Ron Burgundy, The Ex-Files, Wayne's World, and so much more.

• 1:01:17Speaker 00

  We're not worthy!

• 1:01:19Speaker 03

  The best things on TV really are free.

• 1:01:21Speaker 01

  You just have to know where to look.

• 1:01:23Speaker 03

  Escape to Pluto TV. Stream now, pay never.