#316 Robbie Goldfarb: Why the Future of AI Depends on Better Judgment

AI is getting smarter, but now it needs better judgment. In this episode of the Eye on AI Podcast, we speak with Robbie Goldfarb, former Meta product leader and co-founder of Forum AI, about why treating AI as a truth engine is one of the most dangerous assumptions in modern artificial intelligence. Robbie brings first-hand experience from Meta's trust and safety and AI teams, where he worked on misinformation, elections, youth safety, and AI governance. He explains why large language models shouldn't be treated as arbiters of truth, why subjective domains like politics, health, and mental health pose serious risks, and why more data does not solve the alignment problem. The conversation breaks down how AI systems are evaluated today, how engagement incentives create sycophantic and biased models, and why trust is becoming the biggest barrier to real AI adoption. Robbie also shares how Forum AI is building expert-driven AI evaluation systems that scale human judgment instead of crowd labels, and why transparency about who trains AI matters more than ever. This episode explores AI safety, AI trust, model evaluation, expert judgment, mental health risks, misinformation, and the future of responsible AI deployment. If you are building, deploying, regulating, or relying on AI systems, this conversation will fundamentally change how you think about intelligence, truth, and responsibility. Want to know more about Forum AI? Website: https://www.byforum.com/ X: https://x.com/TheForumAI LinkedIn: https://www.linkedin.com/company/byforum/ Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Why Treating AI as a "Truth Engine" Is Dangerous (02:47) What Forum AI Does and Why Expert Judgment Matters (06:32) How Expert Thinking Is Extracted and Structured (09:40) Bias, Training Data, and the Myth of Objectivity in AI (14:04) Evaluating AI Through Consequences, Not Just Accuracy (18:48) Who Decides "Ground Truth" in Subjective Domains (24:27) How AI Models Are Actually Evaluated in Practice (28:24) Why Quality of Experts Beats Scale in AI Evaluation (36:33) Trust as the Biggest Bottleneck to AI Adoption (45:01) What "Good Judgment" Means for AI Systems (49:58) The Risks of Engagement-Driven AI Incentives (54:51) Transparency, Accountability, and the Future of AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  But our expectation is that I can ask it anything and it's going to give me an answer. And it's going to give me an authoritative answer that I can feel like I can trust. And I think that's a really scary expectation to have set. There was, OpenAI had an announcement a week or two ago where they said that every week they're seeing, it was like over a million conversations where the user demonstrated suicidal intent. That just gives you a sense of where we're headed or the health of information people have. Avoiding echo chambers, avoiding misinformation. That's so important to building a healthy society. And AI can go either way on this one.

• 0:42Craig S. Smith?

  Okay, Robbie, can you start by introducing yourself to listeners and give your background? You have an interesting background. And then give us the, tell us what Forum AI is and how you came to found it.

• 1:01Speaker 00

  Yeah, for sure. So my background is in software engineering and product management. I've spent most of my career at the intersection of AI and trust and safety. I've done quite a bit of work in education technology. I eventually went to Facebook where I worked on news and misinformation. I was actually, I was working on misinformation through the COVID pandemic, through the US 2020 elections, through a really interesting period in time that I think opened my eyes to the potential impact of the technology can have on people, both in a good and a bad way.

• 1:45Speaker 00

  And maybe more importantly, our ability to shape what those outcomes are. And so that really set me down the path that ultimately got me working on Forum. I went though from Facebook to Instagram where I worked on similar issues. I also did quite a bit of work on youth safety there. And then most recently was in Meta's AI lab focused on trust and safety more broadly. And then probably about a year ago, I was, I was thinking deeply about these issues and the concept of trust and AI. And then a, a good friend, Chris Struhart, who's actually the,

• 2:30Speaker 00

  he's the head of product for Gemini at Google now, was talking to him about it. And he said, you know, you got to talk to Campbell. So we connected and co-founded Form AI together.

• 2:42Craig S. Smith?

  And, and describe what Form AI is and how it works.

• 2:47Speaker 00

  Yeah. So to put it simply, we scale the world's smartest minds to help evaluate and improve AI systems, particularly in tricky or contentious spaces like healthcare or politics and news. But to understand what we're doing, our philosophy and our belief is that there is a shift that needs to happen in how AI is being developed, and that experts, credible experts, need to play a bigger role in that process. It's not to say they aren't playing any role in it today, but we believe that role needs to be bigger. Because as,

• 3:30Speaker 00

  if you think about it, as AI gets more and more sophisticated, the caliber of expertise needed to further improve it also gets more and more sophisticated. And so that's the future we're working towards is scaling the highest caliber of expertise. And, and so in practice, there's two things that we've done. One is we've built the networks. So we've built these incredible networks of subject matter experts. We work with people like Fareed Zakaria and Neil Ferguson. We work with academics from Stanford and MIT.

• 4:12Speaker 00

  We have partnerships with the Cleveland Clinic and Mount Sinai Health System on healthcare. The people that you want in the room, helping to shape these disciplines. And then the second part of it is building the technology to effectively scale those experts for evaluating AI systems. And so a lot of the work here is about how do you extract efficiently and effectively their knowledge and expertise. And then at the end of the day, what we're building are are judges. So these are AI systems that we've built, which capture the judgment of these experts for factors like identifying political bias or measuring clinical safety that we can

• 4:56Speaker 00

  then go use to help evaluate and shape the broader AI systems. One thing, maybe I just add, and it's still early days, but what we started to see is when you involve real experts in the process of building AI, the results can be profound. And I think, you know, often back to Craig, have you seen the Dario Amade's essay Machines of Loving Grace?

• 5:28Craig S. Smith?

  No, no, but I'm aware of it.

• 5:31Speaker 00

  Yeah, so he paints, I mean, it's this great essay where he paints this almost surreal picture of what the future could look like if we realize AI's potential, economic equality, healthcare, governance, et cetera. And and we can give examples, but you know, as we work with experts, as we evaluate these models, identify the gaps and then improve them, you're starting to see glimpses of that, which I think is really exciting.

• 6:01Craig S. Smith?

  And in in can you talk about how you what the process is of extracting the thought process of an expert, generalizing it and then applying it to other to to or having the the judge, the the AI judge absorb that? I mean, what what's the mechanism and that that process?

• 6:32Speaker 00

  Yeah, so I mean, you hit the nail. I mean, that's the, that's the most, that's the trickiest part of what we're doing and arguably the most important part is how do you effectively and accurately capture their expertise. And so there are several tactics that we use, and I can talk through a couple of them. One, and this is usually where we start, is what we call thought process. So rather than just, you know, hoping that experts are going to spend many tens of hours a week evaluating and labeling tons of data, we take a bit of a step back. And so, you know, let's take a question like, you know, was it ultimately

• 7:16Speaker 00

  the right decision for the Venezuelan people for the US to intervene? Right? We, rather than asking an expert, okay, what do you say? We'll say, how would you go about answering this question? And we'll try and understand what their thought process is to get there. And when you start to do this with several different experts, and across several different topics, you can start to identify patterns and build a graph. And what's interesting that you start to see is you start to see chaining and patterns. You know, one example is if you're dealing with factuality, right? So the question to the expert would be, how

• 7:58Speaker 00

  would you assess whether this is accurate? You know, one of the things they might often say is, well, I would want to cross-reference it with some reliable sources. Then you have a chain there. Okay, how do you think about defining or evaluating what is a reliable source? And you define that. And the result is you get this graph. And what we can do is we can build agentic systems that mirror that graph. So if the first step is, you know, whatever, I would try and find reliable sources. Okay, then we have a system that does that. Of course, in practice, it's a little bit more complicated than I'm making it seem, but we, to put it simply, we try and build these systems to mirror the way that experts think.

• 8:43Craig S. Smith?

  And then that's generalizable to other similar questions.

• 8:48Speaker 00

  Usually it is, but not always, right? And so the only way to figure that out is to test. So, so what we'll do is, you know, we'll ask for their thought process across a wide range of scenarios. But then we'll actually test the resulting judge on an even larger set. And we may find that it generalizes perfectly well. Sometimes we do. And sometimes we don't. If it doesn't generalize well, then that means you need to break, you need to actually have two separate judges here. And we haven't really, it's sort of a little bit of trial and error there right now. But generally, we found that these do overall generalize quite well within a domain. So within, let's say, political topics.

• 9:33Speaker 00

  Obviously, if you wanted to go from political topics to sports, that's, you know, a bigger jump.

• 9:40Craig S. Smith?

  Yeah. And you mean, right now, L L M's and as we spoke earlier, there are people that regard A I as truth engines that believe that you ask an A I model

• 10:01Craig S. Smith?

  and whatever it says is, is the truth. But there are all of these gray areas. And in fact, what the model is doing is just giving it, giving you the highest probability within the distribution and its trading data. And that can be manipulated or it can be biased depending on

• 10:25Craig S. Smith?

  what's in the training data or what's out there on the internet. I mean, I think there are people who are actively trying to manipulate training data by filling the internet up with

• 10:41Craig S. Smith?

  with things that support their, their view. And, and I used to ask people if you could

• 10:50Craig S. Smith?

  get all the data, all of the inputs that would go into deciding

• 10:57Craig S. Smith?

  what the optimal solution to the Russia-Ukraine war would be.

• 11:04Craig S. Smith?

  And you fed it all into a An AI, and this is far too complex for any human to sort of parse through and

• 11:15Speaker 00

  Yeah.

• 11:16Craig S. Smith?

  and make judgments. So could you come up with an optimal solution? So I, I, I think that's really a data question, but, but so in, in these, you're able to generalize because you build this system of agents. Do you end up then with, with different networks of agents for different kinds of questions? And do you then use those to evaluate models depending on what domain you're querying the model about?

• 11:57Speaker 00

  Yeah, I think, well, I'll respond to that. And then I want to touch on your other point because I think that was the, the idea of truth seeking is a really important one. But yes, absolutely. There are roughly three, I would say, dimensions around which we have to design these systems. One is what we call topic. So for example, within, if you consider political topics a category, you might have foreign affairs, you might have US domestic policy, economic policy, social policy. And so those differing topics can often necessitate needing different approaches to, to evaluation. You have different question types or different

• 12:42Speaker 00

  user intent or scenario. Is the user trying to, you know, write an essay or are they trying to get an answer to a factual question? Are they trying to debate with you? You know, what the user wants creates this very, these very different, in the case of LLMs, very different conversations. And then, so, and then the other piece is you have different, sorry, you have different. Topics, you have different different user scenarios, and then you have different dimensions, right? So what are we actually looking to evaluate? In the case of political topics, it might be, you know, we look at, is it biased? We look at, is it accurate?

• 13:27Speaker 00

  We look at, is it written in an understandable, explainable tone? There are all these different factors you can look at. So you can imagine this as sort of a matrix, and you do need different evaluators for different different combinations.

• 13:42Craig S. Smith?

  Right. And and the capturing the thought process, as you said, you can't expect these people to interact with with a model for hours on end. They're they're busy people. Do is it through interviews with them? Is it through surveys? How do you capture their their thought process?

• 14:04Speaker 00

  Yeah, we've we've tried everything, right? And so and what we found, well, we tried, we did try surveys. We also tried a more structured approach where we would almost collaboratively collaboratively try to define these thought processes. What we found works best is just ask them to talk it through. So we'd ask them a question, like the Venezuela question, for example. We'd say, just walk through how you're thinking about it. I mean, usually it's helpful if they have a piece of paper in front of them so they can actually, I mean, depends on the person, but we found that's often helpful. And and they'll just talk through it. And then you do that with several different experts across several different scenarios. And then you start to get a sense of what

• 14:47Speaker 00

  the right graphs look like or what the right thought process looks like. Of course, and I should add, you know, looking, understanding an expert's thought process is one tactic. There are others that we do that, you know, of course, all of them together are what allow us to get to evaluators that are accurate. But that is certainly an important one.

• 15:11Craig S. Smith?

  Yeah. But can you mention another one?

• 15:16Speaker 00

  Yeah, for sure.

• 15:19Speaker 00

  There's one, I'll tell you this one because it's, it's, this is experimental. We haven't formally started doing this with any clients yet, but it's something we're working on internally that I think is really interesting. We're calling it consequence mapping. So in traditional data labeling, you might have experts go label, let's use an LLM as an example, a conversation and they'd say essentially whether it's good or bad, maybe across a few different dimensions. With consequence mapping, the labeling works a little bit differently. Instead of asking them to label or rank or evaluate the conversation,

• 15:59Speaker 00

  we ask them to tell us what the outputs or what the consequences of that conversation would be. And so I'll give you an example, let me just pull it up because I was just looking at it. So this is a consequence map for mental health. So what we would do is you could imagine it's a user having a back and forth conversation with an AI about a mental health issue they're experiencing. What we would ask the experts, we'd say, okay, after looking at this whole scenario, what would, what would the users resulting emotions be? How would they feel after this? What actions would they take about after this? What would their family be saying after this? And so it's, it's, it's, it's essentially just data labeling,

• 16:42Speaker 00

  but it's a much richer way of assessing it that allows us to build a deeper intuition of how they're thinking about it. It's sort of the why behind whether it was good or bad. So that's something we've been playing with as well, that in practice we can use to, we use some of that data for, for, for training, but also just to guide even our prompting strategies in the agents as well.

• 17:09Craig S. Smith?

  Yeah.

• 17:11Craig S. Smith?

  And, and in, in mapping the thought process of experts, you know, you, you gave the example that, well, first I would check trusted sources or reliable sources. And so you have an agent and then you, you know, how do you determine what's a reliable source or trusted source? In today's world,

• 17:36Craig S. Smith?

  you know, there's no objective truth. There's consensus, evidence-based consensus. That's, that's basically what we all agree on. That's what the scientific method gives us.

• 17:55Craig S. Smith?

  But there are people who have extremely different opinion about what is a trusted source. And depending on who your sources are, you can come up with very different opinions, particularly in the political sphere. So how do, how do you?

• 18:19Craig S. Smith?

  I mean, how do you make that judgment that,

• 18:23Craig S. Smith?

  that Fox News is not a trusted source? MSNBC or MSNBC, I guess it's called, is a trusted source or vice versa?

• 18:36Craig S. Smith?

  Yeah, how, how do you? Because that's a bias in itself, you know, giving the model a direction on, on what's considered a trusted source.

• 18:48Speaker 00

  Totally. Yeah. And I remember you brought up this idea of truth before. It's, so what you're hitting on is, I think, one of the most fundamental challenges, I think with AI more generally, and certainly that we're focused on, which is this idea that in subjective domains, there is no ground truth.

• 19:09Craig S. Smith?

  Right.

• 19:10Speaker 00

  There is a ground truth to how many Rs there are in strawberry, right? To use that example.

• 19:16Speaker 00

  And, and, and, but of course, when there's not a ground truth, what are you aligning to? Right? And, and our position is that the best way to do that is to rely on credible experts. And so what we've done, and this is fundamentally our thesis, is we've, we've built this network of experts who, number one, follow strict criteria in that we require the network to be balanced, to be representative and to have a certain caliber of expertise and experience. But number two, it's entirely transparent. Meaning we show our network on our website. We welcome feedback on our network.

• 20:02Speaker 00

  Once you have that, then you can start to defer to the network, or that's our approach. You know, we'll look to the network to define what the ground truth is. Now, to your point, sometimes folks will disagree, right? And experts in the network disagree. The philosophy we have in this scenario is if there are two differing opinions that are from our credible network, both should be represented by the AI. We would encourage the AI, the right answer, quote unquote, or the ground truth, is to present both opinions. Now, this, to be clear, is controversial. If you look at even just today's chatbots, Grok will often just give you an answer.

• 20:47Speaker 00

  Whereas the others tend to sort of more follow this approach where they'll share different perspectives. Our belief is that it is not the job of AI to make decisions for people or to tell them what to think. It's the job of AI to give them the information needed so they can make the most informed decisions for themselves.

• 21:12Craig S. Smith?

  And, uh, yeah, I'm just thinking in areas that, uh, that are science-based, you know, climate change, whether Tylenol causes autism, you know, these things that have,

• 21:31Craig S. Smith?

  have really divided the political world.

• 21:37Craig S. Smith?

  How do you, how do you protect, I mean, from, from being, uh,

• 21:45Craig S. Smith?

  labeled as bias, your, your system being labeled as bias without giving credence to something that you personally don't believe?

• 22:00Speaker 00

  I mean, I'd say first and foremost, and I think this is an important thing for anyone working in this space, what I personally believe doesn't really matter. That's, that's part of the point of this company is, is we want to defer the decisions to the people who should be making the decisions. I mean, I can't speak to specific examples here, but the short answer is we would defer to the experts in the network.

• 22:22Speaker 00

  And we will always be entirely transparent about who they, who those are. And so in that way, I think there's no perfect way to approach this, but I think it's much better than the alternatives, right? Which is, I mean, there's a few, like one is you essentially just rely on internal engineering and product teams or business executives at these companies to make those decisions. The other is you rely on sort of massive scaled teams of labelers to collectively make those decisions.

• 22:57Speaker 00

  And our position is, is, you know, the right answer is to find a transparent and credible group of experts and, and put it on them.

• 23:05Craig S. Smith?

  Yeah, or put it on their process.

• 23:08Speaker 00

  Totally. Yeah. I mean, what I, what I should add, because this is important, is, you know, the thought process is one part of how we develop these judges. One of the other things we do that has proven very interesting, but also very effective is, is separately, so separate from understanding their process, press them directly on edge cases. So the most extreme topics, the most extreme questions, ask them about that. So we have a log of those decisions explicitly as well. And especially when there are disagreements among experts, it's important we get a lot of data there that we can use. So you can imagine if you go back to our system that mirrors this expert thought process,

• 23:49Speaker 00

  there are certain points where it may need to reference very explicit guidance from experts if it's talking about, you know, for example, one of the contentious examples that you, you gave a moment ago.

• 24:02Craig S. Smith?

  Yeah, yeah. And and then the output is used to fine tune the models or to rebalance the training data or even excise some stuff from the training data. How how is the the output, the report that this judge generates used?

• 24:27Speaker 00

  So the report we give to companies gives you a very clear and specific picture of the issue. So it may say, you know, these sorts of questions you are responding to poorly because you're often not representing this perspective. For example, we ultimately then lean on the research teams at these labs to figure out the best way to address it. And that could be, it could be fine tuning, it could be RL, it could be playing around with the training data. Sometimes it's, you know, playing around with the system prompt as well. And that's ultimately where we lean on them. What I will say, and this is an area we're just starting to get to, is developing,

• 25:10Speaker 00

  taking the evaluators that we build and making them available for reinforcement learning. And so that's where those can have a more direct effect on or a direct impact on training the models as well. But short answer is the evaluation highlights the critical issues and opportunities so that the teams can then know what they need to hill climb against.

• 25:32Craig S. Smith?

  And how many different qualities do you do you evaluate? I mean, bias is one. I don't know if quality is the right word, but bias is one.

• 25:47Craig S. Smith?

  I guess factual grounding or at least grounding in in.

• 25:55Craig S. Smith?

  Historical literature or scientific literature or something would be another. Yeah, how maybe sycophancy is is one, perhaps. I don't know. Do you do you judge for that?

• 26:11Speaker 00

  Yeah. So within political topics, there's four dimensions that we look at. I think you got most of them. There's there's bias. Although I will caution that bias actually breaks down into several different things, which I can talk about, but bias is one of them. There is factuality or accuracy.

• 26:32Speaker 00

  There is a source selection. So for you, you know, these AI systems will often, especially when you're talking about news, refer to outside sources like news. And so are you looking at credible sources, a balanced set of sources? Are you using the sources accurately? Are you doing attribution properly? So that's that bucket. And then we also look at what we call tone and language.

• 27:02Speaker 00

  And this is even just from my previous work at Facebook and Instagram. You can say the exact same thing in two different ways, and it can have very different outcomes, depending on whether you're using large language or not. And so that's another thing we look at in in politics as well.

• 27:19Craig S. Smith?

  Yeah. What do you call that?

• 27:22Speaker 00

  We call that tone and language.

• 27:25Craig S. Smith?

  Tone and language, yeah.

• 27:26Speaker 00

  Yeah. And it's, yeah, it's just this measure of.

• 27:32Speaker 00

  You know, this is a common issue is is a user may come into a conversation. We're talking about LLMs with a strong bias. They may ask a question in a very angry way.

• 27:48Speaker 00

  And that's fine, and you can respond to it completely. But sometimes the risk is it tends, I mean, even because of the sycophancy and some of the stuff behind that, it will mirror their language. And and that can be really dangerous, right? To the point before, because the exact same thing, but saying it with some swear words or with hyperbole can mean a totally different thing to them.

• 28:12Craig S. Smith?

  Yeah. And how how large is the expert network?

• 28:20Craig S. Smith?

  And and.

• 28:23Craig S. Smith?

  Yeah.

• 28:24Speaker 00

  Yeah, we have, we have four tiers of experts. Uh, or I should say like four groups of experts. So we have, um, and, and, and some of them are, are larger teams of more scaled labelers. Probably a little bit more similar to what you might see at a, you know, a surge or a scale AI. Uh, and then other groups are, um, you know, we have a group with Fareed Zakaria and Neil Ferguson and that sort of people. So depending on the group, then the, the sizes are, are, are different. But what I would say is our approach is very much quality over quantity.

• 29:02Speaker 00

  We, we are trying to find, and Campbell always says this, you know, who are the maybe a hundred to 500 smartest people in each domain? Let's bring them on and then we can figure out on the technical side, how to scale them through some of the techniques we were talking about, right? Rather than let's get 10 or 15,000 people and then throw them straight at the work. More man.

• 29:26Craig S. Smith?

  Yeah. I mean, do they have to be living people? You, you could turn to, you know, the, the writings of

• 29:38Craig S. Smith?

  philosophers or, or scientists, uh, to follow their thought process.

• 29:45Speaker 00

  Yeah. Have you seen, there's a video of Steve Jobs from 1985 where he talks about predicting this, this future? A writ with Aristotle.

• 29:56Craig S. Smith?

  I, I may have, but I, I, yeah.

• 29:58Speaker 00

  You gotta check it out. It's, it's unbelievable. This is in 1985 and he says something like, um, he believes there's going to be a day where when the next Aristotle, when the next Aristotle comes, we'll be able to capture their underlying worldview in a computer and be able to converse with it. And he goes on to essentially describe kind of what the way these L L Ms work today, but in 1985, which is crazy.

• 30:27Speaker 00

  But anyways, that's more of an aside. Yeah.

• 30:30Craig S. Smith?

  Actually, before you move on from that, but don't forget your point. I, I, I, I know a guy at Boston Consulting, a very senior guy, and he has this idea that, uh, you know, there's, there's all of this effort to, uh, with RLHF and, and.

• 30:51Craig S. Smith?

  Fine tuning to to nudge L L Ms toward certain moral and ethical positions or or

• 31:02Speaker 00

  Yep.

• 31:03Craig S. Smith?

  personas or uh and that the world has 5000 years of of religious texts that that deals with that with moral and ethical questions. And that you could use that to guide an L L M in its uh thought processor and it's not thought process, but in its uh in its

• 31:31Speaker 00

  Reasoning.

• 31:33Craig S. Smith?

  Yeah, reasoning, exactly. Yeah.

• 31:35Speaker 00

  Yeah.

• 31:35Craig S. Smith?

  I mean.

• 31:37Speaker 00

  It's very, it's interesting, like, I mean, because what what this is ultimately getting at is the is the point we were talking about before, which is in in subjective domains, there is no ground truth. So what is the ground truth? And it's a very interesting case to say, I think the ground truth should be, you know, the entirety of these religious texts and ethical like writing or whatever it may be. Um, I mean, our position at forum, once again, is we think the best answer is to take, you know, the top few hundred smartest people in a given area and look to them to define this. Um, but there's certainly something that's a very interesting idea. Yeah.

• 32:16Craig S. Smith?

  Yeah. And but do you go back and use experts through their writing to to map their thought process?

• 32:27Speaker 00

  We just started doing this two weeks ago. Um, we we hired a researcher who's specifically pretty good at at this sort of thing. We're calling it extraction.

• 32:38Speaker 00

  And so still trying to figure it out. But what we are finding is that um, when you get access to certain content, but you can extract certain themes or judgments that the author has in a way that might be generalizable. Um, for for example, for training our judges. I think there's a lot, there's more work we have to do here. There's also a lot of other research in the space that we're looking to. So this isn't something entirely new. But yes, you're, you're definitely right. That's a, that's an interesting opportunity. What we haven't thought of, or we haven't actively been working on, which you're alluding to, which is very interesting, is looking at the texts of deceased experts, right?

• 33:22Speaker 00

  And that could be a really interesting way to approach it that maybe we'll take a look.

• 33:28Craig S. Smith?

  Yeah. I mean, you know, if you could map Einstein's thought process, maybe valuable for, for research systems.

• 33:41Speaker 00

  For sure. I mean, I can just say, just from the work we've done with, with living experts, it's, it's really hard when they're alive and you can actually go back and forth with them. So I imagine this would be pretty tough, but it's probably something there.

• 33:54Craig S. Smith?

  Yeah, yeah. How much of this can you surface using an LLM? Because, you know, the largest LLMs have Fareed Zakaria's writings in their training data.

• 34:11Craig S. Smith?

  And presumably, they would be able to, to, I mean, I, I, as we said, I mean, they're really looking at probability distributions, but in, in the reasoning, maybe they're influenced by reasoning in the training data.

• 34:33Speaker 00

  I mean, totally. And first of all, I'll just say this is probably getting a little bit, you know, this is a deeper research question. I think what is interesting though, because we, I've seen this in a number of the clients we've worked with, and we've had these discussions several times. LLMs, of course, do generalize quite a bit from their training data. I mean, that's essentially how they're designed. And that can lead to a ton of unintended consequences. Sometimes it can be good, like what you're implying, you know? But how do you know that it's going to generalize the content from Fareed versus maybe something erroneous it picked up from a Reddit post that it was trained on?

• 35:15Speaker 00

  And I think that gets into the complexity of thinking through the composition of your training data and how you ultimately go about like post-training and fine tuning your models.

• 35:26Speaker 00

  So.

• 35:28Speaker 00

  Yeah, I mean, it's I think there is though, and I mean, the piece that interests me a lot that you're getting at is how do we leverage these experts and potentially their content too as much as possible to help guide and inform these models. My sense is it's probably going to be more post-training, more about evaluating, having them set the gold standard and then hill climbing towards that than pre-training, but who knows?

• 35:58Craig S. Smith?

  Yeah. And so you've got this this large network, but focused on quality, not quantity. You haven't mentioned the quantity, but I'm guessing that's proprietary information.

• 36:19Craig S. Smith?

  How do you see this affecting the future of AI? I mean, presuming that that your methodology

• 36:30Craig S. Smith?

  becomes widely adopted?

• 36:33Speaker 00

  I mean, the thing I would just go back to here, and if we have a broader objective is trust, right? There was a couple of months ago, KPMG released this report, one of these public opinion polls, and it showed that eighty something percent of people are optimistic about the potential of AI to improve their lives. I don't remember exactly how it was written, but something, yet only forty something percent trusted it. And that is a really, I think that says a lot, right? Because if no one was interested in it and no one trusted it,

• 37:14Speaker 00

  then who cares? But the fact that there is an interest and a demand for it, but the trust is low, suggests that that delta there is going back to Dario's essay where he talks about this amazing, you know, potential future. Trust is a blocker, right? There's actually one other thing that folks should read is that Forbes had a piece a few months ago, and it says, the article is titled, "Consumer Trust Is The Next Battleground For AI." I think it's for that reason exactly. And so going back to your question, I think What we're a, what we're going to be able to do is to build models that people can trust.

• 37:55Speaker 00

  Because number one, people trust people and you want to see that there are trusted people behind them. But number two, and most importantly, the outputs speak for themselves, right? People have a good sense for what feels like a good experience. And what feels like a useful, helpful, reliable, trustworthy, honest experience. And that's what, you know, bringing experts into the mix creates in terms of a product.

• 38:21Speaker 00

  So that, I mean, that's maybe overly grandiose, but at a high level, that's I think about it.

• 38:27Craig S. Smith?

  Yeah. Although the trust issue, isn't that tied more specifically to hallucinations?

• 38:35Craig S. Smith?

  The

• 38:36Speaker 00

  Yeah.

• 38:38Speaker 00

  The way, the way I think, the way I would think about trust is that there's three things here that have come up if we think about trust in AI. One is unverifiable, sorry, verifiable situation. So when there is a right answer. So when there is a obviously right answer and it says something wrong, if you ask it, how many straw, how many Rs there are in strawberry? And it only says two, it's wrong. And that is certainly a major blocker to trust. I think that's probably a little bit easier to fix because there is a ground truth and it's just about, we got to figure out how to get there. But that's where a lot of hallucination falls in and absolutely you're right, that's a part of it. The next part of it is unverifiable situat- scenarios.

• 39:23Speaker 00

  What about when there isn't a right answer? And this is like we were talking about before, when there isn't a ground truth. And that's a lot trickier, right? And that's where, you know, we posit that experts can be the ground truth and that's the right way to solve that.

• 39:37Craig S. Smith?

  Yeah.

• 39:38Speaker 00

  Then the, the other piece that I just throw in there is this concept of responsibility, right? So if, if those things are talking about How AI acts, there's also this idea of AI knowing when not to act, which I think is really important. I think a trustworthy, responsible AI knows when it should take action, when it should respond, but it also should know when it doesn't have the answer, or when it would be irresponsible for it to engage. So it's a long way of saying, I think hallucinations are certainly one part of it, but there is a bigger picture. And experts can probably play a bigger role on the latter two pieces.

• 40:22Craig S. Smith?

  Yeah, and that's why I was asking about Sikkah Hensy, whether that's, whether you score for that, because

• 40:34Craig S. Smith?

  I was telling you when we spoke before, I had a conversation with a podcaster who's a smart guy shortly after GPT-3 came out. And he was arguing that this is a truth machine. And we got into this argument because there's a guy out in Arizona that does these experiments that are have not been repeated and and, you know, are very controversial about essentially

• 41:10Craig S. Smith?

  people showing physiological reactions before the stimulus, which he argues

• 41:20Craig S. Smith?

  proves that there that there's a separation between mind-body and that there is this

• 41:28Craig S. Smith?

  non-physical realm that

• 41:31Speaker 00

  Okay.

• 41:32Craig S. Smith?

  Yeah, so, and, and, you know, it's one guy and, and you know, the, but this guy, when he would ask the LLM. I think it was chat G B two, three, it would say, yes, you're this proves that there is a separation between mind and body. And it drove me crazy because I was saying the L L M doesn't know it's it's just giving you a probability distribution or an answer out of the distribution and you you bias it depending on how you ask the question because if I ask the question, I get no, this doesn't prove anything, you know.

• 42:17Craig S. Smith?

  So that and that is a synchophancy problem. So do you do you guys address that?

• 42:26Speaker 00

  The example you gave feels almost like a borderline hallucination, right? Where it's essentially saying that it's confident in something that it's not. And so this is where there becomes a bit of a blurry line between verifiable and unverifiable scenarios. But I would argue that that is a verifiable scenario in that I think we can pretty concretely say or the A I can pretty concretely say that it doesn't know and it didn't do that.

• 42:53Craig S. Smith?

  You know, I mean, how do you train or judge an A I's answer on that question? Because it depends on what whether you're looking at data, what data you're looking at essentially.

• 43:09Speaker 00

  Yeah, for sure. I mean, a part of the puzzle that we haven't talked about that's particularly relevant here because this is in the news is the retrieval sources, the news it's looking at, right? These models are only trained and updated every few months. And so a lot of the information they're getting is from the news sources that they or the social media sources or whatever that they have access to to reference. And so that's why, I mean, I mentioned earlier, part of what we do in our evaluations is look at source selection. And this is why, because I think so much of how an L L M is going to respond is going to be a product of the sources it has access to and it chooses to use and how it chooses to use them.

• 43:52Speaker 00

  So that is a is a very high leverage way to improve how these models are responding about some of these more newsworthy topics.

• 44:01Craig S. Smith?

  If you If the model looks at the law, looks at the volume of illegal immigrants in the United States.

• 44:12Speaker 00

  Yeah.

• 44:14Craig S. Smith?

  I could imagine an LLM saying, well, yes, this agency is charged with

• 44:22Craig S. Smith?

  enforcing this law.

• 44:24Speaker 00

  Yeah.

• 44:24Craig S. Smith?

  And there are reasons why that law exists without considering the human or moral aspects to the question. So, yeah, I mean.

• 44:37Speaker 00

  That example is so interesting, right? Because like you see what you just did, like that's your thought process, right? Your thought process is that you you do go through and you consider the moral or the human implications of it. And so that's that, I mean, for what it's worth, I agree. But but yeah, it's all about, you know, how do we go about answering these questions as opposed to the answer itself?

• 45:01Craig S. Smith?

  Yeah. Well, let's let's move on a little bit. What does it what does it mean for AI to have good judgment? In your view?

• 45:13Speaker 00

  Yeah. I mean, this this goes back to the conversation we were having before, right, which is

• 45:24Speaker 00

  the verifiable and the unverifiable scenarios, right? Verifiable scenarios, I don't think that that is an important part of the piece of the puzzle for trust. So when there is a right answer, it gives you the right answer and it doesn't hallucinate. I think how it handles the unverifiable scenarios, how it behaves when there aren't rules, when there isn't a clear ground truth, that is what judgment is. Right? And that's, of course, really tricky. What is good judgment? That's then the broader question. And of course, you know, our approach is, well, let's look to the experts to define that.

• 46:02Craig S. Smith?

  Yeah.

• 46:04Craig S. Smith?

  And what do you see as the most critical issues facing the AI industry today? Is it this trust issue or?

• 46:17Craig S. Smith?

  Yeah.

• 46:19Speaker 00

  I mean, so yeah, I'll give you a couple of specific examples and then I can tell you my biggest broader concern. But just from our experience, um, mental health is a really big one. There was, uh, OpenAI had an announcement a week or two ago where they said that every week they're seeing it was, it was like over a million conversations where the user demonstrated suicidal intent. That just gives you a sense of where we're headed, or I suppose where we are with this, right?

• 46:52Speaker 00

  And, and so I think figuring that out is really tricky. We've done quite a bit of work in mental health and the, the term I would go back to here is clinical nuance. Um, and I think the models don't quite have that now. It, it's really important in being able to, I'm certainly not an expert here, but, but work with patients and support patients in a way that's not going to harm them. Classic example here is, do you remember the Tessa chatbot from two years ago?

• 47:22Craig S. Smith?

  Yeah.

• 47:23Speaker 00

  So that was, I mean, this was from the National Eating Disorders Association. They launched this chatbot, um, and there's an article that describes how it didn't have clinical nuance. They ultimately ended up shutting it down, um, because it had adverse effects on users. But, but I think mental health is an area where, of course, there's enormous potential, but I think there's a lot of people using it right now and a lot of risk. Another area is politics and I don't even think we have to go into that too much more because I mean, it's all the stuff we've been discussing, right? So much of this is subjective. And to the point I was sharing earlier, I think the health of the information landscape, the health of information people have, you know,

• 48:08Speaker 00

  avoiding echo chambers, avoiding misinformation, that's so important to building a healthy society. And AI can go either way on this one, it really can. And so I think that's, that's another specific area that we've seen directly. The broader thing I would say to you, and this like Craig relates to something we were talking about before we jumped on this call, but.

• 48:33Speaker 00

  We've set, I think, a pretty scary expectation around AI. That, and I think this started with ChatGPT, it relates to what your friend said about this is the truth model. But our expectation is that I can ask it anything. And it's going to give me an answer and it's going to give me an authoritative answer that I can feel like I can trust. And I think that's a really scary expectation to have set. If you go in to a doctor, there is a natural back and forth. You're not going to say one thing and then they're just going to give you a prescription. There's a back and forth and that's the way, that's the way the world works. There's sort of, you need to absorb context and understand things.

• 49:17Speaker 00

  But with AI, we've sort of said, no, don't worry about it. You can just ask anything and it will give you a response, as we've seen. And I think that's some, that's an expectation we're going to need to shift. We're going to need the models to be more aware about when they need more context and when they need more information. In order for them to be responsible and ultimately drive good outcomes. The, the piece that makes it particularly scary is that maybe at odds with engagement, right? Because if I go to this AI and I ask it a question and then it asks me another question. I'm just going to go to this one that's going to give me the answer right away. And so that's a little bit of a tricky, that's what makes it tricky.

• 49:58Craig S. Smith?

  Yeah, I mean, the whole succumbency disaster was driven by a guy tweaking the, the algorithms to increase engagement.

• 50:10Speaker 00

  Yeah.

• 50:11Craig S. Smith?

  And it created all sorts of problems. Who, who are the customers for you guys?

• 50:20Speaker 00

  All the usual suspects. So we work with the, we work with the big AI labs as well as the large system integrators. We like the, you know, the larger consultancies that are deploying the models through to other companies.

• 50:37Speaker 00

  So mostly them, we'll work, do a little bit of work with some smaller labs here and there. But that's the bulk of what we, where we focus.

• 50:46Craig S. Smith?

  Yeah, and and are you not to get into the business side too much, but are you a subscription model or pay usage based fees or yeah?

• 51:03Speaker 00

  Yeah, we have both. It depends on what the service is. So what we'll often do with clients is we'll have a baseline subscription where once a month or once a quarter, we'll do an evaluation. So let's take political topics, for example, where we'll do a comprehensive evaluation where we'll run all our judges on the model. We'll actually have experts come in and manually review the results. So you just have this continued pulse on on how your model is evolving and a pulse on any potential risks. And then on top of that, we sell more bespoke or a la carte services. The way that would usually work is, and I'm making this up as an example now, but we might

• 51:43Speaker 00

  run an evaluation, a quarterly evaluation. We might identify that, you know, the model is particularly weak in source selection when talking about breaking news related to foreign affairs. We may then engage in developing a benchmark specifically focused on that one problem. And that would be more an ad hoc service.

• 52:04Craig S. Smith?

  Yeah, and we haven't talked about this, but you know, I spent a lot of my life in China and and so I follow what's happening in China fairly closely. You know, and they have a the Chinese government certainly has a very different worldview, but the people as well have a a different cultural view from that of the United States. And there are all these sovereign AI models of being developed in various countries around the world that are trained on local training data. And do you think your systems judgment?

• 52:49Craig S. Smith?

  Is objective enough and general enough that it could apply to models from other countries?

• 52:59Speaker 00

  Yeah, certainly. I mean, we've run, and this is something we're actually hoping to publish at some point in the near future, are the evaluations we've run on the open source models, particularly the ones coming out of China. And I would say definitely you can capture a lot of interesting things there. I think the other thing, yeah, on the, I guess, the national security or foreign interference piece, that there's two things. There's the models themselves and their training data, like you alluded to. But there's also news sources to the conversation about retrieval we're talking about. And so the sources they're use, if you're using, you know, state sponsored sources, you know, that can also have an influence on the models as well,

• 53:44Speaker 00

  which is why it's so important that we look at source selection as part of our evaluation. So I will, I will let you know, because we're going to publish something there.

• 53:53Craig S. Smith?

  Yeah, yeah, because these models just, you know, you were at Facebook and Instagram incredibly powerful in shaping public opinion.

• 54:07Speaker 00

  Yeah.

• 54:08Craig S. Smith?

  And, you know, depending on the trading data, depending on the fine tuning,

• 54:17Craig S. Smith?

  you know, you could have a model that reinforces the government's position on any particular topic. And, you know, a lot of people worry, given the closeness between the big tech leaders and the current administration, that, you know, that there may be some impact on how the models respond to certain questions. Is, do you worry about that?

• 54:51Speaker 00

  I mean, so to some extent, but I would say, and this goes back to what we were talking about before on transparency, right? I think at the end of the day, what people need to know is who is behind training these models. And part of the value that we can offer at Form AI is we can give you a very clear picture of who those people are. And then, and then it's ultimately on you as a user to, you know, evaluate whether you don't feel good about this. And I think certainly in our case, we have a very diverse and reputable group of people that I think people will feel much better about the models and they'll,

• 55:32Speaker 00

  they'll build that trust if they see these individuals involved. But I think transparency is the only answer here, right? Because you may have one opinion, someone else will have a different opinion. And so we just have to be candid about how these are being built, and then allow people to make the decisions from there.

• 55:50Craig S. Smith?

  Yeah. In the notes that were sent over before the call, they mentioned Delphi, which is a market for evaluating models. But it's, as I understand it, it's, it's kind of, you're betting on one model versus another, and there's some financial incentive.

• 56:16Speaker 00

  Oh, well, so there's, there's two different things here. There's like, so Delphi is interesting in that what they're trying to, they're essentially trying to do the Steve, the thing that Steve Jobs said in 1985. They're trying to be able to, you know, take Craig Smith and put, I think they use the term a brain in a box, put your, or a digital brain in a box. And then I can go and I can have a chat with you, even though of course it's not you. And so conceptually, I think that's very interesting. There's another company called Superme that's doing something similar, giving people the ability to clone themselves into an AI. Our approach is very different, as we described, in that we are

• 56:58Speaker 00

  not focused on cloning our experts so you can have a conversation with them. We're focused on cloning them for a very specific use case. So, you know, evaluating the factual accuracy of a claim, you know, of a political claim, for example. Now, the advantage we have in focusing is that we can do this accurately. Right? We can actually measure this. We can build these judges. We can have them, you know, evaluate a thousand things. We can have experts do it and we can tell you, you know, this is 95% accurate to what experts would say. If you're going to just have a conversation with an LLM, back and like, you know, you're not really talking to LeBron James and it's probably not,

• 57:39Speaker 00

  you know, 95% similar to the conversation you'd have with him. So this is a very different model. Of course, very different use cases. We're doing evaluation. For them, it's more, you know, it's just a different user experience and purpose.

• 57:52Craig S. Smith?

  Yeah, but you said early on that you're not a leaderboard. Do you make your evaluations public?

• 58:00Speaker 00

  Yeah. So this is, and this is where there's a comparison to LM Arena, who are, they are a leaderboard and they're, you know, experts, the people they rely on quite heavily is just the general public. The challenge there is it's created this weird incentive system where essentially what the general public defines as good is usually what's most engaging. And what's most engaging, I can tell you as someone who worked at Facebook and Instagram, what's most engaging is not always the best thing or the safest or most trustworthy thing. And so that's the issue there. For us, we don't make our evaluations public. We are thinking about potentially doing some,

• 58:42Speaker 00

  you know, releasing some public facing leaderboards in certain specific areas, like maybe for open source models or for certain areas of more frontier type use cases. But for the most part, you know, our goal is to work with these labs and work with our clients to make their models as good as possible.

• 59:02Craig S. Smith?

  Yeah. You know, I had a conversation with Max Tegmark at MIT during a recent conference and he has a,

• 59:15Craig S. Smith?

  I've forgotten what he calls it, but it's a scorecard for the major model producers. And he scores them on, or gives them a grade on various aspects. And his

• 59:29Craig S. Smith?

  hope is that this will eventually become, you know, the,

• 59:36Craig S. Smith?

  A standard that people is, you know, they, they wanna get a passing grade or A plus grade and all of the different areas that they're being evaluated as a, as a product validation.

• 59:52Speaker 00

  And so that's one of the things we're thinking through is how, what is the responsible way to potentially release, you know, some of our benchmarks so that people can, you know, see these scores and, and use that for them, for their own decisions, but without hurting the integrity of the benchmark.

• 1:00:09Craig S. Smith?

  Yeah, yeah. And I would imagine there's also a little bit of a conflict of interest. Companies are paying you and then you give them a crummy score publicly.

• 1:00:22Speaker 00

  Yeah, I think, I mean, the thing I found, cause, cause I mean, of course that is a real tension, but remember with us, it's not me and Campbell giving anyone a crummy score.

• 1:00:33Craig S. Smith?

  Yeah.

• 1:00:33Speaker 00

  It's, it's our experts, right? And that's a very defensible way to have a lot of these conversations. And I think a way that everyone understands. So, but yeah, that's fair.

• 1:00:47Craig S. Smith?

  Yeah. And in your mind, you have, you don't have to say it openly, but you have certain models that, that you think do a better job at being truth-seeking than others.

• 1:01:05Speaker 00

  Um, yeah, it's interesting. You use the word truth-seeking. This is, there's some, the government just in December released some guidance on, on the way they look at neutrality. And they, they say truth-seeking and ideological neutrality. So we're spending a lot of time thinking about those factors. I mean, no, I, I don't.

• 1:01:27Speaker 00

  It depends. Like, there's not, no model is better or worse. It's so, that's one thing I have learned. Maybe this is the most, the best answer to your question. I think one thing I've learned, having now run so many evaluations across so many different systems is it's often more nuanced than people think. Even just remember the factors, the three areas we were talking about earlier. There's different domains, like, you know, foreign affairs versus domestic policy and different types of users. And then there's different, you know, dimensions. Is it the tone versus, or is it the bias? And so different ones are better and worse at different things.

• 1:02:07Craig S. Smith?

  Yeah. Okay. Well, I think we're over an hour. Let's leave it there. If someone wants to follow up and learn more, where should they go and who should they talk to?

• 1:02:21Speaker 00

  Yeah, I mean, feel free to, so first of all, definitely can reach out to me. So you can find me on LinkedIn at Robbie Goldfarb or on X under the same name. And, you know, feel free to go to our website, byforum.com. We have a, there's a link there where you can request a call and we will get back to you ASAP. Would love to have conversations with, whether you're developing AI systems and you'd want to work together, if you're an expert and interested in being part of the network, or just someone who wants to nerd out on some of this stuff. We would love to chat.

• 1:03:00Craig S. Smith?

  Yeah. And give me that URL, spell that out for me.

• 1:03:05Speaker 00

  Yeah, B Y F O R U M, byforum.com.

• 1:03:11Craig S. Smith?

  Okay. Great, Robbie. It was really fascinating. And yeah, let's have a conversation again in, you know, a few few months or in a year or so, so you see how it's working. Yeah.

• 1:03:26Speaker 00

  I would love to. Yeah, no, really, this is a lot of fun. Appreciate it, Craig.

• 1:03:28Craig S. Smith?

  Yeah.