#290 Joel Hron: How Thomson Reuters is Approaching The Next Era of AI

This episode is sponsored by AGNTCY. Unlock agents at scale with an open Internet of Agents. Visit https://agntcy.org/ and add your support. Joel Hron, Chief Technology Officer at Thomson Reuters, joins Eye on AI to unpack the future of agentic systems and what it takes to build them responsibly at enterprise scale. We dive into the shift from prompt-based AI to true agentic workflows capable of planning, reasoning, and executing complex tasks. Joel breaks down how Thomson Reuters is deploying generative AI across law, tax, risk, and compliance, while keeping human experts in the loop to ensure trust and accuracy in high-stakes domains. Topics include: - What separates agentic AI from simple prompt-based tools - How "agency dials" (autonomy, tools, memory) change system behavior - Infrastructure and architecture required for multi-agent collaboration - Why human verification and user experience design are essential for trust - The future of coding, engineering skills, and AI adoption inside enterprises If you want to understand how a 170-year-old company is reinventing itself with AI — and what's next for agentic systems in business and knowledge work — this conversation is a must-listen. Stay Updated: Craig Smith on X:https://x.com/craigssEye on A. I. on X: https://x.com/EyeOn_AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The ecosystem of tools and capabilities is pretty nascent. And it also, even nine months ago, has changed quite substantially to what it is today. Like what you thought you wanted out of an agent, you know, nine months ago is very different than what you want today. We're trying to automate a tax return preparation. Input documents would be like W2s and 1099s and these kind of things. Output would be like a tax return that's ready to file with the IRS. And, you know, the tools that would need to be used through that process would include tools like, you know, OCR. It would include tools like some document extraction, like entity extraction APIs. In our case, what we've been able to do, we have a tax engine as one of like our core products in the tax space.

• 0:43Speaker 00

  And so we've been able to expose that tax engine as a tool for the agent to go use and run tax calculations and validate tax calculations and inspect errors. And so in many cases, we, you know, those tools that we're building are built on top of our existing products and applications.

• 1:01Craig S. Smith?

  Build the future of multi-agent software with AGENCY. That's A-G-N-T-C-Y. Now an open source Linux Foundation project, AGENCY is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on AGENCY, including robust identity and access management that ensures every agent is authenticated and trusted before interacting.

• 1:42Craig S. Smith?

  AGENCY also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next-generation AGENCY.

• 2:10Craig S. Smith?

  AI infrastructure together. Agency is dropping code, specs and services, no strings attached. Visit agntcy.org to contribute. That's A G N T C Y dot O R G.

• 2:27Speaker 00

  Nice to meet you. Appreciate you having me on today, Craig. I um, uh, so my name's Joel Ron. I'm the Chief Technology Officer at Thomson Reuters today. I joined Thomson Reuters, uh, about three years ago via acquisition. I was the, uh, Chief Technology Officer of an AI company. Uh, that Thomson Reuters acquired and, um, uh, shortly thereafter, I led our AI Research and Development Lab. Uh, which we call TR Labs. It's a group of about two hundred and fifty, uh, research scientists and engineers, really focused on more, I would say, applied AI research.

• 3:06Speaker 00

  Um, and, uh, probably close to a year ago now took over as CTO and, and as Thomson Reuters. Uh, we are, we're really born as a content company more than anything, but I would say, um. You know, over the years, we've been really invested in not just like delivering content, but also selling the experiences of how. Uh, professionals interact with that content. And a lot of that comes down to search and retrieval and, and these sorts of like technical disciplines as well. Um, uh, and we do that across a number of areas, like the, the legal industry certainly is one of our biggest.

• 3:47Speaker 00

  Uh, but also within tax compliance, risk, uh, maybe most well-known for the news and media side of our business as well. Um, but, uh, but we really solve that problem across a wide spectrum of industry today.

• 4:00Craig S. Smith?

  Yeah. And what was the startup that was purchased?

• 4:04Speaker 00

  Yeah, so we were a document understanding company. We really were focused on the legal sector in terms of understanding and interpreting legal documents. And that's what Thomson Reuters had had acquired us for. In particular, one of the things that we had done, and this sort of predates generative AI, which has sort of like superseded a lot of a lot of those

• 4:30Speaker 00

  supervised AI models that were being built back in that time. But we had built an ML Ops platform and an annotation platform that was really tailored to to lawyers in particular, to, you know, allow them to develop ontologies of certain types of contracts and curate data associated with those contracts that would be required for training those models and evaluating them and deploying them. And we built a really good proprietary operational platform for for that type of thing. And Thomson Reuters acquired us to really scale that into our practical law editorial teams

• 5:09Speaker 00

  to take, you know, the practical law editors who have experience across a wide spectrum of legal practice areas and have them do that kind of work across different document types in different practice areas.

• 5:23Craig S. Smith?

  Yeah, yeah. Can you say the name of the company? I talked to a lot of companies in that space and the supervised learning.

• 5:31Speaker 00

  Yeah, it was called Thoughttrace at the time. And we changed the name when we were required to document understanding as a product name. But prior to acquisition, it was called Thoughttrace.

• 5:43Craig S. Smith?

  Right. Well, congratulations. Jumping to CTO, I'm sure that's a pretty, pretty exciting thing for a company.

• 5:52Speaker 00

  Yeah, it's it's been a big change over the last few years, for sure.

• 5:56Craig S. Smith?

  Yeah. So we're going to talk about what agentic AI really looks like in production.

• 6:04Craig S. Smith?

  And one of the things I'm interested in is what separates true agentic systems from prompt based assistance.

• 6:14Craig S. Smith?

  Uh, how AI is planning reasoning and executing multi-step workflows autonomously? And what it takes to build that responsibly. It's a mouthful, but.

• 6:26Speaker 00

  Yeah.

• 6:27Craig S. Smith?

  If you could just talk about that, I'm sure listeners would be interested.

• 6:31Speaker 00

  Well, yeah, I'm sure and a lot of threads that we can pull on. Uh, throughout that that question is pretty broad. I mean, I'd say maybe the. The the best place to start would be defining like what an agentic system is. And I think. Um, when I think about agency, I think about it on a spectrum. It's not a binary. It is either agentic or not, but it's really. A spectrum of agency and what agency really means in this case is. Uh, uh, the level of autonomy that you give the system to sort of plan certain actions. Um, the the tools that you give that AI system to to then take an action.

• 7:16Speaker 00

  Um, and the ability to coordinate like memory across. Either one sequence of actions or maybe across like a full history of a user's. You know, interaction with the system. So like those are kind of three. If you can think about it, agency dials that you can move up or down to give. Uh, an overall AI system more or less agency. So if you start on the very low end of agency. You know, this might, this might be like a traditional, like you said, prompt based rag system like. It it's got no agency to, uh, to write its own prompt. It's got no agency to select what data sources.

• 7:58Speaker 00

  It uses to retrieve information like it literally just takes an input and processes. A discrete step or a sequence of steps even. Uh, to to go, um, to go execute it. Whereas if you move to the like high agency. Uh, these systems are more goal oriented rather than like a discrete number of steps. They they have a goal in mind and they have the ability to sort of plan. Uh, a sequence of steps that they think will help them achieve the goal. They have the ability to go execute. Tools to, you know, along that trajectory of steps. And then they have the ability to react, you know, through the middle of that as they see they've uncovered more information. It might reinform.

• 8:42Speaker 00

  The plan and maybe they replan additional steps. And so that would be like a higher level of agency. And I think depending on the the the capability that you're trying to build into the application, you can move that agency dial up or down. In some cases, like particularly in in certain aspects of law, like people appreciate high precision, repeatability, consistency. And so you might want to dial that down to be like less agentic, if you will. But in other areas like legal research, for instance, this is a this is a discovery problem. It's often very open ended and you want to follow the breadcrumb trails. And

• 9:24Speaker 00

  so you may want to dial up agency in that case to to give the model the freedom to go out and sort of learn and explore multiple paths.

• 9:34Craig S. Smith?

  Yeah, although the the dialing up and down, that's done at the creation of the agent. It's or are there agents today that you can, you know, there that you can set parameters after the agent's built?

• 9:52Speaker 00

  Yeah, so it's a good question. I think in general, yes, it's done at the time of creation or how you design or architect the agent. I think there are some good corollaries of how this can be done in the user experience itself, though. Like, if you look at an application like a coding application like Cursor, for instance, I can constrain the agency of of Cursor by constraining the context over which it works. So, for instance, I can highlight just a line of code that I want it to explain to me, or I can highlight a whole function or maybe a file, or I can give it access to the whole repo and tell it to just like

• 10:35Speaker 00

  go to town. And so that would be a way to on the fly, control the the agency dial of context. And thereby, you give the the AI system more or less freedom to go out and create. And and that's really a function, you know, in the coding example, is a function of Uh, the use case and the user and are they working on a mature code base that has hundreds of files and you know, tens of thousands of lines of code, or are they building something new and just trying to vibe their way through it? Like, that would determine whether they want to go high or low agency in that, in that case. And, uh, and I think, I think those systems and those user experiences kind of lend themselves to allow the user to adapt

• 11:19Speaker 00

  it to what they need at that point in time.

• 11:22Craig S. Smith?

  Yeah. Uh, and, and these are L L M based agents that we're talking about. Uh, and they're making tool calls and giving instructions to various tools that then go out into the world and execute, uh, and then, you know, come back with, with information that the L L M either returns to the user or does, passes on to another agent. Uh, which, uh,

• 12:00Craig S. Smith?

  at Thomson Reuters, do you guys have a preference for which models you're using to build agents on?

• 12:10Speaker 00

  Yeah, it's a good question. So, um, they are L L M based in general and in particular, like the, the orchestrator, like sort of the, the master agent that's sort of planning and reacting along the way, uh, is, is one of these, um, uh, what I, what I would say frontier models in, in most cases.

• 12:32Craig S. Smith?

  Yep.

• 12:33Speaker 00

  We evaluate that pretty consistently. Uh, you know, we, we're close partners with OpenAI, Anthropic, Google in a number of areas, and we use all of those models and more in, in certain situations. Um, the, uh, the tools themselves that the, uh, agent system is using may or may not be large language model based. In some cases, they could be a web search. In some cases, it could be another L L M, maybe a smaller L L M that's fine tuned to a particular task. It could be some other type of model entirely. So the, the tools themselves can be a variety of things. Uh, generally at the orchestration or planning layer, there's, there's kind of like a master model that's bringing it all together.

• 13:18Speaker 00

  And again, depending on the use case that we're working with, we've we've found that we've had good success working across all three model providers and and they each, you know, tend to be good at at particular things and we pretty continuously evaluate that as well.

• 13:36Craig S. Smith?

  And do you build your own agent building platform or do you use,

• 13:46Craig S. Smith?

  you know, commercial

• 13:50Craig S. Smith?

  solutions? I just, I've been getting, I've been having call after call with with companies that have agent building platforms and it's very difficult as a journalist. They all kind of sound the same to me, but

• 14:09Speaker 00

  Yeah.

• 14:10Craig S. Smith?

  Yeah. But but everyone I talk to says they have the answer and I'm curious who company like Thomson Reuters.

• 14:19Speaker 00

  It's been a bit tricky because, you know, this development pattern or design pattern for AI has really only existed for nine months, you know, maybe 12 months at best, but

• 14:34Speaker 00

  so I would say that the ecosystem of tools and capabilities is pretty nascent, to be honest with you. And and it also from even nine months ago has changed quite substantially to what it is today. Like what you thought you wanted out of an agent. You know, nine months ago is very different than what you want today. So we have used, you know, particularly like MCP, for instance, is something that that we leverage. Like, I think there's a lot of good industry gravity around around that. Other sort of frameworks, like Langchain has their own, Crew.ai, etc. Have their own. I think they're all good at certain things. We found that like building some of those components ourselves has actually

• 15:18Speaker 00

  made things a little bit easier for us that, you know, 12 months from now, when the space is a bit more mature and some of these design patterns are a bit more stable, you know, maybe we would go to a third party. But right now, we've kind of built a lot of this ourselves for the moment with, like I said, the MCP and other examples where we we do leverage industry standard.

• 15:46Speaker 00

  From a, from a tool building standpoint, um, you know, one of the interesting things that we've found as we've done this development is that like the, you know, the same way you would design a UI for a human user to use it, or you would design an API for like a computer to computer interaction. Like designing tools is very much a, a, a, a thought process of how is an agent going to interact with this? And what is the sort of modalities of information that it is going to provide? And how does information need to be presented back to it in, in terms of a tool output?

• 16:27Speaker 00

  So that the reasoning model can sort of act or react on it, you know, most appropriately. So there's a whole nother like level of kind of, um, design consideration for how you build tools that allow agents to execute and exercise them, uh, more efficiently and more effectively. And again, that's a very nascent space as well that we're, we're still learning every day on.

• 16:51Craig S. Smith?

  I see. So you're building your own tools. You're not, uh, sending it out to, uh, to call, yeah.

• 16:59Speaker 00

  In some cases, you know, like if you were gonna go, um, do a web search, for instance, there's a lot of good third parties that provide like web search tools that, that will sort of leverage out of the box.

• 17:10Craig S. Smith?

  Yeah.

• 17:10Speaker 00

  Um, in some cases, like a tool could be accessing our content systems. And so in that case, we will, uh, build a tool that leverages our search APIs in the, in the appropriate way to, to search and retrieve content from our systems. In other cases, it may be like taking action in the application itself. Uh, so for instance, uh, and a good tax example would be if we're trying to automate a tax return preparation. Like in, input documents would be like W2s and 1099s and these kinds of things. Output would be like a tax return that's ready to file with IRS.

• 17:49Speaker 00

  And, you know, the tools that would need to be used through that process, uh, would include tools like, you know, OCR. It would include tools like some document extraction, like entity extraction. APIs. In our case, what we've been able to do, we have a tax engine as one of like our core products in the tax space. And so we've been able to expose that tax engine as a tool for the agent to go use and run tax calculations and validate tax calculations and inspect errors. And so in many cases, we, you know, those, those tools that, that we're building are built on top of our existing products and applications that we've been maintaining for, for, for many years now.

• 18:33Craig S. Smith?

  Yeah. You know, Thomson Reuters is 170 years old or something.

• 18:45Craig S. Smith?

  You know, how did companies move from content or software to delivering intelligent systems? I mean, that to me is, uh, it sounds like an enormous task because it's not simply building the systems, it's getting the enterprise to, to change behavior. And, and this is something these guys that are building platforms for agentic workflows that I've, I've been talking to them about. If I were young and, uh, and had some smart friends, I would pick an industry and

• 19:30Craig S. Smith?

  use one of these platforms to, to field, you know, agent after agent. And you, you could basically challenge the behemoths with a company of agents, couldn't you?

• 19:47Speaker 00

  Yeah. And I mean, look, I mean, competition in the space today is fierce and it's never been easier to, to kind of get started, as you say, uh, like developer tools and things like that, you know, make the, the time to ship something that's meaningful. Uh, you know, orders of magnitude shorter than what it used to be. I think, I think for us. Though content and not just content, but the the human experts that generate that content. We see as a real sustainable and durable advantage for us.

• 20:26Speaker 00

  Like, it's one thing to just build a um a superficial agent that goes out and drafts an NDA. But, you know, quite honestly, I could just go use ChatGPT or Claude or Gemini out of the box to do that quite well. And it's seen a bunch of the internet and is pretty good at that. Uh, but I think what really separates um us i- in that regard is the fact that we are not just using those models to generate this, but we're using those models with the context of what practical law, for instance, says about drafting an NDA in certain jurisdictions, uh, in certain industries.

• 21:07Speaker 00

  And we've had lawyers for many, many decades now who literally draft know-how, draft sample documents, draft how-to articles on how to do this well, what pitfalls to avoid, et cetera. And that content goes into grounding and steering the agents to to doing that task better.

• 21:26Craig S. Smith?

  Hmm.

• 21:26Speaker 00

  And then the second thing I mentioned, which is not just the content, but the the people who have drafted that content for many years. They are critical in the alignment and evaluation of our AI systems as we build them and as we operate them. And uh, their ability to provide feedback and steer and uh, uh, continuous critique of what's working well, what's not working well. Make sure that like we get the best performance that we possibly can out of these systems and that we're continuing to keep them up to date with current law and and current industry standards and and practices. So I think, you know, it's very easy to build a very simple AI system today.

• 22:10Speaker 00

  I think it's very hard to actually ground it in, you know, a a true and verified information and also to provide the the transparency to end users that you that you're doing that and they can they can trust the those outputs in that way.

• 22:26Craig S. Smith?

  Yeah, uh, but on the other question of how do you? How do you get the organization uh to adopt?

• 22:36Craig S. Smith?

  Agentic workflows, or is that easier than I imagine?

• 22:42Speaker 00

  I wouldn't say it's um, I wouldn't say it's easier than you imagine. I mean, it's a hundred and seventy year old organization, as you said. Uh, years of, of, of legacy uh, behind us that certainly we're very proud of and for good reason. I think what we've done though is one um,

• 23:06Speaker 00

  demonstrated real success early on, even for those who have like been in this world for a long time and may like be uh, slow to change. Like when they see this happen, it's almost undeniable like that this is a better path, right? So part of it has just been like really leaning in and showing success early, and I think that brings a lot of gravity. Uh, the other thing we've done is we've been quite acquisitive over the last couple of years, particularly in, in early stage companies uh, like CaseText, like Materia, who we've uh, acquired SafeSign. And I think bringing new talent into the organization who has kind of a fresh, fresh perspective on

• 23:48Speaker 00

  this problem and partnering them with people who've done it for a long time, like creates a really good outcome at the end of the day. And I think, I think we, we've done that quite well and quite aggressively and certainly will continue to do it. And then thirdly, it's just been organically uh, upskilling and, and integrating uh, talent into the organization who has experience in these kind of skills. I think uh, that's been something that we have been super focused on for the last two years. I mentioned I came from the TR Labs, like that organization in the last two years has more than doubled in size. We were about a hundred and twenty people. Uh, two years ago, we're two hundred and fifty plus now, uh, and continuing to grow.

• 24:32Speaker 00

  And I, so I think our, our talent composition of people who have, who have experience building and architecting AI uh, applications uh, has grown as a percentage of our overall engineering workforce. And as you get more and more of this talent into the organization, uh, I think it, it becomes more and more part of your DNA and, and that's certainly something that has happened over the last two years and, and I think will continue to happen over the next several.

• 25:00Craig S. Smith?

  Yeah, and then there's, I mean, you're talking about building the products, right? The, the, the, the part of the organizations that's building these agentic products for,

• 25:13Craig S. Smith?

  for, for use by other organizations. But within Thomson Reuters, I mean, it's a very large organization. Is there an eager uptake of agents? Or is that not really your responsibility to build agentic products or workflows for Thomson Reuters more broadly to use?

• 25:40Speaker 00

  It is and it isn't. So, so my team also owns our internal AI platform, which we call Open Arena. And, and I would say that has been one of the, like, we, we, we built and shipped that within like two months of ChatGPT, you know, coming on to the, to the market, you know, two and a half years ago or so. So we were very early in terms of building our own platform and driving experimentation into the company. Uh, and I would say we have huge adoption. So I, I think our month, our, our weekly active users on Open Arena today are more than 16,000. It's probably, probably more than 60% of the company who uses that on a weekly basis.

• 26:21Speaker 00

  And so, uh, we've seen really strong adoption of just AI in terms of day to day usage. And Open Arena is not the only tool. We're, we're very eager and, and, and leaned in towards, um, you know, using third party tools in sales or in marketing or, uh, um, in other functions of the business where there's best of breed kind of domain tools. But we also have this platform that, that has just general usage, uh, that it enables. Um, and I think we've seen a lot of uptick. As you think about agents though, that's really like what I would say the next level of productivity from an internal optimization standpoint.

• 27:03Speaker 00

  And I think we are still on the path of figuring out how to do that best. So today, if you think about where AI was. A year ago, it was all about AI augmenting or being helpful to a human. You would go in there and you would like say, hey, help me rewrite this email and it would like rewrite it for you. It was helpful. Well, with agents, maybe now you're saying, go write this email and send it, you know? Or maybe you're not even telling it to do it. It just knows that it should go write that email and send it. And so there's, there's this level of sort of autonomy and automation that you're shifting to AI.

• 27:39Speaker 00

  And, and that requires, I think, more technical depth and certainly more uh attention from a, from a uh uh uh a process standpoint about where you want to do that. And again, what level of agency you want to give the system to do that? And those are areas that we're building around today. I can tell you, like for my teams on the software development side, you know, we're not just sitting there letting AI write all of our code, but there are certain parts of the coding process, maybe around test development or accessibility or other like uh aspects of the development process where we can achieve higher levels of automation. And, and we're trying to drive that forward.

• 28:22Craig S. Smith?

  Yeah, uh, and, and in building products or, or uh agentic AI for um production, uh what skills, you were saying that you're, you're acquiring new skills in, in the workforce or what skills are you looking for? Cause it's another question I get from a lot of young uh software engineers, like, you know, am I obsolete? What skills should I be developing to keep myself relevant?

• 28:57Speaker 00

  Yeah, I actually, you know, I, I know a lot of people are concerned about that and what it'll have uh impacts to the workforce and. And I'm I'm sure there will be a workforce transition. One of the things I see is that AI amplifies the best engineers orders of magnitude more than it does the rest of the population like. So for great engineers, it makes them like ten X greater than they were before, right? And I think, you know, and that that doesn't mean that they are a scientist. It doesn't mean that they've like trained and deployed AI models for their whole career. They could be, you know, a standard backend software engineer.

• 29:38Speaker 00

  Uh, so, so I don't think there's a specific, like necessarily AI skill that's necessary to be really good at this. Certainly you need some of those. You need some core engineers, but for all of those people. I think the most important skill is actually adaptability. Um, the, the pace at which technology is changing right now. We talked about like agentic patterns and frameworks, like there's a new one every week. And how do you know whether you should use it or not, or whether this is better? Like, you need to be very adaptable in this environment. You need to be comfortable, like picking up something new and learning it and trying it and failing it and moving on.

• 30:18Speaker 00

  And that level of adaptability to just pick up anything and get a very quick sense for it and be able to apply it if it makes sense. Uh, is a really important skill. I think the second skill that's super important in engineers uh, is understanding the lack of determinism in AI systems. I think traditionally for engineers, they've thought about like, here are my requirements, like when I click this button, like this screen better show up and it better look like this. Whereas AI systems are not deterministic in this way. And, and so building for non-determinism requires you to, to think about like, well, how do I eval something? And how do I like,

• 31:00Speaker 00

  build validation into the application? Or how do I build rubrics? That support the validation process that needs to happen? That are stochastic and, and not deterministic. And, um, and I think that's also a skill that some people have, and, and certainly I think any engineer could develop, but is an important thing for, for engineers to develop, to, to get comfortable in building AI systems like this.

• 31:27Craig S. Smith?

  Yeah. So, so you think, um,

• 31:31Craig S. Smith?

  Coding is still an important thing for kids to study. Because in order to become a software engineer, you need to understand the structure of software and understand the structure of software, you really need to be able to code.

• 31:47Speaker 00

  I I think I think it is exceptionally helpful and I don't, like I said, I think the best engineers are even better than they were before. Uh, with these tools and I think that will continue to be true as we go forward. I think the the the population of people who know how to code because of these tools will also grow. So I think there's tons of people out there who didn't go get a computer science degree, but will go learn how to code uh in a way that that they become great engineers themselves, right? And uh and I I I think to do that well, you don't just sit there and like

• 32:27Speaker 00

  let the AI model run wild on everything. I think anybody that's done that and like written an application knows that like they can go off the rails pretty quickly and and so understanding logically, like how do the applications uh get built, like what does service architectures look like, like how do I keep this thing on a leash as I walk it through this process like? Those are how I think really good engineers get a lot of efficiency out of these tools. So I I don't think the the engineering discipline is going anywhere. Uh, like I said before, I think there are certain aspects of the discipline that lend themselves to really

• 33:06Speaker 00

  uh full or substantial automation in a lot of cases, but the core aspects of engineering, I think, will will continue for a while.

• 33:13Craig S. Smith?

  Yeah, and um I've also been asked by software engineers that don't are not AI uh engineers. Uh, you know, how how should they, what should they study to become an AI engineer, but would like in this agentic age that we're entering? The AI uh is uh accessed through an API call. I mean, it's the building an agentic system is really an engineering problem, not an AI problem. Is am I wrong on that?

• 33:50Speaker 00

  No, I think you're right. It's an engineering problem. What makes it an AI problem is this aspect of non-determinism, like I mentioned, like I think. M L engineers historically would have been trained and well versed on like how to build a good testing and validation sets and how to look at things statistically and say am I hill climbing this thing to get better or worse or on what dimension is it getting better or worse? Like they think statistically in that sense about the system. And I think that's what every engineer will need to learn how to do to be able to build A I systems well. Um, I think today we call that A I engineering is probably

• 34:34Speaker 00

  the same analogy to to fifteen years ago when we thought cloud engineering was like a separate thing. Now it's just engineering. And I think probably the same will happen with A I engineering today is somewhat of a unique skill, but I think you know, five, ten years from now, it'll it'll need to be ubiquitous for for all engineers.

• 34:53Craig S. Smith?

  Yeah. Um, why uh is infrastructure and not just models uh important uh going forward in A I?

• 35:07Speaker 00

  Yeah, well, you know, I think I think I would separate that maybe for a company like ours or like the hyperscalers. You know, infrastructure for the hyperscalers is like

• 35:21Speaker 00

  a fuel for them. This is like if they're an electricity provider, like a utility provider, like this is their power plant like that. That is what um, uh, right or wrong, I think is necessary to create more advanced uh reasoning models, more advanced versions of these models that keep coming out, you know, month on month. And so I think that's where you see a lot of this infrastructure investment going. It's really around um, uh, training infrastructure. But I think what you've seen more recently is this concept of like test time compute where the inference in infrastructure actually gets quite large very quickly as you start to expand the cycles of inference that happen on any individual call. Um.

• 36:07Speaker 00

  And and I think this is another thing that's driving a lot of the big infern or infrastructure uh spend that that's going on uh within the industry. I think what you've seen over the course of the last, you know, several years is like uh the the both the cost to to run and the speed at which these models uh perform has just precipitously dropped like orders of magnitude from two years ago. Who knows whether that, you know, curve will continue on on that particular trend or not. Um but again, what what what that really allows you to do is not to necessarily deliver the same output for a lower cost.

• 36:49Speaker 00

  What allows you to do is spend more cycles running the model so that you get a better answer, and that's really what this test time compute kind of concept speaks to uh is being able to to exercise the model over more cycles and and get the better answers. And certainly, as we look at areas like law or tax or otherwise, that that's the most important thing for us, like our users demand accuracy, uh they demand uh confidence, uh transparency, these kind of things. And so any anything that we can do to to to drive accuracy and and confidence up in the answers that our AI systems are given is is what we're aimed to do.

• 37:31Craig S. Smith?

  Yeah, and what in in this case, when you when you when you're talking about infrastructure, you're not talking about uh data centers and and chipsets and things like that. You're talking about architecture, yeah, in effect, right? How you stack uh, you know, orchestration layers and proprietary tool chains and uh domain-specific log logic.

• 37:59Craig S. Smith?

  Is is that right?

• 38:00Speaker 00

  Well, yeah, I'm sorry. So when I was talking about test time compute and and the the training side, I was talking about core like chip infrastructure. But yeah, there's another side of infrastructure, I guess, which you which you would call architecture, which is, you know, how does the agent reason? How does it get access to tools? How does it maintain memory? How does it know when it's done? Um, which is a hard problem. Uh, how does it uh verify information like like the citations it provides? For instance, how does it? How does it verify that if the AI generated this statement? Where did that statement actually come from in its memory?

• 38:41Speaker 00

  So all these are hard problems that speak to the architecture of the AI agentic system itself. Um, in terms of in terms of how you build it, how you set up prompts, how you set up memory, these kind of things. And um, and again, I think that that's a super nascent space right now. That's probably where we've spent the majority of our time over the last nine months, building and designing these systems. Um, not on the core model itself. It's it's really on all of these kind of surrounding bits of infrastructure or architecture that uh that that really drive model performance.

• 39:18Craig S. Smith?

  But if you're working with a probabilistic uh foundation model as sort of the core reasoning uh logic engine, uh can you ever satisfy that demand for uh accuracy? Or or do you just uh keep narrowing the the uh error?

• 39:46Speaker 00

  At this point, it's a great question. Uh, I think it in an auto regressive system like you will never satisfy to uh.

• 39:58Speaker 00

  To a hundred percent accuracy, like it it's just, I I don't think that it that it's possible. Uh, and so the name of the game is to con- continue to narrow the the error bars uh and to design good evaluation systems and to have good human in the loop interaction with those evaluation systems to uh make sure that that you're you're doing this well. I think the other thing that's super important is this um uh user experience design of how the user interacts with the agent itself. Um, and and Andre Karpathy gave a a a talk on this at Y Combinator, I thought was really good. And he spoke about um the the

• 40:42Speaker 00

  the speed of the loop of the uh generation and verification. So if you think about the agent generating something and the human verifying it, like the faster and tighter you can make that loop, the better the overall system is. And the user experience uh that that you build in in an application needs to facilitate a fast verification. And so, for instance, the way that we do that in legal research is uh, you know, our deep research system uh provides clear like descriptions of what it's doing, what cases it's searching, what it found out of this case, what it's doing next and why. And a user can follow that if they want.

• 41:22Speaker 00

  And at the end of a a a research task, you know, we provide deep links into Westlaw. We provide key site flags as to the cases we're saying. So that experience itself is what drives confidence in whether the answer is correct or not, or what aspects of the answer might be wrong that they might want to do further investigation into. So the goal is to get to a hundred percent. But like I said, I don't think you'll ever get there. And and the the job is to build uh as close to a hundred percent as you can, but to build user experiences that drive easy and fast verification by humans in into the process.

• 42:02Speaker 00

  And uh, and so I think both of those things together is what ultimately builds builds user confidence in in the system.

• 42:10Craig S. Smith?

  And in your experience, uh, law firms, for example, do they have paralegals going through and verifying the?

• 42:21Craig S. Smith?

  The output, checking the references. There's this thing called automation bias, you know. That, you know, something works.

• 42:34Craig S. Smith?

  Once it works, twice it works, 10 times, pretty soon you're you're just taking it for granted that it works.

• 42:40Speaker 00

  Yeah, yeah, we certainly, uh. When we train users to use our application, this is how we treat me. We train them on how to verify, how to how to look at something quickly and to know whether it is right or wrong. So for instance, I mentioned. Uh citations in our answers for deep research. So like a a a big thing that's been in the news is around like. Uh AI systems that hallucinate cases, like they make up court cases, right? So one of the very like easy things that we do in our UX is like we have the richest repository of case law in the world, right? If we get an answer and we're able to resolve it to a case.

• 43:24Speaker 00

  Like we create a blue deep link to that case in Westlaw. You can very plainly see that this is a real case because it's tied to a real case that exists in Westlaw. If it's a case that doesn't have a blue link, like that that would be a concerning thing for a lawyer and they should they should see that and they should do something about it, right? So there's very, it's a very simple like UX thing, for instance, that. Uh helps users to hone in on where they need to spend their time looking versus like maybe be more trustworthy of of things. So these are kind of the kind of ways that we try to train users in in terms of using our system.

• 44:03Speaker 00

  Um, I certainly hope that it, you know, in all cases, users are applying some level of verification. To the process. I don't think our intention is that people are just like sitting there clicking the button and sending to the to the. Uh to the judge uh at this point.

• 44:20Craig S. Smith?

  Yeah, uh, yeah, that's interesting. Um, so, uh. Looking ahead, uh, where are multi-agent, uh?

• 44:34Craig S. Smith?

  Where is all multi-agent collaboration going? I mean, you, you, well, first of all, just on that issue of, of, you know, the

• 44:45Craig S. Smith?

  providing as much accuracy as possible. I, I assume then you have sub L L Ms that are doing what a paralegal would do, look up the case law, verify that it's a real case and, and

• 45:01Speaker 00

  These are all, these are all tools, again, that the agent use. Like, there's a, a case reader, there's like a key site investigator, there's like a verification. Uh, like, so all of these, again, are, are tools that the agent is able to use to, you know, assess its answer and the correctness of it.

• 45:22Craig S. Smith?

  Yeah. Um, so on, on multi-agent collaboration, I mean, uh, as I said at the beginning, people are talking about building organizations that are essentially agentic with, uh, you know, thousands of agents doing, uh, each step and then passing work back and forth among themselves. Uh, where, and then structured memory and user feedback loops. Uh, where do you see this going? Uh, and, and how do you see knowledge work of the future more broadly?

• 46:02Speaker 00

  Yeah. Again, I think the analogy to, to APIs is a good one. Like, in the same way, I build APIs so that different, you know, applications can connect to them, connect to each other. Like, I think, uh, we will build applications for agents to be able to, to use and talk to each other. I think the part that's somewhat unsolved to me at the moment, particularly as you think about like multi-party agent systems. So, you know, vendor A, vendor B, vendor C, is this concept of memory. And because at this point, there's sort of a single orchestrator of of flow. And generally, that orchestrator owns the memory across.

• 46:46Speaker 00

  And so, you know, as you think about that, like, context is really what drives the performance of these systems. Like, as they run and as they build memory, that is then reinjected into the context. And the model becomes more aware of what's going on and what it needs to do differently and how to adapt and this kind of thing. And so, memory becomes kind of the most critical, not the most, but one of the most critical, like, parts of executing the flow correctly. And so, who owns that memory? How does that memory get shared between these agents? Is something that I don't think is quite well solved yet. Again, like, self-contained within Thomson Reuters and the agent we build, like, we can solve that problem.

• 47:31Speaker 00

  But but if you start, like, bringing in vendor A, B, C, D into that equation, I think that still needs to be figured out. I think MCP is a good protocol in terms of, like, the communication pattern between them. But memory is is another thing. And certainly, other vendors have tried to, like, throw their hat into that ring, like, A to A and things like that, to help facilitate that. But I think that pattern will evolve in the same way that, like, API standards evolved for how applications communicate with each other. I think you'll see similar patterns with agents. Again, I think, I don't think that this verification loop is going anywhere anytime soon, though. I think the

• 48:18Speaker 00

  this idea that you just have this, like, autonomous systems of agents that that is going out and doing all these things without any human ever seeing anything is probably far off. Like, if you look at self-driving, like, self-driving was really good a decade ago, but like, I was I was in the first Waymo in my life just a few months ago, you know, so like it's taken a decade or more for people to get like really to the point of comfort of this. And I think the same will be for AI that like the the human verification will be an important part of like the software experience. For quite a while, I think, until we develop a level of comfort and consistency with this level of automation.

• 48:58Speaker 00

  Uh, for for some use cases, again, for others where they're just maybe lower risk and. Uh, um, there's less, less, um, maybe criticality, maybe you'll see those get automated fully more quickly, but. But for many things that really matter, I think this this human verification loop will continue to be an important part of the software design experience.

• 49:20Craig S. Smith?

  And and, uh, I mean, that's interesting about context and memory because I just had a conversation with a guy. Talking about the problem with transformers, uh, that the larger the context gets, the slower the system gets and the more expensive it gets and.

• 49:41Craig S. Smith?

  How do you guys handle, um, context and memory? Do you, do you, do you, uh? Use like a scratch pad where you have. Everything that's happened in a kind of a knowledge base that then the orchestrating agent can refer to. Or is it being held in? Uh, the context of the model?

• 50:09Speaker 00

  Yeah, I I would say, um, this is also an area that is under development and. Kind of continuously changing as the models change. I think if you were to. I guess the short answer to your question is, I I think some external system is generally the best approach right now, whether that's a scratch pad or. Whether it's actually a database or an index or a graph or something like this, some external source of knowledge. Uh, that the model can can go references is, um, is generally better than trying to like keep it all in the context window. I think. Going deeper on that point, if you look back a year or or a year and a half ago,

• 50:53Speaker 00

  um, when this was still quite early, there was sort of like everybody was, I I remember I was on several of these interviews. They were like, what? What's sort of the next big thing with AI? And a lot of people were talking about context window and like,

• 51:05Craig S. Smith?

  Yeah.

• 51:05Speaker 00

  At that point, you recall, context windows had grown from like one hundred and twenty eight K or even like like thirty two K. And one hundred and twenty eight K was like a big deal. And then it got to like a million or two million. And everybody thought context windows are going to go infinite and and um, not a lot of people were talking about agents at that point. I I I remember having a couple of conversations about agents and and how profound they could be. And I think what's happened is agents have like not removed the need for infinite context windows. I mean, big context windows help, but there's certainly still diminishing returns with larger and larger context windows.

• 51:43Speaker 00

  And having agents in test time compute have allowed you to kind of minimize the level or or the the amount of the context window you use and store some of this information offline and organize it and let the model go retrieve what they need when they need it, like in a more targeted way. And certainly the model is better when it does that. If you fill its context window with a bunch of information that doesn't matter, then uh then it's going to get lost at some point, right? Or it's going to follow a wrong path at some point. So I think agents allow a pattern where you're not just flooding the context window all the time, but you're being more intelligent about how you use the context and and use it in a more targeted way.

• 52:29Craig S. Smith?

  Yeah, uh, I'm coming up to an hour. Is there anything I haven't brought up that you want to say to listeners that that you guys are doing?

• 52:41Speaker 00

  Yeah, i- it's a good question. I mean, I think uh we we covered a lot of ground today. I I I think one of the like most important things that we touched on a little bit early on is around um uh like these validation uh exercises that our our human experts do. So a lot of people talk about content and how important it is for AI systems, and that certainly is true. Um but I would say we have seen a tremendous shift. I mean, we have forty five hundred uh domain experts, lawyers, tax analysts, etc., that we employ at TR who have who have spent their careers like generating content for the industry. And these people like know more about their domains or their practice area of law than than maybe anybody else in the world, right?

• 53:26Speaker 00

  And so being able to leverage these people to design and build these AI systems, I think, has been one of the most like profound things for us because it's it's allowed us to sort of like take their experience and and their knowledge and their feel for uh the the certain practice area that that they work in and really align and and and train our systems to to to look, feel, behave like like those people do. And so I think that's been one of the like really most interesting shifts that we've seen over the last two years, and we're still making more of those shifts as we go through time. But uh but that's been just a critical part of our development, and I think a critical part of the the quality of the systems that we're able to build is being able to to really bring those people together with our engineering and science teams to to to build great products.

• 54:21Craig S. Smith?

  And and is you're you're talking about using those people in an RLHF, is that right, uh loop? Or are you talking about uh th- them evaluating just as sort of as uh external evaluators that can watch the model and and flag when it's wrong?

• 54:45Speaker 00

  Yeah, really all of the above. So if you think about what an evaluation is, I mean, in many ways, like it's a preference judgment that can be used in an RL routine, for instance. And so really, you can leverage this data in a number of different ways, but from evaluating our systems to writing prompts themselves because they understand how they want the thing done to to doing actual annotation or RL data for us, also to like designing rubrics for evaluation, like so one of the things that gets a lot of attention is like LLMs as a judge, where LLMs sort of assess their right answer.

• 55:27Speaker 00

  And the best LLM as a judge systems are really rubric based, which means don't just like give me the right answer and then ask the LLM, is this answer right? You need to design a rubric that says, what about this answer makes it right or wrong? It references this case, it reaches this conclusion, blah, blah, blah. And so these are rubrics then that the LLM can evaluate an answer on and say did it meet the criteria or not? So designing those rubrics also is quite a domain specialized task. And so we've been able to leverage our SMEs across across these industries for for all of these kinds of tasks. And it's been a really huge enabler for us, I would say.

• 56:09Craig S. Smith?

  Wow. And I've got to ask, do you, Bill, do you have any personal projects that you're building on the side? Do you have a Joel Horan chatbot that people can talk to or something like that? A digital twin?

• 56:27Speaker 00

  I wish I did. I'm sure there's many people on my team who wish they had that digital twin too. It'd probably be pretty helpful.

• 56:36Speaker 00

  Uh, not too much. I I've I've nothing like personal project that like I'm building for any like material use, but I I have like done personal things just to try to understand where the AI dev tool market is going. So I've built things that like like uh interact with my email, for instance, and like help me track personal finances, personal vacations, like

• 57:02Craig S. Smith?

  Yeah.

• 57:03Speaker 00

  um, these kind of things. And and I don't do that because I like want to use them. I I do it so that I understand where AI dev tools are going. For two reasons. One, I I think it's important for me, like as I think about the workforce and the talent and like how they're going to be working in the future, that helps me get a better sense of it. Um, but secondly, I think AI dev tools have been like eighteen months ahead of the rest of of the market in terms of AI, particularly because code lends itself to AI particularly well. The language is well structured. There's a lot that's publicly available.

• 57:43Speaker 00

  It's testable, like diffs work really well. Like, there's a lot of things about code that make it very conducive to AI. And so you've seen that space really uh move way faster than any other space. And so I like to look at AI dev tools as kind of like a foreshadowing of maybe what a lot of these other industries are going to look like in in a couple of years as models improve. And so I I spend a lot of time working with these tools just to try to get an insight and feel for for that as well.

• 58:14Craig S. Smith?

  Build the future of multi-agent software with Agency. That's A G N T C Y. Now an open source Linux Foundation project. Agency is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on Agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting.

• 58:55Craig S. Smith?

  Agency also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next-generation AI infrastructure together. Agency is dropping code. Specs and services, no strings attached. Visit agntcy.org to contribute. That's A-G-N-T-C-Y dot O-R-G.