Tool calling and agents

It seems like everyone is uses the term “agent” differently these days. In this episode, Chris and Daniel dig into the details of tool calling and its connection to agents. They help clarify how LLMs can “talk to” and “interact with” other systems like databases, APIs, web apps, etc. Along the way they share related learning resources. Notion http://notion.com/practicalai – Notion is a place where any team can write, plan, organize, and rediscover the joy of play. It’s a workspace designed not just for making progress, but getting inspired. Notion is for everyone — whether you’re a Fortune 500 company or freelance designer, starting a new startup or a student juggling classes and clubs. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: smolagents https://github.com/huggingface/smolagentsHugging Face agents course https://huggingface.co/learn/agents-course/unit0/introductionUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Speaker 00

  Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, and accessible to all. If you like this show, you will love The Change Log. It's news on Mondays, deep, technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at fly.io. Launch your AI apps in five minutes or less. Learn how at fly.io.

• 0:44Daniel Whitenack?

  Welcome to another Fully Connected episode of the Practical AI Podcast. In these Fully Connected episodes, Chris and I keep you updated with everything that's happening in the AI world, if we can. There's a lot. And we try to give you some, some learning resources to level up your, your machine learning and AI game. I'm Daniel Whitenack. I'm CEO at Prediction Guard, and I am joined, as always, by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris?

• 1:18Chris Benson?

  I'm doing great. I just don't know what we're going to talk about because just nothing ever happens.

• 1:22Daniel Whitenack?

  There's nothing, there's nothing happening.

• 1:24Chris Benson?

  There's never anything going on.

• 1:26Daniel Whitenack?

  Elon has done nothing. Ignore it.

• 1:29Chris Benson?

  Oh, my gosh. Elon is throwing spitwads at, at people again. He's, let's say, you know, he's been suing OpenAI, and now he's put his bid out, you know, in the last few days for OpenAI. That's.

• 1:44Daniel Whitenack?

  Yeah, we, we are, the, the article that I saw was, "We are not for sale," ChatGPT Boss says.

• 1:51Chris Benson?

  I know. Good old Sam. Sam said, we're not for sale because you know, the two of them really love each other.

• 1:58Daniel Whitenack?

  Oh, yeah, definitely.

• 1:59Chris Benson?

  Elon Musk and Sam Altman, they are best friends.

• 2:04Daniel Whitenack?

  Best friends. That's how we'll report it here.

• 2:06Chris Benson?

  That's how we're reporting here because we always look for the upside in the AI world here.

• 2:11Daniel Whitenack?

  Yeah, I, who knows the motivations behind billionaires. It's a, it's an interesting thing to watch. It certainly spices up conversations in the workday and is a, is a nice point of discussion with friends, you know, so.

• 2:28Chris Benson?

  Yeah.

• 2:29Daniel Whitenack?

  There it is. That's about how I'm taking it.

• 2:32Chris Benson?

  Yes, that's, I would agree with that. It's the spats between billionaires just doesn't quite make it to my, not being a billionaire myself, it doesn't quite make it onto my list of concerns.

• 2:46Daniel Whitenack?

  Yeah. What do you think is kind of the trajectory with players like OpenAI? You know, they're, is someone, someone was, well, the other thing that happened in the US was, this fits into maybe the other thing I was going to ask as well, although I got sidetracked in my mind because I remember that there was a Super Bowl and there was a Super Bowl commercial that OpenAI had, which, you know, it was a, it was a cool commercial. I think, I didn't know what it was going to be at first because it's just the artistic dots around, you know, forming scenes. And then I think I gradually realized that this is those little

• 3:27Daniel Whitenack?

  dots on the OpenAI ChatGPT app that, you know, expand and cool ad. And someone, someone commented to me, they spent like $14 million or whatever a Super Bowl ad costs. I don't, I forget how much it was, but I was like, that's, that's really nothing compared to what OpenAI is, is losing generally on hosting models and infrastructure. So, yeah, that's what circles back to my other question, which was, yeah, what are your, what are your thoughts on, I mean, if Elon doesn't buy OpenAI, what's, what's the future?

• 4:09Chris Benson?

  I honestly don't know. And I gotta, I gotta be honest with you. I'm not sure that I care a whole lot. I was thinking about that as we were leading into this is that, you know, I mean, There's not a protagonist here from my standpoint. There's not a side, there's not a side that I'm for or against so much. You know, you have Elon with all the, you know, the, the adventure around Elon Musk. And I say that word kind of tongue in cheek. And then OpenAI and, you know, you know, there is a kernel of truth to what Elon says when he talks about it being, you know, going from being the nonprofit with the grand

• 4:49Chris Benson?

  vision that it started out with in the early days, and then it has increasingly gone commercial and become for profit. You know, so it's another big, it's another big giant AI company, you know, like the others and stuff. It's, I'm watching it with half an eye, like everybody else in the world, but not sure. I just don't, don't know. And I'm not terribly sure I care.

• 5:13Daniel Whitenack?

  Yeah, yeah.

• 5:13Chris Benson?

  Is there somebody out there in our audience that is, that is deeply concerned about this? I would love to, I would love to hear somebody who is not Elon or Sam Altman, tell me why this is a big deal.

• 5:26Daniel Whitenack?

  Yeah, maybe we'll leave it at that. It's a, it's a good point. I, I am interested in, you know, some of the dynamics like OpenAI released their deep research.

• 5:40Chris Benson?

  Yeah.

• 5:41Daniel Whitenack?

  Product. So if you kind of look at the trajectory of what they're releasing, what they're doing, there's this deep research product, which is really geared towards this, you know, multi-step online information research type of task.

• 6:00Chris Benson?

  Yes.

• 6:00Daniel Whitenack?

  So, you know, going and looking at various, you know, trends across various sites with various data, reasoning certain information, consolidating that, you know.

• 6:13Daniel Whitenack?

  Contributing to some sort of research project. And I, I find it interesting that, you know, OpenAI introduced this. One of the dynamics I love watching is OpenAI releases the, the application level products. So like deep research. And then, so I see the blog post by Hugging Face. It was, it was like the day after. So they say yesterday, OpenAI released deep research. So this is a, a blog post that I'll, I'll link in the, in the show notes from Hugging Face. And, and basically they, they just decided to make sure that they could reproduce the functionality

• 6:55Daniel Whitenack?

  with open source code, uh, maybe some recently released models like DeepSeek models or others in 24 hours. And then they wrote the blog post and, and released it. Um, I don't know how long of a 24 hours it was, but you know, you see that dynamic happening. So you see that with deep research and then you have, you know, the open deep research thing. You see kind of the operator stuff where it's operating your, your screen, your, your browser window. Now, earlier today I was running Hugging Face Small Agents. They have a web agent, which is essentially that it spins up a browser window. It does certain tasks for you in the browser window.

• 7:38Daniel Whitenack?

  Like it, you can type a, a prompt like, hey, find the most recent episode of Practical AI, summarize the, the topic and then find, you know, seven other articles of a related topic, um, list them out in markdown format and, you know, output that, you know, so something like that where it requires this sort of agent operating over the internet. Super slick, super fun. I would definitely recommend people, if they, if they want to try that sort of thing, try the, uh, try the Small Agents web agent. But yeah, you see this kind of trend where at the application level, some of this is just.

• 8:21Daniel Whitenack?

  You know, it seems like you can't develop a moat generally there. Now, you might be able to develop a kind of moat as a company in a specific domain or a vertical or with certain knowledge or proprietary data, right? But it's very hard at that kind of general, general application level, I would say.

• 8:42Chris Benson?

  I think, you know, I keep wondering, as, you know, OpenAI had had a substantial lead and it was taking quite a period of time for a while for open source options and application, you know, level of things to come about. And we're, and we've seen that the, you know, that time interval shrink tremendously here. So, you know, and ironically, at the same time that, that Elon makes his $97 billion effort to buy OpenAI, but you can't help but wonder a little bit about what the future business model looks like, you know, to your point there about, you know, if, if it takes so, you, you never have time to create a moat, you know, if, if you're one of the

• 9:27Chris Benson?

  main players. Now, you can, you know, there's, there's certainly business models for other players to come in their industry, as you just mentioned, and create capability because that's their thing and it's not something that the big boys are going to go after. But as we've seen this interval between the commercial players and open source shrink to almost nothing, how does that, what do you think that means for the business models going forward for the Googles and the OpenAIs in the anthropics of the world?

• 9:57Daniel Whitenack?

  I mean, I think part of, part of it is maybe this sort of integration in the kind of enterprise stack. And what I mean by that is, is the kind of bundle effect that you get from something like offerings from Microsoft. So, you know, absolutely no one in the world wants to use Teams because it's absolutely terrible and I will go on record as saying that. Uh, sorry for those that work on it.

• 10:26Chris Benson?

  I have to use it. I have no choice.

• 10:27Daniel Whitenack?

  I guess, you know, you have a podcast, you have an opinion, but that's. That's my opinion, but you know, I'm also not gonna pay hundreds of thousands of dollars to Slack if I can just flip on Teams in, you know, in my Microsoft tenant and they already have all of my data and all this stuff. So the fact that they're tying in, you know, Copilot and those licenses around Copilot in an ecosystem that's already so embedded in the enterprise world, there is a very strong bundle effect there. And yeah, it's very real, right? And it doesn't mean that it's necessarily the best solution, but it is, it is a solution depending on what you're looking for, right?

• 11:11Daniel Whitenack?

  At that kind of generic Copilot level, in a case where you have, you need kind of single tenant, meaning in theory, the terms and service are, my data is not being used in, in certain ways. That kind of gets that generic case, but again, like the real business value that a company has, the way I see it is you've kind of got these generic cases where someone random is gonna want to find a Word document or paste in an email. Then you've got like the core business value, right? So a pharma company that has their most sensitive tiers of data that are the, you know, lifeblood of their company,

• 11:51Daniel Whitenack?

  or a healthcare company, or a finance company that has certain classifications or regulatory burdens around certain tiers of data. It's a whole nother thing to think about integration of those tiers of, of data into a generic system like that. Cause they're not, you know, they're a generic Copilot system for those kind of less sensitive tiers of data. There's still something that needs to be solved at those other layers, which is where I think, you know, vertical AI players, but also, you know, tooling and infrastructure players can, can still make, you know, a lot of progress.

• 12:30Chris Benson?

  Do you think that the bundling that you're describing that's occurring between the the vertical capabilities where they're producing these and you know and and OpenAI going and doing deep research or Google integrating Gemini into you know the Google Suite, which they've been doing and trying to drive a premium, you know, from users for that, is that bundling going to be critical to them going forward? Or do you think that the OpenAI's of the world and and you know, and we've seen this historically with Google, maybe not in an AI context always, but driving into specialties where they, you know, they open up a new vertical underneath the umbrella and stuff. Do you, you know, is OpenAI going

• 13:13Chris Benson?

  to have to do that to survive? Because since it's going to have open source chomping at its heels, coming down the the general path?

• 13:21Daniel Whitenack?

  Yeah, I don't know. I it could be by vertical. It could be, I mean, you look at Palantir, for example, you know, stock price soaring. Most regular people aren't using a Palantir copilot, right, in their in their day to day. But they have that a certain market, particularly around, you know, DoD or defense or other areas. They have really put a lot into serving that well with the less generic, but still fairly generic across different use cases, set of functionalities. And that that, you know, has has served them well, at least from an outsider's perspective, if I'm if I'm looking

• 14:06Daniel Whitenack?

  at that. So it may be a specialization in terms of tools or vertical. It might also just be a a segment of the market that you choose to to focus on and is kind of the bread and butter. It's interesting because you've got all of these really end users, direct to consumer traffic on OpenAI and these things now, where a lot of what we had talked about before with data science and AI and machine learning was really enterprise focused, not direct to consumer.

• 14:35Chris Benson?

  So allow me to throw one other layer onto this conversation as we circle back around to AGI ideas with, you know, kind of having artificial generalized intelligence being bantered about. Sam Altman was just saying that he was expecting GPT-5 to be smarter than he was. And so as we look at that, you know,

• 15:00Daniel Whitenack?

  I think GPT-3 is smarter than I am.

• 15:04Chris Benson?

  Me too. I agree with you.

• 15:07Chris Benson?

  But, but with that, you know, with us, with that, the AGI chase continuing at this point. And, you know, we've heard, you know, with DeepSeek and all, all these others going in and talking about business models and bundling and such, and exploring new verticals. How do you think that the AGI race fits into that?

• 15:26Daniel Whitenack?

  Yeah, maybe that's the piece that in my mind, I'm not, isn't really entering into my mind much in the same way that you don't think about Elon so much, which is, is probably good. Yeah, I think it's an interesting question and there's implications. The, the questions that come into my mind at a more general level, which is what you could talk about it as AGI or not, I don't know. But the questions that come into my mind are more the downstream effects of some of these things. Are we building systems that enhance human agency rather than replace it? Are we building systems that allow us to trust more in human institutions or fear and distrust

• 16:08Daniel Whitenack?

  them more? Are we, you know, are we building systems that actually drive us more into isolation as individuals or into community together? I think those are, those are interesting kind of directions that, that are on my mind as I think about the more general side of this.

• 16:33Speaker 03

  Well, there's no shortage of AI tools out there, but I'm loving Notion and I'm loving Notion AI. I use Notion every day. I love Notion. It helps me organize so much for myself and for others. I can make my own operating systems, my own, you know, processes and flows and things like that to just make it easy to do, checklists, flows, et cetera, that are very complex, and share those with my team and others externally from our organization. And Notion on top of it is just wow. It's so cool. I can search all of my stuff in Notion, all of my docs, all of my things, all of my workflows, my projects, my workspaces.

• 17:17Speaker 03

  It's really astounding what they've done with Notion AI. And if you're new to Notion, Notion is your one place to connect your teams, your tools, your knowledge, so that you're all empowered to do your most meaningful work. And unlike other specialized tools or legacy suites that have you bouncing from six different apps, Notion seamlessly integrates, it's infinitely flexible, and it's also very beautiful and easy to use. Mobile, desktop, web, shareable, it's just all there. And the fully integrated Notion AI helps me and will help you too, work faster, write better, think bigger,

• 17:59Speaker 03

  and do tasks that normally take you hours to do it minutes or even seconds. You can save time by writing faster, by letting Notion AI handle that first draft and give you some ideas to jumpstart a brainstorm or to turn your messy notes, I know my notes are sometimes messy, into something polished. You can even automate tedious tasks like summarizing meeting notes or finding your next steps to do. Notion AI does all this and more, and it frees you up to do the deep work you want to do. The work that really matters, the work that is really profitable for you and your company. And of course, Notion is used by over half of Fortune 500 companies and teams that use Notion.

• 18:41Speaker 03

  Send less email, they cancel more meetings, they save time searching for their work and reduce spending on tools, which kind of helps everyone be on the same page. Try Notion today for free when you go to notion.com/practicalai. That's all lowercase letters notion.com/practicalai to try the powerful, easy to use Notion AI today. And when you use our link, of course, you are supporting this show and we love that. Notion.com/practicalai.

• 19:14Daniel Whitenack?

  Well, Chris, we talked a little bit about tools and agents. Well, agents generally, the web agents, the deep research things. And we've kind of talked about tool calling and the connection to agents at certain points on the show, but I don't think we've really dug into, you know, the detail in a, in a way that, that maybe will make things clear for people. I still see a lot of confusion around this. Even, you know, in my day to day, as I'm, I'm talking to customers, the question of, well, how do I make an LLM talk to this system, right?

• 19:57Daniel Whitenack?

  Or how do I, you know, that deep research tool? How do I make an LLM go and do a thing, right? That's often how the question comes. And what I think I realize when I'm hearing those questions is there's kind of a fundamental misunderstanding of what the LLM does and how it's tied into a framework, um, which you might call tool calling, you might call agentic. The, the names kind of get mushed around a lot these days, unfortunately.

• 20:32Speaker 03

  They do. I was thinking that as you were saying all that, and then you got, that's literally what was in my head in terms of the, the misuse of different names of this technologies and what's, what's doing what, so.

• 20:44Daniel Whitenack?

  Yeah, yeah, exactly. So in my mind, so this is, I'm feeling very opinionated today. I don't, I don't know.

• 20:52Speaker 03

  Go for it. Excellent.

• 20:54Daniel Whitenack?

  In, in my mind, how I kind of draw the lines here, there's, you know, of course, models, large language models. They predict probable text, they generate, they generate text or images or whatever you want them to generate.

• 21:09Daniel Whitenack?

  Then there's other systems kind of over on the other side. So you could think of, you know, your email or your bank account or an external system like an Airbnb where I might want to make a reservation or my company's database, right, which contains transactional data or another system that I use like HubSpot or all of these types of things. There's all of these other things. And to ask a question, well, how how could I, how could an LLM go and create a new deal for me in HubSpot, right?

• 21:46Chris Benson?

  It just hurts me when when you phrase it like that, it causes pain in my head. Okay?

• 21:52Daniel Whitenack?

  But that's how that's how people phrase it. To be clear, like I get, you know, these questions are the questions that come up every every day. Right? So how the question is often phrased, how do I make the LLM create a new deal for me in HubSpot? So right in that phrasing, to your point, I don't know, what makes you, what makes you cringe about that?

• 22:16Chris Benson?

  It's just, that's a fingernails on the chalkboard kind of moment for me is, is, you know, to answer that question, in the six and a half years that we've been doing the show, and we have evolved through a number of technologies, you know, that at each point in time were the hot thing. And inevitably, people focus in on just that for a while. But right now, we're at a point where generative and LLM, the last few years have been the hot thing. And we forget that they don't necessarily do everything out there. There it's not, you know, people will say LLM.

• 22:50Daniel Whitenack?

  In fact, they only do one thing.

• 22:52Chris Benson?

  That's exactly right. And not only that, but but there might be an AI architecture that does, that could do the thing that they want to talk about, but it's not necessarily the thing that they're talking about and they're mislabeling.

• 23:04Daniel Whitenack?

  It's not the model.

• 23:05Chris Benson?

  It's not the model. And so that's the the fingers on the chalkboard of, oh, wait, there's, we've kind of talked about this over the last year, the tunnel vision of the generative AI era, you know, in terms of everyone focusing on that. But It's to the point that there are other technologies in the mix and there is a technology that will do the thing they want to do. They're just not picking the right one in the way that they're verbalizing it. So.

• 23:29Daniel Whitenack?

  Yeah, yeah. So let's, let's maybe break this down into components. So, so let's say there's the LLM, you know, we'll just talk about text now. Certainly there's multimodal and all that stuff, but just think about text. There's the LLM, which all it does is complete probable text. So I could, you know, ask it to autocomplete. I could ask it to write something for me. I could ask it to generate something for me. That, that's what it does, right? Then there's the system. Let's say, we'll take the HubSpot example, since I use that. HubSpot, for those that aren't familiar, it's a popular CRM solution for those that maybe aren't, you know, don't, don't want to mess with Salesforce and, and all of that

• 24:14Daniel Whitenack?

  world. So HubSpot, I can create a deal associated with maybe a sales lead I have, right? That is its own software system that's hosted by HubSpot, right? And I, I actually, I don't know this, but I assume HubSpot has an API, a REST API, meaning you could programmatically interact with HubSpot. This is how apps on HubSpot work, right? An app on HubSpot is regular, good old fashioned code that maybe allows you to add a feel, add these fields to these records or retrieve this data or report on this data. That's just good old fashioned code. It uses the API. So this is a separate system. And so there's really

• 24:56Daniel Whitenack?

  no connection between, there can be no connection directly between the LLM, which generates text and this other system out there that's a CRM that does certain things. There's no connection between the two. Except in the middle of that, there can be this process, which I would generally say, I would categorize as tool calling generally, or function calling.

• 25:24Daniel Whitenack?

  Which, let's say that you wrote a good old fashioned software function that creates a deal in HubSpot via the REST API of HubSpot, right? That has nothing to do with AI. It's just a software function where you tell me the email of the person, the name, the company, I'm going to go in and create the deal in HubSpot via the API. So there's a function, you give me these arguments, I'm going to create the deal in HubSpot. Okay, still no connection to the LLM. But if I then ask the LLM to say, hey, I have this customer information, email name,

• 26:06Daniel Whitenack?

  et cetera, generate the arguments for me to call this function, which takes these specific arguments, then the LLM could generate the necessary arguments to call that function. And if you create a link between the function and the output of the LLM, so the LLM is still not really doing anything other than generating text, but in your code, you literally take the output of the LLM and you put it into the input of that function. Now, you could put something on the front end into the LLM and have the result be a flow of data out of the LLM into the function and then into the HubSpot API.

• 26:49Daniel Whitenack?

  So that's sort of how this tool calling, function calling thing works.

• 26:54Chris Benson?

  Which, which makes perfect sense and that's standard software development. You know, that's the only thing that is different there is the fact that the, the function parameters that you're using have been generated by the LLM, which is a generative model. Perfect. That's what it does.

• 27:10Daniel Whitenack?

  And, and there are some special things related to this in the sense that, you know, if you look back in time at LLMs. First, we had kind of really good autocomplete models because that was a meta task for people, you know, training language models. Then people figured out, oh, I kind of want to use these as general instruction following models, right? And so they developed specific prompt formats and prompt data sets to fine tune LLMs for specifically instruction following, right? So here's your system message. Here's the message I'm providing you. Give me the assistant response.

• 27:52Daniel Whitenack?

  And they trained it on a bunch of general instruction following things. Well, they've done the same thing now because they've realized, oh, a lot of people want to do this tool or function calling mechanism. So certain people, including OpenAI in a closed sense, but others in an open sense, like News Research, who we had on the show, they have a data set called Hermes. This includes a set of prompts that is, that are related to function calling specifically. So they've given a huge number of examples of function calling prompts to a model that they would train like a Llama model. And now you have Hermes, Llama 3.170B.

• 28:35Daniel Whitenack?

  It's been fine tuned to follow that Hermes style prompt format for function calling, which means it kind of has an advantage, if you like, or certain models that have been trained with these examples have an advantage specifically for that function calling task, right? So there is an AI element in the sense that some models are better at this than others because of the way that they've been trained. And there's certain prompt formats that are special and you'll get better performance if you use those prompt formats or if you use a model server like VLM that supports or has the inbuilt translation to those prompt formats, et cetera.

• 29:18Daniel Whitenack?

  So there is an AI element of it, but it's only in the sense that you're preparing the model for this type of use case rather than, you know, connecting, there's some inbuilt connection of the model to something external.

• 29:31Chris Benson?

  So I'm curious, can you tie in the, the tool calling into what would be, you know, might be considered a full, you know, agentic implementation? What's, what's the leap there, if any?

• 29:44Daniel Whitenack?

  Yeah. Interesting question.

• 29:47Daniel Whitenack?

  Because people use the term agent very loosely. So some people would say what I just described, even just that chain of processing. So I put something in the front end of the L M. Deal is created in HubSpot. That might be considered a, an agent, my HubSpot deal creation agent. I would say that's really just a tool calling example of how to use an L M. In my mind, what separates out the agentic side of things is where you have some sort of orchestration performed by the L M.

• 30:30Daniel Whitenack?

  So what I mean by that is you have a set of tools. So let's say I have access to Airbnb's API and Kayak's API and United Airlines API, or like whatever other travel things I need to do. Maybe my Gmail for various things. And I say, hey, I need to book a car next week for my trip to wherever, right? That input could then be processed through the L M, not to call a single tool, but first as an objective to determine what tools to call and in what sequence with what dependencies.

• 31:13Daniel Whitenack?

  Try to do a first step of that and then reevaluate and then do the next step until you reach the objective, right? So first, in order to book my thing, I need to know when my flight is. So I go to my Gmail and I look for the confirmation, right? Or, you know, second, I use that date in the Kayak API to look for choices. And then I evaluate those choices and then I use it to book the reservation. So there's a series of steps that might call different tools or systems. You know, it could be data sources, unstructured or structured data sources, like a database or a RAG system. And so that thing that I talked about, like that HubSpot deal creation

• 31:54Daniel Whitenack?

  tool, might be one of those tools in an agentic system where an agent could choose to use it at certain points. And I'm being, I'm anthropomorphizing here. It's not choosing anything, right? But it's useful to talk about it sometimes in that way. So forgive me. It's choosing to use that tool in one case, and maybe other tools in other sequences in other cases. In my mind, that's what really distinguishes the agentic side from just the tool calling side.

• 32:37Daniel Whitenack?

  Well, Chris, it's fun to talk about some of the the agents thing. Normally, we wait till the end of the episode to share some learning resources. But since we've been talking about tool calling and agents, I just wanted to mention this new course by Hugging Face. So they now have an agents course, which I think was just released and is coming out live on YouTube, if I understand correctly. And so in the course, they talk about studying AI agents in theory, design and practice, using established libraries like small agents, LangChain, Llama Index, sharing your agents, evaluating your agents.

• 33:21Daniel Whitenack?

  And then, at the end, you earn a nice certificate. So plug for the Hugging Face Agents course, if those of you out there intrigued by some of the tool calling and agent stuff. It seems like a good one.

• 33:36Chris Benson?

  Yeah, as we record this, yeah, they're actually doing it in about an hour and 20 minutes from right now, as we record the show. It'll be passed by the time you're listening.

• 33:44Daniel Whitenack?

  If you're listening, you missed it.

• 33:46Chris Benson?

  You missed it. Sorry. You're gonna have to get the replay.

• 33:49Daniel Whitenack?

  Yeah, but you can do the replay. Yeah. And it's interesting, you know, one of the packages there that they mentioned is called Small Agents, which is, is really great. I love using that, that package. It's a lot of fun. And, you know, I've even used it in a, in a couple of really interesting internal, internal use cases at Prediction Guard.

• 34:12Chris Benson?

  So, so do me a favor here. And depending on, if, so long as there's no secret sauce moments there for Prediction Guard, can you, can you plant a couple of seeds on things that you've done, you know, that, that people could explore in terms of what you found useful and I, hey, I did this thing. And just kind of let people get a sense of how you're looking at it and what things they might be able to do so that they can ideate on their own.

• 34:37Daniel Whitenack?

  Yeah, yeah, definitely. So I'll, I'll speak somewhat generically here. So I don't reveal certain things, but, you know, customer things. But one of the cases that we actually experience fairly often with customers is they want to build, you know, maybe it's they want to build a chatbot that has access to some, or, or has access to some special knowledge or can access special knowledge in one way. So on the one hand, if you have a bunch of unstructured text, right, that's a typical case where you would use a RAG workflow. And you would put that into a vector database, you can retrieve it on the fly. That's a RAG chatbot.

• 35:22Daniel Whitenack?

  On the other side, there are text to SQL methods, for example, or, you know, API calling methods that could allow you to interact with your database, right? So there's those methods. Sometimes though, you have a source of data and there's been a couple of times for us where it's maybe a, a web app that doesn't have a really convenient API, but has a really complicated and annoying user interface. And the company has this web app that has a bunch of knowledge in it, right? But there's really no good way to extract all of that content from the web app. It has an annoying interface, so no one wants to use it, right?

• 36:02Daniel Whitenack?

  And so something like the Small Agents web agent, like a system like that. And what the web agent does is it's, it executes a series of tool calls that leverage Helium under the hood, which is a package that allows you to automate interactions with a browser. And so if it's a web app, It can basically spin up the application in the browser and then interact with certain elements like search for a certain thing or find a certain component or an object, summarize that output and output it from the web agent. So, one of the interesting cases where we're thinking about that is, is these cases where

• 36:44Daniel Whitenack?

  a company has invested a lot of money in some system or application that's maybe a legacy system that they have to keep on using, right? But, no one really wants to engage it with it because the UI sucks, but it also doesn't have a really nice API or way to access the data in there. So, actually using an agent as a kind of extra user that you can control programmatically to interact with the application is really, really an intriguing kind of prospect to tie in that knowledge and extract things from the app. The other one that I think comes up a lot for us because we work, you know, we work in a lot of regulated, security, privacy conscious contexts.

• 37:28Daniel Whitenack?

  That's kind of the, what we do is prediction guard and deploying secure infrastructure for AI in people's companies. Often, people will want to, once they now have a private secure system, tie in their transactional databases to their queries, right? And that's often a text-to-SQL type of operation where you're querying a database, you're generating a SQL query. That can be error-prone, right? Like, you can generate SQL queries that don't execute, or potentially problematic SQL queries, or ones that are very computationally expensive.

• 38:07Daniel Whitenack?

  And so, you can tie in other elements, agentic elements into that where you kind of try to answer the question iteratively with different SQL queries until you reach an objective, having the agent, that's, this is kind of an agentic way to go about the text-to-SQL. Or you could tie in other tools like SQL query optimizers and that sort of thing to help in that process as well. So, on, more on the enterprise kind of business side, those are a couple of things that have come up for us.

• 38:37Chris Benson?

  No, that sounds interesting. It's a- I'm just kind of curious what you're thinking is, as how does this change the human side of the workflow as you've, as you've seen, you know, in recognizing these are some small use cases and everything, but you know, this is the beginning of the agentic wave as we go forward. And I guess especially prompted with the kinds of things that we're seeing in the news these days, you know, about evaluation of, of government departments and, and just that general, that general notion of reassessment for better for worse. How do you think that that's going to, you know, be taken into, into commercial spaces in terms of deploying these agents? Will

• 39:20Chris Benson?

  it change jobs significantly? Do you think, or do you think it will just be adding in without that kind of, I'm kind of curious what your lay of the landscape is?

• 39:27Daniel Whitenack?

  Yeah, I mean, I think there will be a shifting of jobs. I think some of the things that we've talked about specifically in those examples are actually good examples of expanded human agency, because a lot of times people don't do certain tasks or can't do certain tasks that they would like to do as a part of their job because of limitations of, you know, really complicated UIs or that this, you know, doing this and then this and then that will take me a ton of time and I've got to jump to this meeting, right? And so I think there's a lot of those things where that is expanded human agency of that. And so it's amplifying the effect

• 40:08Daniel Whitenack?

  of that worker and helping them feel like they have superpowers because they really didn't want to log into that application and use it one more time, right?

• 40:18Chris Benson?

  Yeah.

• 40:19Daniel Whitenack?

  So I think there's an element of that. Now you could make the argument, well, maybe they've hired three people under them because of those inefficiencies to do some of those tasks, which in some ways is a shame because if, if they're really just grunts, you know, cranking through.

• 40:42Daniel Whitenack?

  Extraction of data from horrible API or horrible user interfaces. Like you could, I mean, maybe there's people that enjoy that all day. I think generally that's not a very dignified sort of way to go about it. Now I'm realizing that I'm kind of making generalities here and there's the reality of people's work. Not everyone kind of gets to do the work that they, you know, they might desire to do or would give them most dignity. So I want to recognize that. And I think there will be a, there will be a negative impact for portions, but I, I'm hopeful that there's also this positive impact.

• 41:21Daniel Whitenack?

  And even for people that are maybe in less skilled professions, if there's more of a natural language way to access skilled knowledge and kind of these amplifying effects of AI, it could hopefully open up new types of opportunities within the market as well. So.

• 41:43Chris Benson?

  I would hope so. I mean, I think that's certainly, I think, I suspect that we'll see all, just as we do in life in every other aspect, we will see people enhancing human agency on that kind of to the use cases that you're talking about. And we'll probably see people with, uh, that, you know, that would, that would rather take alternative paths to that as well. I think it will be a mixture of the whole thing. So.

• 42:06Daniel Whitenack?

  Yeah, yeah. As we kind of close out here, and I guess we're talking already about, about new trends and other things. One thing I wanted to note is Deloitte just put out there in January, they put out the state of Gen AI in the enterprise quarter four report, um, which I've been going through for, so for those maybe business leaders or managers or other people that are wanting to get a sense of some of the things that, that are being tracked across different industries in, in the enterprise.

• 42:40Daniel Whitenack?

  There's a great report there. I see, you know, for example, that you're tracking barriers to developing and deploying Gen AI, worries about complying with regulations, difficulty managing risks. They're tracking certain use cases, volume of experiments and POCs or proof of concepts, benefit sought versus benefit achieved, which is an interesting one. And also Gen AI initiatives, where they're most active within certain job functions, all of these sorts of things and many more. So if you're interested in those sorts of insights, which I do think are interesting to track, then that's a, that's a great learning

• 43:24Daniel Whitenack?

  resource that we'll link in the show notes and hopefully people can find and peruse if, if they're interested.

• 43:30Speaker 00

  Definitely.

• 43:31Daniel Whitenack?

  Yeah. Well, Chris, it's been, uh, it's been a great time. I felt like I, I functioned well in my tooling as a, as a podcast agent.

• 43:45Speaker 00

  You did good. You did so good that who knows Elon Musk may, uh, may be coming after Prediction Guard any, to any day now. So.

• 43:52Daniel Whitenack?

  Yeah. Or maybe what I'm saying is just being generated by Notebook. I'll, um, that could be too.

• 43:58Speaker 00

  Yeah. Okay. Well, good, good conversation today.

• 44:02Daniel Whitenack?

  All right. Yeah. Thanks, Chris. Um, have a good one.

• 44:05Speaker 00

  You too.

• 44:13Speaker 00

  All right. That is our show for this week. If you haven't checked out our change log newsletter, head to changelog.com/news. There you'll find 29 reasons, yes, 29 reasons why you should subscribe. I'll tell you reason number 17, you might actually start looking forward to Mondays.

• 44:33Speaker 04

  Sounds like somebody's got a case of the Mondays.

• 44:36Speaker 00

  28 more reasons are waiting for you at changelog.com/news. Thanks again to our partners at fly.io, to Breakmaster Cylinder for the Beats, and to you for listening. That is all for now, but we'll talk to you again next time.