The Evolution of AI Agents: Lessons from 2024, with MultiOn CEO Div Garg

In this episode of The Cognitive Revolution, Nathan welcomes back Div Garg, Co-Founder and CEO of MultiOn, for his third appearance to discuss the evolving landscape of AI agents. We explore how agent development has shifted from open-ended frameworks to intelligent workflows, MultiOn's unique approach to agent development, and their journey toward achieving human-level performance. Dive into fascinating insights about data collection strategies, model fine-tuning techniques, and the future of agent authentication. Join us for an in-depth conversation about why 2025 might be the breakthrough year for AI agents. Check out MultiOn: https://www.multion.ai/ Help shape our show by taking our quick listener survey at https://bit.ly/TurpentinePulse SPONSORS: Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers13. OCI powers industry leaders with secure infrastructure and application development capabilities. New U. S. customers can get their cloud bill cut in half by switching to OCI before December 31, 2024 at https://oracle.com/cognitive SelectQuote: Finding the right life insurance shouldn't be another task you put off. SelectQuote compares top-rated policies to get you the best coverage at the right price. Even in our AI-driven world, protecting your family's future remains essential. Get your personalized quote at https://selectquote.com/cognitive Weights & Biases RAG++: Advanced training for building production-ready RAG applications. Learn from experts to overcome LLM challenges, evaluate systematically, and integrate advanced features. Includes free Cohere credits. Visit https://wandb.me/cr to start the RAG++ course today. RECOMMENDED PODCAST: Unpack Pricing - Dive into the dark arts of SaaS pricing with Metronome CEO Scott Woody and tech leaders. Learn how strategic pricing drives explosive revenue growth in today's biggest companies like Snowflake, Cockroach Labs, Dropbox and more. Apple: https://podcasts.apple.com/us/podcast/id1765716600 Spotify: https://open.spotify.com/show/38DK3W1Fq1xxQalhDSueFg CHAPTERS: (00:00:00) Teaser (00:00:40) About the Episode (00:04:10) The Rise of AI Agents (00:06:33) Open-Ended vs On-Rails (00:10:00) Agent Architecture (00:12:01) AI Learning & Feedback (00:14:01) Data Collection (Part 1) (00:18:27) Sponsors: Oracle Cloud Infrastructure (OCI) | SelectQuote (00:20:51) Data Collection (Part 2) (00:22:25) Self-Play & Rewards (00:25:04) Model Strategy & Agent Q (00:33:28) Sponsors: Weights & Biases RAG++ (00:34:39) Understanding Agent Q (00:43:16) Search & Learning (00:45:39) Benchmarks vs Reality (00:50:18) Positive Transfer & Scale (00:51:47) Fine-Tuning Strategies (00:55:16) Vision Strategy (01:00:16) Authentication & Security (01:03:48) Future of AI Agents (01:16:14) Cost, Latency, Reliability (01:19:30) Avoiding the Bitter Lesson (01:25:58) Agent-Assisted Future (01:27:11) Outro SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://www.linkedin.com/in/nathanlabenz/ Youtube: https://www.youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Div Garg?

  I think OpenAI has definitely lost a lot of the lead they used to have. The G P T four were like they were kind of the sole winner and no one could catch up with them. At this point, it does seem like a very homogeneous market where everyone is kind of close. When anything is disruptive, I think it takes a lot of time for that to catch on. And I think we are at the start of the disruptive era where like a lot of the online communication and interaction will get disrupted by humans. He's pretty good at like navigating websites with U I. And so like theoretically, like A I can also become very, very good. Even better, I think I would go to explosion of applications, which I don't think has happened so far. If you think about agent applications, I think it's still very good.

• 0:40Nathan Labenz?

  Hello and welcome back to the cognitive revolution. Today, Div Garg, founder and C E O of Multion, returns for his third appearance on the show. A lot has changed in the A I agent landscape since I first spoke with Div in mid two thousand and twenty three. As you might remember, at that time, the A I community was abuzz about the potential for A I agents. With projects like Baby A G I giving large language models access to tools and really only very minimal guidance, and then stepping back to watch and see what they could accomplish on their own. Of course, as it turned out, they didn't accomplish all that much. While there were many amazing moments, those were the outliers. On average, the step-by-step error rate, including on very mundane microtasks,

• 1:21Nathan Labenz?

  was too high for agent frameworks to successfully string many-step sequences together all that often. And we also learned that while large language models can improve in many areas through self-critique, they have a tendency to get stuck on obstacles that humans quickly find ways around. For that reason, much of the last eighteen months of work on agents has gone into developing better and more prescriptive scaffolding, with many companies ultimately delivering platforms for what I call intelligent workflows. That is workflows that a human has designed and where the A I is needed to do some important sub-task which requires intelligence, but where the A I is not given freedom to choose its own adventure. As of January this year, Div and the Multion team were still among the most bullish on open-ended agents. And as you'll hear in this conversation, they have continued,

• 2:06Nathan Labenz?

  at least partially, to buck that trend. They have built some new scaffolding, and they have developed interesting techniques for domain-specific fine-tuning. But their agent continues to take arbitrary natural language requests and gamely does its best to fulfill them. The progress I found in my testing is pretty obvious. And in some contexts, the company claims human-level performance. But still, the system as a whole is not a viable substitute for a human assistant. With that in mind, I was excited to pepper Div with questions about what he's learned from all of this activity. And so in this conversation, we unpack the latest in agent development, including the company's data collection strategy, the seemingly missing market for human-computer use data, and the role of synthetic data in bridging that gap. The company's model strategy, including

• 2:50Nathan Labenz?

  what models they've chosen as base, what fine-tuning techniques they're using, and how their computer vision approaches have evolved over time. Why benchmarks so often show human-level performance while the real-world results are clearly not as strong? The future of agent authentication, as well as which parts of the internet at large will compete to serve agents versus which parts will try to exclude them. And finally, what sorts of customers Multion is looking to partner with now, as well as how they're thinking about competing with hyperscalers in light of Claude's new computer use capability?

• 3:22Nathan Labenz?

  Overall, it's clear to me that while it's taken longer than I had expected, reliable agents that can perform a very large percentage of routine computer use tasks are coming. It's only a matter of time. And as you'll hear, Div agrees with recent suggestions from both OpenAI and Anthropic that 2025 will be the year. That, of course, makes Div a very busy man, and so I very much appreciated his time and how open he was willing to be about the path that Multion has taken and the lessons they've learned along the way. As always, if you're finding value in the show, we'd appreciate it if you'd share it online, write a review on your podcast app, or leave a comment on YouTube. Of course, we welcome your feedback and suggestions via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, here's my conversation with Div Gerg

• 4:07Nathan Labenz?

  of Multion, catching up on the last year in AI agents. Div Gerg, founder and CEO of Multion, welcome back to the Cognitive Revolution.

• 4:16Div Garg?

  Yeah, thank you, Nathan. I'm excited to be back.

• 4:19Nathan Labenz?

  So agents, agents everywhere, the last eighteen months has been quite the saga when it comes to AI agents, their promise and whether or not they've fulfilled that promise. And obviously, you've been right in the thick of it as the founder of this company. How would you tell the story of agents over the last eighteen months since like the launch of GPT-4? And where do you think we are now in the grand saga of AI agents?

• 4:45Div Garg?

  Yeah, it's been interesting to see. I was just saying very early, I would call it like it's similar to the internet explosion. So we are kind of seeing like the first wave of the internet in a sense, where it's still like glitchy, it's slow, most people have not used it. But the next thing that's gonna happen is like this will become like more mainstream, like the quality and reliability will go really up. And over time, this will become like, uh, brought to a lot of like opportunities, well, a lot of like the things that happen in the daily life, so become more agent taking over time. So I do think like agents is still very early. I think it compared to GPT-4, definitely, I think at that time, it was kind of a promise that okay, like it just kind of things are gonna happen, you'll have this kind of automatic capabilities. Now, I think we're starting to see the early versions of that, where like maybe people have found some like use

• 5:29Div Garg?

  cases and like specific verticals, some business use cases. I think it still is very early. Most people have not created by the agent product. So I think it's still like haven't explored it fully. And I think that will take more time.

• 5:40Nathan Labenz?

  Tell me about your agent lifestyle today. In the past, you've posted some viral demos on Twitter of like having an AI agent order your lunch, or do these kind of various transactions for you. I think you had one of the first demos of it, booking a flight. How is your day to day experience of AI agents? Like, what are they doing for you right now?

• 6:06Div Garg?

  Yeah, I think that's been pervasive. I think like I'm still using agents a lot for like, uh, shopping has been a big use case, like Instacart, groceries, definitely calendar invites, and then maybe a bunch of stuff on like LinkedIn or Twitter or a bunch of this kind of places where like you want to have bots that can automate things. For me personally, I definitely have this, like a lot of my scheduling is now based on agents, a lot of my groceries and there's some of my online shopping and the app are also based on agents, huh?

• 6:34Nathan Labenz?

  So let's talk about kind of a few different approaches to building agents. It seems like in the beginning, the first wave of things that were launched, often it's just like open source, check this out and mess around with it, kind of projects I'm thinking here of like Baby AGI and that sort of early wave, were very much just like, hey, GPT-4 looks really smart. Let's let it figure everything out and kind of give it very open-ended environment, very sort of open instructions and kind of hope for the best. It seemed like by and large, with notable and kind of exciting demos as the exception rather than the rule, those approaches led to disappointment when people found like, oh,

• 7:17Nathan Labenz?

  well, it doesn't really work most of the time. It gets, you know, stuck in various ways and whatever. And so the counter narrative, I would say, has been, or the counter, you know, the sort of compensating approach, maybe is a better term for it, has been to put the agents on rails, to give them much less sort of autonomous decision-making authority and instead build out like very prescribed workflows where the step-by-step is kind of designed by the human workflow designer and the AI is kind of there for the key bits that require intelligence, but is not like left to choose its own adventure. You shared a, you know, preview version of the latest

• 8:01Nathan Labenz?

  multi-on agent and I was messing around with that. It seems like you guys are still more in the open-ended frame of development, but I'm sure, you know, you've tried a bunch of stuff and have perspectives on those two different approaches. So tell me everything about the open-ended versus the on-rails approach.

• 8:20Div Garg?

  Yeah, I think that's a good question. I totally agree. I think on-rails is the right way to start from, especially if you know exactly what use case you want to target, if you can do anything like enterprise or B2B SaaS, whatever. I I I don't think like this kind of like more linear constraint workflows. I think it's the right paradigm to start with. And I think over time, I think you can have more open ended use cases, which allow for more general purpose experiences. Now, I I think for us also, we are trying to take a road in the middle. But we don't want to be too constrained, because I think we're still targeting a lot of everyday use. And what happens, we have several, like, I don't know, like, at this point, five thousand plus people from all the user data we have. And I I I do think the consumer behavior is just very different for every person. So depending on the demographics and age groups and like where you live, a bunch of things.

• 9:05Div Garg?

  I think consumer behavior changes a lot, so you can have very concentrated linear workflows. Because then you have to build like millions of these, and like, that's not possible. And you also want to be fully open ended, right? Because like if you have fully managed something like the computer user API that came out from Anthropic. It's it's very general purpose, but like, it's like, it's very hard to find a utility for that. And and and so I think we're trying to take a road in the middle, where like, how can you build more constrained workflows for like a task that you can define? And then we are building a lot of like verifiers and like one thing in the direction was the agent Q research paper we did. Where like the agent can learn to improve and like improve its behavior on new websites it has never seen. And so we're starting with like more domain specific models. And the models like know how to like do maybe like vertical workflows or tasks to develop.

• 9:48Div Garg?

  All these models can improve and like generalize over time, but we don't want to start with models that are fully general. Because that seems to be like a very heavy effort and also doesn't seem something that will, I think, for a startup like us will pay off in the next six months.

• 10:00Nathan Labenz?

  How does that look behind the scenes? I mean, in in my experience of the product, it does still present as a very open ended thing when you talk about kind of having these linear or closer to linear on rails workflows in the background. The Voyager project comes to mind from Nvidia, I'm sure you're very familiar with that. Where they would have the agent in a like a virtual video game environment, I think it was Minecraft. Go out and like figure out new skills and then sort of cache those skills for future use, so it didn't have to constantly reinvent the wheel. Obviously, that's like a pretty big open ended environment, not nearly as big and as wide ranging as doing stuff broadly on the internet. But what have you found to be the right architecture for

• 10:46Nathan Labenz?

  or the right balance point between like what is prescribed versus what is left to the agent to decide at runtime?

• 10:54Div Garg?

  Yeah, that's a good question. I think what paradigm we have landed on is kind of like user choices. So it's like you don't want the agent to be fully autonomous. I think an example I like to use a lot is kind of like fact booking. So you don't want to have the agent go and book you like a random flight and waste thousands of dollars. Uh, that like sort of like checking in with you, like do you actually, uh, are you fine with this time? So do you want to find the evening? Do you want non-stop or like, uh, this airline is to play that? So preferences matter a lot. And and I think like the paradigm we have landed on is like choices where like you might give some sort of potentially ambiguous, uh, like user query to the agent, like, okay, go book me a flight this weekend for a trip to Paris. And then and then you want the agent to kind of fill in the blanks. Where like, okay, like that requires maybe like a lot of choices that have been implicitly implicitly being made.

• 11:40Div Garg?

  And and then like we are building a lot of these workflows where okay, like how do we cast things as preferences? How do we make like involve the human in the loop? And I I think that becomes that starts becoming complicated where like you do it are not relying on just the actions, but also maybe like how to personalize, how to maybe like really get to know like what to do. And and then based on that, like build like a really sort of product experience.

• 12:02Nathan Labenz?

  One of the experiences that I recall from an earlier version of the product that I don't know if it's still there, maybe I missed it this time around in my testing, was the opportunity to essentially correct the AI if it made a mistake, you know, to sort of demonstrate or like teach, I think was the word that you used in the past. The AI how to do a particular skill, uh, and I think probably most people would find it reasonably intuitive to imagine how that was supposed to work, right? You would have the agent go do stuff when it makes mistakes, the human could teach it what to do. Then, of course, it would learn what to do and you'd maybe fold that into your training data later. I wonder if like that has played out as you've thought it would, or are you now doing that in a more implicit way?

• 12:46Nathan Labenz?

  Or do you just find that like that human sort of teaching paradigm is just not so useful as anticipated? And if that's not working, like, what is the alternative in terms of how do you source data to, you know, to teach the models what to do?

• 13:01Div Garg?

  I I think we had a good success with that. I think we collected like some millions of trajectories using that method. The problem with any sort of crowdsourcing is like you can't really trust the quality of the data. And I think Tesla is a good example for that. I think Tesla has been trying this for the most having vehicles where they crowdsource a lot of data. But like filtering like the quality of the data is like actually good data versus bad data. And that that has been like a thing where like you have to build a lot of filtering pipelines and build a lot of like AI's logic around that. So I don't think we have been doing a lot of that. We've seen some pay off. But I think it's still like, I think it's just like a very noisy. So that makes it like a hard problem. The second way we is like this is just working with like high quality annotation data where like like I won't say like too much here, but like, like if you have annotators or like people who can like sort of like

• 13:46Div Garg?

  have to collect expert data. And I think that seems to be like one paradigm, maybe like more frontier labs are pushing towards. But like you can go and like train these agents on a lot of these general purpose use cases and then like train different models that are working in the early volume.

• 14:02Nathan Labenz?

  I hear you on things being noisy. I also imagine, of course, the counterpoint would be like human annotators are expensive and limited in supply. One good question I have is why is nobody propositioning me to pay to watch me use my computer all the time? Like, I feel like there should be a market now in install a general observer that just kind of watches a person use a computer. Maybe get up a couple of different modes of it, including one where you like talk through what you're doing and explain yourself. Uh, because obviously, most of the time, the chain of thought is is not actually said out loud and and recorded.

• 14:45Nathan Labenz?

  You know, you'd have to clean up that data. Of course, people are doing random stuff. You'd also have to anonymize that data. I would want to know who you are and I'd want to make sure I trust you before I let you install that on my computer. But I feel like with the the cost of annotation is people get into like higher and higher end knowledge workers or even like scientific experts in some cases for the sanitation stuff. It feels like that's pretty valuable and I feel like I should be getting like a thousand dollars a month at least for somebody to watch me use my computer. So why isn't that happening? Or if it is, you know, point me to where I sign up?

• 15:17Div Garg?

  Yeah, I totally agree. I think the market is there now. I think it wasn't there maybe like even a month or two ago. I do think it's a new market. I just, I think this depends like who's willing to purchase the data because I think at the end of the day, getting this kind of data is not that expensive. Because I think there's enough people who are willing to train this data for very cheap costs, especially if you can like use other countries and like whatever, like outsource a lot of this kind of data collection. So I do think a lot of this data can get very cheap. It does become like if the company is really interested in your personal data. And then they might be going to offer like more like whatever thousand dollars a month, just to be able to like get access to be very like more like personal, okay, like what makes Nathan, what does it that look like when he's using his computer?

• 16:00Nathan Labenz?

  Yeah, I mean, it's an interesting paradigm. It kind of reminds me of, we did two episodes that were separated by some number of months on MindEye and MindEye 2. And in the first case, and this was like reconstructing what somebody was looking at, like what image they were looking at by the FMRI data scan from a scan of their brain at the time that they were looking at this image. In the first version, it was patient specific modeling that was being done. And you had to have something like, I don't know, 20 hours or whatever of of these scans with a person looking at a new image every few seconds to then finally get to the point where you could train a model on that person. And then in the second edition, they had figured out how to kind of

• 16:46Nathan Labenz?

  let's say, map all of these different users, each of which have like a different anatomy of their brain. Literally like different brain sizes. There's a lot of course, difference between people. Map that all into essentially a shared space, train one model on all of those data points. And then from that base model, it would only take like an hour of one individual's data to kind of fine tune it to their particular anatomy and activation patterns. So I can see that here too, where you might say, you know, there's vast bulk data to be collected out there and then we just on the margin, it's really about fine tuning to each individual person. So I guess what you're saying is like that market is

• 17:32Nathan Labenz?

  International and it's just a it's a lower price point because I guess people feel like they want to separate out like scientific expertise from just like general routine computer use and it's better to buy those separately at different market prices.

• 17:49Div Garg?

  Yeah, because I think what the general computer use, it's not that expensive. It's also like I think like a very common patterns. So I started thinking you can just outsource a lot of that at the very cheap. I do think there's value of personality, which is like if you're willing to like share all your personal data and like exactly how you are doing things and you don't care about privacy. Then I think people are willing to figure a lot where like like if they can get access to a lot of this private data. And then they can like maybe use that fully. And then maybe like I I do think like that is a market that doesn't exist, but that can be like very well, a market where people are willing to pay a lot of money if they can get access to this private data. And then you're willing to like sort of like live with that kind of like loss of privacy in a sense.

• 18:28Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:33Nathan Labenz?

  Do you know what those data, I assume you're not doing this directly yourselves to the degree that you are sourcing this kind of data. I assume you're working with partners to do it. Are there like marketplaces that exist or you know, go to companies? And are there, I'm also curious as to what degree people are sort of logging in and doing like discrete tasks, like here is your task, go do it on the browser, tick, tick, tick. Versus just kind of open ended observation that would be more of like an imitation learning paradigm. Well, how would you describe the you don't have to give us any secret sources unless you want to, but let's say I wanted to buy this data. What what's out there, you know, that I can buy and who do I buy from and how much do I expect to pay?

• 19:16Div Garg?

  I would definitely say there's a lot of like data leaving from these, a lot of them we are working with. Again, I can't tell really like it's like who we are working with just for like more confidential purposes. But I would definitely say like you've tried a lot of things, like even if you use some like M2 or or whatever, like online solutions. I don't think you can like create task. Okay, like I want someone who's doing this particular whatever like a computer use workflow. Collect a lot of data and then get a lot of like data that you're going to train on. I'm pretty sure that's what maybe like Anthropic is doing, which that's what they're doing. We have definitely done that, but then we're also like thinking about a lot of like smart tricks where like how do we find data in like the regimes we care about? So how can we incentivize people to give us like the data that we don't have? And then make sure that we can like keep improving, figure out like where

• 20:01Div Garg?

  Is the agent most efficient on and get the right quality of data and keep making it better and better.

• 20:06Nathan Labenz?

  Are we at the point of self-play? Another thing that I sort of expect we'll have to tip soon is just object feedback from reality, right? I mean, we've obviously seen that work in all these game playing environments. It has worked, but hasn't quite maybe hit the critical tipping point yet for code generation. But I would assume that there's some analogous version of feedback from reality that you could do in a open-ended computer use context too, right?

• 20:37Div Garg?

  Not at all. Yeah, I think the thing that's missing the most right now is just having a really great quality to report that you can use from the environment. So when you're in a game-based setting like Minecraft or something, you're pretty good reward or like even like when you're playing chess like Azero. You have like, you know, exactly like, like what are the rewards for any moves you make in the game space? And then it's easy to optimize a policy and make it like, like, yeah, this is what we want to do to then. What happens though is like if you are in like a more of a real world scenario, it's there's no environment to reward. And then you basically, it that makes it challenging because you don't know what is the objective that you're optimizing. And then you maybe need to obtain another like model that is the kind of like giving like some sort of proxy award. That again, you use to like sort of like optimize the model and make it better and better.

• 21:19Div Garg?

  I think the coding, I think it's easier because you can create like unit tests or you can have some sort of like static checking. Like so you can get a numeric score on the quality of the code and like does this code actually function well? And then you can do some sort of self-play, okay, like based on this kind of like score. Can we optimize this code to actually become better and better and optimize it? I I don't think it's possible like there will be like some like the the thing with this kind of algorithms is like there's always ways to cheat in a sense. So whatever metric or score you come with, like the algorithm might find a way to cheat. Like it might find some sort of like, I just find out like you can just create a lot of like null strings or something and like that actually solves the problem and like gets you like infinity score or something like that. And people have seen that reinforcement learning a lot when you use that in game playing environments. So yeah, so I do think it becomes a question of can you obtain like high quality rewards on the setting you are in?

• 22:04Div Garg?

  And I think computer use is like a hard one there, but like maybe a human can give you like the right proxy various if you're doing things like human feedback. What are the things that's possible, but I think it's a harder problem than just like doing self-play on a game based model.

• 22:19Nathan Labenz?

  I always remember that visual of, I think it was a DeepMind video game player from like 2017, where the little boat is circling around and around the same space and like picking up infinite points, but not actually advancing in the game in the way that it was meant to. So yeah, lots of vivid examples of Mode Collapse or other strange solutions in the reinforcement learning world. So what's your model strategy today? I read through the Agent Q paper that you guys put out a couple of months back. At that time, you were reporting results on a Llama 3-70B, which you were doing a number of things to enhance, obviously, from its base version.

• 23:03Nathan Labenz?

  Is that still the baseline for what you guys are using in production? Or how would you characterize the available, I guess, commercial and non-commercial, open source options that are available to an agent developer today?

• 23:18Div Garg?

  Yeah, at this point, I think we've got a lot of different models. Close service, open source, commissionable. Like Llama 70B has been, I think, a good compromise in terms of speed versus the model reasoning capabilities. And I think right now, I think it's starting to go more towards like a GPT-4-01 style, where can you do like inference time compute? And can you optimize the models to do like chain of thought, doing inference and like become a better reasoner? So I don't think that seems to be the way to solve a lot of these complex reasoning problems. So that's also what we're going towards. But like, I think at this point, the base model doesn't matter that much, because I think there's enough base models, most of them are forming like similar-ish. The recipe and the data they train on are also like more or less the same.

• 24:01Div Garg?

  So the trick kind of becomes like, okay, like what is your application? And how do you optimize a model on that application? And I think that's what we did with Agent Q, where like we cared about this kind of web interface, that's kind of like a shopping environment. How can we get the right environment like feedback? How can we tune on the feedback? How do we make sure that this base model can like optimize and become better and better there? And then can you combine that with some sort of like inference time compute tricks to make it like a better reasoner? So I don't think at this point, architectures seems to start becoming irrelevant. Where like I think we might see some sort of like, at at least at the current scale. It's possible like if you are able to like ten X the parameters of a model, then a different architecture might like start shining. But right now, I think like most architectures seems to like level out at a similar performance where like there's not too much diversity.

• 24:46Div Garg?

  In in terms of the models out there, whether it's close to a certain source and like how much difference you can get. So even if you look at the leaderboards, like everything seems to be very tied up. It's everything's starting to come very, very close. No one has, I think OpenAI has definitely lost a lot of the lead they used to have. The G P T four where like they were kind of the sole winner and no one could catch up with them. At at this point, it does seem like very homogeneous market where everyone's kind of close. And the mod, the base model architectures kind of seems to become like not relevant as long as it can use it.

• 25:14Nathan Labenz?

  Interesting. Oh, one would seem to be a

• 25:20Nathan Labenz?

  possible exception to that or no? I mean, obviously, oh, one is like expensive if you're trying to do mouse by mouse click. And I guess there's also a question right now of what is available is a one preview, which doesn't have the vision aspects enabled yet. But does that sort of, does that summary of everything sort of being on a roughly even level include a one or not include that one?

• 25:46Div Garg?

  Like I'll say, Owen is interesting. I think I would say it's very good map. People have found out like it's not very good outside the methodology domain, but I think it's very highly specified. So so also to give it no one, I think it's not too different from what you have right now. Maybe like at least for the one preview, maybe like the full one model is much better. And then it can shine and like all a lot of different regimes. I I do think like this kind of paradigm where you instead of like training then compute where like like last year, G P T four, everything was just throw more compute during training. More parameters, more compute, more data, and they get a better model. That's more better reasoner. And I think we're starting to see this type of like instead of training computer during training, it's for computer during inference. And then like the more change you do, the better you get. And so for Owen, it's kind of like,

• 26:28Div Garg?

  if I throw less compute, maybe it's kind of like maybe even worse than GPT four, but if I throw a lot more compute, maybe it's much, much better. And then so it's kind of becomes like, how much time and compute can I throw, bringing inference depending on application. But I do think that becomes like a different class of how to think about these problems. Especially if you think about this as kind of like, how much latency can you tolerate? Adding friends and like, how much resources are you willing to like give to the model?

• 26:51Nathan Labenz?

  Yeah, I'm interested. Maybe we'll come back in a little bit to talk about kind of user experience and how much you think some of that stuff matters. Like I'm always unsure about, you know, how much cost matters, how much latency matters. Obviously it depends on the user experience and depends what the agent is trying to accomplish, but let's circle back to that. For a moment, let's stay on the agent queue research that you did. I've got a table here that I pulled up from the paper. There's basically two kind of environments, right, that are used in the paper. One is a e-commerce benchmark that was created by academics and put out there. And it's sort of a self-contained, you know, Amazon-like environment for shopping.

• 27:36Nathan Labenz?

  And then the other one is actually going and having the agent do things on OpenTable. If I understand correctly, in both cases, you get to about ninety-five percent success through a variety of techniques. And I thought maybe you could just like walk us through this graph, which people can pull up from the paper. But it's a nice sort of explanation of how you build up to the success, starting from a Llama three seventy B instruct base model, which for the graph is coming in on the OpenTable task at not even twenty percent success. That compares to GPT four O, which is over sixty percent,

• 28:15Nathan Labenz?

  but still leaves like six more bars on the graph where you're layering on these additional techniques and showing what the contribution of each one is. So you want to take us from, again, under twenty percent on Llama three seventy B to a little over sixty percent GPT four O to, through six, you know, alternatives all the way to the ninety-five percent.

• 28:36Div Garg?

  Yeah, no, totally. Um, yeah, I think this is what I'll say, like, is possible with vertical uh specialization of models, because I think what happens is like, most of these models are just trained on very generic internet data. And then, and they're not like really, really super great at one thing. And I think that's very obvious when you look at like Lama, like Lama's like this performance like twenty percent, because I think it's probably never been trained on this kind of like tasks before. But then you start looking G P T four, I think G P T four maybe has more browsing kind of like capabilities, or I think they're often each other browsing too. So they maybe have more data on this kind of interfaces and that makes it like a better reason and a better like, better accuracy on this kind of like uh action tasks.

• 29:16Div Garg?

  But again, I think what happens is like, just because this is very gen- generic, there's no like specific fine tuning towards a particular vertical. I think we'll see a lot of loss of accuracy. And I, I think that's one thing we proved with the queue, where we're like, like now, suppose like you have a level model that performs twenty percent. But then based on like a lot of the data that we collect, can we actually boost the performance? To make it like essentially goes off the environment almost all the time. And, and then we tried this experiment where like, let's use a lot of like techniques like D P O. That we're doing reinforcement and using like human feedback. Let's combine that with maybe like stuff like uh Monte Carlo pre-search, where can we search the space of the websites?

• 29:57Div Garg?

  So we can figure out, we can explore the websites, figure out like, like if you go down this route or this link, will it work or not? And if it like, okay, this didn't work, this one worked, and then keep doing that. Until we can kind of like, uh, keep fine tuning, make it better and better. And, and then we, over time, we have like a very specific like vertical agent capability, where like, okay, now this agent just knows like really how to navigate this website. And, and the, the great thing is like, uh, this thing just took us one day, so it was very fast. And it's, it's kind of a self improvement cycle. So though you can just keep doing it more and more. And so there's no limitation on like, like how much can you improve the performance as long as you have a good. Quality feedback. So as long as your feedback signal is very, very high quality, I think you can, you can just like keep making this better.

• 30:40Div Garg?

  Um, I mean, I think that was like a very interesting learning, like we were actually also very surprised like, like that's almost like a four point five X improvement, close to that. You were like, like, yeah, that's kind of uh uh crazy. Like just in one, you get able to like boost a model. That's forced twenty percent to like to get this all the way to to where we were able to push it with the tech is used. And I, I think that's kind of like the same print here where how do you explore these moments? And then then how do you have this capability where you can like self learn and optimize?

• 31:10Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 31:17Nathan Labenz?

  So let's go one by one. I, I think it is worth just breaking it down. The first one on the chart is Lama three seventy B instruct R F T. Is that a reinforcement learning fine tuning or a different kind of usually it's in just simple like instruction fine tuning is like the first layer of post-training, right? I wasn't clear to me what the R F T stood for in that case.

• 31:42Div Garg?

  Yeah, I appear it's like a older algorithm. I think it's a research paper. I do. I do think it's a reinforcement fine tuning. But I think it's a research paper, like some of the algorithms I can read that came out last year. So we use this as a baseline. And then you compared to the bunch of other math who telling there's a couple of work from like Salesforce, the groups. And then the interesting thing was like no one kind of like thought about like the direction that we thought about where like how can we do more efficient search and like a mind that fill up this kind of like learning? And then like, how do you self optimize?

• 32:11Nathan Labenz?

  So that's reinforcement learning. D P O is next. I know you're right in the heart of D P O country there in Palo Alto. As maybe an aside, can you help me develop my intuition for? The DPO algorithm, I feel like I'm on a quest, you know, I can look at the equation and that doesn't jump out, you know, is like super intuitive to me. How would you describe in a qualitative or intuitive sense? What DPO is doing and like? How? A set of preferences on different generations is ultimately being translated back into like updates to the weights of the model?

• 32:50Div Garg?

  Yeah. No, I was like, DPO is, is it a very intuitive algorithm at the end of the day? So when you do like supervised fine tuning, what you usually do is like you have a bunch of expert data. This is like kind of the optimal data. And then training the model to be like, like, here's my ground truth optimal data. And then you want to like, sort of like, make the model predict behavior that's similar to it. And then you're doing this learning where you're doing like gradient descent. But like go more closer to imitate like what the data looks like. So the model's predictions are similar to the ground. I think DPO, I think the thing they do is I'll call it like, they also use a negative data in a sense. So they do like this kind of contrastive learning where they're like, like we have some positive feedback data, we have some negative feedback data.

• 33:31Div Garg?

  And then you want to do gradient descent towards the positive data. So you want the model to like go closer to the behavior. Uh, in terms of its predictions towards the, the positive outputs. But we want to do like gradient ascent on the negative data. So you want it to go away from the negative like behavior. I think that actually works better because like if you're just kind of like, if you think about like maybe like the positive behavior is kind of a circle and the model is kind of like trying to get close to it. It's possible like the model might be very widespread. So it's like it can cover the circle, but might also be very widespread outside. But now when you're doing DPO, you're saying like, here's a positive stuff, there's a negative stuff outside. And you're taking the model, the model has to kind of become, close the circle. So it will basically have, all predictions will be here. So it can be widespread.

• 34:13Div Garg?

  And, and so I think that's a one good way to think about it. Like you're kind of like giving it more what to do and what not to do. And the what not to do, I think that kind of like thing becomes very useful. Especially when you're doing this kind of reinforcement learning where you're like, like this plus one, this minus. And then so I think at the end of the day, it's a very intuitive algorithm. And I think just beautifully formulated using like a lot of reinforcement learning, like, like algorithm, literature and like principles. And, and then overall, I think it's like kind of like gradient descent towards the positive samples. Gradient descent away from the negative samples. And then you kind of like counterbalance that by summing the sum of like, like all the samples and dividing that to normalize the factor. And I think that's basically it.

• 34:54Nathan Labenz?

  I know in like typical instruction tuning, much like the pre-training, it's literally just token by token evaluation, right, of the output. And then for PPO, I know there's like a reward model that is responsible for sort of scoring all the tokens and that reward model ultimately gives a signal of like this token is really good, this token is not good, whatever. With the DPO, I understand that there is no reward model.

• 35:26Nathan Labenz?

  And yet, if I understand correctly, it's like, it's not just training the model to predict like exactly what the tokens were, right? Because with the negative examples, like you can't just say, don't do that token. You have to say like, okay, well, so what, right? So can you give a little more intuition for how, if I have whole generations that are scored positively and negatively, how is the model understanding, or how is the algorithm translating a score that is like not token by token into something that can be ultimately applied token wise to update weights through back propagation?

• 36:06Div Garg?

  So I would say first is the initial DPO that was formulated, I think that's token marks. So the first, the original DPO paper from Rafaeli Co. So I do think that that one's, you're kind of giving a positive and negative feedback for each generation of the model. And so it basically becomes like supervised fine tuning. So you're saying like, here's this generation, this gets positive score. Here's this generation, this gets negative score. And then you can kind of like just directly

• 36:33Div Garg?

  put that in like a simple loss equation and train on that. So you don't have to think about trajectories in the original DPO. The one modification we did in our agent Q paper was kind of like come with a trajectory level DPO. So when you're like working on environment, you're taking multiple steps. So you are not like to start putting like maybe a single token and like finishing that. You have to like keep taking a lot of steps until you reach your end state. And then, and then you can kind of generate a lot of trajectories. And like once you've generated a lot of these trajectories, sub trajectories and false scores and negative score. And it's more close to maybe what PPO does usually in the ring for signing. And then for this to work, then you have to think about like, like, how can we apply this DPO algorithm on a more like a trajectory level? And like all the scores are for trajectories, not for individual steps. And I think then we propose this like trajectory level DPO that you can find in the paper.

• 37:19Div Garg?

  Where like, like, how can this work? And like, how can we balance? So if you have like different attractors. And I don't think so. There's like, yeah, so the original DPO, I think I was calling like very simple, where like it's working on like word token or post up. So you don't have to actually think about directions.

• 37:34Nathan Labenz?

  So in the original version, you're saying basically my positive score or my negative score is just applied to all tokens equally and then that signal is propagated and that's it. Can you give a little more intuition for how the next generation works? You know, this is maybe more tedious than some more interested in, but I really do want to continue to develop my intuition for what exactly in this, as precisely as I can understand it, what is the signal that we are actually sending into the model? Cause I feel like that's really helpful for

• 38:14Nathan Labenz?

  at least having some foundation on which to make like guesses or on which to interpret the downstream behavior that comes from that.

• 38:21Div Garg?

  Yeah, no, that makes sense. Also, at the end of the day, the signal is kind of like the positive or negative score for a generation. So I suppose like you're applying DPO to a large image model and then you say like, maybe like, I want you to maybe like, say like, who's the current president of the United States or something? And I suppose like the model knows the right answer and you can give it like, like this was a generation, so you give it a plus one score. But if it comes with the wrong answer, then you'd be like, like, this is not correct. And then you can give it a negative one. And then you can do this multiple times where like you can ask here, you can generate 10 generations and then you can have different voter, different people who are like giving the score or it can be the one person who's giving the score. And once you have enough of this, like this for all the positive generations, here's all the negative generations. And you put that in DPO and you're like, like now the goal is to

• 39:05Div Garg?

  improve the sort of like optimize the probability of outputting the positive generations and minimize the probability of outputting the negative generations.

• 39:14Nathan Labenz?

  Okay, I still want to keep studying this a little bit more, but maybe that I'll leave the rest for another day. Returning to the techniques in the paper, can we first just talk about like, what is the definition of agent Q? Like, the, just to be sure I'm clear on that. And then I see, you know, it jumps off the chart that enabling the Monte Carlo tree search drives a big boost in performance. But maybe you could sort of talk about what are the, in addition to that, like, what are the biggest drivers of improved performance? But first, just give us like, what is the bundle that constitutes agent Q?

• 39:51Div Garg?

  Yeah, like I would say, agent Q, again, is a simple idea. I think it's kind of like it borrows from, I think, Richard Sutton's bitter lesson. Where like, the only thing that seems to work is like, search and learning. And I think like, we were kind of very inspired by that, like, even if you look at like Norman Brown's work, like, what he did on diplomacy at Meta, and then like, the stuff he's working on currently at OpenAI. A lot of that is like, like, how do you combine search and like, how do you make that, like, use that learning process to make it more intelligent, like, algorithms? And so we did a similar thing, where we're like, like, like, search just seems to be very underexplored, like, no one has really thought about it, like, how do you do search for agents? Where like, that seems to be like an obvious thing.

• 40:30Div Garg?

  And we're like, let's, can we now, like, combine all this, like, Monte Carlo tree search, where like, it's easy for the agent to go explore the state of different environments? And then once we can explore these environments, can we get a lot of like, this kind of like, positive and negative reward? We're like, it's for the positive generations, positive like territories, which are actually used the gold, for the negative territories, which fail to reach the goal. And once we have that, then can we put that in a learning process? And then like, keep optimizing, so like, maximize the probability of positive territories that actually use the code, minimize the probability of territories that don't use the code. And how the MCTS helps is it's like, we could be kind of giving like, explore as much as possible.

• 41:10Div Garg?

  So we're kind of like, adding a lot of entropy, where like, it's possible, if you're not exploring environment, maybe just like a very narrow path that the model has learned, and then you just, things like, correct, this is just the one thing that you do on the environment space. And so it just doesn't know how to navigate the environment. But we tried to add a lot of entropy to the initial exploration process, so like, the model is trying to go on like, as many routes as possible. But then over time, it's finding out, okay, like this route was like that and so, okay, this didn't actually reach the goal. So let's avoid that in the future. This were the positive ones. Let's keep doing that. And that's where we like can train a model that has kind of seen the whole environment's bits. And has learned to exactly like how to reach the goal. And and then we keep doing that and again and again until we can make this model like self optimize and better and better to maximize the positive.

• 41:54Nathan Labenz?

  So by applying all these things, and I I was, I didn't realize until this conversation that this was a relatively quick project with like not a ton of compute presumably put into it, although maybe, I mean, you can, you could spend a lot of compute in a day, but I assume you didn't spend that much compute in just a, you know, a quick sprint of a project like this. You gradually climb all the way up, you get to ninety five ish percent on the one benchmark that is better than human performance. And this sort of gets philosophical pretty quick, but we see this all over the place, right? Where it's like, oh, it's the AIs beat humans at this benchmark and that benchmark and all these benchmarks. And I do take that very seriously as a signal of

• 42:38Nathan Labenz?

  how far we've come. And yet, at the same time, if you just looked at all the benchmarks, you would think like we definitely have some sort of AGI uh, running around and yet somehow we don't quite yet. So I guess, first of all, how do you think about that divergence between the AI performance on a benchmark versus a human performance on a benchmark? Like it's still, I assume you would agree, and I would say candidly in my testing of the agent, like it can go do stuff, you know? It can accomplish some tasks, but I'll bet on myself in a Paul Bunyan first machine style competition between me and the multi on agent still. So why do we not, why do we see these benchmarks with the AIs winning when it, you know?

• 43:24Nathan Labenz?

  Obviously, every, you know, people design them to try to represent the real world and I I actually did go look at the the webshop thing and it like, it it looks like a pretty, a little bit bare bones, but like a pretty normal ish e-commerce environment. Are people not trying? I don't know. I've come out, you know, one idea that comes to mind is maybe people aren't trying that hard on the benchmarks, but I don't think that's probably usually the case. I don't know. What's your theory of the apparent divergence between what the benchmarks seem to be telling us and what we actually then see in practice?

• 43:55Div Garg?

  Again, call it like a difference between our shield world scenario or the mock scenario. There's like a benchmark is a more like a narrow domain, like maybe a couple of, like the web, the virtual environment has like maybe like five, six different like environments. And what we also do is we're training on each different realm and each different site. So we can optimize the model to be very good on that side. When we do more like general preparations and that's working in the real world, like until we have the leeway to go and like optimize this agent on every single website. I think like the performance won't match, right? Because like a benchmark is just a very narrow domain, so it's easy to set straight the benchmark for us to set straight the entire internet. That is just a much more challenging problem. I think it's possible, I think about it. But I, I think it's just much more complex and like to pass more resources.

• 44:38Div Garg?

  Now, I, I do think that's kind of like where this thing starts emerging, where like a benchmark is a, like a simplified version of what you will see in the real world. Um, but if you think about it as a vertical, like, here's this one narrow domain that I really care about. And then we take a look at the average human accuracy on the domain. And then we take a look at like, what can we do with agent Q style method? And then the surprising result is that like in this narrow domain, we can actually be like human. And then, and that becomes like sort of like a very interesting learning. Okay, like now it can be like a human in a very narrow domain. Then, okay, like over time, you can keep generalizing and generalizing. That you can like start being humans and like a more generalized domain. But the question then becomes like, okay, like how much data do you need? How much feedback do you need? How much skill do you need?

• 45:23Div Garg?

  How much resources do you have to throw up? I think that's kind of like an interesting skill problem. They're like, they're like. I, I don't think we've kind of figured out like the fundamental ways to solve the problem, but then you have to apply scale to make this actually really shine.

• 45:36Nathan Labenz?

  Are you to the point now, I mean, of course, this is like the big, there's obviously multiple big stories, but I feel like a good candidate for what's different about the current era of AI versus previous eras of AI is that we see positive transfer across lots of things, right? Like it used to be the case that if you tried to train a model on two tasks, it wouldn't be as good as two models trained on one task each. Now you have with foundation models, this sort of like foundational ability that seems to be, you know, rather quick to generalize to other things. And training on a million tasks turns out to make the million and one easier to learn.

• 46:21Nathan Labenz?

  Do you have a sense for where you go from negative to positive transfer as you scale up? An agent like this, like I presumably doesn't happen if you go from just open table to just open table plus Amazon. But if you keep, if you just kept adding, and I'm sure you've, you know, explored this sort of thing, if you just kept adding different sites. Is there a point where you start to see positive transfer? Like, do you have a sense for where the the threshold or the tipping point is there?

• 46:52Div Garg?

  Yeah, I definitely see this, right? Especially, I suppose you have like a lot of shopping domain websites. You know, it's like Amazon or Target or Best Buy, Walmart, whatever. The most of these websites actually look similar because it's kind of like a search bar, product catalog. When you have some sort of like detailed product view checkout. And and and there's a lot of positive transfer, but like even if you're playing on one website and then you were like trying to generalize it to like similar domains. I think you actually get a very good transfer. So I think within each category, I think there's a really good positive transfer. And overall, to I think like the internet is not it's, I think it's made for humans to use. It's not that diverse. If you think about like the different UIs and like elements that you encounter, you basically see like a lot of like dropdowns or drawers or like

• 47:35Div Garg?

  navigation bars and whatever like text fields, but I think like there's a lot of commonality. So the model can like learn how to work on like a lot of these interfaces. I think there's a lot of positive transfer where it can work on new websites and then like even touring. And that's also like one notion that you're very bullish on in what you're doing with that agents, that they can keep making this page and spatter and butter. And over time, I think it, I, I do think there's a scaling law here. Well, like once you do hit enough scale in terms of like the number of websites you've trained on, then you can turn this to the remaining websites.

• 48:08Nathan Labenz?

  Let's come back to the, the scale, the question of scale and the scaling laws and the possibility of better lessons and for whom the lessons will be better again in a minute as well. Just staying on your model strategy for a minute longer. I'm curious what you've learned about fine tuning. I feel like there are, you know, obviously lots of different strategies for fine tuning. Laura seems to have kind of become the default thing because it's efficient. I was a big fan, or at least I was like very excited to read a paper called Mora from not too long ago, which was an alternative that was like similar number of parameters, but higher rank.

• 48:49Nathan Labenz?

  And the math on that escapes me a little bit, but conceptually they reported sort of a denser fine tuning, denser use, more intensive use of the available parameters. And they did report that it was better at learning facts, which I thought was really interesting. So interested if you've experimented with that or if you're just biting the bullet and doing like full weight training.

• 49:11Div Garg?

  Yeah, I, I don't think we see like a lot of improvement, improvement just with Laura. I think Laura is like a very efficient, it's a very good way to train these models and funding them on, on, on your application. I, I do think there's also like a lot of variants for Laura. I think there's like PAF to Laura and like there's much of like this kind of like variations and like a lot of this work really well. I don't think like Laura works really well. There's definitely a lot of this new innovation that's happening where people are coming from more alternatives. So I don't think that's an interesting space to watch for. Um, I think for training can help, but again, is that about how much data you have? So if you have like. Crazy amount of tokens, you have like billions of tokens, then definitely you should be like fully fine training. But if you have like hundred thousand tokens or in that manner, I think then low rise actually like a better method. Because you have less parameters to optimize.

• 49:55Div Garg?

  And so if you have less tokens, less data, then I think that's actually a better use um of of that data. And then you actually get better performance. Um. And but if you have like billions of tokens, then I think like you should be optimizing more parameters. Yeah, I think the good bit also thing about this is change your last killing lots. Like, what is the optimal data for optimal number, like the size of a model? And so if you have like this many parameter model, this is the amount of data you should be throwing at this model. And I think that's a good way to think about like lower versus not lower.

• 50:25Nathan Labenz?

  I guess I would maybe expect you would have billions of tokens. It sounds like you're not, it sounds like you're not using huge data sets. Obviously, quality is a super important dimension. Should I infer from this that you have been just trying to optimize the quality of the data set that you're working with? Over quantity and it's just not that big that you the Laura approach still is enough?

• 50:50Div Garg?

  Obviously, we've tried a couple of things. One thing we definitely care about is speed. And then if you're training over like billions of tokens, that is a very long process. And I think so, so that we have some experiments have been running. But the part is we care about like quick iterations. And so what can we do with like a lesser amount of data? And how can you make more sufficient use of models? Let me deploy in production and then like make his offer for our customers. So so in in that scenario, we have found like if you look at eight queue, like we did like a couple of days of training and we were able to get very good performance on this like narrow domains. It actually is a better thing if you want to build a product. For like doing like pure service, then I think we should be just spending like six months. Train like hundreds of years of tokens and like kind of train like the best general agent possible.

• 51:35Div Garg?

  But yeah, but the learning so far for us, sorry, I also will say for the spaces, it's like. It's much easier to build a narrowish and it's also very easy to much, much easier to prioritize it. Unless you want to be like a foundation, a model company.

• 51:49Nathan Labenz?

  Yeah, okay, gotcha. What is your vision strategy right now? Is that all? I mean, of course, by vision, I mean, like, interpreting what the agent encounters on the computer screen. In the early days, we were seeing lots of examples where people were trying to parse the DOM of the website and figure out how to, like, strip all the crap out of the HTML so that it would fit into the context window. And now, of course, we've got much more in the way of multimodality that we can take advantage of. I mean, you know, in my testing, I did find a couple of weird places where the agent sort of got stuck and like was saying that it couldn't find something that was like just plainly

• 52:32Nathan Labenz?

  on the screen. It wasn't clear to me if that was like, just, you know, I don't know, I don't know what was happening there. Like, was the model not able to see well? Or was there some sort of weird situation, like an iframe, that would could possibly be causing some visibility issues? So I guess two parts of that question, like, what's your overall strategy for interpreting the visuals? And maybe what are some examples of just weird challenges that pose problems, you know, relative to kind of the happy path that people would think of first?

• 53:02Div Garg?

  Yeah. So I would say at this point, like, I think we're doing a lot of hybrid pipelines, but I think there's a lot of, like, useful structure you're taking up from the top, where there's a lot of, like, meta tags and other labels, which is kind of like optimized for bots. And then you have a lot of, like, the visual data, which has a lot of layouts and stuffs. And we're doing a combination of both. So we try to be hybrid, where like, can we accept the past of both forms? And then can we use it to train the models and do things? One challenge we always, I think this is something that throws off a lot of people, is because if you're a human, you kind of expect these things to fully work like a human, right? But what's happening is like the representation of the screen on the UI that the agent is saying is different from what you're seeing.

• 53:46Div Garg?

  And then and that makes like the behavior of the agent could be like a bit weird where maybe it's able to click on invisible elements or it's able to make like, it doesn't need to scroll down, it can just like kind of like see the whole like, if there's a website with ten pages, maybe just is able to see like all the ten pages without needing to scroll down like a human. And I think that kind of like throws off like people from our experiments where you have to artificially figure out how to constrain it to be more like a human. And and then there's also this uh things where maybe like you might have some sort of elements which are tricky to detect or identify which maybe like are very obvious for a human, but like like maybe the agent is not able to see that, maybe it's some sort of like uh complex service issue or some sort of IP issue. I think like right now we don't see a lot of that. I do think like in our new prototypes, I think we have been just like trying a lot of different things.

• 54:30Div Garg?

  So they're not, they might have some deficiencies there. But I do think like a lot of the mature products we have, I think like we have been able to solve a lot of these things over the last uh couple of months where like now the our pipeline for processing like the information on websites, I think it's it's very robust.

• 54:47Nathan Labenz?

  Interesting. I I realize too, there's probably somewhat of an adversarial situation happening because what I was trying when I observed that was and just to to give people a little bit of a sense for the feel. So you added me to a test flight where I could go, you know, download the in development multi on app. And then basically open the app, tell the agent what you want it to do. It then has kind of a dual interface where it's like telling you what it's doing and allowing you to, you know, pause it or give it additional instructions, feedback, whatever as it goes. And then if you minimize that, you're just watching the screen and it can also talk out loud to you. So you can basically hear it narrate its progress.

• 55:29Nathan Labenz?

  And you're watching the, you know, the state of the browser evolve as it goes on and and does its thing. Well, I'm in Detroit and it's the day before Thanksgiving as we're recording. So we've got the Lions playing their annual Thanksgiving Day game tomorrow. And Michigan Ohio State is also this Saturday. So I just asked it to go get me tickets for each of these games. It tried both. And it was it was successful in terms of like understanding my query. It was successful in terms of doing like an initial search. It actually did a couple of different approaches, you know, and different trials. It in some cases like. Went to Google and searched. In other cases, it went like direct to a ticket site from just

• 56:14Nathan Labenz?

  prior knowledge. It was able to search successfully on the ticket sites. And then there were a couple of instances where it was like, those tickets were now on the screen, but it would say, I'm having a hard time finding the tickets. And especially if you're saying that's rare overall, I suspect that possibly part of what's going on is, you know, obviously these ticket companies have bot wars going on for, for many years now at this point, where they're trying to prevent people from buying up and reselling and whatever. I don't know, no expert in the tick market, ticket market, but I know it's complicated. I guess all that is to say, what do you think is the, how do you describe the dynamic right now?

• 56:58Nathan Labenz?

  Maybe I didn't, I wasn't thinking about this at the time, but as I am thinking about it now, I'm thinking, you could imagine this cutting multiple different ways, right? You might imagine new off frameworks for agents coming online. And by the way, I also, another test, I didn't have to log in for these ticket sites, but another test where I did need to log in to create an account. I tried shopping for Thanksgiving dinner on Instacart as well. To do that, I had to, you know, at some point I got to a, you must create an account screen. And I just kind of paused and like, I put in my email and then, you know, when it sent me the confirmation, I had to go get that out of my email and provide that to the agent. So it could provide it. It was able to fill that, to do those steps with me putting that stuff into the chat. But you could imagine like different

• 57:41Nathan Labenz?

  parts of the world evolving very differently where Instacart might say, we want to be an aggressive early adopter of whatever sort of off paradigm is coming for agents. Because we obviously want people to shop however they want to shop. And if they want to have an agent shop for them, more power to them. Whereas a Ticketmaster or a StubHub might say, we want to ban all agents effectively because in some way or other, they're eating into our margins. And so whatever countermeasures they might put up. That's a long prompt, but I'd love to hear your thoughts on, if you see any good off frameworks starting to emerge? If you are seeing like countermeasures in the wild that you feel are

• 58:25Nathan Labenz?

  kind of anticipating an AI agent wave and, and trying to resist proactively. And I guess just kind of generally, you know, where do you see that sort of thing shaking out?

• 58:36Div Garg?

  So let's see, first one, definitely authentication, I think it's getting more mature. From I think there's a lot of authentication providers, I think like uh. Um, Anon is one, I think that's been trying to build a lot of agent identity for browsing kind of stuff, I think. As uh, there was a new one that came out, agent auth or something for the APIs and like plugins integrations. So I don't think that's uh, it's starting to get mature, but I've been thinking about it a lot. And I I don't think that's a major, become major enough that like everyone can go and like work the likes of authentication provider and like. You're good to go. But the second question becomes like, are the current services and websites gonna be adversarial or cooperative? I think long term they will be cooperative, because I I think it's kind of a win-win over time.

• 59:20Div Garg?

  A short term, it's hard to say, because it's possible like some people might just take their um. Uh, direction or the positioning, more to this is kind of like the harmful behavior, it's kind of like a more like spamming. And then that kind of becomes like, how do you build trust? How do you make sure that you're like, you are actually helping uh, the website owners and the services? And the merchants. And I I don't think there's a win-win situation where like, as this is mature, I I think it's transforming a lot of the experiences. But it's like, if if a merchant is too care about like the revenue and about users. And I think that's something that begins to help. It's just like the, I think it's like, and when anything's disruptive, I think it takes a lot of time for that to catch on. And I think we are at the start of the disruptive era, where like a lot of the.

• 1:00:03Div Garg?

  Online communication and interaction will, they'll get disrupted by it. And I think like, there might be a lot of like initial backlash, where like, like this is kind of dangerous and this is not safe and this is uh. Causing whatever these issues and but I think over time, I think it's a win-win. Because it's just a different paradigm and it's a tennis better paradigm, uh, and long.

• 1:00:23Nathan Labenz?

  So yeah, let's look ahead a little bit to the future. I've got kind of multiple related questions. I mean, you guys have been. One of the most. You know, quick to put things out there and try anything with this and see what happens and it's been very open-ended. As you've described in this conversation.

• 1:00:43Nathan Labenz?

  You know, you've not bucked the trend when it comes to getting somewhat more narrow and being like, okay, you know, let's really nail some dialed in use cases. So where are you today on that? Are you working with like businesses? And you know, if so, I'd be really interested to know what sort of use cases they are finding agents to be valuable for. I think that's probably like the the probably the most important question for all of this stuff is like, where are people finding value today? And then, and maybe that also kind of answers like, why the off stuff isn't so much of a focus at the moment? If it's like, well, if we're working with enterprises, we're in their off environment and it's a whole, we don't have to worry about signing into StubHub or whatever.

• 1:01:28Nathan Labenz?

  So yeah, I guess I'm really interested in like where you see, you know, you've mentioned like a six month timeframe. What bets are you making right now to, you know, start to bring real value to the world in the not too distant future?

• 1:01:43Div Garg?

  So I'll let you say like verticals, I think like more like vertical use cases, that's what we're also doing. Internally, I think I don't think that's the actual ambition. I think the agency was kind of like our first time and the direction, okay, can you have vertical agents that we can like train and learn on? So you don't want like something that's purely hardcore, it's you don't want like they write script or like sort of like scripting language. That's it. Because it's purely hardcore, it's very like brittle, it will just break all the time. But you don't want like some sort of like, like if you want this model to be able to adapt. So maybe like the interface change, it's able to like recover, it's able to like, like improve. And but like, I still wanted to be like narrow now. And I think like agent queue was kind of like showing, okay, this is kind of possible.

• 1:02:23Div Garg?

  And that's one thing we've been working with a lot of our clients on, where like, can we now ship this kind of like more narrow agents? For specific use cases, which could be maybe, I don't think it's explore a lot of.

• 1:02:37Div Garg?

  The sectors, so let's say like a good example is maybe like restaurant reservations. Or if you want to do like travel kind of things, if you want to do like like scheduling. And I think there's a lot of these kind of sectors and I can't say too much about the batch we're making right now. And we feel like it's like, like, how can you kind of build this kind of like agent, uh, like verticals. Where like this age gives becomes very current, this particular vertical. And then keep optimizing it and making it into like a solid product experience. And then there's a lot of you to think about, you know, sort of reliability and then like optimizing for the user attraction and the human that do open much other things. I think people are starting to think about it, but I think it's still very early. And I think like building a product is just a complex thing because it's just not the agent. I think there's just like a ten or kind of more things.

• 1:03:20Nathan Labenz?

  Yeah, I think when what I'm kind of inferring from your comments is that you're kind of finding a a third space that I hadn't really considered as much where one approach would be to say, here's a, here's, you know, consumer, here's the, you do whatever you want. Here's our open-ended agent. Another one would be to say, we want to kind of compete in like the business process space where it's like back office, not visible, super structured. And then OpenTable kind of represents a sort of a middle ground where you might say, we want to, and maybe Instacart would be another, you know, great example of this, or Kayak.

• 1:04:02Nathan Labenz?

  We want to be the partner that creates an agent for your site that could still be consumer facing. But because we can partner with you in a deeper way, we can really dial in that reliability and get that product experience to be exactly what you want it to be for. The kayak assistant agent or the Instacart agent or the OpenTable agent, or obviously people can fill out the list of your CRM for themselves. But am I headed the right direction there? That's, that's not what I'm reading between the lines from the agent queue paper.

• 1:04:37Div Garg?

  Yeah, yeah, I do think like this can be applied even to business processes of the industry. Because they're there to, I think, like, if you look at the current issues people have with UI path or other automations offer, I think it's just like, it's very brittle, takes like a lot of like onboarding RAM, but requires like special like engineers who are like just really good at building this kind of like automations. And I think if you can decrease the complexity, that also knows a lot of use cases where like, if we can enable anyone to build this kind of like automation, even for business processes, and this automation is a resident, then I think that also is a big enough. So I don't think it can be applied there, and I think there's like, like, interesting use cases there that also we might look into. But we do, at the end of the day, I think, like, we're just trying to build something

• 1:05:20Div Garg?

  that's more like, like, every, everyday purpose, in a sense.

• 1:05:24Nathan Labenz?

  When you think about being that layer for some of these complicated products, you know, like a kayak, right, there's a lot of UI. If you were going to do a partnership with a kayak or with an open table, would it still make sense to work through the UI? Or would you start to augment the UI action space with like, a specific sort of more API like, you know, or tool kind of modality? Because you could imagine the agent might want to, you know, again, thinking of kayak, right, all the little UIs in that sidebar. It seems like the AIs would have a better job if they know that they're the kayak agent,

• 1:06:08Nathan Labenz?

  and they don't have to, you know, generalize beyond that. It seems like they might not want to go like, drag the sliders for the time interval that the user requested, but rather just like, make a sort of declarative statement to the system that like, I want to narrow the search in this way, more kind of function calling like. I realize I'm kind of, a couple of layers of maybe speculation deep here beyond what you've explicitly confirmed, but how do you think about the hybrid between the fully general UI path to, you know, taking the next step and, in some cases, what would seem to me more reliable of making like a function call?

• 1:06:49Div Garg?

  Yeah, I I think we we have been thinking about that a lot. I I think we have some things we're working on. I I don't think that's an interesting action, but I think it just takes a lot of time. Also, a good analogy here is like you think about self-driving cars, right? So it's you can imagine like if I was to build like a self-driving car from like sketch today. My I can just say like, let's why not just go and construct special roads on every highway? And this is special, like this is special lane, and that's the lane is for like the self-driving car. And the self-driving car just use the lane and the the lane has sensors and everything ready and the self-driving car just has to follow the lane. And that's it. Like, you basically don't have to do any R and D. You basically just build the lane and you kind of put to go. It's a very simple problem in terms of engineering research. The thing is like, when you want to do something like that, it's uh just like very complex, because if you change a lot of behavior, it's a big infrastructure project.

• 1:07:35Div Garg?

  Everyone has sort of this kind of like new thing and that will cause like, if you think about this, it will take like ten years and like a lot of like politics and convincing people and like getting there. And only that can do that. And then like a lot of self-driving cars, startups decided like, okay, like that's probably not gonna happen anytime soon. Let us build a car that can like, like, let's build Optimus car that drive like a human car on any lane. And let's go solve this problem using the car of R and D and like make that fun. And because that's how humans like navigate the world. I think we have a similar analogy here, because I think like over time, you might have this kind of like a spatial internet infrastructure. That allows agents to communicate to websites and then like do more reliable function calling. But again, that becomes like, like, who is incentive? So people like move over that. I think that it just takes a lot of time.

• 1:08:17Div Garg?

  And like, I think I do think this is like a multi-year thing, even if you were like someone was to start this. And I think we've actually been looking at that. So I do think that could be like what the future looks like. But in the meantime, until like everyone agrees on like the here's this new protocol and like, or here's how we'll do this communication. And like convince everyone to adopt that protocol. I think like like basically being able to like use these websites, you know, to how humans use them. I think that's the right way to start from, because I I think that's also the benefit of AI. The benefit of AI is like whatever a human can do like for the AI, that enough data and learning can be able to do that. And so that's how you're trying to do where like like like humans are pretty good at like navigating websites with UI. And so like theoretically, like AI can also become very, very good, even better.

• 1:08:59Div Garg?

  And so if we can have this kind of AI, which is like as good as human or better. I think that's a good way to solve the problem immediately. And it's just like, how fast can we get there? Once you get there and then you're like, okay, like now maybe it's time for something like different, where we just want to take change and flip the paradigm. And then I think like that will happen. But I I think that becomes like a buying problem. They're like. It just like a massive change in behavior and nothing that takes a lot of time.

• 1:09:24Nathan Labenz?

  A lot of these sites though do have like a, I'm just pushing a little bit in the middle space. I mean, I hear you on like the road, you know, the dedicated lane for the self-drivers. I've been calling for that since I was in school and maybe that'll be one of the pleasant surprises that we get from a new Trump administration, although I'm not holding my breath. But there's been at least a little talk. I totally get it when you map that onto an enterprise. It's like, hey, guys, who wants to implement this new agent protocol? You know, no hands go up or maybe one does, but you know, the boss doesn't like it, whatever. I get that complexity, but a lot of these sites do have like an internal API, right? I mean, is, isn't there something in the middle in a lot of cases that does exist?

• 1:10:07Nathan Labenz?

  Like what the UI talks to in many cases is an API, right? So is there a, does it make sense in some cases to just be like, okay, kayak? You've got a real bear of a UI, but this data structure that manipulates is easier. And this is like an actual live question for myself too. I mean, I'm, I actually have an episode coming up that I'm still putting some final touches on for people that want to do what I call building software that uses itself. Kind of trying to coach people that are building apps to create like the, you know, the magic AI functions that, that their users have always wanted, even if they didn't know to ask for. And that's kind of my paradigm.

• 1:10:49Nathan Labenz?

  So I wonder how you react to, you know, I sort of see this equivalence between UI and AI, not everywhere, but in like a lot of places and encourage developers to take advantage of that. To say like, okay, instead of asking your user to use all this UI, have one button, ask them what they want, and then have the AI translate that to the structured data that sits behind that UI. Now that doesn't scale super well, right? Because it, that wouldn't be something you could do for every website. But if you did want to do a kayak or an open table partnership, it seems like it would be viable. And it's certainly if I'm like talking to developers that are working on their own app, you know, it should be viable for them. So I don't know, any other reactions to that? That sort of attempt to find the Goldilocks zone?

• 1:11:33Div Garg?

  Nah, I I do agree like this is again something I thought about. Yeah, I've been thinking about like actually proposing some sort of like open standard around this. Where you can have like like actually like a sort of like standardized uh way to communicate between like agents and websites. And I I I didn't think that can make a lot of sense where like all of these websites have back in APIs. That will like directly interface with. I think the biggest issue becomes like security. And like okay, like I was like, how do you use the APIs get used? Who's going to call these APIs? Identification, where like how do you identify who's calling these APIs? And there becomes like patterns of use. Because like I think if you're a website like Airbnb or DoorDash. I think they have all the patterns of use where like they know like this is basically what the user to like use a website and it's optimized the

• 1:12:15Div Garg?

  website for that particular pattern of use. And then the what happens is like if you transform yourself into like an API first business. I think that just changes a lot of paradigm because now you have to kind of fight against the product teams and maybe like a lot of your front end teams. And I think like the backend in your house don't want this, like you're exposing a lot of like security loops, you have to think about talk to a bunch of things. So it becomes very hard for enterprise to navigate that landscape. But I don't think like if you think about engineering wise, yes, it's possible. But I think it's just like a very different business so patterns and I think very different security and like identity patterns, which just complicates the landscape a lot.

• 1:12:47Nathan Labenz?

  Yeah, gotcha. How about I I put a pin earlier in just like cost and latency? This is a very common question for kind of anyone uh building in this space. What have you learned about what really matters? And you said you care about speed, certainly in the context of like your own development iteration. Uh, how much do you think this matters for the end user, you know, for your customers, if they're partner businesses and the end users? Um, what's the sort of optimization problem look like? Cost, reliability and speed?

• 1:13:24Div Garg?

  And it's make for our agents or like, oh, I'm I'm just curious exactly what it might not.

• 1:13:30Nathan Labenz?

  Yeah, I mean, you can contextualize it however you want, but in some sense, that's the, you know, the oldest question in software, right? Good, fast, cheap, pick two. Some would say in AI, the magic is you can maybe have all three, but you're still obviously making some trade offs. And, you know, you said kind of 70 B is like a good compromise. Is that because you like have users like sitting there waiting and you want to return for them quickly? Is it? Yeah. So I'm just kind of curious as to how you're thinking about, you know, what you're willing to pay in terms of financial cost and latency for, you know, marginal improvements to reliability.

• 1:14:07Div Garg?

  Yeah. I think it comes down to the use case. If it's a high risk case, like you want to do some sort of purchase. And you're like, I really want to confirm and make sure that we buy the right thing. You don't actually make the wrong purchase or like, like you're doing travel or something, just make sure that everything's correct. So how much does it matter? And then because like it's a, is it a soft boundary versus a hard boundary? So a hard boundary is something like if you do it, it's irreversible. And if something's irreversible, it's very costly to the user. And you want to be like, very careful. Like, like maybe you don't really care about how much time it takes. Maybe it takes you like, maybe like 20 minutes or something potentially. Maybe it's, it can be like more costly if you have to do more reasoning through more compute. But the job gets delayed every done all the time.

• 1:14:49Div Garg?

  Then I think a lot of people would want to live with that world compared to just like, yeah, maybe it's very fast, but it's very low reliability. For like higher, like this kind of like hard decision boundaries. There's some things which are like more soft decision boundaries. Like chat is a good example. Like if you're taking a chatbot, it's a soft decision boundary. There's something irreversible about it. And there's like some patterns like that, like suppose maybe like you're like doing some scheduling kind of things, maybe like occasionally have some small things that you can like fix.

• 1:15:15Div Garg?

  But a lot of agitatic behavior is like hard decision boundaries. And then so you want to be careful about, okay, like before you fully actuate the action and like complete the task for the user, that the task is actually like correctly done. And like before you do something irreversible. And detecting is something is reversible versus irreversible. But that is also something that I think I've spent a large amount of time thinking about. Because like if something's reversible, we can just have like a week, like a small model and stick it to that and then it's fine. But if something is reversible, then you're like, you need to build a lot of verification, make sure that everything is like as good as possible. And there's no edge cases. And then we then take that irreversible action.

• 1:15:51Nathan Labenz?

  Yeah, that's makes sense to me. Okay, one more big question, then I'll invite any other comments on things we didn't get to. You know, and I should have probably said this at the top, but full disclosure, I'm a, you know, very small investor in the company. And I mostly don't really invest for returns so much as to support things and people that I think are cool and want to see come into existence. So I've been fascinated by what you've been building basically since the beginning. I do wonder how you think about kind of positioning yourself to avoid being a casualty of the bitter lesson here. We see like Claude, of course, now has computer use out and it is still not super reliable

• 1:16:35Nathan Labenz?

  when it comes to step by step advancing through tasks on the web. It does bring though, some like really nice advantages in terms of being like an outstanding writer, you know, there, there are some like qualities of Claude that are really hard to get anywhere else. And then of course, OpenAI is always rumored to have something coming soon. And right now, the rumors and hints, I would say, are suggesting that there's an agent framework coming soon too. So what is an agent company to do that hasn't raised, you know, $10 billion to compete effectively when, you know, they're clearly coming for you with incredible scale on some timeline?

• 1:17:18Div Garg?

  Like, again, I would say it's kind of like finding the niche that you are, that you want to focus on and then like going after that really well. I don't think that's how, like if you look at a lot of the successful AI products, perplexity is a good example, and they just chose like, like search of the niche and they just take first and hallucination is the one problem they were trying to solve it. They're like, yeah, like this problem, models have issues with hallucination, that is all fits hallucination. And that's basically what they were focusing on fully more or less, and then building a product experience around that. Curse is a good example where they're like, yeah, like that's a builder's coding ID and like that's free for us or not. And and they they kind of do the same thing with agents, because computer use at the end of the day is capability to the product. And so you will have this action as capability, but then it's kind of like, what's the one thing that you can really clearly do to leave out?

• 1:18:01Div Garg?

  And then how do you optimize the loop, how do you build a product experience? And the thing is, like, most frontier AI labs are actually not interested in this, because like, if you have hundreds of billions of dollars, so you could, you don't want to go off like a narrow domain. It's like, you're like, I just want to throw in a computer and like, sort of like, build a general thing. And it is great, because that's kind of like a foundation block. But then I think like, someone needs to do, take that general purpose thing and build like the right product experience. And I think that that kind of becomes like, that unlocks a lot of like, market opportunities where like, like, there'll be like, a lot of like, different businesses and like, different use cases that are built. That satisfy actual user need or like, product need that is missing right now, using this as like a building block. And I do, I do think that's something that will start to happen maybe like early twenty twenty five, because I think like, this year, but it probably was too early. Uh, I think like, we are probably like, one of the first companies that were providing again, this kind of any kind of like agent capability.

• 1:18:46Div Garg?

  And there, no one was able to build an application, where like, unless we are there in terms of the technology frontier. Like, it is hard to build a product, right? It's kind of like, if you want to build a a card, you have to first invent the view. But if you, if you're like, okay, like, we don't know how to build a view properly, or like, it's like, uh, it's just not there. Then it's like, you can't really like, uh, like, build up a product experience on that. And I think that's where we have been, like, the technology has been like, just very new. And then that makes it hard to build a full experience, which is like, all this lab. But now, I think that as the technology becomes better and better, I think like, early twenty twenty five, we've seen like, uh, like, uh, I've got an explosion of applications, which I don't think has happened till five, I think, I think about eight, I think it's still like, I go.

• 1:19:29Nathan Labenz?

  So does that suggest that you would be open to starting to use these? Like, is the future of multi-岸 like potentially powered by Claude computer use? Uh, you know, with that being sort of, you know, the core capability and you being the product around it?

• 1:19:47Div Garg?

  Yeah, again, I won't say too much here. I think there's a lot of things that are possible. We have a close relationship with an anthropic team and a lot of the whole there and the research team and the marketing teams. Yes, I think I will say another day we are a product company and it's kind of like what gets you across. And I think we are also very innovative, like so if you look at agent Q, I think those are the kind of things that we are able to come up with. So I do think we'll always have this kind of like a R and D advantage, like we are able to come up with a lot of new things. And then like the question for us becomes as the ecosystem matures, how can we combine a lot of our expertise to have like a like a strategic advantage and then use that as a way to like sort of like solve a lot of specific problems.

• 1:20:29Div Garg?

  And I like to call this a product focused research, which is like you have an identified problem and then you're doing the research to solve this problem. Because what happens like most of the research you do, even a foundation, I have that in vacuum. It's kind of like, here's a research problem, let's go solve it. Research is not directed towards anything. But if you're like, here's this one problem that you really care about, it's just like there's a missing block, we just don't know how to fill the block. And like, let's go and do the required R and D to build that block. So we can fit it in and like sort of like complete the piece of the puzzle. So we can reach to where we want to go. And I don't think we, that's how we think about it. Like we have a lot of this capability where we're like, a lot of things are missing. Even if you think about right now, the models and even the computer use, there's a lot of things that are missing. And that's like, how do you solve the problems?

• 1:21:09Nathan Labenz?

  Cool. Well, never a dull moment. It's a fast evolving landscape. I think that's all the questions that I had for you. Is there anything else that you wanted to touch on before we break? Great time to pitch if you're hiring for any particular roles or, you know, looking for any kinds of customers or just anything else that, any other wisdom you want to impart?

• 1:21:28Div Garg?

  Sure, no, we definitely are open for hiring. I think we're always looking for like greater people who can like raise a bar of talent. Also, always open for researchers and engineers and product folks who want to like sort of work in this kind of patient domain. And I think I'm just very excited about like how to like application, especially like a lot of the things you're working internally. I think I give access to one of the prototypes. That's still like a very early prototype, but I think I'm very excited about like what will the right user interaction and the right ways to like use this products to look at. Cause I think that's just the next wave, right? We're like, it's like now you have the agent take product, we're like, okay, like things are happening automatically and you don't have to watch or do things all the time.

• 1:22:08Div Garg?

  But maybe you sometimes you want to take control. Sometimes you want to like, maybe do things manually. Sometimes you want, maybe like you want to like learn and improve. And I think just, that's just a really new paradigm. And I think building a really sort of product there, I think that's a hard challenge. So I think that's something that we are very, very um spending a lot of time on in a sense of like, how do you actually solve this problem and like, how do you make this into the best interaction and like the best experience? And like, how do you fit everything together seamlessly?

• 1:22:32Nathan Labenz?

  Maybe one more question. Where are we a year from now? That could be a multi-specific answer. It could be an agent's generally answer. You kind of teased that a little bit by saying, you know, twenty five is the year. But it's what it's the year for what? You know, can you paint a picture of my agent assisted life a year from now?

• 1:22:54Div Garg?

  Yeah, it's, I was thinking much sad to see like something like, I said, like Jarvis from Adam and then kind of things where like, okay, like now you have this assistant. It's directly like, okay, like look up my files or find me this document. So maybe like book this like whatever, like a flat for me or like do this sort of a boring job and just find it. I don't think like people kind of trying this, say even like last year, people were very bullish on like trying this kind of things, but just again, like the technology was not there. And now if you fast forward to twenty twenty five, I think a lot of these things might start working. And and to me, I'm starting seeing that actual assistants that are like useful for you are helping you in a lot of your day to day. And so I'm sort of like a mainstream product, but it's on agents and a lot of like this kind of like explosion, explosion of like vertical applications, which has not happened yet.

• 1:23:39Nathan Labenz?

  Devicard, founder and CEO of MultiOn, thank you for being part of the cognitive revolution.

• 1:23:45Div Garg?

  Thanks.

• 1:23:46Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.