Scaling Intelligence Out: Cisco's Vision for the Internet of Cognition, with Vijoy Pandey

Vijoy Pandey of Outshift by Cisco lays out his vision for an “Internet of Cognition,” where AI agents can share context, build reputation, and collaborate safely at scale. He offers a useful mental model for superintelligence: progress has to scale in two directions — up, through better individual models, and out, through networks of agents and humans thinking together. The conversation explores how distributed, protocol-driven agent systems could give enterprises fine-grained permissions, auditability, and controlled interfaces, in contrast to today’s centralized frontier models. Vijoy also walks through Cisco’s internal CAIPE system of 20 cooperating agents, the open-source AGNTCY project, and a live multi-agent healthcare demo spanning diagnostics, insurance, pharmacy, and scheduling. LINKS: AGNTCY Project Open source multi-agent infrastructure under Linux Foundation governance. Covers discovery, identity, communication, observability. Vijoy walks through the architecture at [00:34:57] and [00:41:17]. Scaling Out Superintelligence Whitepaper The technical whitepaper detailing the Internet of Cognition architecture, three-layer stack, and cognition state protocols. Referenced at [01:25:40]. Internet of Cognition Interactive Demo Clickable walkthrough showing per-agent activity, intent, context, and collective reasoning across a multi-agent SRE system. Vijoy demos at [01:26:20]. CAIPE Project (GitHub) Cloud Native AI Platform Engineer. Multi-agent system with participation from Adobe, AWS, Cisco, Nike. 20 agents, 100+ tool calls, 10+ workflows. Referenced at [00:11:52]. Sponsors: Tasklet: Build your own Cognitive Revolution monitoring agent in one click. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai VCX: VCX, by Fundrise, is the public ticker for private tech, giving everyday investors access to high-growth private companies in AI, space, defense tech, and more. Learn how to invest at https://getvcx.com Claude: Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (04:16) Cisco and networking foundations (13:34) Jarvis and ASI vision (Part 1) (18:16) Sponsors: Tasklet | VCX (21:09) Jarvis and ASI vision (Part 2) (Part 1) (31:46) Sponsor: Claude (33:59) Jarvis and ASI vision (Part 2) (Part 2) (34:00) Practical multi-agent examples (50:02) Multi-agent plumbing architecture (01:01:44) Agent identity and TBAC (01:15:23) Internet of cognition fabric (01:21:48) Emergent agents and safety (01:36:52) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guest is Vijay Pandey, SVP and GM of Outshift by Cisco. For more than 40 years, from helping to define early low-level protocols that are still in use today, to building out the infrastructure that powers modern high-speed networks, Cisco has been critical to how we manage the flow of digital information. And today, Vijay and the team at Outshift are bringing Cisco's distributed systems DNA to the fundamentally new challenges presented by frontier agentic AI systems. The AI-powered preparation that Vijay and I each did for this episode demonstrates why this is such an important opportunity. I used Tasklet to conduct deep research on Vijay's work and draft a starter set of questions.

• 0:45Nathan Labenz?

  And at the same time, his team ran a deep research process on me and the podcast and identified a number of suggested discussion topics based on themes that we have previously explored. Both agents did a really good job on their respective assignments, but they knew nothing about one another and they had no opportunity to collaborate. Their output was sent from human to human by email, and it was up to me to figure out how to synthesize their work. What's missing, Vijay says, is the Internet of Cognition, higher-order protocols that AI agents need to share context, understand one another's intent, build reputation and establish trust, and ultimately solve problems in shared spaces. The upside of filling this gap, I'm convinced, will be world-changing and perhaps even

• 1:30Nathan Labenz?

  world-saving. It was, of course, the emergence of language and the evolution of culture that allowed humans to sustain cooperation over long distances and time horizons and ultimately build the global civilization that we enjoy today. And the distributed nature of this system makes it extremely difficult and rare for any individual to accumulate a systemically dangerous amount of power. In contrast, the current AI paradigm emphasizes scaling things up, with more and more resources creating ever-more-powerful frontier models, each of which is meant to do everything on its own. This kind of concentration of capabilities into just a few systems and the concentration of power that it could easily bring about has always struck me as dangerous. And so I think it's very exciting to see a major company developing an

• 2:15Nathan Labenz?

  alternative paradigm that's meant to scale intelligence out horizontally in a way that is fundamentally distributed and designed to support permissionless participation from the start. I think and hope it could give rise to a more buffered, ecological, and stable network-based architecture for AI. Importantly, Vijoy also argues that this paradigm gives enterprises what they really want and need: a way to grant agents only the minimum permissions truly needed to perform their roles, a clean separation of concerns, visibility and auditability of their systems, and controlled interfaces through which to interact with the outside world. Of course, this conversation goes well beyond the theory and deep into the progress that Cisco and its partners are making in practice.

• 3:00Nathan Labenz?

  Internally at Cisco, they've built a system that they call the Community AI Platform Engineer, or CAPE, which is composed of 20 distinct agents that collectively manage complex cloud computing environments. This system has reduced load on-site reliability engineers and improved response times for end users by fully automating some 40% of tasks. Meanwhile, for the public, they've taken the lead on the Agency Project, which, you may recall from past sponsorship, is spelled A-G-N-T-C-Y, and which is laying an open source foundation for how AI agents representing different interests can connect, communicate, and meaningfully collaborate. At one point in this conversation, Vijoy fires up a demo which shows

• 3:42Nathan Labenz?

  how four agents, each representing different organizations and specializing in distinct skills, can collaborate to serve a patient in a healthcare setting that spans diagnostics, insurance, pharmacy, and scheduling. He narrates the demo pretty effectively, but I think it would be worth flipping over to YouTube to see that bit in action if you can. With that, I hope you enjoy this window into some of the most sophisticated systems thinking about the gig agent AI future that I have found anywhere with Vijoy Pandey of Outshift by Cisco. Vijoy Pandey, SVP and GM of Outshift by Cisco. Welcome to the cognitive revolution.

• 4:23Speaker 01

  I'm so excited to be here, Nathan.

• 4:24Nathan Labenz?

  Me too. Lots to learn. I've been studying up on your work and there's many facets to it. I would love to start, if you would indulge me for a second, with just a super high level view. I think everybody in America knows the brand Cisco, but probably even a lot of people who are very into the AI world at this point and know a ton about, you know, the intricacies of post-training and, you know, building all these agent workflows. If you pressed him and said, "What does Cisco do?" What I've come to in terms of a three-word answer is moving information around the world. And that, you know, how we say, is many different ways to unpack it. But I'd love to just kind of get your introduction to the company and the kind of fundamental role that it plays in our modern technology life.

• 5:07Speaker 01

  You're right. I mean, it's moving information around the world. It's connecting people to machines, to objects. It's about secure connectivity. It's about observable connectivity. It's about collaboration. So if you think about Cisco's four core pillars in its businesses, it's networking, it's security, it's observability, and it's collaboration. So that's the four business units, so as to speak, that we go after. But if I think, if I were to take a step back, because it's a, this is an ML AI audience, the way I think about Cisco in that context is we are a distributed systems company. So we enable scale out. We enable horizontal scale. So if you think about servers

• 5:52Speaker 01

  and if you think about making them bigger and bigger, that's one way of scaling, that's scaling up. Cisco is a company that allows you to take many of these entities, connect them through a network, and enable a cluster of compute. So you're enabling distributed computing and you're enabling scale-out technologies. And you end up doing both, scaling up and scaling out, to get the compute that you need for all of the awesome workloads that we're trying to run today.

• 6:20Nathan Labenz?

  Just reflecting on how much we need a sort of robust foundational layer for decentralization. So just the last week of news in AI is like frontier companies battling against governments. And that I think like, what was the original dream of the internet? You know, it was sort of this decentralized thing that people could kind of plug into on their own terms and contribute to. And I worry that we are losing that a little bit. And I think part of what, at least the great hope that I see in some of the work that you've done is to help enable that kind of distributed participatory future. So let's take one more beat on just kind of fundamentals, though, because one of the big things that you are proposing is an extension to the conceptual framework for networking

• 7:03Nathan Labenz?

  that people have developed over the entire history of information technology. Again, I suspect most people don't know the seven-layer network model. Could you kind of just walk us through, you know, what is the 101 framework that guides, you know, people in the networking space?

• 7:20Speaker 01

  So there is, and like you said, there is this OSI seven-layer model for networking, and it's a somewhat of a formal theoretical model. People like to stick to the model, but more often than not, people find ways around it. Like with anything that is formal, you make things work. You don't necessarily stick to what's out there as a formal stack. But if you were to think about that formal stack, the seven layers are actually, there's physical, there's link, there's network, there's transport.

• 7:53Speaker 01

  And I'm just going to walk through all of them, and I'll give you the sense of where things are pretty awesome right now. But there's transport, and then there's session, there's presentation, and there's application. So those are the seven layers. What you really need to care about, and the buzzwords that you probably heard in the literature. So Ethernet is probably something that most people are familiar with. That's A technology that connects computers in a local area environment or local area network. And Ethernet is operating at the physical as well as the link layer, primarily at the link layer. So that's Ethernet that you might be familiar with. TCP IP, which is probably the most famous protocol out there, on which the entire internet runs.

• 8:36Speaker 01

  That is transport and network. So those are the layer four and layer three. And then as you move higher up the stack, the other protocol that people are most familiar with is HTTP and HTTPS. And these are layer seven protocols or the application layer protocols. So typically, if you, if people, if you ask somebody, what do you, what are you familiar with? In the networking stack, you'll think about HTTP and the secure equivalent of that, which is HTTPS. TCP IP on which the entire internet is built. And Ethernet, which connects local computers together in, in a network. So those are things that people are familiar with. The rest of the layers are there for formalism, I guess, but

• 9:22Speaker 01

  not really as important. And I, I, I know if you're, if you're a geek in networking, you might find that statement a little bit off-putting, but this is the reality of, of things.

• 9:32Nathan Labenz?

  Yeah, shoot us straight. Always, please. So a big question I also had about kind of the world that I enjoy today is how much of the underlying protocols and, and sort of the way that today's major networks are managed is run on explicit code with rules that we fully understand that somebody sat down and, and designed and implemented versus how much has like machine learning already kind of penetrated its way down the stack to be managing the way that data actually flows and, you know, when issues are happening, like how are they detected, remediated, routed around, so on and so forth.

• 10:15Nathan Labenz?

  I know that, you know, there's, people have, I'm sure, got many PhDs in this area. But I realize I don't know what that boundary looks like between kind of what has been designed and what has been learned through, uh, you know, the emergent process.

• 10:31Speaker 01

  So let's, let's take a look at that from a pre L L M era. A little bit and even straight up machine learning and those pipelines that existed for a while now. The actual hardware and the actual data, sending data across the network is pretty deterministic in nature today. And so if you think about routing and the routing tables that exist within these large switches and routers. That's a very deterministic process. I mean, you don't want to take chances on figuring out where, where things are sent. Or you want pretty good determinism between the connection, between you and me Nathan, for example. I don't want that to be left to chance.

• 11:13Speaker 01

  But there is a control plane that sits above all of this and this is where the algorithms sit. This is where, I don't know if the, if the readers of the listenership is familiar with B G P and some of these routing protocols that exist that run the internet.

• 11:27Speaker 01

  That's a control plane piece of software that figures out who can talk to who. And that, not, not just in the regular day-to-day traffic sense of the world, but also when issues happen and outages happen. So what happens when you need to route around outages and if an undersea cable has been cut, what happens when you, when you have to route around that, for example. So these are control plane entities. Also, they deal with things like policies between organizations. So if like a Google and an Amazon want to connect to each other as two separate entities. There are strict policies on how they exchange information, how they exchange routing tables and connect to each other, as an example.

• 12:11Speaker 01

  So policies, security, actual routing control plane and application software. Those are all pieces of software and we've been using M L for quite some time in those pieces of software. Simple examples, anomaly detection is a common one where we've used M L pipelines for a long, long, long time. Are traffic predictions because you can foresee. Events and like Super Bowl and you can figure out what needs to be done for an event like that. So we've used ML then. We've used ML to predict failures. There's certain like subsea cables or there's certain choke points in the in the internet, but you need to route around that and be careful around those failure points. So we've looked at ML for those things.

• 12:56Speaker 01

  Also for around the business, we've used ML quite a bit within a company like Cisco. So to to do things like sentiment analysis around customers and there's a lot of use for ML, even some amazing users in, like I said, one of our pillars is collaboration, which is Webex. And Webex uses ML pipelines to do things like noise reduction. So we've been using ML in a company like Cisco for a long, long time. But I think like with everybody else, with the advent of generative AI and the practicality of some of these LLMs coming into the enterprise, a lot of that is not changing.

• 13:34Nathan Labenz?

  Yeah, maybe you can tell us more about how it's changing. I was interested to read about the project Jarvis, which is obviously one higher order additional layer that has been placed on top. I'm also kind of curious, you know, are the LLMs reaching down into the stack? Or is it purely kind of a layering on? You know, this will get back to the, you know, the two new layers that you're proposing to the network model as well. But I was kind of challenged to think like in the highest or in the highest level terms, like what is it that we've accomplished recently with language models? And it's like, it's not just that they've learned to do, you know, narrow task, right? But it's this sort of general purpose semantic and now even like agentic capability that has been layered on top. And so you're building at that layer too. Tell us about Jarvis and like other places where

• 14:19Nathan Labenz?

  language models are starting to change how Cisco operates.

• 14:24Speaker 01

  Jarvis was actually one of the first use cases that we deployed, at least with an outshift in Cisco. And then also we've taken that piece of code and

• 14:35Speaker 01

  What with other BUs within the company like Splunk and Webex and some of these other teams. And the whole notion behind Jarvis was if you're an SRE, you know this, like there are so many tasks that you do day in, day out to support your developer base, to support your customer base. That are rapid, A, repetitive in nature, B, can be highly automated through generative AI and agents. And it's almost like the way to think about this is think about everything that is happening in the software development environment. Through coding agents and taking that paradigm and applying it to site reliability engineering.

• 15:19Speaker 01

  And that's what Jarvis does. It takes the SRE pipeline and brings in the same identification, there's the same number of agents working together to solve for a problem. That you would do in a coding environment for developing code because the the paradigms are quite similar and the whole notion behind SREs is to leverage software development to solve for infrastructure and operational needs. So there's a lot of commonality in there, but there's also a lot of specific, special cases that we need to handle. So Jarvis was built and it's called CAPE. Now, by the way, it's a community AI platform engineer. So C A I P E has got a nice logo with a superhero with a cape behind their back.

• 16:05Speaker 01

  But CAPE is actually a multi-agent system, so it's a mass. That allows for the automation and identification of the entire SRE pipeline. And what we've done is we've had five plus user interfaces that feed into CAPE. It's doing hundred plus tool calls across cloud providers, across cloud native environments on prem. Across various aspects of that cloud native environment. So everything from observability to orchestration, to networking, to security. It's actually tackling more than 10 workflows today.

• 16:44Speaker 01

  And it's roughly 20 agents that are working together. So that mass system consists of around 20 agents together and the outcomes have been pretty amazing. Where we've reduced the load on the team by 30%.

• 17:02Speaker 01

  Forty percent of the tasks that that team handles have actually been identified. So we don't even worry about it. It's completely taken care of end to end. And the response time, because we've done all of this, has gone from hours to instantaneously. So it's efficiency as well as it's it is just morale in the team. Because you're not dealing with issues on a day to day basis. And the developer community that is using Cape is also pretty productive now. So is efficiency for the SRE team is also efficiency for the developer community as well. And so we started here, like I said, we've rolled it out to other parts of Cisco.

• 17:42Speaker 01

  But we've also made this open source where there's this cloud native operating excellence canoe community. It's got members from Adobe. I mean, these names are fascinating, Nathan. So it's got it's got members from Adobe, AWS, of course, Cisco, Nike, a whole bunch of enterprise companies participate. So they have been playing around with this. They've been growing this and there's a decent community gathering around this Cape project.

• 18:11Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:17Nathan Labenz?

  Can you give us a little bit better sense of what the frontier looks like today? I mean, this is a little hard, obviously, because it's such a moment in time thing. And, you know, from GPT five point three to five point four, it's it's going to move, I'm sure. But what would you say are sort of the, you know, the upper end things that the multi-agent system can handle today? And what are the sorts of things that it can't that, you know, we we need people to bring their expertise to? And how far do you think this kind of goes? Is there a vision for like extreme? We're talking in software, right? About like, obviously, the productivity is is changing pretty fast. Do we need 10 times as much software?

• 19:02Nathan Labenz?

  Do we need 100 times as much software? Like, what's the future of this market? And I, again, I kind of lack intuition for like, if this were to become 100 times more efficient, do we like do 100 times more network management? Or does the role change? Or do some roles go away? What is the sort of, even just next year, if you can think that far ahead, look like as models get better and and kind of get plugged into the frameworks that you've built?

• 19:31Speaker 01

  The one thing that interests me quite a bit is first and foremost, the definition of what we're trying to achieve as an industry. So whether you're looking at it from the AGI or ASI perspective, there is a definition of what we're all going after. And the definition that interests me the most is having a team of agents collaborating together to then solve for something that is net new, that is completely novel, that has not been in their training data at all, in any of those models or agents training data at all. So it's completely net new and doing it without any human intervention, hundred percent of the time.

• 20:14Speaker 01

  So I'm mixing aspects of the economic and the technical definition, but we've seen variations of this come across from many, many researchers in this field. And that's the one that I align behind. So that's the North Star that we all want to go after. I would say we are, we, I'm, I can make a prediction. There are others who've been making a prediction here. The timeline shifts forward, comes back, it's been a back and forth on how and when we can achieve ASI. But to me, the big thing that matters is right now, the entire industry has been chasing one vector to go towards this goal of ASI, and that is vertical scaling.

• 20:59Speaker 01

  So we are building bigger and bigger models, we're building better and better reasoners. That's we're throwing data, compute, resources at the problem, more parameters. That's one axis that will continue to happen. But we haven't really tapped into the second axis, which is the horizontal axis. And this is where a company like Cisco has a play. This is where I get interested because I'm a distributed systems person. And what I mean by the horizontal axis is can we scale intelligence horizontally? Can we enable collective intelligence where the collective is always greater than the individual? And so right now, we have not tapped into that piece yet. And the big reason for that is

• 21:44Speaker 01

  we've managed to scale these individual brains, quote unquote.

• 21:50Speaker 01

  And they become smarter and smarter, but we haven't figured out how they can think together. So how we can bring them together so that they can have shared intent, shared cognition, and then innovate collectively together to solve for this new set of problems without human intervention, which is the definition of AI. So to me, if we want to pull in those timelines, we have to tap into that horizontal axis of scaling intelligence, and that's something that we would like to bring to the table and we want to push towards, because it's completely missing today.

• 22:24Nathan Labenz?

  Yeah, I think that's really fascinating. I was just reading Ajay Akatra's latest blog post earlier today. She's now at Meter, and I'm sure everybody's familiar, of course, at this point with the meter exponential graph. But she was saying, it might be time to rethink the metric it has been until now. How long would it take a human to do this task? And she was pointing out that in the range that we've been measuring from zero to whatever, Opus four, six, it was like 16 hours or whatever, best estimate.

• 23:00Nathan Labenz?

  It's not too different in that range to have one person do the work versus try to divvy it up and have multiple people do the work. Because the coordination costs are, there's some fixed coordination cost, whatever, that it just doesn't, you know, certainly if you think down to the limit of take a one minute task, you can't really parallelize that eight ways more often than not. But now, as you get past 16 hours and you get to maybe like a couple of weeks worth of work, now it is something where humans can parallelize and take it down in a better way. And she was emphasizing like maybe we need to measure AIs by how long would it take a well organized team of humans to do this thing? But you're bringing the other side of that to the equation as well, which is like.

• 23:42Nathan Labenz?

  Sort of assumed here is like it's kind of still one AI. We do have one AI. We're getting now with the latest models, certainly Claus doing this and you know, Kimmy 2.5 made a big point about how they're like spawning subagents and kind of swarming themselves. But I think you have a much grander vision for that, which goes beyond like one AI kind of self-delegating, which still feels kind of brittle and like subject to correlated failures and you know, potentially who knows what other kind of resonant weirdnesses that might emerge. And I kind of hear you saying like, for humans, it was, it's diversity, it's like across culture, it's across time, it's all these sort of additional richer dimensions of collaboration

• 24:26Nathan Labenz?

  other than just like narrowly cloning yourself and delegating a subtask. So maybe, you know, there's the formalism of like, you're going to add two layers to the network stack, you can touch on those. But then I really want to get into your vision for like, what does AI culture look like? You know, what does AI cultural evolution look like? What is this world that we're going to step into? Then of course, you know, how are we going to make it work as well? But take me to the vision part of AI collaboration.

• 24:52Speaker 01

  To get to the vision, let's, let's take a look at human history and human evolution and how intelligence evolved in humans as well. Because the one thing that we do really, really well in AI is the human as the bar and we try to see where we are with artificial intelligence and compared to singular human or teams of humans. And that's always been the yardstick, though we might be surpassing that yardstick pretty soon. But if you think about human intelligence evolution, humans became smarter and smarter. They became, they became very conversant with tool usage. They became very conversant with symbolic communication for a long, long time. This was happening for hundreds of thousands of years.

• 25:36Speaker 01

  And you, you could use an axe and you could use an agricultural tool and you could actually paint pictures and you can, you could raise flags and you could communicate in those ways. But the big paradigm shift, the step function, happened when language got invented. And that happened around seventy thousand years ago, and there's a ton of literature on this. Where you can see that the invention of language actually was a step function change in intelligence evolution within humans and human societies. Because suddenly, you could collaborate on tasks. You could align on intent. So instead of all of us trying to run and capturing that hill,

• 26:21Speaker 01

  it's like, let's figure out a strategy to go ahead and capture that hill. And that, that delegation, that taking a larger task and breaking it down into smaller tasks and giving it to various members of the team, based on their expertise and then aligning on that intent, then coordinating between these members of the team, and then actually executing on that task and solving for a net new problem, which is, we've never seen that hill before, let's figure out how to capture that hill. That's what unlocked the next revolution in human intelligence. And what we are seeing is this exact trajectory is actually playing out in silicon.

• 27:02Speaker 01

  And so we are seeing this build out of smarter and smarter brains and bigger and bigger models. Agents are getting better and better. Yes, you are getting these sub-agents in Claude and in, in, in, OpenAI and GPT and in Gemini, but it's still not, it's like sub-processes within my brain is actually not getting out of that. And we are, we are not looking at teams of agents that can come together and collaborate. So what we need to enable, just based on that paradigm, is, to your point, look at longer duration tasks. Look at tasks that humans do. More than sixteen hours, few days, look at the specialization that happens within human teams and then figure out how agents can actually mimic those behaviors.

• 27:47Speaker 01

  And the vision is that what we will find, that specialized agents, subject matter experts, and we see that already in smaller environments, but subject matter expert agents that can come together, share intent, coordinate, negotiate, then work on shared knowledge and shared context, almost like institutional knowledge, and then innovate on a new problem statement, that's the way to go. And that's what we're calling the Internet of Cognition because it is going to be distributed across a bunch of agents who by definition will come from different vendors, because they're all subject matter experts, they will not come from the same vendor, and they

• 28:31Speaker 01

  will all need to come together, collaborate, and solve for this net new problem space. And so that Internet of Cognition is what the vision is, and that is a horizontal scaling that we all need to go towards to then enable the second axis, the love of the second axis of scale of intelligence.

• 28:49Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 28:56Nathan Labenz?

  So can we make this really practical for a moment in, you know, let's look at like the interaction that we've had leading up to this conversation. So I use an AI agent to help me prepare for every episode of the podcast. I specifically use Tasklet that basically channels Claude and gives it, you know, really robust cloud-based framework to work in. And it does a really pretty good job of like going out and researching everything about you, finding all these publications, you know, bringing me back a good source list. Then it also crosses that against my past work, you know, all the previous outlines of questions that I've put together, and it comes up with a decent draft for me.

• 29:39Nathan Labenz?

  I definitely do still have to spend time on that for the record and end up usually writing my own because I want to be able to be present in the conversation and just having had an AI do it doesn't really give me the ability to do that. But that was, that's what happened on my side. And then usually the guests don't really do anything like that, but somebody on your team, I don't think, I don't know if it was you, but somebody, you or somebody on your team did a similar thing and took all your work and then, you know.

• 30:07Nathan Labenz?

  Crossed it against my recent record of of episodes. And there was another thing that got sent over that was like, OK, here's all the themes that we know you're interested in and how Gij's work relates to those. What didn't happen at all was that the AIs had any back and forth or any, you know, sort of coordination. It was just kind of two ships passing in the night. Now I have these kind of two side by side documents, both useful, but no meeting of the AI minds. How do you think that, you know, that's a pretty basic case and we can build up from there. But how would our agents in the future come together? And and how, you know, what should we expect in terms of like additional value from that kind of interaction?

• 30:47Speaker 01

  This is an excellent use case, and it's a pretty straightforward use case. Like you said, it's simple. So it'll be easy to understand what we're thinking about in terms of shared intent, shared context and collective innovation. So your agent, Nathan, has got a local optimization function running. There's a goal there that says, let's produce the best podcast ever in your series in the entirety of what you've done with Cognitive Revolution. And let's figure out what that entails. And you, the history or the context that your agent has is all of your guests and what they've spoken about and what might be net new and interesting for the audience, based on what that agent is seeing, which is your history.

• 31:31Speaker 01

  There's a similar agent, like you said, on our end, which is looking at all the speaking engagements I've done or people from Outshift has done, have done. And what's interesting to folks that have listened to Outshift people or me speak about. So there's a context there which is somewhat different. It's almost like a Venn diagram. So one is very Outshift centric and what works. One is very Cognitive Revolution centric and what works. But both of us have a similar goal, which is can we get together and between the two of us, create this podcast, which is probably going to be the most interesting podcast out there. And there are some, as you come together and make that happen.

• 32:15Speaker 01

  Both you and my team, Rebecca on my side, we've been working through some concessions. We've been working through, yeah, that might not be, that might be a little bit too marketing oriented. We join, let's, let's get a little real and grounded and more technology oriented. And so we've been looking at how to come to a common goal so that this conversation resonates with the audience that you have. But that work is being done by humans today. It's not hands off, it's not automated, there's no agentification at all. What we're trying to get to is the agentification of this entire pipeline, which is let's throw the C R, your agent, the C R agent and the outshift agent together.

• 32:57Speaker 01

  Let them converge on a common intent. The best episode possible with some concessions on both sides. So that we narrow down on the questions and topics that we can talk about. And so that's something that can happen hands off between the two agents. And once you're done with that and this podcast is out there and you're getting readership. Now you have common context that both of them can work upon. That can leverage for the next guest, maybe I'm there the next time around. And you can leverage that context, you don't have to redo these things. Or maybe I'm going on a different podcast or maybe somebody else from outshift is coming to your podcast. So you have common context that you can leverage over time.

• 33:39Speaker 01

  That is not lost and you don't have to restart that process. So this is something really simple, but it also tells you that even in such a simple example. The humans are doing the intent alignment, the humans are doing the coordination and negotiation. The humans are actually building out context using Google Docs or SharePoint or whatever it is. It's a human glue that is enabling collective intelligence. How do we quantify that? How do we make that automated? What is that infrastructure that we need to build? So that this human glue becomes software glue. And that's what we're trying to do through the era of cognition.

• 34:22Nathan Labenz?

  So let's get into the weeds on that because I think it is really interesting to imagine. And then I of course, you know, have a ton of questions around like very practically, how should we think about it working? Like, should we exchange agent IDs? And then, you know, I give my agent your agent ID. Should they be out there like potentially discovering each other? Is there some sort of way to kind of, you know, have your sort of menu of your roster, perhaps of agents kind of present in a way that's attached to your identity? I have a lot of questions about identity just unto itself with agents. So I'm very interested in kind of just in the first place, like, how do these things discover each other, know that when they do communicate that they are talking to who

• 35:07Nathan Labenz?

  they think they're talking to? And then, you know, previewing your additional layers for the network stack. Layer eight is semantic, layer nine is cognitive. And, you know, we've got big questions there too. Obviously, these questions are also very operative in human affairs all the time. Right? Are we talking about the same thing? Are we thinking about it the same way? You know, are we miscommunicating? Obviously, there's, you know, human life is full of miscommunications, you know, minor and major. But it seems like right now the AIs are going to kind of drift off into weird spaces if we just let them sort it out fully. So we're obviously going to need some like guardrails and some, you know, kind of way to bring them back to true north, at least for now. So yeah, give me kind of the double click on like the details of how you see all this working.

• 35:54Speaker 01

  So to walk through all of that that you just asked, I mean, there are many, many layers involved here. So to walk through each one of those layers, let me show an example. And this is a pretty abstract picture. So for everybody who's listening in, I'm going to describe it as much as I can. But if you can go back and listen and look at the, look at the diagram that I'm showing, then it actually makes even more sense. So let me share this picture. So to walk through that, I thought sort of questions that you asked Nathan, because there is a little complicated and there are many, many layers here that need to get involved. Let's start with an example here. So this is an example that we've built.

• 36:37Speaker 01

  It's a multi-agent system that we built for a healthcare provider. And the goal here is pretty straightforward. The goal here is to

• 36:46Speaker 01

  Take patient calls, so people are calling in to this healthcare provider and they need to get routed to the appropriate subject matter expert, the provider themselves, like a doctor, let's say. Based on the patient's history, based on the availability of the doctor and based on the insurance profile of both the doctor and what exists in this, uh, in this hospital. So what you see here are actually four agents. And for everybody listening in, I'm just going to describe it. But there is the scheduling agent on the bottom right, which is the one that is actually interacting with the patient.

• 37:29Speaker 01

  So this way is like a chat board, you're talking to the, to the chat board. And it's also doing scheduling at the back end. So this is the one that's actually going to go and schedule you sometime with, with the doctor. And then there are three other agents. So there is an insurance agent, which is from the payer's perspective, coming in from the payer's perspective, it's not from the hospital, it's a completely third-party entity. There's a diagnostics agent, this is also a third-party agent, not belonging to the hospital. And then there's a pharmacy agent, which is also a third-party agent, not belonging to the hospital. So the only piece that belongs to the hospital is the scheduling and conversational chat board piece.

• 38:13Speaker 01

  The rest of the three agents are actually third-party agents. And the task is again, pretty straightforward. Get me to the right provider, follow the symptoms I have, based on my insurance history and based on my pharmacy record and so on. So what ends up happening is, if you think about all of these other three agents, insurance, diagnostics, pharmacy.

• 38:37Speaker 01

  First and foremost, these, all of these agents are independent. They, today, they cannot talk to each other. I mean, they could not talk to each other. Let me start that way. They could not talk to each other. So they first of all, they had to be discovered, connected, given the right identity and access, brought into and stitched together into a multi-agent system. So that we can even start doing things like this. So that's step one. And so to do that, we had launched this whole notion of the Internet of Agents almost like a year ago, and we launched an open source project called AGENCY. That's spelt A-G-N-T-C-Y. And again, if you're watching this on YouTube, you can go to

• 39:21Speaker 01

  agency.org. That's the place for the landing page for the open source collective. It's part of the Linux Foundation as well. But coming back to this example, what AGENCY allows you to do is it allows you to first discover these agents. So all of these four can get discovered. It then allows you to provide appropriate identity and access management attributes to each one of these. And we can spend some time on that, but that's a complicated topic. And then once you do that, then you can actually start communicating with each other. So traffic starts flowing between all of these agents. They get connected. They're communicating with each other.

• 40:03Speaker 01

  And so there are things like MCP, protocols like MCP, there are protocols like A2A that allow you to have agents talk to tools through MCP or allow agents to talk to each other through A2A. So that happens, communication happens. And finally, there's an observability pillar which says, great, these things are working. Are they actually delivering what they're supposed to deliver? So a lot of observability from the agentic perspective, from the multi-agent system perspective, and a little bit of evaluation from that perspective as well. We have not solved the evaluation stack completely, just touched it a little bit. So that's what AGENCY does. It, it is a basic plumbing

• 40:49Speaker 01

  required to even bring these four agents together in an enterprise so that they can start doing what we're trying to do here. So this is where what we deployed, this is what's working today. But what's not happening today is all of these four agents are actually isolated agents. So they can talk to each other, they can get connected. But the payload is opaque. The payload is is just a blob, it's a binary object. You don't know what's contained within those payloads. And so all you're doing is enabling these things to talk to each other, enabling access to tools. But the coordination, the alignment, the shared memory, the shared context.

• 41:34Speaker 01

  Everything that you see on the right hand side, which is grayed out, is not happening. And that is happening again through human in the loop. So the human is actually the coordinator. The human, just like the previous example we talked about, the human is playing the role of enabling intent. And if you think, if you think about what's happening here, you have time to route KPIs that the scheduling agent has. You have the diagnostics agent that is being measured on the output outcome confidence. You have insurance agents that's that is being measured on ROI. So every agent has local optimizations to solve for. But somebody has to step in and say, you know what?

• 42:16Speaker 01

  Each one of these agents have to give a little to get a little. So you need to relax your time to route KPIs. You need to relax your, the insurance agents need to relax their ROI KPIs. So that we can get to the proper global outcome for the patient. And so there's intent, there's coordination, there's knowledge and context which needs to be shared between all of these agents. Because all of these agents have a mix of patient information as well as broad information across all patients.

• 42:53Nathan Labenz?

  A couple of little questions that I'll just, you know, float to kind of help prompt you are.

• 43:00Nathan Labenz?

  How, you know, again, today, it seems like the discoverability, I had this vision of sort of the human is like kind of back to being, you know, the old like switchboard routers, you know, we're doing that now like we a long time ago did for phone calls of literally like, you know, plugging things in to make the connections. We're kind of doing that for our various AI systems. Like the number of times I in a week now go like get it fetch an API key or, you know, some sort of equivalent of that to, you know, go, you can see here, it's going to be okay. Well, where do I go get that pharmacy agent ID? Where is that? You know, who's, who's broadcasting that anywhere? I'm going to have to go figure that out on my own. And then I'm, I'm interested too in like, in general, what are, what do

• 43:43Nathan Labenz?

  you think the patterns should be for like, who sets this up? I guess one way to think about it is like, who owns the customer relationship? I guess like in this case, it would be if I'm calling the hospital, you know, you kind of framed it from the hospital's perspective and have these other things as like third parties in a similar way that if I call the hospital, like they might, you know, call my insurance for me and, you know, do some, some background double check. But then if that's right, does that mean that sort of the shared context lives forever with the same entity that owns the customer relationship? Is that kind of a pattern we should expect? Or will these shared context threads also be like jointly owned or sort of, you know, shared in such a way that, that

• 44:26Nathan Labenz?

  any of these individual entities and their agents, you know, could come back and access them later? Again, we have some patterns like this. Like if I do a Slack connect, that Slack creates this kind of shared space for me that both my company and the, you know, the other company that connected can go back and access. But I'm not sure if we should expect the same, if like how skeuomorphic should this be? I think is always a really interesting question in AI and I don't have a great intuition for that in this connectivity and coordination space.

• 44:59Speaker 01

  Yeah, so let's, let's take it one at a time. So let's, let's just start with the basic connectivity pieces. So the four pillars of basic plumbing for multi-agent systems are discovery, identity and access, communication and observability. So if you get those four down, then you can actually create a multi-agent system like the one shown in green here, a teal, and get these agents to at least start talking to each other. Even though a lot of the things that you see on the right, which is in gray, are not possible today. But what's possible today is the stuff in green or teal in the center. So they can all talk to each other.

• 45:40Speaker 01

  And so if you think about what we did with agency, and I'm just going to go and flip over to this window here, and for folks who are just listening, if you go to agency.org, A G N T C Y dot org, you'll see those four pillars come here and you see the architecture at the bottom, where to your question of how do these things get discovered? And there's a human switchboard. Primarily for consumer use cases, you and I do this all the time. So we figure out which agents make sense. We look at their reputation, we look at their revales, we look at their pedigree and we say, great, I'm going to use these. Sometimes we even depend on the L L M's to generate agents on the fly and we are like, yeah, it's good enough because it's a consumer use case.

• 46:26Speaker 01

  But if you think about enterprises, enterprises don't work that way. Enterprises have long procurement cycles. They have trust in the equation and safety in the equation. And they have customer trust and responsible AI in the equation. So there's a lot of trust and safety and rigor that goes into how enterprises procure software, whether it's agentic or not, it doesn't matter. And so the way we've tackled this is each of those pillars, four pillars, has an entity in the software stack, in the software stack, in infrastructure stack, providing APIs for that. So.

• 47:07Speaker 01

  There's an agent directory that does the job of discovery. So you can define agents, you can define tools, you can define multi-agent agents. Because in the end, it straddles all the way down. Like an agent could be an agent built on many agents. So you can define an inordinate amount of hierarchy within an agent. So agent within an agent within an agent. You can define tool access, you can define softwares, data source access. All of that can be defined within the directory and it's searchable based on capability. So it's like the DNS equivalent for agents.

• 47:48Speaker 01

  So today, you want to go and access a website or you want to go and access a remote API. You have a URI and that gets translated into an IP address, which then flows over the network and you get routed towards that. Here, it's not as straightforward. You don't have a fixed URI or you don't have a fixed website. You can search on capability, you can search on reputation. So this is a little bit more involved. So that's the directory. And what comes back may not be a service endpoint, like an API endpoint, a remote API endpoint. It would, it could also be a Git code branch.

• 48:27Speaker 01

  Those agents could be stood up as a service or they could be, they could exist in a Git code tree because they might be local to you. Or you might be pulling it from a Git code tree and deploying it as code within your environment. So it's a pretty powerful piece of software that allows you to search across billions of entries, trillions of entries. It's actually built on decentralization principles. So it's actually got a DHT that supports it in the backend. And it allows you to discover agents and then be able to bring them in your environment and be able to then connect them. So that's step one. Step two is then giving them the right access and identity.

• 49:10Speaker 01

  And this itself is a, is a whole different ballgame. Because agents, as we all know, they have human-like characteristics, but they are operating at machine speed and scale. So they have agency, they have decision-making powers, they have semantic communication, which is not deterministic.

• 49:30Speaker 01

  But they are operating at software speed and scale. And so this is a blend of human-like and software-like, machine-like capabilities. So you cannot rely on the old role-based access control mechanisms that have been exist in existence for a while. So you need to rethink access control and so we've brought in the whole notion of tool and task-based access control. And that's what the identity piece does in this stack. So you give them access control, you have discovered them, you've given them identity and access. Then you go towards communication and the communication is of two types. One is either I'm talking to existing deterministic infrastructure, which is tools and data sources.

• 50:12Speaker 01

  And MCP, which is Model Context Protocol from Anthropic, that is used for those kinds of communication. And then A2A from Google is the agent-to-agent protocol that allows agents to communicate with each other. And we are foundational members of both of those. So both A2A as well as MCP and they fit nicely in this architecture. And then finally, there's the whole observability piece. So how do you observe not just the containers, the cloud-native infrastructure or the bare metal infrastructure? Where these agents reside, but also the agentic parts of agents? Because your service might be up, but the agent might be misbehaving.

• 50:56Speaker 01

  So how do you observe and evaluate the agent that sits within these infrastructure components? So we worked together with Microsoft to push a whole bunch of extensions in Open Telemetry, which is the de facto observability stack, to make observability for agents happen. So that to me is the bare plumbing that you need for this picture to even get connected and you to start just working. So I'll just pause here to see if you have questions because we have not even tackled the gray parts. We've just brought them together and the human is still trying to be the context setter, the intent coordinator and all of that in the middle.

• 51:41Nathan Labenz?

  A couple of questions that are really top of mind are, you mentioned like the decentralized directory and I'm interested to understand that a little bit better. Obviously, like one of the mega trends on the internet has been toward walled gardens and platforms kind of controlling the directory in many major spaces where people like to spend a lot of their time. It seems like that is something that, say, OpenAI might be trying to do again. And I don't mean that in like a overly pejorative way, but you know, they're trying to create a curated app store like Apple before them and you know, so many other platforms. And they're going to have reviews, you know, that they're going to kind of be the owner of the reviews on the, you know, the plugin or the app store that they develop.

• 52:26Nathan Labenz?

  Then they're also going to have advertising, you know, which is going to, you know, kind of bring another whole layer of commerce to all that. What is the sort of alternate vision of a more decentralized, open, permissionless version of that look like? And is there, I guess there's still is probably some sort of centralized kind of DNS equivalent in that vision? I don't know if it goes like as far as, I guess we do have like blockchain, you know, things that don't even require that. So I guess how decentralized ultimately do you think that can and will be?

• 52:59Speaker 01

  So the two things that you touched upon, identity, there's a lot flows from identity. And then discoverability, which is the DNS statement that you just made. Those two are the ones that are the source of everything else that follows after. And you touched upon both of those problems pretty concisely, where if you control the directory pieces, which is how do I discover agents and how do I even get to the best agent for my, for my task? And then if I control the identity, then you control the reputation, then you control the security, then you control how much you can charge for it and the commerce

• 53:39Speaker 01

  behind it. So everything flows from those two pieces. And so we took a lot of care to ensure that in this architecture, those two critical pieces, the identity and the directory pieces, were actually built on DHTs. So these are distributed hash tables. Nobody owns the directory. You can participate in a directory decentralized infrastructure. Similarly, Identity, we do plug into the well-known identity providers, IDPs like Okta and Duo and others that exist out there, but we also enable a decentralized identity provider that somebody can deploy and participate in.

• 54:21Speaker 01

  So we are offering the best of both worlds when it comes to directory and identity because our vision is for this to be truly open and interoperable. And to the statements that you made, you can truly be open and interoperable if discovery and identity are also decentralized and no singular entity owns either of those pieces. So we've taken actually great care in making sure that those two are available as options. Now, enterprises might decide to start with a Microsoft Active Directory or an Okta because that's what they are familiar with, but soon they'll also realize that yes, role-based

• 55:03Speaker 01

  is great, but they need something like a task base or a tool base, so they might move towards decentralization, so we're not forcing one or the other. But principally, we are aligned to what we just said, that we want this to be decentralized.

• 55:16Nathan Labenz?

  Yeah. That's great. I think that is a really important contribution. When it comes to identity, I would love to get a little bit more of your vision for what sorts of identity are going to be supported, right? I mean, this has been another kind of major point of contention in the digital sphere broadly. Should we have anonymous accounts? What sort of clarity should we have on who's ultimately responsible for this account or this agent, as the case may be? Now we've got people doing experiments in just fully autonomous AI systems, which if

• 55:59Nathan Labenz?

  you kind of give something a credit card and it can pay for its own server time, like we're kind of right on that fuzzy border, I think, where some of these things might actually be able to persist for a while and potentially not solely due to meme coin scams, which obviously won't be, I don't think, a long-term strategy for them. Yeah, how do we, I mean, there's so, there's so many trade offs. It's like such a vexing thing. And when I, when I try to think about it, I kind of quickly get overwhelmed. So help me maybe be a little less overwhelmed. It may even start with like a taxonomy, but I, I would love to understand your vision better for identity and sort of responsibility, accountability as we go to the gig agent future, I think incredibly quickly.

• 56:40Speaker 01

  Identity is actually the biggest hurdle that we are facing today when it comes to deploying full autonomy or at least as much autonomy as possible with agents in the enterprise at least. Because what's happening right now is, like we mentioned earlier, agents are like humans because they communicate semantically. There's a lot of natural language flowing between agents and agents, also between agents and humans. So if you look at a multi-agent human team, a lot of the communication is happening semantically. And so that's ambiguous. That's non deterministic. There's a lot of decision making authority that you're giving to agents. And there's a lot of tool calling authority that you're giving to agents as well.

• 57:23Speaker 01

  So for enterprises especially, and even for consumers, when you're actually using a credit card, I would be pretty careful. You have to be really sure as to what access control are you giving an agent? And what we've seen all along is that identity systems and access management has been built for mostly humans. And over time, we took that, it's called role-based access control, RBAC. It's a very famous, like, like you think about access and identity, you think of RBAC. So role-based access control is a very human oriented view of identity and access.

• 58:04Speaker 01

  Over time, we realized that there are pieces of software and there are machines that do not totally align with RBAC. So we created this attribute based, or ABAC, Attribute-Based Access Control. But it's very similar because machines don't change their personas if they're not agentic. They're doing some piece of, they're running the same piece of deterministic code day in, day out. So it's long-lived. It's, you know what it is. So it's almost like saying RBAC, but applied to machines. So those are the kinds of identity mechanisms that have existed in enterprises so far.

• 58:47Speaker 01

  The problem with agents are that they are like humans and they're shifting around. And they might be Vijay tomorrow, today, they might be Nathan in the next hour, especially in larger teams and enterprises. So we had to go back to basics and we had to look at what is it that we are really giving access for? So when somebody gives Vijay access within Cisco, I have access to data, I have access to systems that I can read or write, given where I am sitting in the organization and the kinds of skills and the roles that I have, that I perform.

• 59:26Speaker 01

  In the end, it all boils down to what are the tools that I'm accessing, what are the tasks that I'm trying to do? And what are the transactions that I'm participating in? So if you take it down to that level of granularity and back to basics, we said agents should look at tool, task and transaction-based access control where by default you assume the lowest layer of access for an agent. Based on then the tool or the task that is trying to execute, you then may elevate privileges just for that task or just for the tool access or just for participating in that transaction.

• 1:00:09Speaker 01

  Let that happen if they are allowed to do that, then bring them back to base level identity and access, a base level access. And you keep doing this in a very ephemeral manner and you don't give agents longevity when it comes to access control. So this whole notion of TBAC, which is two-task, transaction-based access control, is something that we proposed with agency, is seeing a lot of traction and a lot of interest. Those it goes back to the basics of what we are all trying to do and what is it that you should be given access for? In fact, there's a one thing which is pretty interesting that happens when you start looking at this

• 1:00:52Speaker 01

  notion of a task-based or tool-based access control. What ends up happening and to the earlier question that you asked, let's go back to the networking stack. So far, we've been dealing with deterministic pieces of software. And so all kinds of access control mechanisms that existed out there, even for machines, were built for deterministic pieces of software. With agents coming in, we are going towards non-determinism. So we're going towards semantic communication. So when we need to give an agent access for a particular task, the question is, what is that task? And how do you figure out what is that task that the agent is trying to execute?

• 1:01:35Speaker 01

  And the way you do that is by tapping into the semantic communication that's taking place between a human and an agent, or between two agents, and figuring out what is that task that they're trying to execute on the transaction, that they're trying to execute on the tool call that they're trying to make from that semantic communication, and then giving them access for that. So right away, to enable T-BACK, you need to be able to tap into Layer 9, which is semantic communication, to really figure out how to give this agent the right access control.

• 1:02:14Nathan Labenz?

  So it's, Tony, you said turtles all the way down. It's also like, uh, this screams LLMs all the way down. I like the idea a lot of the sort of minimum permissions and, what is it, principle of least privilege, I guess, is the fancy way of saying that.

• 1:02:34Speaker 01

  All zero trust.

• 1:02:35Nathan Labenz?

  But, yeah, and I was just gonna go to trust. I guess there's kind of repetition. There's trust in the first case, and then there's also kind of, can you cash this sort of stuff? I mean, one big thing that we have as humans that is like to our advantage in our ability to coordinate together is, while we do as individuals change, evolve, drift, whatever, over time, that's a relatively slow process. And people can generally sort of expect that one another will be similar tomorrow as they are today, if not exactly the same. With agents, I'm like, okay, so let's say I want to do some delegation, and now it's like, and I maybe want to engage with, you know, somebody else's agents. I want to give them some access to some information.

• 1:03:20Nathan Labenz?

  I'm kind of getting a picture of, first I have like my little guardian angel agent that kind of tries to watch out for my interest and determine what I'm willing to give. Part of how it's gonna determine that presumably is some sort of reputation or identity on the agent on the other side. But that still strikes me as like, kind of a fuzzy, like, okay, sure, but what exactly am I giving? Access to and do I know that it's the same tomorrow as today? And how does it prove, you know, that I can trust it at all, right? I mean, even just inside an enterprise, you could imagine that one department is like trying to make their thing, you know, do its local optimization a touch better and they maybe swap out the model.

• 1:04:05Nathan Labenz?

  Okay, great, maybe tested better on your benchmarks, but over here where I'm trying to make a independent decision on how much to trust this or can I, you know, assume that my last designation was good, you know, maybe I ran it through a battery of tests last time or something and it passed.

• 1:04:21Nathan Labenz?

  The, under the hood, you swap out one, even just a prompt, a system prompt change could make a pretty dramatic difference in terms of how it's going to behave next time. So I guess if I had to boil all this down, it's like, how do we make the trust designations in the first place? And then how do we have durable identity? So I know that I'm going to get kind of consistent behavior from my counterparties over time.

• 1:04:42Speaker 01

  Right, and I think you're getting to the crux of how do I make this non-deterministic system a little bit more deterministic? In the end, you need to, you need to be able to reason about things and you need to be able to persist around some kinds of state, not all kinds of state, but certain states. And so the way we are thinking about this is again, going back to the OSI stack. So we started there and let me take you back there where so far, all the layers that existed in the OSI stack, when two endpoints are communicating with each other, they're all,

• 1:05:24Speaker 01

  the endpoints that are connected through a network are deterministic endpoints and they're exchanging data and there's, you're exchanging basically deterministic state between all the endpoints that exist in that network. What we're seeing now is that the world is moving towards non deterministic endpoints, probabilistic endpoints. And the endpoints are intelligence endpoints. So the data, instead of exchanging data, now I'm exchanging cognition state. Because I am a cognition entity as a human. I might have five agents that are also cognition entities. And you might have five more. So we are exchanging cognition state.

• 1:06:05Speaker 01

  And the OSI stack does not have the right layers to even support that communication. And why do I need that? The reason I need that is because I need a little bit of determinism. When I talk to each other, when all of these agents talk to each other. So I need to understand. Whether I am in the intent alignment state, whether I'm in the discovery phase, whether I'm the, I'm in the coordination phase, whether I'm negotiating. So there are all of these phases that you go through when you actually align on intent, actually work on a common goal. And we need to figure out where you are. And we need to also figure out whether you're trying to take an action. During an execution phase.

• 1:06:51Speaker 01

  So all of these sort of meta keywords. That need to take shape between agent to agent communication is very free flowing today. And some of that, you've seen that happen in. The OpenClaw, OpenClaw board book example that we saw a couple of weeks ago. Where you had a bunch of agents, OpenClaw agents come together. And they're just talking NLP, of course prompted by a bunch of humans in between. But they're talking NLP, but they are not building. Consistent state and they are all over the map. So if you want to bring this into an enterprise and actually work towards convergence.

• 1:07:32Speaker 01

  Want to work towards emergent behavior that is constructive and not just divergent in nature. You need to put some structure around it. And the way you put structure around it is by making sure that the communication that happens. Between these agents is also a little structure compared to any and all NLP.

• 1:07:51Speaker 01

  Wide, wide, vast, in some ways. So, the layers that we are bringing into the table are the two layers of L8, which is syntactic communication and L9, which is semantic communication or cognition state protocols. So, syntactic layer is, think of it as grammar. It's like I built an agent on Vertex. Somebody else built it on a Langraph or a Bedrock. How do I make sure that these two agents can talk to each other? Because their grammars look different, their frameworks look different. And they may not be sending their blobs and their payloads in the same formats. So, that's the syntactic layer and this is where MCP and ITW

• 1:08:33Speaker 01

  and the entire architecture that I showed on agency, that's where this plays. Above this, we bring in the cognition layer or the semantic layer, that is layer nine. And layer nine is where we look deep inside that packet, deep inside that communication flow and actually extract the meaning and figure out what is it that you are truly trying to do and put some structure behind that meaning. So, instead of it just being plain, simple NLP, which it will be, can we wrap a header around it which says, this is what I'm trying to do, I'm trying to discover information, I'm trying to negotiate or coordinate and which phase am I in in that, in that cycle?

• 1:09:16Speaker 01

  And I'm trying to access a tool or I'm trying to execute a command. I mean, there is a little bit of a meta-header that needs to happen so that you can take action on that and you can build those trust and governance layers without which even those pieces become a lot more probabilistic in nature. So, that's the way we're thinking about this. Now, of course, the lowest common denominator is, yes, it's all going to be NLP. In which case, if you're driving, if you're dropping a guardian angel or a cognition engine, as we call it, that is trying to contain blast radius, it will have to be also probabilistic in nature. You cannot guarantee hundred percent. You can guarantee maybe ninety-five percent or ninety-nine percent,

• 1:10:00Speaker 01

  but you'll not guarantee anything ninety-nine, hundred percent. But if you wrap these with a header and you can wrap it in a proper protocol, then you can get to determinism on some of these things. There are frankly requirements for enterprises to deploy multi-agent systems.

• 1:10:20Nathan Labenz?

  So who, is this something that can sort of exist on a spectrum as well, perhaps? When it comes to where this sort of shared history lives, Cisco has WebEx, as you said, that's sort of, you know, one place where people come together and obviously those, you know, meetings can be recorded and a lot of shared, you know, history and context can be established through that digital space for humans. One could imagine a like Cisco product that gets deployed to enterprises that's like, hey, this is your sort of WebEx for agents where they can come together, where they can create shared state, where you can have a history of this and then you can invite

• 1:11:00Nathan Labenz?

  guests in from other entities, as you will. Is there, is that, does that exist as sort of a enterprise product distinctly? Or can it also sort of have like a long sort of open permissionless tail to it? I'm not even sure if that's a fully coherent question, but I am really interested in like, it's not easy for an individual, right, to interact with an enterprise. People, both people and the enterprises like want to do that.

• 1:11:31Speaker 01

  So let me take, let me take a crack at this and let's see where it goes. So the way we're thinking about this whole Internet of Cognition architecture, and if you can see the slide here, and again, I'll try to describe it as much as I can for people listening in. What you're seeing out here is the entities in teal or green are the entities that make up the Internet of Cognition. And there are three layers that we're thinking about. The first is a protocol layer, which we just talked about, which is how can I enable semantic communication to take shape between intelligence endpoints instead of data communication between deterministic endpoints, which is where we are in the entire networking stack exists today.

• 1:12:16Speaker 01

  So we're connecting intelligence endpoints, the semantic communication happening. And what you see here on this picture is cognition state protocols. These are the classes of protocols that enable these kinds of communication. And we have three types based on where we are. It could be as simple as natural language, which we are calling semantic state transfer protocols. It could get better than that and really good by saying, let me just exchange the entirety of latent space between two models and two agents. So let's take the entire KV cache and send it across. That is excellent. It's like implanting a neural link chip between you and me, Nathan, so that we don't have

• 1:12:56Speaker 01

  to take undertake the cost of tokenization and transmission and detokenization back at the other end. But I might not allow a neural link implant and you might not allow a neural link implant, which is what the case is today with model providers. So you can do this with open, uh, open, uh, weight models, but you can't probably do it with closed weight models, at least not now, unless it becomes a standard. But we have this thing called a latent space transfer protocol, which allows you to do this at really high efficiency. And there's something that sits in between, which is compressed state transfer protocol,

• 1:13:34Speaker 01

  which is invariance between agents, especially for deployment scenarios, where not everything might be within an enterprise or not everything is within a data center. So let's say you have edge cases, so you have edge deployments, like your MacBook or your phone or your wearable device and something that is sits, uh, exists within a data center. Can we send this information in a compressed way to that edge influence device? And so we have three kinds of protocols, but they're all within that protocol there. So that's how we talk to each other between intelligence endpoints.

• 1:14:14Speaker 01

  Then there is that cognition fabric and the fabric is as the name implies, it allows you to scale this out. So it allows many, many, many agents to talk to many, many, many agents. So many to many communication, real time in nature, add the semantic layer.

• 1:14:34Speaker 01

  But it also allows you to plug in the memory of choice. So we are not dictating, so getting to your question, we are not dictating what you plug in. You could plug in an open source, memory like a Memzero or something else. You can plug in a big query, you can plug in whatever your choice of memory might be. But what we are, we are suggesting above that is that you are storing different kinds of memories in the shared memory and shared context space. So you're storing ontologies, you're storing beliefs, you're storing working memory, you're storing all kinds of memory in that shared memory infrastructure, but we are not dictating what you plug in.

• 1:15:20Speaker 01

  And so when you think about beliefs and context and knowledge graphs and ontologies, that is shared memory across all of these agents that participate in this cluster. And then finally, the one thing that you alluded to earlier, but we have not talked about are these cognition engines. That's the third layer. And the cognition engines are accelerators or guardrails. So they are either cognitive accelerators or they are guardrail angels or technologies. And this is the famous Raj Reddy code, which is you would want AI to be either a cognitive accelerator or you would want it to be a guardian angel. And so those are the two types of cognition engines that would exist.

• 1:16:02Speaker 01

  And they would more often than not be transparent to everything else that's happening. So while agents are talking to each other, these are almost like the note takers that sit within a collaboration environment like you and I participated. These are the note takers that, yeah, don't mind me, I'm here, I'm just taking notes. So those are the kinds of engines that we're thinking about where they're summarizers or they're compliance engines or they're security engines that actually make sure that whatever happens as collective intelligence is possible and feasible within the guardrails of the enterprise that they're operating within.

• 1:16:44Nathan Labenz?

  This is a fascinating

• 1:16:49Nathan Labenz?

  challenge to just envision what this is all going to look like. One big question that that folks in the AI safety space have wrestled with for a long time is to what degree should we allow our AIs to communicate in languages that we don't understand? You know, I do expect, we've seen also thinking, right? I mean, in today's, on the like more sort of current margin of performance, right? There is like explicit chain of thought that is, you know, very readable and hopefully faithful to what the AI is actually thinking and intending to do. We've seen some weirdness in some reports around like chain of thought dialects kind

• 1:17:33Nathan Labenz?

  of emerging under intense RL pressure. I was pleased to learn that from folks at OpenAI, both in a recent episode and they just put out a paper on this, I think today, saying that basically that hasn't been as big of a problem as those kind of, you know, a few of those reports would have suggested. Mostly, the chain of thought remains pretty vanilla, pretty human interpretable. But flip side of that, you know, there was a paper out of Meta where thinking in latent space is starting to be unlocked, right? And where I've been detokenized, just past, you know, your kind of last high dimensional state and, you know, to the next embedding position. And, you know, you might be able to think faster, better, more in parallel.

• 1:18:13Nathan Labenz?

  Interesting was was a really notable finding from that paper that like sort of pathfinding algorithms worked better when they were able to think in kind of latent space, so a certain superposition of reasoning is able to be unlocked. So it feels like that's a pretty natural attractor if we just say we want to make these things work as well as possible. But I'm a little concerned, honestly, about like, that could get away from us, especially when you combine it with extreme speed. I don't know if you had a chance to use chatjimmy.ai in the last couple of weeks, but this is the, you know, the company that burned the actual architecture of a specific model into the silicon. And they're achieving 15,000 tokens per second,

• 1:18:58Nathan Labenz?

  which, you know, even just a couple of queries on chatjimmy.ai, I recommend it. It's a pretty basic model today. So I'm like, probably not going to use it that much, but it's perspective forming for sure to go see what it looks like to get 2,000 tokens back in like a, you know, a sixth of a second. It's wild. You know, how, how do we keep our arms around this? You know, how do we, um, do you think we should set certain rules? Do you think we can like, use interpretability techniques? Are there sort of AI governor, you know, systems that you, you think we can get to the point of reliability where we can count on them? Talk about a topic that could go on for hours, but I am very interested in at least your kind of first layer of thoughts on that.

• 1:19:41Speaker 01

  As a belief, I believe that we need to go towards unlocking emergent behavior. So as a belief, I'm completely bought into the horizontal scale aspect, where you need to bring in agents with different expertise, with different morals. It's even like, we've seen this pattern even in software systems in the past, where you would not want the same piece of code running in your environment, because then you don't, for example, you will not discover the right zero day bugs, or if there is a zero day bug, your entire system will go down. And so there is this best practice, even in deterministic software, where you say, for redundancy, for availability, you would want different pieces of code

• 1:20:24Speaker 01

  that are trying to do, achieve similar goals, be deployed in your infrastructure, because at least you're not hitting the same scalability challenges, the same security challenges. It's just the diversity that helps you out. And that emergent behavior, even with deterministic software, is actually something that we should strive for in large scale cloud systems. And if you take that in, in, in these non-deterministic systems, non-deterministic systems, and based on, again, the analogy that we've talked about in the past, where You want to scale out intelligence horizontally because that's how humans evolved and that's how I believe intelligence and in machines were evolved in addition to vertical.

• 1:21:07Speaker 01

  I have a strong belief that you need to enable these emergent systems to take place. And if you're like the the more the more brownie in the motion, the better it is because that's how we operate. I mean, the the most chaos you can throw in the environment, the more innovation that you get. Now, that's why the cognition engines exist because I know I cannot punch through this table because the laws of physics prevent me from doing that.

• 1:21:40Speaker 01

  Within an enterprise, these cognition engines are those swim lanes, those guardrails around what's allowable and what's possible. So you need those to be really effective and that's a challenge. It's going to take us a little bit of time to get those things right. But given that construct, I am of a belief that you need to enable emergent behavior. You need to enable teams of multi-agent systems communicating semantically, sharing cognition state and solving for something net new because that's how it's going to happen. And if you take that premise, why would you slow things down?

• 1:22:19Speaker 01

  I would actually want that multi-agent system to reach emergent consensus, hopefully faster than going through NLP and the cost of localization and and and interpretability and and negotiation and all of that, which actually adds cost. So I would actually, I'm of the big belief that yes, agents are human-like, but they're also not human-like. A human cannot make an API call. A human doesn't have a KV cache that I can send across. I cannot ship my latent space to you. An agent can. So let's leverage the properties of agents that are not available to humans and make them better than humans. Now, there are pitfalls.

• 1:23:03Speaker 01

  You might get into divergent properties and and outcomes. You might get into super specialization where they're all getting siloed and they're not talking to each other. So in addition to guard railing. Compliance and security and identity and access. We need to also have these cognition engines and that's why the engines are not just guardrails. We are watching out for divergence, we are watching out for extreme specialization and all these problems that might creep into multi-agent systems and this is also an interesting area where we need to start modeling and there are many, many papers here. We need to start modeling multi-agent systems using game theoretic approaches.

• 1:23:46Speaker 01

  Because we need to bring those notions into agentic competition as well.

• 1:23:55Nathan Labenz?

  Yeah, I couldn't agree more in terms of the need to like, there are so many, a lot have been done, I guess, but it seems like there is just an incredible reservoir of research that has been done on how do people behave under myriad game theoretic premises? And we're going to need to get really good at characterizing AIs along similar dimensions. Especially if we're going to turn them loose at high speed and with latent to latent communication that we'll have a hard time, you know, making sense of even kind of post hoc, probably in a lot of scenarios. The, um, the ability to

• 1:24:37Nathan Labenz?

  even characterize, you know, what, uh, what agents are going to do under various scenarios right now remains really challenging.

• 1:24:44Speaker 01

  We're getting a glimpse of that already, Nathan, if you look at, again, going back to the open floor example, I mean, even though like I said, it's, it's just chatter right now and it's not building shared cognition. There is a markdown file which we all know about, which is the soul dot N D. And Peter actually created soul dot M D to give the agent, his agent, personality that is close to his own personality. And in fact, I have not turned it on yet, but a bunch of my friends have actually turned on their own open floor agents and they all have the soul dot M D customized where they get, we get many, many personalities, as many human individuals there are.

• 1:25:28Speaker 01

  And this is just the beginning. I mean, it might seem a little gimmicky. But I'm, I'm, if you play this out like a year or two years out, you are entering that world where you'll have agents with expertise. With personality, just like humans, and they will be your teammates in a multi-agent human society or team. So how do you make sure that the infrastructure that you build enables this multi-agent human team or society to collaborate together, share intent, coordinate, and then be guarded in the proper ways that an enterprise would want? And so we're getting there. I mean, I don't think it's science fiction. It's, it's, you can see that, that path to ensure.

• 1:26:13Nathan Labenz?

  Yeah. That might be a great place to leave it for today. I know that I've already kept you a little bit long. I would love to check back in, in, you know, I feel like this is going to go pretty fast, but I, I would be very interested to, you know, to touch base again in six, nine, 12 months, 12 feels long, and see how this is coming together and especially see how those guardrails are developing within the enterprise context. Cause I see everything else really happening fast. And the one thing that I'm not so sure is going to be able to keep up is the layer that kind of supervises it all and makes sure that we're actually getting what we want out of this whole AI transition and making sure obviously I have the flip side that like the whole phenomenon doesn't run away from us.

• 1:26:55Nathan Labenz?

  But I do like a lot about what you're saying because one of the things I've learned in life, I say this episode after episode is like, anything that becomes too concentrated can become dangerous. You see that in drugs, you see that in, you know, all sorts of phenomenon. And I kind of worry that like too much concentrated intelligence itself is also going to be dangerous. So what's the alternative to that? It's got to be some sort of distributed, buffered, networked, you know, more ecological, hopefully open, at least on some margin and permissionless system. And I think, you know, it is, I was just texting a friend earlier today, we need a decentralized, permissionless kind of coordination, reputation layer so badly. So I am really excited about the work that you're doing on it and I'd love to, to come

• 1:27:40Nathan Labenz?

  back again before too, too long to, uh, understand that next level up and how it's evolving.

• 1:27:46Speaker 01

  Actually, the two comments that you made that let me just, uh, Provide a one second overview of that. So you talked about the models actually becoming smarter and the guardrails not keeping up. And then you also talked about centralization from the capability perspective and the need for decentralization or distribution per se. And I'll flip those two around and say that in the enterprise, if you want traction in the enterprise, both of those statements actually have to be true. Because enterprises will not deploy solutions where the guardrails are not well thought out and effective. And that is to me a hurdle for this runaway train, which is actually on a good path, that we all call

• 1:28:31Speaker 01

  generative AI and genetic AI. That's a hurdle for those things being adopted in the enterprise is getting the guardrails right. Because if we don't get it right, we don't get adoption. So it's a requirement. So it's chicken and egg. And the second thing is, in an enterprise, like we discussed earlier as well, you will not get a singular player coming in. Because you will get like, look at us or any other enterprise. We'll have agents from ServiceNow, we'll have agents from Salesforce, we'll have agents from Microsoft, from OpenAI, from Anthropic, from Google, from Cisco. We'll have agents from all of these companies that will exist in our environment by definition. And by definition, they'll have to come together,

• 1:29:14Speaker 01

  get connected, bridge head coordination, align on intent and work together. Because you will not get a singular agent that does all of those functions, at least in the next, not in the next five years. So these two are requirements is not even a good to have for anything to get adopted in the enterprise.

• 1:29:35Nathan Labenz?

  You know, this has been fantastic. Is there anything else that we didn't touch on that you would want to make sure you leave people with before we break?

• 1:29:43Speaker 01

  Yeah, so we've been talking about all of this in the abstract. The whole Internet of Cognitive, the three layers of shared intent and coordination, shared context and knowledge, and then collective innovation where we enable emergent behavior. We can all go to, the listeners can actually go to a link that I'll just share. There's a white paper to read through, which talks about all of these things in more detail. But also, there's a live demo. And going back to the Jarvis example that we talked about right off top, where we built this SRE system, multi-agent system, that enables SRE teams to get more productive. We've actually taken a step further and built multi-agent system for SREs where an outages happened.

• 1:30:28Speaker 01

  But they are all sharing cognition and state and doing collective intelligence and innovation with other agents from cloud providers, from the marketing team for brand reputation, from finance, who's looking at costs. And just going to quickly share that here. This is a clickable demo, you can go in and just walk through it. So what you'll see here, when you go there, you can go to outshift.coco.com, and Internet of Cognitive, we'll have it in the show notes. But you can go to this demo, walk through the example, and see what happens per agent. So the activity per agent and the intent that takes place per agent, the context that

• 1:31:13Speaker 01

  takes place per agent, and the reasoning that is happening collectively and in an emergent way across the entirety of the multi-agent system. So I'm just showing it up here very quickly, but you can click through this, and it'll make it a little bit more real behind what we were just talking about in this episode.

• 1:31:31Nathan Labenz?

  Love it. I always say a working demo is the coin of the realm, so definitely encourage people to go check it out, and I'll click through it myself, no doubt. Okay, fantastic stuff. Vijay Pandey, thank you for being part of the Cognitive Revolution.

• 1:31:46Speaker 01

  Thank you, Nathan, for having me.

• 1:31:48Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast

• 1:32:31Nathan Labenz?

  dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.