Intelligence on the Edge: Liquid AI's Ramin Hasani on the Search for Device-Native Foundation Models

Liquid AI co-founder and CEO Ramin Hasani joins Nathan to make a technically grounded case against the idea that scale alone defines the future of AI. Drawing on Liquid’s path from MIT CSAIL work on liquid time-constant networks to Automated Foundation Model Design, he explains why efficient, hardware-aware architectures can look very different from frontier-scale attention models. The conversation centers on device-native foundation models for phones, laptops, cars, and wearables, including Liquid’s open-weight LFM family and production deployments at Shopify and Mercedes-Benz. The stakes are whether useful intelligence can move beyond the data center into local, privacy-sensitive, low-latency applications—and which model and chip companies will own that on-device intelligence layer. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/intelligence-on-the-edge-liquid-ai-s-ramin-hasani-on-the-search-for-device-native-foundation-models/ Mercury: Command is Mercury’s new conversational interface, giving you natural-language access to your finances and helping you take actions within your existing permissions and approval policies. Visit https://mercury.com to learn more and apply online in minutes. Sponsor: Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (03:53) Special Sponsor (05:41) Liquid AI origins (22:35) Neurons versus parameters (Part 1) (22:40) Sponsor: Claude (24:32) Neurons versus parameters (Part 2) (30:51) Scaling liquid networks (40:09) Automated model design (52:04) Gating and input dependence (01:01:16) Architecture bias spectrum (01:09:17) Device foundation models (01:18:16) Hardware intelligence layer (01:30:01) Local agent setup (01:36:20) Miniaturizing intelligence limits (01:40:40) Curiosity driven AI future (01:43:45) Episode Outro (01:46:46) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello, welcome back to The Cognitive Revolution, and happy 4th of July to everyone in the United States. Today, my guest is Ramin Hasani, CEO of Liquid AI, a company founded by MIT researchers that's developing device-native foundation models. I'll say upfront that just before recording, I encouraged Ramin to go deep into the weeds on the technical details of Liquid AI's work. And as you'll hear, he did a truly excellent job, demonstrating a mix of technical sophistication, differentiated vision, and a contagious passion that, in my humble opinion, makes this episode an instant classic. We start with an overview of the team's research into tiny, biologically-inspired,

• 0:44Nathan Labenz?

  differential-equation-based neural networks that Ramin and team developed at MIT and which inspired them to start the company. Some of the capabilities they demonstrated, such as parking a car with a control module that consisted of just 12 liquid neurons, still sound a bit like science fiction today. And while those systems haven't scaled up to today's capabilities frontier, the company has maintained the liquid philosophy, which today means taking a neutral empirical approach to designing and optimizing neural networks to perform under all sorts of exotic constraints, including, most commonly, the need to run on edge devices with limited memory and processing power. Considering the fact that the global smartphone

• 1:27Nathan Labenz?

  and laptop market is roughly $800 billion per year, a number which the global AI data center buildout is only now surpassing, that demand for inference threatens to price much of the world out of the frontier model market, and that so many enterprises and individuals value privacy and the ability to control their own information, this is an absolutely massive market opportunity unto itself. And Liquid has serious proof points, including holding the number five spot on the Hugging Face United States Downloads leaderboard, plus notable partnerships with companies such as Shopify and Mercedes-Benz. Anyone who doubts can do a quick download and demo of Liquid's Apollo app, which shows, in my experience, that even a 1 billion parameter model, which combines a small number of attention layers

• 2:12Nathan Labenz?

  with a very simple gated learned convolution, while admittedly far from the frontier, can run fast enough on an iPhone to be a real option for basic use cases, such as privately searching through and classifying one's own local documents. Perhaps most interesting is the network architecture search process that Liquid uses to develop networks for particular use cases and runtime environments. Having found that proxy metrics too often lead the process astray, they now evaluate models on real downstream tasks, on the actual target hardware that their customers intend to use. Ramin shares a lot more detail on their findings. But in short, while attention-based architectures continue to generalize better than any known alternative

• 2:56Nathan Labenz?

  and therefore continue to dominate the frontier, the more specific your use case and the more limited the compute resources you have available, the more likely their search process is to land on an exotic architecture. This, for now, is where architectures like Mamba and other subquadratic innovations really shine. Toward the end, Ramin teases a platform that Liquid will soon be introducing to allow customers to fine-tune small models for their own use cases on a self-serve basis. For multiple reasons, including the potential to ease demand for frontier models and improve access to AI globally, I for one will be very excited to see that come online. And so, without further ado, I hope you enjoy this high-energy look

• 3:41Nathan Labenz?

  at how Liquid AI is squeezing as much intelligence as possible out of any given computational resource with co-founder and CEO Ramin Hassani. The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. I've wired AI into nearly every corner of my life: my email, my messages, my calendar. I even gave Mercury virtual cards to my agents with low limits and category and merchant restrictions for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me.

• 4:26Nathan Labenz?

  But for real-time, up-to-date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard, too slow, and too error-prone to be worth it. And that's why Mercury's new conversational interface, Command, is such a big deal. It's built directly into Mercury, which means you get natural language access to your finances without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way. And it can already help you take actions, too, with everything bound by the permissions and approval policies that you've already set up in your account.

• 5:10Nathan Labenz?

  I am genuinely impressed to see this level of AI integration in banking in 2026. And so, I invite you to join me in the future. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting The Cognitive Revolution. And now, on with the show. Ramin Hasani, co-founder and CEO at Liquid AI. Welcome to The Cognitive Revolution.

• 5:47Speaker 01

  Thanks for having me.

• 5:49Nathan Labenz?

  I'm excited for this conversation. I've been following Liquid AI from afar for a number of years and fascinated by some of the architectures that you guys have developed, going back to your time at MIT together. Also fascinated by the trajectory that I perceive the company to have taken over the last four years as it's become more customer-focused and commercial entity. So maybe for starters, how could you tell the kind of broad story of Liquid AI leading up to what you're doing today and what the company's mission is today?

• 6:25Speaker 01

  Yeah, absolutely. So Liquid AI, three and a half years ago, we spun out of MIT, Ccel, building on a technology that we have actually have been working on it like 10 years before, like basically like a decade before when we started the whole company. We've been, our objective function like at MIT has always been like maximizing the amount of intelligence that we can into a smaller format of algorithms. So efficiency has been like the cornerstone of our research. And we have been working on, you know, like specifically thinking about robotics and systems that were like coming into the real world. You know, like the idea of liquid neural networks, that was kind of discovered as my PhD thesis and together with my co-founders, the four co-founders that I have,

• 7:09Speaker 01

  we've been researching ideas about how can we build machine learning solutions that can go on robots and doesn't have like millions of billions of parameters so that we can actually host them directly on, let's say, CPUs, NPUs, or let's say like, you know, like smaller GPUs that are mounted on top of like physical systems while delivering the reliability of much larger kind of instances of artificial intelligence systems. Essentially, what we try to do, we try to, you know, like build alternative algorithms in order to get creative in the algorithmic space, to see how can we build machine learning systems that can generalize beyond the data that they have seen. Because when you go in the real world, the distribution shift becomes like a real thing. You know, like you imagine you deploy a robot

• 7:52Speaker 01

  in an open world, kind of like, let's say, an autonomous car, a flying drone, a fixed-wing vehicle, you know. So all of these systems are going to go and very, very rapidly, you're going to get out of distribution. So you have to actually build systems that are really comfortable around getting it, being able to get out of distribution. You know, humans are extremely good at it, and humans do that extremely, I mean, natural learning systems, animals also like the same way, right? So they, they actually follow a really nice trajectory out of distribution as well. You know, so that means like the learned concepts can generalize to data that you have not seen before, right? So our idea was that like can we build systems that can have that kind of properties, you know, better out-of-distribution generalization?

• 8:36Speaker 01

  This has been not just our goal. I think an entire artificial intelligence field has been like working towards this thing, and especially like in the robotics and closed-loop environment, you know, like closed-loop beginning environment where you have an agent acting in an environment, you know, you want to have that kind of properties. So naturally, the place that we started looking into was brains, you know. We started looking into animal brains, you know, and we started looking into how, one thing that I really liked, I wanted to look at it from a first-principle kind of approach, like how neurons exchange information with each other. You know, we started looking into brain of worms, you know, small animals, and why worms, this specific worm, sea elegants. We started working on the brain of the worm. The reason behind it is that it was the only animal

• 9:18Speaker 01

  that in 2015, when I started this type of research as part of my PhD with my co-founder Matthias Lechner, we, this was the only animal that we knew the entire nervous system as a whole. And this animal exhibits massive amount of kind of sensory reactive kind of behavior, like amazing levels of control with 300 cells in its nervous system, you know. And that's what's fascinating for us because this is much smaller than any neural network that performs control at that time, like on, let's say, like autonomous systems, you know. And this worm, it can do better, you know, dexterous movements and stuff like better than the best robotic systems that we actually had in the world, you know.

• 10:01Speaker 01

  So we thought that, okay, so let's start understanding how neurons exchange information in the brain of the worm. From there on, let's start building nervous system more complex and complex kind of neural circuits so that we can get to the stage where we can, let's say, like build the next animal, build the next, you know, like basically follow the path of evolution of nervous systems, you know, and see how we can, how we can rise and evolve as part of this thing. So. There are equations that describe the neuronal neural dynamics of, let's say, like two neurons. In the brain of C. elegans, because the worm is very small, the neurons do not spike. They are graded kind of neurons, like they they they behave like in an electrotonic way, you know.

• 10:44Speaker 01

  So they're they're they're very similar to artificial neural networks that we have, because they're also very differentiable, right? Because you don't have spikes in the activity of neurons, they're very differentiable. That's why, like, it was even even nicer for us, more attractive for us, because we could apply learning theory to this type of neural networks. Once constructed, like, let's say, two neurons, four neurons, eight neurons, like, let's say, a hundred neurons. Next to each other, and then you start, like, training them, you can apply backpropagation, like, as a differential kind of programming. On top of a a a a system that is built by by these type of uh uh inspirations that we got from nature. The type of differential equations that were there were also very well behaved, you know, you can make them as complicated as you can, you know, and.

• 11:25Speaker 01

  And and and then when they get more complicated, they mimic their biology much more. And, uh, you know, of course, like, from a computational footprint, they become more complex to scale, you know. But it's still like you can sacrifice, you can actually tune how much complexity you want to encode inside a neural architecture. You know, in artificial neural networks, like, we we try to abstract away, like, those, uh. Complex differential equations that describes the neural dynamics between two cells. And we just show them with a sigmoidal function, gated sigmoidal functions, and now we have, like, matrix multiplications that are coming, like. You're capturing the impact of, like, the inputs that are coming to a system. With a transformer-based architecture and attention mechanism and all of those things.

• 12:07Speaker 01

  But these are all simplified kind of computations for us to be able to scale these machine learning solutions. For us, from a neuroscience perspective, it was very interesting for us to explore, to see. If we actually start making, like, bringing back those differential equation-based computation and a little bit more elaborate form of computation. Into the behavior of everything a neuron and mimic the behavior of, like, how two neurons exchange information with each other. Maybe we can unlock something like greater than uh what what we have seen from an artificial neural networks perspective. So this, I mean, early on, the results were fascinating, like, we saw that with twelve neurons, with with actually twelve neurons, you could. You could parallel park autonomously like a car, you know, like a small car. With nineteen neurons, you could drive a car.

• 12:51Speaker 01

  With hu- with thirty neurons, you can fly autonomously, like navigating kind of a drone. You know, so and you can get sensory information and process this information with this. A little bit more complex and elaborate version of these neural neural dynamics, you know, which we called liquid neural networks. And liquid for adaptability, I called it liquid liquid time constant neural networks, like that was the LTC kind of paper that we we we got out. And we coined the name liquid for the fact that the. Dynamics of these systems are staying flexible even after training, you know. So the models would be able to react to new type of inputs that they receive. And even learn during backpropagation, how to be adaptive towards the inputs that you're receiving, you know.

• 13:35Speaker 01

  So it actually encoded a little bit more degrees of uh freedom on flexibility of the learning dynamics of a neural network. Compared to artificial neural networks, compared to other systems that you've seen. That form of dynamics actually allowed us to scale this technology into not just, you know, not just in robotics, but also like. Applying it to predictive AI in in financial services, in medical domain, and in many, many different places that we applied in. Like, let's say, audio modeling, you know, multimodal video understanding, you know, like, we applied this technology over and over. And we saw promise that this technology is actually very, very amazing, like, you know. Every neuron is much more complex than a normal kind of neural network.

• 14:16Speaker 01

  But but at the same time, like, you don't need that many uh uh points of computation in order to actually get you get to the results that you want. We also saw that the out-of-distribution performance of these models are extremely good. So and and they they were suited for robotics applications, so, I mean, we had interactions with with the United States. Of course, I mean, there there was, like, at that time, Boeing was actually hosting my postdoc at MIT. And and and and there, we showed that you can actually fly even jets, like, with these type of neural networks, with, like, a handful of these neurons, you know, like, which was. Very interesting kind of topic to see how far you can push, like, let's say, out-of-distribution generalization. With a handful of, like, a very, very small set of system, you know, a system that is like powered by these liquid neural networks as opposed to artificial neural networks.

• 15:02Speaker 01

  Then, so as I told you, that each node in a liquid neural network is very complex, you know, like, it's it's a differential equation, you have to solve that, and you have to. Solve a set of different, the more neurons you have, the more complex is a forward pass and a backward pass to a network. Therefore, we needed, we needed to think about efficiencies, you know, these thi- these systems are highly nonlinear, you know, like, we tried not to sacrifice on nonlinearity. Because adding nonlinearity to learning systems allows you to build more expressive systems, you know, this is a a proof that actually I have in my. PhD thesis, where I showed, like, nonlinearity actually directly contributes to, especially on the smaller size of models, you know. But then the idea here for us was that, like, how can we scale these systems, you know, you have nonlinear systems, but it is extremely difficult to actually scale nonlinear systems.

• 15:46Speaker 01

  So you have to solve, the reason behind scalability is the computational complexity of the models are are hitting, like, even cubic complexity, you know, like, it's a. It's not even quadratic, you're talking about cubic complexity, which is like too much. And then you have, like, this, like, you you usually have, again, you have a set of differential equations, you can roll them out with a numerical solver. You run the numerical solver and you can step by step compute kind of the the outputs that that are desired, and then, you know, like. The the more steps you you want to use in order to actually compute, then you would get more and more accurate kind of results at the end, right? But then the problem becomes scalability again, you know, like, you you cannot really get to infinite precision with these numerical solvers. One idea that came to us was that, what if we just solve the whole system in closed form?

• 16:31Speaker 01

  You know, like, literally, like, just, you have a differential equation system with these liquid neural networks. Each of them representing neurali- that neural dynamics of, let's say, two cells that are exchanging information with each other at a certain level of abstraction. And we thought that, okay, let's take this system and try to solve it in closed form. Turns out, the closed form solution for this type of equation has hasn't existed since nineteen oh seven. You know, so nineteen oh seven, there was a scientist called Louis Lepic. That actually modeled the membrane potential, like, how to model mathematically membrane potential kind of in in in in cells. And that format of equation became like a fundamental of channel modeling, like, how.

• 17:12Speaker 01

  Information propagates through ion channels inside a bit, like, in inside a cell, and then how neurons, how neurotransmitters are getting propagated to the other one. So nineteen oh seven, this was Louis Lepic's uh kind of, you know, membrane potential equation that is like an open differential equation. And then we have seen, like, some scientists called Huxton and Huxley, they uh they started working on. Really biological kind of grounding this type of differential equation and adding a little bit more complexity into the model. They developed a neuroscience model of one neuron, how how actually a neuron reacts to, let's say, a membrane potential. Nineteen fifty-three, they started, nineteen sixty-three, they won a Nobel Prize for the fact that they actually.

• 17:56Speaker 01

  Developed a better and a more accurate representation for a neu- neurali- that neuronal dynamics, right? And then from there on, so that if in every textbook that you read, these type of formats of equations, they do not have a known closed form solution. You know, and this format, like, liquid neural networks were also part of that type of equation. Really, for the first time, we actually solved that, and in this was like twenty twenty-two. Around twenty twenty-two, we solved the liquid neural network kind of interaction of neurons with each other in closed form for the first time. This became a Nature Machine Intelligence paper, uh, published in November of twenty twenty-two. This is called the closed form continuous time uh systems, you know, like, this is, this liquid neural networks in closed form. And the closed form, I mean, it has massive implications, why?

• 18:41Speaker 01

  Because now I don't need to use any numerical solvers to actually run a liquid neural network. I can now have not only hundreds of neurons, but now I have, I can have billions of neurons next to each other, and I can actually scale these computational. Still keeping the non- nonlinearity as part of this thing, you know. Then, then, uh, in in January of, in in actually, actually, beginning of February of twenty twenty-three. An article came out of a Quanta magazine, you know, about, let's say, me and my co-founder, Montias. We, um, it was about a profile about, like, all the implications of having some closed form solution finally on. On on a on on neural dynamics, you know, like, and how important this can be for both a machine learning and also, like, for brain science as a whole.

• 19:26Speaker 01

  And then, so my inbox was full of VCs, you know, Silicon Valley started, like, talk, like, everybody wanted to draw it, like, a term sheet of us, like, to really get started. And and really scale this technology, because this was fundamentally different than a transformer-based architecture and attention-based architecture. This was grounded in biology, like, the type of math that we have. Around the same times, we have alternative models are coming out, like, state space models, like, you know, like, like the, you know, like, you've you've seen, like. Faster iterations of status models, like convolutional neural networks that came out, like, to be, like, scalable. All of them in a linear form, because the scalability of alternative models, you have to linearize them, and then you can scale them, you know.

• 20:05Speaker 01

  And we have something here that is also another category, you know, like, it adds some operators that we learned from biology and from physics. As we we grew the research, and we got to the point where we can now become very competitive and and and, you know, like. Building models, like, for solving more and more general-purpose tasks. And when I say more and more general-purpose task, I'm talking about modeling signals beyond predictive performance. Modeling signals for language, for audio, and for vision in a way that human humans understand, right? That that's the form, that's the complex format that humans understand. So the mission of Liquid AI became building efficient, general-purpose AI systems at every scale.

• 20:46Speaker 01

  And we we coined the word efficient in our mission, and we we care about it so much. Because I think we are, from a foundation model company's perspective, we are probably the most efficient foundation model company on the planet, I'll tell you why about this in a in a minute, but. But the reason why, you know, like we we we set the mission into like efficient machine learning because we start at how much of the considerations that we had to do in order to really build a closed form solution, then taking this closed form solution and scale it to, let's say, a a a system that can model language, a system that can model audio models, a system that can model video and vision in general, you know, how much considerations they had to go into these things so that we can computation in a computationally tractable

• 21:31Speaker 01

  way, scale our neural architectures to the point that that where we are today. Today, the the base of our technology, I'll talk about like the core technology of the company later on, but but I will tell you today, the technology became liquid foundation models, liquid foundation models or elephants are pretty popular right now, like we are ranked number five actually in the US from the number of downloads from the popularity perspective. We have over one million downloads per week on the high face like that's a lot of downloads of these small models that you're building. They're very popular, like the the top top ones from the US side are Google, Meta, Microsoft and Nvidia, and the fifth one is Liquid AI. So from the US side. So I mean, we we got ourselves there by consuming about 1,000

• 22:14Speaker 01

  GPUs, having 1,000 GPUs in house. So that's like from a foundation model perspective, like getting to that level of popularity and releasing more than 50 models, instantiation of models that people are using in enterprises. That's kind of the place where we brought liquid, building on top of inspirations that we got from nature and and and the past that I actually like portraying for you.

• 22:35Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 22:40Nathan Labenz?

  Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts, going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with

• 23:23Nathan Labenz?

  the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai slash T C R. That's claude.ai slash T C R. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai slash

• 24:08Nathan Labenz?

  T C R. Brilliant. So many different directions I want to go in follow up there. Maybe for starters, one thing that just on a very fundamental level jumps out at me is when you describe the liquid neural networks, you describe them in terms of the number of neurons, as we're used to hearing about the number of parameters. And something tells me there's like a kind of paradigmatic difference there that underlies the difference in description. So yeah, help me develop my intuition a little bit more for, I guess it probably connects to nonlinearity and and this is so fundamental, right? Because there's so much incredible

• 24:49Nathan Labenz?

  progress that has been made, but on things like really robust out of domain generalization, we've still got a lot of work to do on things like adversarial robustness, we still have a lot of work to do in the mainline paradigm. So yeah, tell me, unpack a little bit, why are we counting neurons versus parameters? And what does that tell us?

• 25:09Speaker 01

  That's a great question. So I think early on, we we actually started talking about the number of neurons as the unit of computation like for in our research, because that was like very interesting. Like you can associate, you're like every neuron is a process that can be modeled by a mathematical equation, you know. Now, you can allocate basically like in a sigmoidal kind of from gates or the more functions, you have the more parameters getting added to that single cell, you know. And the cell that now in a liquid model in terms of parameters, you can say multiply by seven, you know, and that will be the number of parameters of, you know. So that in in that differential equation. But that cell itself, because of the equations that it has, it it computes more than just, you

• 25:52Speaker 01

  know, like a forward pass competition. It has like internal feedbacks, that's three degree feedback kind of mechanisms in inside. All those parameters are actually like one to one analogous like to any artificial neural network system that you would you would build like in terms of parameters. So a a a rule of thumb for like when I say number of neurons, you multiply by seven, you would actually get like number of parameters of the of the system. But this is for early on version of liquid neural networks, you know. As we started standardizing parallelization schemes for the type of mathematical functions that we want to explore, and let's say inspirations that you're getting, we just converge to the number of parameters. Like, of course, like today, when we when we are talking about a liquid foundation model that has, let's say, one billion parameters, we are literally

• 26:37Speaker 01

  meaning one billion parameters in the sense of like the any other kind of GPT that is getting built out there. Does that make sense?

• 26:46Nathan Labenz?

  Yeah, but let's zoom in maybe a little bit more on that neuron. And I guess I'm wondering a little bit too, what is the fundamental limit or what is the what are the first bottlenecks that you hit when you try to scale the original liquid approach? With the sigmoid type functions that we have today, those are, I think, certainly carefully selected to be easily run on GPUs. I'm guessing that these sort of multiple degrees of freedom inside a single neuron maybe present challenges in terms of executing that process on available

• 27:26Nathan Labenz?

  hardware, like what, how we can get these amazing results from tens of neurons. So the obvious bitter pill question would be, how do what if we take that exact paradigm and go to millions, billions? But what you must be hitting some bottlenecks along the way. What are they?

• 27:42Speaker 01

  So the main challenge is turning sequential computation into parallel competition. So when you have degree, like when you start like going from a single neuron dynamics to multiple neural dynamics, weight parameter, like parameters of your system, instead of being a scalars or vectors, they become matrices and tensors. So now you're talking about matrix multiplication, you want to turn the scalar computations into tensor computations, right? You want to be able to, this is how you parallelize kind of, let's say, sequential computers. The problem is that not all the time, like if you have nonlinear relationships

• 28:25Speaker 01

  in an equation, you cannot trivially create a one to one map between a vectorized computation and a tensorized computation. So those nonlinear kind of nonlinear attributes of liquid neural networks and any other nonlinear recurrent neural network, I mean, recurrence itself is adds like some degrees of complexity into into the math of the whole equation. I mean, let alone if the if the recurrence has has a nonlinearity on it, it becomes like a lot more complex to disentangle vectors or write them with with typical linear algebra kind of methods that we know into tensors. And you have to be able to turn these competitions into tensors to be able to compute in parallel, you know, like you compute them once at a

• 29:10Speaker 01

  time, you see. So that's that's kind of the the the mathematical, it's not related only to liquid models. Any mathematical operation that you want to tensorize, if it is nonlinear, you're going to have the troubles that we're talking about. That's why the state space models that actually came out, these are all about around linear dynamics. You know, we're talking about like linear dynamical systems, right? And why those linear dynamical systems are important is just pure fact that we do not have proper ways to scale nonlinear systems. You can apply nonlinearity to a not to a linear dynamical systems. For example, in a, let's say, like a gated, or let's say, you're like, you can, you can, you can add

• 29:51Speaker 01

  a sigmoidal function after you perform the dynamical system like in a linear way, you know, like you compute the, you compute all the matrices and everything like in parallel, but then you can do a point wise kind of application of a nonlinear operation on top of the entire tensor. That's what you can do. But what if the relationship between the parameters themselves are governed by some nonlinearity? So you cannot really use typical linear algebra. You you gotta, you gotta always approximate that nonlinear system into a linear system, then you would be able to actually parallelize the systems, right? Does that, does that make sense? So that's the fundamental bottleneck.

• 30:29Nathan Labenz?

  So how much does the closed form solution address that? And how far has the with available computing resources? How far has the original liquid network paradigm been able to scale so far to present?

• 30:50Speaker 01

  Fantastic question. So one of the properties of liquid neural networks was the fact that they have multiple feedback mechanisms. You know, they're not just one format of feedback. You know, like they have like three layers of feedback of between two cells between in the synaptic dynamic itself. Because we wanted to mimic like how how it is done in the brains, right? So it has multiple degrees of feedback. It has then a non basically nested nonlinearities on top of each other. That nested nonlinearity itself would also add a degrees of complexity. Even the closed form solution version of a liquid neural networks from a dynamics points of view, you can now instead of hundred neurons, you can have hundred thousands of neurons, maybe one million to 10 million neurons.

• 31:35Speaker 01

  But the nonlinearity, you still have to compute the the the models sequentially because of this nested nonlinear relationships that actually happens in the circuit. So even the closed form kind of solution would be limited in a scalability in the sense that you cannot paralyze them very effectively. But there are also like people are people are also like working on these topics a lot like paralyzing, but not just paralyzing, but the speeding up sequential computations. So we're talking about sequential scan, you know, like scan is an operator that you can run on top of like, let's say, like doing matrix multiplication instead of doing it in cubic or let's say quadratic, you can do it in sub quadratic kind of time, right?

• 32:16Speaker 01

  So you can actually reduce the complexity of computations even on nonlinear operators without paralyzing them, without linearizing them, you know? So there are there are research ongoing research going on like we are doing like some of that that kind of research ourselves to really find ways to to really sequentially run those nonlinear systems. Because clearly there is an advantage to have like those form of nonlinear nonlinearities when you do function approximation, you know? We are talking about also scaling laws becomes also very important here. I believe like the scaling laws define architecture, you know, how the how how does it do? When you're talking about transformers being this revolutionary thing, we're talking about maximum scale.

• 32:59Speaker 01

  The reason why transformer architecture and attention mechanism is such a brilliant architecture is the fact that it is unstructured. There is no structure when I when I'm talking about nested nonlinearities and all this crap that we have like in liquid neural networks. You don't have that in transformers, right? You have basically just matrix multiplication as the core functional kind of things that represent. And it is unstructured. You can literally make multiply any matrix matrix into any matrix of any size. So the whole idea here is this, the larger neural networks that you make into infinite size, the larger neural networks you make, the more want you want them to become less and less structured.

• 33:42Speaker 01

  And we've seen like the success of transformers at the trillions of parameters. Now we are talking about tens of trillions of parameters, like, I mean, the next generation of models that are going to come. We're talking about trillions of and you can do that with a transformer architecture. As soon as you start adding a little bit of bias in that architecture at scale, things become completely messed up. So we are talking about right now liquid neural networks, exactly these alternative architectures that we are we are talking about. They have a scale to hit, you know, like there is a regime of parameters that you can just do better. You know, like, let's say up to, let's say, 100 billion parameters, up to a trillion parameter, you know? Like, that's kind of the range we are we are operating right now.

• 34:23Speaker 01

  In this range, the smaller is the model architecture, the more you want to specialize them for a certain application to solve. And they are actually like mathematically, they are biased to actually solve a certain type of tasks better than any better than other types of architectures. So I would say biases on algorithms, you know, the more the more kind of biases you put on there. And what by biases, what I mean is this adding a lot more nonlinearity, adding like multiple gating levels on top of a neural network, you know? Adding like recurrence, you know, like multiple different types of recurrence, adding, let's say, convolutions like as a, I mean, convolution itself. If you just keep the convolution or kind of neural networks, they're also pretty unstructured because you can apply convolutions on any size of neural networks, you know?

• 35:07Speaker 01

  So that that that that question becomes like how much bias and what kind of problems you want to solve? So that the bias in the neural neural architectures becomes kind of a function of, I would say, scale of the neural networks as well as like the the use cases you want to solve them all. You see, so the spectrum would be the more you go to the larger size, the more unstructured kind of mathematical operators. Use pure matrix multiplication, use pure convolutions, use pure, let's say, sequential scan, pure, pure operators, you see, it's like you you get you get much more like and then forget about gates, forget about like adding like, you know, like fancy like a control, like forget gates and all those things.

• 35:50Speaker 01

  And at scale, this is something that we we have seen like because we have scaled like neural networks and we have like a really good understanding of that architectural kind of variation at different scales.

• 36:00Nathan Labenz?

  Just for calibration, when you talk about the liquid neural networks that have gone up to hundreds of thousands or pushing a million neurons, what kind of hardware does that run on? Is that like a CPU supercomputer?

• 36:13Speaker 01

  You can do that on a CPU, like on even even simple CPUs or simple GPUs, you know? Because they don't have that much of probably like one of these liquid neural networks would fit on, I don't know, one to 25 megabytes of, you know, like file size, basically, like you can you can literally like put them there. And then, you know, a CPU would be enough, like a Raspberry Pi would be enough like to perform computations. Yes, we have shown that on a lot of predictive specialized applications of AI that these systems can be like very, very powerful and do stuff. And and they can be as as powerful as the systems like these closed form variants could be as powerful as the open differential equation version of the neural networks in a speech synthesis. You know, like that's that's one of the applications.

• 36:56Speaker 01

  In, as I said, predictive sequence, like imagine you have complex formats of sequences coming in, multivariate kind of sequences that are coming in and you want to perform some sort of a prediction on top of them. These neural networks are actually pretty good. And then if you're talking about like more, let's say, out of distribution generalization to to the extent that you are like within a bounded period, you cannot have like an open ended continual learning. These are not liquid neural networks are not continual learning systems. They are more adaptive formats of computations because of the many gateings, many feedbacks and many, let's say, input dependent parameters that they have in their in their mathematical kind of organization.

• 37:34Nathan Labenz?

  What is the definition of continual learning that you're using there that they don't satisfy?

• 37:40Speaker 01

  So continual learning would be a system that is continuously receiving new data and retunes itself. So it changes also the parameters of the system. Liquid neural networks are the dynamics of the systems are input dependent. So that means new data comes in, the dynamics of the system would behave to those inputs. But the parameters of the systems, like any other neural network is fixed. It's just because the the format of every neuron or every node in a liquid neural networks was like a differential equation, you would have a different dynamics than you. So dynamics, you know, it's another dimension that you add to number of parameters. It allows you to compress more information, compress kind of more knowledge.

• 38:21Speaker 01

  If that's why with the smaller instance of the models, there's no free lunch or there is no magic here like we are doing like also with liquid neural networks. It's just that the access of dynamics has been something that we added to the neural networks for what? For like being more adaptable. So I'll give you a very tangible example here. Imagine you're driving and all of a sudden it starts raining, you know? So depending on the format of the rain, you know, your autonomous driving system that is actually taking control of the car could react to those kind of driving scenarios. And if you have like a traditional neural network that hasn't seen that kind of environment, it might actually get like biased, like really, because that rain that actually hits the camera, it's noise. There's a certain type of noise, you know, or certain type of adaptation of the input that is coming.

• 39:07Speaker 01

  But but the reality hasn't changed. The confounding variables of the whole environment hasn't changed, right? So that's why liquid neural networks, because they react to the input completely differently, you know, like they they they absorb the input, they apply the low pass filtering on top of the input, you know? They are much more adaptable to those kind of scenarios. That doesn't mean that they change the parameters of the system to be more adaptable because there are two accesses here, one changing the parameters of the system, the other one is changing dynamics of the system. I would consider continual learning to be attributed to where you continuously also change the parameters of the system. In liquid neural networks, we don't do that.

• 39:47Nathan Labenz?

  Gotcha. Okay, helpful. Okay, so let's fast forward to the present. You guys are now in the market, working with customers. One notable conversation I heard with one of your customers was with the CTO of Shopify on the Laten Space podcast, who had some very good things to say about you and your your technology. I was struck by the fact that you've gone neutral in a sense. Now, excuse me, what I, what sort of jumped out at me in terms of your approach is that you've developed a architecture search process where the promise to customers is not that, hey, we developed this one paradigm and old calculus teacher used to say, when all you have is a

• 40:32Nathan Labenz?

  hammer, everything looks like a nail. So you're explicitly promising to people that even though we came from this lineage of this particular kind of network, we're not just going to blindly apply it to your problem. Instead, we've created this sort of higher abstraction or more meta process for searching through architecture space to find the thing that's going to work best for you. And the two notable details of that are, one, proxy metrics you found don't work that well, just measuring perplexity or whatever, you found you needed to actually go farther and test models on the actual downstream tasks that they're going to be asked to perform.

• 41:14Nathan Labenz?

  And second, hardware in the loop, actual target hardware in the loop to test the architecture, subject to the very real and physical constraints of the robot, the sensor, the phone, whatever it's going to be running on. Maybe, I understand that the LFM model came out of that, but maybe before we even get to the LFM, could you sketch out the range of different types of problems that we're putting into this architecture search? And then maybe also, subject that implies, of course, what kind of hardware are we targeting? And then what kind of different architectures are winning for different kinds of problems under different kinds of constraints?

• 42:00Speaker 01

  Absolutely. So there's a system that we developed in-house, we call it Automated Automated Foundation Model Design, you know, AFMD, you know, like that's a meta, meta learning system that puts a hardware in the loop and then tries out like many different operators with an evolution strategy, the criteria as an evolutionary strategy, optimizing for a couple of things, optimizing for memory consumption on that device, optimizing for latency, optimizing for speed, while no sacrifice on quality. When we talk about quality, perplexity is not the measure, it's actually the application to downstream applications that we care about, you know, it's not just also public benchmarks, we're talking about 100 different benchmarks. So the problem space becomes, from a meta learning perspective, becomes like a very,

• 42:43Speaker 01

  very complex kind of problem. Now, I'll tell you that, why did we take this approach, you know, like to design an architecture? Why? Because we wanted to remove all the human biases early on, as we are actually like building architectures. One of the things that we realized, like culturally at companies, like this is what I can tell you, like even at largest foundation model labs in the US right now, Anthropic and OpenAI, there are a bunch of people, like people that are coming from the science, like I call them the Avengers of the architectures, or Avengers of the, you know, like post-training, or let's say pre-training. These groups of people, there's usually like a very small set of people that are calling the shots on like, oh, you know what, you're going to tweak this portion of this architecture so that it performs better.

• 43:27Speaker 01

  Why? Because in my personal experiences, it has started working better. If you're really, truly think, and this is something that is broken in all the foundation model labs, you cannot say that like somebody has a fix to this, but now their recursive self-improvement kind of process is actually fixing for that, because now people are just finally realizing, you got to give it to the algorithms, you know, you have to be better lessons, you have to, you have, better lesson, you got to, you got to be giving it to a systematic way to actually find out what is the true architecture for the, for the problems that you want to solve. You can build like a general purpose computer. The insights that are shared with you in the format of the scaling laws of neural networks is coming out of our massive exploration of

• 44:10Speaker 01

  the space of architectures, you know, the fact that in the smaller kind of category of models, some biases on the architecture help. In the larger instances of the models, you don't need to bias the systems, you can actually go pure convolutions, pure transformers, and you can do pure, let's say, reconnaances that are like very simplified. You don't need to add any form of kind of, you know, specialized kind of treatment, like getting there, you know, gated delta nets, you know, and then there's the mambas, and there's the jambas, and there's like, there's a lot of different variations of these architectures that are coming out. What you want to do though, you want to, you want to be completely unbiased. Even us ourselves, you know, day one, like when we actually started Liquid, we had one

• 44:54Speaker 01

  of the inventors of SSMs, actually, as our founding scientist, Jimmy Smith, you know, he invented S5, you know, like the simple, he basically brought the parallel scan idea to the SSM world, you know. Then we had Stefano Mascheroli and Michael Poli, they were actually designing like the, you know, like the hyena hierarchy, you know, there was like a convolutional kind of path that is going, you know, all of these are linear systems, you know, like within the realm of linear systems. And then we came out of like this nonlinear, like hyper control theory kind of enabled foundation models, which is liquid neural networks. A lot of big heads in the room, and you got to, you got to call the shots around, again, the same problem in all the labs, you know, like you got to, you got to figure out, we

• 45:36Speaker 01

  thought that, okay, let's fundamentally solve this problem, let's really like systematically first principle, let's put all the operators of interest, whatever we think that could be an operator that can give rise to a general purpose computer, like a system that can, let's say, a liquid neural network on its own is a general purpose computer, you can actually scale it, if you can theoretically, you can get it. So you put all those kind of equations, all those kind of things in a unified kind of theory, you know, like with linear input varying kind of operators, you know, like operators, all the, let's say, space of operators can actually be encapsulated inside this linear input varying, or let's say, in general, liquid input varying kind of operators, because input dependence is what's something that we

• 46:19Speaker 01

  have been like talking about this for the last 10 years, you know, like that's, that's extremely important. And we see that right now in transformers is also extremely important to have, that input dependence is actually coming naturally with the attention architecture as well, you know, but in a very, very unstructured way, it's not as structured as, let's say, like what is done in RNNs, in liquid neural networks, in, you know, like in SSMs, you know, like in all of those things, you don't see these gated variables like to be very structured in transformers, you know. Now, we actually have various variants of a convolution operators, various variants of, of, of a recurrent operator, you know, and then the attention themselves, you know, there's like, there's group query attention, you know, there was the original transformer,

• 47:01Speaker 01

  there's like many different things. Then, on all, some of these variations of dynamical systems, and also various types of convolution, I said convolution before, so, but then, then what we brought, we brought also the liquid computational blocks, like for example, the gated, the double gated kind of variance of gating, what like a certain type of biases on, onto like different dynamical systems or different operators, we added them and probably the space becomes like something around 50 to 100 different operators. And then you want to build hybrid models, so that you can reuse, basically, what's the goal here? The goal here is that maximizing efficiency of computation without loss of accuracy. That's kind of the goal that we set. The objective function of the search space is also this,

• 47:44Speaker 01

  let's run, you know, so let's have like all of our, let's say, all of our compute actually thrown at this problem, and let's, let's try to see like, how the system is going to design this, you know, and we started doing scaling laws on this, like we went as small as a neural network as, let's say, 10 million parameters, to running the scaling laws up to like 72 billion parameter models, you know, like in these hybrid kind of structures. So we have done a massive, early on, early days of Liquid, like 2023 and 2024, has been always like proving out, on a certain type of processor, what's the most efficient type of architecture that can come out. And turned out, as, as, as all of our, you know, when we put all of our biases

• 48:26Speaker 01

  away, you know, all the gating stuff that we are putting around operators, you know, they got simplified. So, you know, like in Mamba style architectures, like you have a bunch of gating mechanisms in, let's say, gated delta nets, you know, gated delta nets, you know, you got, you have like a bunch of operators in there, you know, in linear attention, there are like gated variants, everybody's like tweaking a certain parameter in the network like by hand, because in their own experiments, actually, they observe something. Turns out, all of this has to go away, if you want to get to the most efficient format of architecture. And it became the double gated convolution that actually came out of this, this massive search space, like AFMD, you know, like this, this original kind of system that we designed.

• 49:09Speaker 01

  And this was one of the candidate architectures that came out, that turns out to be very good on general purpose computer that I call CPUs. On CPUs, because CPUs don't have, like, they have a special kind of structure, right? On CPU, on all the CPUs coming out of AMD, Qualcomm, let's say Intel, you know, everyone is building processor, ARM, all the ARM processors, we tried to get to the place where tested different operators, what's that generic kind of structure? And, and, and let's say computational graph, that gives you to the most simplified, no, you know, no added hand tune, like there's no hand tuned features to anywhere, you know, it's just literally coming out of like the test, the semantic test that we've done, and

• 49:52Speaker 01

  this became like the de facto kind of architecture elephant two structure that we actually announced. But while exploring these things, like we have observed like many different candidates also popping up, you know, like there is like too many things. And then we tried to like really like figure out, for example, for, let's say, an NPU, a neural processing units that is like inside an AI PC powered by AMD or powered by Qualcomm or powered by Intel. Which of these variants would be like a better neural architecture that gives those hardware providers and silicon products a boost on the amount of efficiencies that they are unlocking? Or let's say, a speed of competition latency that they get, memory footprint that they get, while having a computational graph that doesn't sacrifice any format of quality. So this was like the whole thing.

• 50:38Speaker 01

  Removing all the human bias with a systematic approach to architectures, even our own biases, you know, and the only thing that actually remained in elephant two on CPU competition is this double gated. As I told you, we have nested competition in liquid neural networks originally. This nested format of competition was something that actually became very interesting to be part of these very simplified kind of neural networks that we built. And the fact that we have unstructured 1D convolutions as the layers of choice, 70 to 80% of our networks are structured by these gated convolutions that we have, you know, double gated convolutions that we have. They're extremely simplified and they replace attention. They reduce the computational complexity by a lot. They reduce the memory footprint by a lot.

• 51:22Speaker 01

  They maximize kind of the the speed of competition by a lot and at scale as well. And at the same time, on the quality, I mean, you have seen like some of these models are actually like extremely kind of competitive to the transformer based alternatives. I think you asked a bunch of other follow up questions as well, but I would pause here like for any follow ups here.

• 51:43Nathan Labenz?

  Yeah, let's get into use cases in a minute. I'm definitely interested in that. And I also want to talk about the future of hardware and what your work implies for the future of hardware. But I think it would probably be helpful for a lot of people, including myself, although this is something I've studied in some depth, I still would like to grok it better than I do. This concept of gating seem, I'll give you my rough and ready understanding and then you can improve it and deepen it. I especially noticed this with Mamba, when I went down that rabbit hole and became very excited about it. It seems that the kind of trick that is played over and over again with these gating mechanisms is

• 52:26Nathan Labenz?

  we want the transformation that is done on the data to be input dependent. So it's not enough to learn a transformation. We want to learn a transformation, but then also have some relatively, typically the gate is relatively low dimensional. Sometimes it could just be like a scaler that's applied to that transformation. It could be obviously more complicated than that. But it's some relatively simple mechanism that says for this learned transformation, here's how we're going to modify it, given the input currently under consideration. And that seems to be great. It seems to unlock a tremendous amount. Help me understand more, anything you think I'm missing there?

• 53:07Nathan Labenz?

  Or help me deepen my intuition for why that is such a powerful and recurring theme in all these different architectures?

• 53:15Speaker 01

  That's a liquid, that's a liquid structure. That's when I say like this input dependent kind of thing that is actually coming into Mamba. So right before Mamba actually got out, you know, like actually not right before, but a year and a half before Mamba come out, we released a paper called Liquid S4. You know, this paper was, if you just read the abstract of that paper, you know, like we are actually for the first time, we're introducing this idea of input dependent SSMs, basically input dependent SSM. The idea there was that let's bring the fundamental, let's say, building block that we found like has a lot to do with representation learning, with capacity of learning, this input dependence kind of element

• 54:00Speaker 01

  to really bring that format of gating to the neural architectures, to SSMs, you know, and then bringing that to convolutions as well with the other kind of systems that we are designing today, you know, liquid foundation models as well. That kind of format of gating, it actually adds a lot of, like you should just think about it. Like naturally it makes a lot of sense. If the neural networks, it's not about the forward pass of the neural network that it is adaptive, you know, like because when your input depend, you have, you know, like input dependence in the learning, from a learning theory perspective, all the magic happens in the backward pass, you know? If you think about when you're computing the gradients backwards, that input dependent kind of operator itself, it's going to represent itself.

• 54:45Speaker 01

  So when you're learning from that data that you're seeing, you're also learning some sort of dynamics, input dependent dynamics there. So that that that second axis that I was talking about is not just the number of parameters of the neural networks. It's also the dynamics that the neural network actually learns. You have some representation of that dynamics being learned in the backward pass. Now, the complexity of that gate is going to make a huge change. And also turns out like that gate is extremely important also like in language modeling, especially when you have sequence models like RNNs, like let's say classical control, like let's say continuous time RNNs, let's say discretized RNNs like LSTMs, and then you have liquid neural networks, like again, not very nonlinear versions of these

• 55:29Speaker 01

  linear versions of a recall model is an SSM. And when you add this type of gating, you can improve that the language capabilities as well. It seems like for, let's say, discrete sequences, this gating also helps a lot. You know, it's not just for the continuous time sequences, but also discrete sequence. Because I would consider language to be like a discrete sequence, right, of information. So you would have like that sequential dynamics that is kind of added by this gating. So it is a very right frame of mind to think about that kind of feature. And for us, like we think that this input dependence is one of the fundamental discoveries that we have done, like that we learned also in biology, it happens, you know?

• 56:11Speaker 01

  Like it comes out of physics, like if you just put the math together, you're going to realize like, oh, what is special about this dynamical system is the fact that it has this nonlinear input dependence. That's kind of a structure. And by biases, I mean the complexity of that gating. Even existence of that gating is a bias that you're adding to your system, you know? Now, is it needed or not at scale? That's a big question mark. Like you're going to see when neural networks add 100 trillion parameters, is matrix multiplication is enough like to really like perform like general purpose computer, like get to AGIs of the world or not? That's like a big question mark. But also, you need to also put this discussion about architecture and the obsession about architecture in perspective.

• 56:55Speaker 01

  Another perspective we have to put this thing is the learning theories themselves, you know? Right now, we are talking about different learning schemes that are coming. For example, you can train a model with next token prediction. You can train them in a word modeling kind of context, you know? You can train them in a sequential, in a long term horizon and long term history. There's so many ways you can construct also the objective function of the learning algorithm itself, you know, that that would also contribute to a lot to the learning process. Architecture, I'll tell you that we found the purpose of the architecture and the most important application of architecture or kind of research has been in efficiency, right? Really getting to efficient formats of competition without loss of quality. That's an extremely important thing because we're talking about the resource allocation problem right now.

• 57:40Speaker 01

  As the models are becoming bigger and the demand of AI is exponentially going higher, you want to have more efficient versions of the systems to actually run the systems at scale. Otherwise, like how can we actually provide access to AI to all, you know? Like and we are seeing like early instances of this thing that, you know, like the larger labs are wiping out the compute off of the planet. Why? Because they have to train and they have to then host these models for all of us, right? So efficiency becomes a fundamental property of these model architectures that we are doing the research on. I would say if you want to get into architectures that enable next generation of neural networks, sorry, next generation of intelligent systems in a way that they can become like human brain

• 58:20Speaker 01

  that actually performs, you know, like performs competition with 20 watts of power and it's already an AGI that does that between, that's like the massive exploration that goes beyond architecture. It goes into memory research. It goes into learning algorithms kind of feature. It goes into data, data feature, you know? It goes into prior research, you know? Like looking at it from a statistical learning perspective, that becomes kind of a and then learning theory itself, like the limitations of learning theory itself, it also get imposed on the today's learning systems. Because the definition of learning theory itself is actually broke like at scale. Like what we have right now, like everything is IID, you know? Like we are, we are building averaging machines, you know?

• 59:03Speaker 01

  You've seen the quality of writing and the sycophancy that actually comes out of AI systems because of the fact that, you know, we haven't gotten like this, maybe like to some extent, like multi agents has been becoming the solution, you know? And test time kind of scaling became kind of a solution for the caveats that you're seeing from just learning like systems with auto regressive kind of modeling as a pre-training kind of method. But I think there will be innovations needed, not just on architecture, but the whole thing as a whole, like you have data, you have data, you have models and you have learning algorithms. All together, they can design basically a future, that ultimate kind of holy grail, basically, in this space.

• 59:43Nathan Labenz?

  Yeah, that's very interesting. So basically, your perspective is that as we get into the recursive self improvement era, the real advances will come more from new learning paradigms, new objectives.

• 1:00:01Nathan Labenz?

  And then those advances will be made efficient through architectural optimization. But the architecture comes after the paradigm question of what exactly are we learning? And how are we giving a signal?

• 1:00:15Speaker 01

  It is one component of it. That's what I would say, like looking at it from a data problem, data representation from a, let's say, an axis would be architecture. And the other axis would be just learning algorithms themselves, right? And then we get into that recursive self improvement, which is kind of the continual learning kind of characteristics of what the new system, the fancy name for the, for the, for the, for the continual learning kind of research that has been happening like for, for more than probably four, four decades now. I mean, and, and, and, and that's, yeah. So, and, and that, that, that's how I would actually characterize the whole space, you know? So we cannot just look at architecture in isolation as like the fundamental thing that actually changes everything.

• 1:00:54Nathan Labenz?

  Right? Yeah. You remind me a little bit of Ali Beiruz's The Illusion of Architectures. If we have time, maybe we can touch on nested learning. But, okay, well, let's stay focused on your work for the moment at least. So in LFM, this is the result of this architecture search. And the kind of surprisingly simple thing that comes back is some reduced, but still critical number of attention layers. And then the other layers are, and this is, we've seen this kind of with SSM attention hybrids as well.

• 1:01:34Nathan Labenz?

  But I think the kind of surprising revelation from the result of this search process is the non-attention layers can actually be extremely simple as long as they have some gating. So you've got the gate and then you've got just a real simple convolution that I, I think I read only considers a very short span of tokens, right? Is it like just?

• 1:01:56Speaker 01

  They're short.

• 1:01:57Nathan Labenz?

  Back?

• 1:01:58Speaker 01

  Yeah.

• 1:01:59Nathan Labenz?

  And so if we were going to update the headline, attention is all you need from however many years ago now, we would maybe say, attention is something that you really do still need, at least at certain scale. But you also need gating on your other layers. But you don't actually need anything like super crazy fancy sophisticated, the state space model, all that kind of stuff. It turns out that keeping the gate is actually the part that really drives the most value. And then you can have a really simple mechanism behind the gate. And then you can have 70% of that and 30% of attention. And subject to obviously resource constraints, that ended up being the winning formula.

• 1:02:43Nathan Labenz?

  Am I getting anything wrong there?

• 1:02:46Speaker 01

  No, I mean, you're touching on the right things. And this is basically, it's like the regime you're operating in, the goal of your system. Are you trying to build super intelligent? Are you trying to build like the most powerful version of the AI system? You need the most unbiased version of an algorithm. Now, attention is an extremely rich, unbiased format of algorithms. Even if the computational complexity of attention is N to the power two, maybe we really need N to the power two to really get to that kind of level, you know? And maybe even we need like more complex architectures. We've always like tried to actually reduce the complexity of architectures for the sheer purpose of the fact that we are resource constrained. As humanity as a whole, we are resource constrained right now, you know?

• 1:03:29Speaker 01

  So I would say the discoveries that we have right now, it just shows that there is a gradient on architecture that you can follow as you scale models. The gradient that you're following is the fact that for smaller kind of models and specialized models, you can put as many biases, like these gating mechanisms that we are bringing in, you know? And as you can, you can play around with it, with as many operators of interest like in the, in, in, in, in your competition graph, you know? And it is going to work and it is going to give you some sort of a boost if you're really maximizing for like linear, um, let's say linear time complexity.

• 1:04:06Speaker 01

  Like you want to implement linear attention systems, you know, like just the fastest kind of, if the speed is like so important and you're actually wanting to even sacrifice a little bit quality, you can bring in like linearity and the whole system could be linear, you know? You don't even need some of those hybrids. What hybrid is just boost that accuracy because as we see there, as I said, the O N two, like basically the complexity that that computational complexity at a certain level, at a certain scale, it is needed for us to really like get to those performances that you want to. The larger your network becomes, the more unstructured you can make it. That's kind of the learning from that, that whole kind of algorithmic approach that we started designing in our own architectures.

• 1:04:48Nathan Labenz?

  Yeah. Very interesting. Okay, let's look at the other end of the spectrum then. As you work with customers, what are some interesting examples of when, given a resource constraints, given the narrowness of the domain of interest, other kinds of bias are actually winning in the architecture search process?

• 1:05:11Speaker 01

  Great question. So, for example, if you go to, let's say biology, you know, and you want to, you want to model sequential data, you know, in biology, you're talking about DNA data in biology. DNA data, they, from a vocabulary kind of perspective, they're very limited, right? They're not like language that is like maximum kind of amount of vocabulary and stuff, you know? In, in DNA language, like they're very simplified, but the lengths of those sequences you have to actually process are, let's say, for a human being, or maybe for a bacteria, I heard, it's, it's somewhere between one to 100 billion sequence, sequences, like basically elements in a sequence, you know? For that long context kind of things that you want to perform, when you do not have that much of a large vocabulary, you don't need attention.

• 1:05:54Speaker 01

  So you can, you can actually run on, on, on these kind of data, you can, you can run pure convolutions, pure SSMs, your pure liquid neural networks, like, you know, system, like recurrent neural networks and parallelized version of these reconnaissance and, you know, linear attention for extremely long context. You cannot do it any other way. The reason behind it is because context has become so large that that quadratic cost of attention just kicks in, you know? So let's say on biological data, you would want to have like some, some sort of a structure there. Then there are places like video modeling and stuff, you know, like in those kind of places, let's say you might want to have like different architectures and even different learning algorithms. You have seen like the success of diffusion, for example, there, right?

• 1:06:36Speaker 01

  Like you've seen like diffusion is also like a, I would say like this format of a prior that you put on a certain architecture, but it's still like you, there's debate between like diffusion and learning algorithms or diffusion is actually part of the architecture. You can actually make that connection and you can make that distinction as well. So you have, you have seen like on, on, on video and, and seen understanding probably an elements of diffusion would be needed, you know, like we, some people still believe that with autoregressive kind of modeling, you could actually get there anyways, but, but we will see, we'll see if, if that stays true.

• 1:07:06Speaker 01

  Then when we are talking about audio signal as well, like if you audio alone, for example, if you're talking about audio alone and language is not part of the whole thing, it's just, just pure like voices, like voice to voice, or let's say noise to clear kind of signal, you know, signal to signal. These are places where recurrent neural networks are still very, very powerful. They're extremely powerful, especially in the low data regime, you know? I would say models that have a lot more biases in them, you know, like in these smaller kind of regimes, when you do not have that much data, this is kind of the place where you can, where you can, where you can bring a lot of value, you know?

• 1:07:45Speaker 01

  So, and, and, and, and, and recurrent neural networks and biases in, in architecture can help you in the low data regime to really like fill out that gap with the feedback mechanisms that they have, you know? The more complex architectures, the, the, the, the more you would be able to handle, like the closer the architecture is to the dynamics of the data sets that you're, you're trying to solve, the close, the, the better of a learning system you're building. In fact, you were, you know, like, like in, in the physics modeling, you know, like when you're talking about, there was like a class of models, like physics informed neural networks, you know, like these are kind of, again, another architecture that we are talking about here. They would be really good for physical kind of simulations, you know?

• 1:08:26Speaker 01

  And, uh, yeah, so this is the other side of the spectrum that I'm talking about, like the property of the data, the, the lengths of the, the context lengths and, you know, like, like all the other considerations that I said, would change the architecture by a lot, you know? And then there, again, you want to scale these to the largest kind of regime, make it again unbiased. Again, a transformer would be able to at scale to beat this, but at a smaller scale, transformers would not be able to beat any of the, the other formats of the analytical systems that we describe.

• 1:08:55Nathan Labenz?

  And how about the differential equation inspired original liquid neural networks? It's clear that the worm is running on a very small amount of watts. Do we, in practice, do you encounter things that are so?

• 1:09:15Nathan Labenz?

  Resource constrained that you have to go to these extreme, extremely specific or extremely biased architectures to actually deploy today?

• 1:09:25Speaker 01

  Yeah, a hundred percent. Think about places where you have to bring in latency to my, you know, like competition has to happen like in microseconds. So you cannot really afford to have like larger computational complexity. You want to have the most simple type of system that actually handles that. And you know, like there you can have, you can have adaptive systems, like you can have like many different formats of systems. And there, this is like one place. The other thing is simulation in, in, in a whole. As I mentioned, like in physics, you have like physical data that is coming like you want to build a digital tween of a, let's say, you know, like a physical, a chemical reaction that happens at a, at a factory, right?

• 1:10:05Speaker 01

  So in those kind of places, you would go towards like an extremely biased and even maybe like just differential equation based models, like liquid neural networks today, like they're, they're getting applied to many, many different applications. I see that because the, the repository, the original repository from years ago is still like open source. You know, like people are still like building predictive kind of machine learning models on sequential data or physical data or sensor data that, that, that, that it makes sense. Because the systems are continuous time and it makes sense to have like a continuous time dynamical system to apply to, to modeling those kind of behavior, you know? And then you can, like today with this larger instance, like clouds of the world, you know, you could, you could also like direct your agents to actually try out a bunch of like in an auto research kind of format.

• 1:10:50Speaker 01

  You know, you can, you can just direct them to say, hey, you know what? Go pick like the neural networks, like the best neural networks that would be the best fit for these type of data set. And then here is kind of the space of possibilities you want to, you want to explore, you know, like based on the biases that you have. You know, like machine learning people are now mostly like orchestrating kind of these automated kind of agent pipelines as we're, as we're building. So that's what's happening. In terms of liquid ourselves, like we are, we are trying to, we are trying to contribute to the open source. We are open sourcing some of the instances that are coming out of our, let's say, search spaces, like let's say LFM two, LFM three is going to be like, again, you know, the objective is that the next generation should always beat the previous generation on that criteria that we care about, you know,

• 1:11:35Speaker 01

  without sacrificing quality, you know, like that's extremely important for us. And, and then the range of possible, like we, we decide, for example, how large of a neural network we want to make based on these neural networks that we have. And then this becomes kind of the open source version of our models. Then we work very closely also with silicon companies, you know, we work with AMDs of the world, Qualcomm's of the world. And what we do with them, we try to understand the silicon roadmap that they have, you know, the hardware roadmap that they have. And based on that roadmap, we try to, let's kind of inform also like what they have to do even in the next generation of the basics. You know, because when we have what this understanding of the algorithmic aspects of intelligence and the variety of things that it could be supporting

• 1:12:18Speaker 01

  in order to reduce, dramatically reduce the cost or satisfy the constraints of the use cases that they want to enable. This is, these are the considerations that you have to do. Or even building like to the point that we can build also like, you know, like specific foundation model graphs, like graphs for, for, for, for them, you know, like that's, and that's, that's kind of the projects that we do with semiconductor companies. In terms of like on the commercial kind of side of things, we also take models, we apply them in many different. I call the category that Liquid AI is very enabled, we call the device foundation models, you know? Anything, all the processors that are outside of data centers, we try to apply our technology to those kind of places. This is outside of data centers. Inside of data centers, we apply to constrained use cases.

• 1:13:04Speaker 01

  Use cases like low latency, like ultra low latency applications of AI, extremely long sequences and wanting very, very small memory footprint of your AI system. You, you want to have one cost efficient implementation of these foundation models that you have, you know? We contribute to memory, speed, latency, without sacrificing quality and cost, you know? So these are, these are kind of the elements that defines use cases where Liquid can, can come in. In terms of like access to our technology, you know, be any new generation of our architecture that comes out of our efforts of let's say architecture search or new modalities that we change, we, we check. For example, we Shopify, we are exploring many different kind of, you know, like range of applications, you know,

• 1:13:47Speaker 01

  across recommendations, search, product catalog, like understanding multimodal kind of systems very, very well. That's, that's something that we do. Our models are in production like we Shopify and right now, and, and they are, they're improving the quality of the click through rates and, and, and, and you the, the criteria, internal criteria that Shopify actually cares about. Our models outside of data centers, they go inside cars. You know, some of the use cases that we enable is like basically in car intelligence. For example, we power that recently we have like a contract design. It's actually a historical one with Mercedes Benz where we, where our models like is going to be powering kind of the audio and also the visual kind of elements inside the car. So whatever, whenever you want to talk to your car, basically the new voice would come out of a liquid foundation model, you know?

• 1:14:32Speaker 01

  And we control that with a model that is like giving you the quality of the best models that you have seen so far. What like, let's say like the audio models that you've seen like all around. But at the same time, it is 600 megabytes. It can actually fit inside like the smallest processor that is inside that car, you know? So like, and that, that, that's kind of, that's, that changes the game, you know, like because you're enabling local AI at scale, like on places that matters, you know, on cars, on, let's say on mobile phones, because you can do that. There's a lot of devices, like you're talking about billions of devices in the world that mobile, mobile devices can, can, can be extremely useful. And you've seen like that, that market itself, by the way, just the mobile kind of business is $500 billion market is absolutely insane.

• 1:15:15Speaker 01

  And it is as, as big as kind of the data center kind of market. So you can imagine like there is, there is a power here to be made, like, you know, the efficient markets, you know, and also constrained intelligence market. It's something that Liquid AI is going after. Laptop markets, you know, like the, let's say, wearables, robotics, manufacturing, IOT kind of systems. Anywhere that we have like a processor inside a system, Liquid can bring in intelligence on top of that. So the aspiration or goal that I have is that like we really build an intelligence layer on top of the diverse formats of hardware that is out there and, and processors that are available in the world. You know, that, that would be something that I would want to have.

• 1:15:54Nathan Labenz?

  I'm glad you said that stat. I had that noted to make a point of it's a point, it's worth repeating. The global annual smartphone market is about $500 billion for all of the hundreds of billions that are going into the data center build out. And it will get bigger than the smartphone market. But that's, it's only now getting to the scale and getting bigger than the, the annual smartphone market. So that is a, that there's a lot of dark compute out there from an intelligence perspective that is not being anywhere close to maximum. And that does not include the laptop market. So there's a trillion dollars, generally speaking, worth of compute going out into the world on an annual basis that are sitting in people's desks and pockets and it's, there's a massive substrate there to take advantage of.

• 1:16:40Speaker 01

  And we need to do it, you know, like we need to, we need to take advantage of it. I mean, because we don't have enough energy to host it. I feel like, like we are realizing it right now, like how difficult it is like to really power this format of intelligence that we have like at scale. We really have to have, we work smart. We cannot just do the simplest format or maybe more, more constrained use cases. Like, let's say, if you have like a one-shot, let's say predictive task or maybe data extraction task and stuff. You don't want to call the fanciest type of intelligence, like the metas level kind of model to perform data extraction for you. You can have like a variety of different intelligent systems that are doing many, many different things, you know? And then for, let's say, the most sophisticated problems in the world,

• 1:17:24Speaker 01

  you can go to the most sophisticated AI systems that are existing in the cloud, right? So that, that it's inevitable that the, the world, the, the processor world outside of data centers should get enabled like as we speak right now. Like, and, and I think there, there is actually some talks like coming out, there's a lag in Silicon Valley that even investors like that are actually like just, oh, okay, now it makes sense, you know? There is that lag that is getting gap, that is getting answered right now. And I think, I think we are in exciting times also for efficient AI in general.

• 1:17:55Nathan Labenz?

  Okay, so on the one challenge, obviously, with this like other trillion dollars worth of compute that's going out into the world is it's super heterogeneous, right? There's like many, many different devices, different chips, et cetera, et cetera. So how much of the work that you're doing is about trying to get closer to optimal use of that compute? Obviously, we talked about scans earlier and the transformers and the data centers. That has been not, I'm sure there's still more optimizations to be made, but there a ton has gone into it to make sure that you can get as close to max throughput as as possible, right? When you go then do an architecture search and you're targeting some random cellphone

• 1:18:40Nathan Labenz?

  or sensor in a factory somewhere, how much do you have to work on like kernels and scans as kind of foundation to even be able to do that search?

• 1:18:52Speaker 01

  Yeah, well, I mean, that's, that's one of the, you see, there's like different layers of abstraction, you know, like, and we're, we're, we're testing out like also like the, the AIs, see like how good they can get like on the kernel design kind of side of things, you know? For the things that they have seen, like, you know, in the open source, like the kernels that are available in the open source, like this, let's say the GPU structures that they have seen like out there, but not the NPUs, like the most hidden type of NPUs that, that their, their IP is actually not disclosed, those require like, let's say, another architecture search, because we don't have those knowledge. But I would say like kernel design is at a level that we can actually start automating it, you know, so you can, you can definitely have like loops where you can do, like you

• 1:19:34Speaker 01

  can juice out like the best kind of post hoc optimization, because some of the things that we were, like, whatever we talked about so far has been optimization that we do prefer design of the foundation model, like all sort of considerations before we start pre-training your foundation model, right? The approach that they could take before getting into, let's say, like pre-training a model, run this massive architecture search before getting a bit of hardware in the loop. But then right after you design the architecture, then there is like all these sort of like quantization of hardware training, changing the bits of the system, like going into the kernel level and try to even juice out even more off of the system, you know? All those post hoc stuff is also something that is like, you can automate, automatically

• 1:20:17Speaker 01

  kind of orchestrate it with a kernel engineer that does that and, and performs those optimization and we do that. We, uh, we do, we do those post hoc, we, we do that as a post hoc optimization step. That being said, we have the capability in-house to define architecture, define operators at a kernel level as well. So, for example, you could say matrix multiplication have like all this format of kernels, you know, like 100, 100 ways that you can actually structure matrix multiplication with caching, with not caching, like, you know, like how, distribution between CPU workload and then GPU workload, you know? How do you actually do the entire forward pass, you know, as you, as you perform?

• 1:20:58Speaker 01

  All of these optimizations could also in principle become part of that massive search space as well. And these are kind of the questions that some of the silicon partners are asking us, you know, how can we, how can we even before design of foundation model, figure out the competition graph first, like while having like the guarantees that when the training is finished, because that's a costly process, you know, multiple millions of dollars has to go into full training of a neural network, you know, even in a neural network is small, you want to have like all sort of like inference optimization before the post hoc, like quantization aware kind of optimizations, right? But yeah, in principle, we can definitely launch something like that as part of our search.

• 1:21:38Nathan Labenz?

  So what do you think hardware makers should be doing differently? What are you telling them to prioritize to help us realize this more efficient distributed AI future?

• 1:21:51Speaker 01

  I feel like even every iteration of a technology, software technology, like the level of abstraction that the hardware providers should be building for, it's coming, is coming up. For example, we've been talking about CUDA mode for a long time now, you know? There is no CUDA mode anymore. Why? Because if you look at the AMDs of, there are like agents at the kernel level, at that level, now they can automate that stack. The stack that foundation model companies should, sorry, hardware companies should start like working on is the intelligence layer, like what Nvidia is doing with their Nimatron project, you know? If you, if you see how, this, this project is actually quite successful and probably a couple of billion dollars actually went into design of this Nimatron projects.

• 1:22:34Speaker 01

  And, and if you really look at the, like, what does it do for Nvidia is just like basically building an intelligence layer on top of Nvidia's compute, so that you have like an easiest barrier of entry as an enterprise to buy Nvidia solutions, you know? So that's, the sales of this solution is always like around those things. So from the, from it, I'm just talking about the business of like enterprise business of these hardware providers, you know? If they want to sell more hardware, you know, and they, they, they're gonna up their stack from that kernel level optimizations, those are kind of post hoc things and they get into the intelligence layer. Now, does that mean like they have to become a foundation model company? To some extent, yes, and they have to be able to train that intelligence layer themselves.

• 1:23:17Speaker 01

  And Nvidia is like a success example there. How, how, how these things actually like, you know, paying dividend for them, you know, like building the Nimatron project? If you look at the other foundation model companies, like the other hardware companies, they have not done this yet. They have started like strictly like optimizing their models for the open source architectures that are out there. That's also extremely valuable, you have to do that, that's like a given. But again, if you want to create a differentiated value and you want to be successful, I would say you gotta be able to bring this stack to the intelligence layer. And that intelligence layer should naturally fit on top of your hardware. If your hardware have limitations, let's say in terms of like to your competition, software can always help to give you that juice.

• 1:24:00Speaker 01

  Like, for example, if you're talking about maximum token speed on an AI PC between an Intel computer, between, let's say, an AMD computer and let's say a Qualcomm computer, they might have like their own trade-offs. And the company that is going to have an edge is a company that actually owns that efficient intelligence layer on top of their hardware, so that they can say, at the end of the day, you're gonna run tokens on top of this system, you know? It doesn't matter what's the bandwidth of my, you know, CPU, if it's like a little bit lower than you or higher than you, or the amount of memory that I'm actually having in there is lower or higher, with the intelligence layer and software optimization kind of techniques, I can actually get myself to the place where I'm actually the winner here.

• 1:24:41Speaker 01

  And for example, all of my laptops or all of my PC solutions that I'm actually building as a hardware provider, they are naturally coming with an intelligence layer on top of that. Nvidia is entering this game like also heavily, you know, as a competitor in the CPU space now, you know, as you're seeing like devices, like Google actually entered with, with Android, with the Android ecosystem that they have, like they, they're replacing Chromebooks with Androids, right? Like with Android, like laptops, you know, like they're, they're, this is kind of what they're, what they're, what they're putting out there. I think it's called Aluminium OS or something like that, I don't actually know what, what exactly it is. And then Meta is gonna come with a sort of devices themselves as well. So all the processor builders, I have a feeling they need to get a little bit closer to that

• 1:25:25Speaker 01

  intelligence layer, you know, and try to leverage software and efficiency kind of considerations into account when they are going for their, for their, for the planning, you know, like the planning ahead, you know, where things are going. And why that intelligence layer is important? Because right now, agentic AI is something that everybody else wants. All that applications you want to do on silicon, you want to do that on a base of intelligence that is already provided to you, you know? You want to build harnesses, you want to build like all sort of, the harnesses are basically applications that you can build for solving a certain problem in, in, let's say, in a traditional sense of an application, right? And, and that, that, the market can get enabled on top of chips if you have a very nice stack, you know, that enables intelligence on top of what, what you have.

• 1:26:10Speaker 01

  The other thing that I would say is that like silicon is always, if you're going outside of data center, I mean, there is a, there's a massive diversity of silicon, but then the, the resources are constrained no matter what, you know, you don't have that much power, you know? So you gotta, you gotta be very careful like what, what sort of applications like are the most important uses of your hardware medium and platform? And then those use cases, if you already have an intelligence layer, you can identify when is the time to sunset basically a set of, let's say, devices. For example, I, I strongly believe, I think glasses could be like a format of competition for hardware later on, like devices, I think glasses would be like very, very interesting kind of medium, and I think somebody's going to get it right, you know, eventually, you know?

• 1:26:54Speaker 01

  And, and it might actually be a replacement for, let's say, for, for, for laptops, right? And then, but is it, you know, like, we need to, we need to really like figure out like what is it? But if you really want to define a market and define kind of where the field is going in from a hardware perspective, I would say you gotta up your software game by a lot and really get hands-on on, and investing a little bit more, more energy and, and, and capital into, into, into the R and D side of things on the foundation model side, you know? Because that's the new base of software, you know, like inevitably.

• 1:27:23Nathan Labenz?

  So does that imply a future where we have just a lot of vertical integration and a lot of coupling? The model will come with the hardware that I buy and it may not be so swappable in the future because the model is heavily optimized for the hardware and vice versa, such that these things are not so modular in the future as they are today?

• 1:27:45Speaker 01

  Or why do you want to change the model? You know, like it brings it to that choice kind of question. You know, you're given basically a default. Now you want to switch this thing by all means, but why do you want to switch it? You know, if this model that the intelligence layer that is in there, it's not fixed. It's kind of an adaptable kind of system. It is a, let's say, self improving system, like with a, with a call, like to your data sets and stuff. You can actually have like a platform that does, let's say, full fine tuning of that system. You know, it, it is enabled. And there is not just one model that you can load into your system. And there's not just one cloud model or one on device model that you're going to use. You've got to be able to orchestrate between, let's say, like many different instantiation of this model to be able to build application.

• 1:28:30Speaker 01

  We're talking about a model class that would actually like sit on top of, let's say, hardware on a laptop, for example. And, and, and yeah, so, so I would say, I would say you need to have that. But I mean, if the default is just giving you that efficiency and that, like it's, it's ready to go. That's like the, the choice, you know, like that's, that's something that I think Nvidia is trying to propose. Like in enterprises, when they go and sell, like the NIM project, N V and NIM projects, when they go out there and sell these things, I mean, it is, it makes a lot of sense because you, you already have like a project. You already have a multimodal model that is loaded on top of your, let's say, the PC that you want, you know, and, and, and why should I switch? Because they already have done all sort of optimizations for me, you know, and it is running like extremely fast. Why do I need to change it?

• 1:29:14Speaker 01

  And the model itself is tunable. I can use their Megatron kind of framework to actually tune the model. Now, if you don't want to do that, and you want to just choose like another model to host it. It, as I said, this has to be given, you know, you have, your hardware should be already, you have to be optimizing for the entire open source ecosystem and you know, like models that are available. But at the same time, it would give you an advantage, like, you know, like to yourself and to your customers.

• 1:29:40Nathan Labenz?

  Okay, maybe in the last few minutes, how about a practical application of this? You have a blog post on local cowork. No cloud, no waiting, tool calling agents on consumer hardware with L F M two twenty four B A two B. So that's a, I think people know that's a mixture of experts, two billion active out of twenty four billion total parameters. Let's say I want to make that a part of my life. I am interested in how you would coach me on setting this up. Today I have for reference, and I, by the way, try to make the, sometimes I try to make the transcript of the podcast, something that I can feed to my agent. So you can think of this as partly coaching me, partly coaching my agent.

• 1:30:23Nathan Labenz?

  So I've got this deep context database that is the last five years of my digital output. This podcast will be recorded, obviously recorded, transcribed. It'll go into this database. So everything you said, everything I said will be in there and searchable. And it's got all my email and Slack messages and everything. Okay, cool. So now I've got Claude on my desktop that can call tools locally to get data back. But then it sends all the results to the cloud to decide which of those results are actually the right ones to be looking at. And so far I've been okay with that. The benefits are certainly worth whatever risk I'm taking, I feel. But I would maybe love to run that data through a local model first so that I don't have to send all my data to the cloud every time.

• 1:31:08Nathan Labenz?

  Now notably, what gets filtered through, I'm probably still going to end up running through a foundation model. So it's not going to entirely skip the cloud. But also I'd like to save some tokens too, because I'm going to be, I'm going to use Fable whenever I get it back for whatever it's most appropriately used for. So how do I get really good performance on my local computer with this model? Do I need to be doing fine tuning? Do I need a distillation strategy? How do I actually take the base model that you have and recover as much of, let's say, Opus or even Fable performance in terms of searching through, understanding my data as I possibly can? And how much is that?

• 1:31:51Nathan Labenz?

  What should I expect in terms of how far I'll be able to push that process? How close to parity with frontier models can I get? So tell me everything I need to know and I'll go have my agent do it.

• 1:32:03Speaker 01

  Yeah, well, I mean, that's a, that's a, that's a great question. Obviously, like the, the local coworkers, like as it stands today, like it's, it's obviously just to open minds like, hey, you know what, like this type of applications you can enable. The class of, the class of like what this format of local agents are going to enable is like basically a local computer, you know, like you have an orchestrator. If something is so complicated, it should be able to send it to the cloud, fetch the answers for you if it's not necessity. We have like, let's say, smaller models that are P I I models, like, you know, it should be able to use the P I I models to really like filter out all the personally identified information, send it to the cloud for you. You don't, you don't need to even see these models. I mean, these models should be like on the background.

• 1:32:46Speaker 01

  The model that you have to tune is that orchestrator that be able to route between like many different services or even smaller specialized models that are doing stuff and also some of the cloud models that are out there, right? That router is the computer, you know, like that's like the local computer. That's like a definition of, like, when you open your laptop, it should be just that, you know, like you literally have like just that. And then you start like working with all the services you want to have. It's just the same way that you communicate with your, let's say, assistant, you know, like it should be like in the format of an assistant that does all sort of those jobs, you know? Like it's, it takes, it takes a while for people to get the, get not weirded out by the user interface, like just being that

• 1:33:27Speaker 01

  and not seeing like all the like file formats and you know what, what, what, what you have to do. But you're going to get used to it, you know, like now cloud code is like even as for the I D. Now, is, is the twenty four B off the shelf is going to be there? Like on the quality that does all of those stuff for you? No, it's not. None of the local models today are there. You have to, on the local models today, you've got to fine tune them. You've got to, you've got to, you've got to, you've got to get them to be specialized for the stuff that you want to do with the proper explanations for your Claude agent, you know, like you, you basically like give it these things. But is Claude being able to go out there and actually build this model for you? Like, let's say, like do the fine tuning and, and, and getting it to that place?

• 1:34:08Speaker 01

  No, because today, like even Fable level kind of models, they would not be able like on the, first of all, like you wouldn't have access to that because Anthropic is actually like having access to the, like, you know, like auto tune and stuff like automated kind of models, models generating models kind of platform. There are companies and also ourselves, we are building platforms that enables you to do fine tuning of this whole thing that would cost you between tens of dollars to, let's say, like low thousands of dollars to actually get you to the cloud quality for your, and beat all the checks, production quality kind of model. And it's not going to cost you like tens of thousands of dollars. It's going to be like between tens of dollars to actually like those thousands of dollars, you know?

• 1:34:52Speaker 01

  And that's kind of the, the, the scale that we are thinking about because again, efficiency here matters a lot because we don't want, we don't, and we want, we want this fine tuning to happen on the compute that you already have. Or if you don't have it, you would actually put it like in a secure and proper kind of data center that you would, you would, you would actually like host it like from the, from the providers, basically. And you would, you would be able to get to that quality of the models. There are platforms that hopefully like as we go forward in the next few months, we are going to announce like some platforms that would, you can hook it just directly in your channel, you don't have to do anything, you just say, hey, you know what? Go call this platform, you know? Now for fine tuning this thing, go call this platform. This would give you a production grade foundation model that you can deploy it now for yourself.

• 1:35:36Speaker 01

  Either fine tune on that data or depending on the use case, depending on what you want to do. It doesn't even have to see that data, you know? It can even synthetically generate data on its own and actually train the model to be like just the perfect, reliable tool caller and understands when is the shortcomings of it and it can actually go away and deliver it to some other places. Yeah, so I would say you're very soon close to get to that.

• 1:35:58Nathan Labenz?

  I'll wait for your platform to be ready. I don't have to D I Y is the, is the message.

• 1:36:03Speaker 01

  Yes.

• 1:36:04Nathan Labenz?

  This has been brilliant. Maybe one last question and then I'll just kind of give you the floor to close however you'd like. How far do you think this goes in terms of kind of miniaturization of intelligence, if you will? One intuition would be like the biological world that we see is maybe on some sort of Pareto frontier already. And so the watts that go into our brain is maybe like getting us close to the maximum that we could get for that amount of power. But maybe you have a different intuition. What would you expect in terms of?

• 1:36:41Nathan Labenz?

  Upper limits of intelligence that I could have on my phone or on my laptop as we really get to the kind of physical limits of the technology.

• 1:36:52Speaker 01

  You know, when you think about intelligence, like the way I locate it right now is that like transformer-based networks and also like our type of, let's say, the current architectures, the landscape of architectures that are available, they gave us with scale, they gave us in-context learning capability. The thing that actually emerged from next token, you see, the word is important, emerged from next token prediction. Intelligence for me is an emergent property. If you want to get into, like, if you want to miniaturize kind of intelligence, bring it like to the, to the physical world, I don't believe with the current set of algorithms, like, we would be able to get close to the, let's say, intelligence per what a human

• 1:37:32Speaker 01

  brain is actually like providing, right? You're not going to get there. Why? Because I believe human brain over, we also have to like consider the amount of energy that went into design of humans, you know, as a whole of biological evolution is actually a very, very long kind of process. A lot of the pre-training process that people say, oh, a lot of, you know, like, you're going to see the entire world, humans do not need to see the entire internet to be able to reason about something, you know? No. But humans have gone through like years of evolution, you know, so I would actually attribute it a lot to the evolutionary kind of aspect of things. But again, what I would say, human intelligence came with multiple mechanisms of in-context learning, you know? You just

• 1:38:13Speaker 01

  don't have, when our current AI systems do in-context learning, they learned a vague representation of one algorithm, which is least square. You know, it's basically least the square, you know? So what they figured out is basically gradient descent in a mushy way, you know, like for some use cases, with some examples, you would be able to make the system understand and give you like kind of the next example. These are kind of beautiful properties. This is an intelligence, this is what I would call emergent property of the systems. You set the algorithm to be next token prediction, you got a vague version of a gradient descent in context learning capabilities. For humans, you just, you don't

• 1:38:56Speaker 01

  have only emergent gradient, you can learn by examples, but you can also do reinforcement learning. You can run simulations in your head. You can do all sort of algorithm, you can do Bayesian statistics in your brain, you know? You can, you see what I'm saying? So you have like a diverse set of algorithms emerged from the way that humans actually got designed and got intelligence. For me, I believe if you start optimizing for those, if you try to force your way into the system to become a reinforcement learning, not learner, but learning from trajectories, that's not the right way to actually get to emergent property of intelligence. I believe the format of intelligence that is going to be miniaturized,

• 1:39:37Speaker 01

  you know, like being like in the smallest amount of time, maximum amount of intelligence, like the human brain, we need to come up with these identifiers, like these confounding kind of algorithms, which is next token prediction is one of those. What else do we have to do at the beginning of the design of these systems so that reinforcement learning, like curiosity-driven, like intelligence, drives with limited amount of energy from that final system that actually comes out of it? So I would say that emergent property is something that we need, there's a lot of research that has to go in that direction, I would say. And multiple kind of variations of way of learning, that would like enable the next generation of artificial intelligence, I think.

• 1:40:19Nathan Labenz?

  That could be a great note to end on, but let me just give you one more opportunity. Anything else that I didn't touch on or anything else you just would want to leave people with before we break?

• 1:40:29Speaker 01

  No, I think, I think we covered, we covered a good lot of information, I think. I think we went into some things that we, I don't, I usually talk about, you know, now in my, in my current role as the CEO of the company, you know, I get to talk a lot more about like the business opportunities and the market aspects of the intelligence and stuff. But it's still like, you know, at the core, like we are scientists and we are really like pushing the boundaries, like we are actively thinking about it. Day to day, you cannot imagine, like, even right now, if you're talking, like, I'm already in the backend, like, I'm actually like training some stuff. And I really don't want to get away from, from that kind of mentality, because I think everybody can actually be able to now build, you know, like, and I think that the times couldn't be better.

• 1:41:15Speaker 01

  The agents are pretty amazing, you know, at giving you like this opportunity of like really doing frontier, frontiered form research, you know, if you're curious about something, it's just that we need to find people that are a lot more curious, you know, and improve the curiosity in people as well, to really get, instead of just, you know, like fear of like, you know, everything. So I'm a techno optimist, but probably you've, you've observed that, like, I usually like talk about technology, really in a very, very positive light. And I want people, like, one, one thing that I always say is that we need to really get to the level where people are as curious as the scientists, where, like, from, they want, like, what gets us like into, into science, and the purpose of science has always

• 1:41:56Speaker 01

  been satisfying our own curiosity, you know, and also understanding the world around us, right? So that's kind of the purpose of science. And I feel like that, with the current superpowers that are given to us, like with these AI systems, I think everybody could actually start, like, really contributing to understanding the world around us, and also generating more value, and also just satisfying our own curiosity. But also, like, it requires like a certain degree of rebasing our biases towards like, oh, what is work, what is not, and what is automation, what are, what is an army of agents working for me, you know, what's that definition even look like? And then a change in culture and change in like way of thinking about this thing, not just for individuals, but also for enterprises

• 1:42:38Speaker 01

  we are dealing with as well, like as Liquid AI. And yeah, so I think that change is something that I'm looking forward to, and I think this is something that would happen. The pace of it is extremely fast, but I think we got to do it one way or another.

• 1:42:50Nathan Labenz?

  I love that. I consider myself extremely fortunate to live, to a really remarkable degree, a curiosity-driven life these days. And that's one candidate for a positive vision for the future that I think should inspire a lot of people, especially because, as you note, there's an opportunity to move into that phase already with the AI systems that we have. This has been excellent. I really enjoyed it. Thank you so much for the time. Ramin Asani, co-founder and CEO of Liquid AI. Thank you for being part of the cognitive revolution.

• 1:43:21Speaker 01

  Thank you.

• 1:44:49Speaker 02

  Just enough to make the whole horizon sing.

• 1:46:25Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement

• 1:47:06Nathan Labenz?

  at ai podcast dot I N G. And thank you to everyone who listens for being part of the cognitive revolution.