⚡️ A Worm With 302 Neurons Inspired Their Architecture — Ramin Hasani, Liquid AI

Watch the full show on YouTube: https://www.youtube.com/watch?v=V_9TCu_21SE

• 0:03Speaker 00

  Okay, we're in a remote studio with our new head of editorial, Richard. Say hi. Hi, everyone. Dialing in from the UK, and Ramin, CEO of Liquid. Welcome.

• 0:14Speaker 01

  Thanks for having me.

• 0:16Speaker 00

  Most people, when they've heard about Liquid, they know about LFM, so they've heard of LFM. They maybe haven't personally tried it. The other interesting, I guess, public data point is that it is very inspired by worm DNA or something. I never really pay much attention to that because obviously I think that's an analogy, but you can tell us how seriously to take it.

• 0:36Speaker 01

  Yeah, absolutely. Well, you know, like when we started our kind of research about 12 years ago, we wanted to see if we can bring inspirations from brains back to machine learning. So that's where everything got started. And then when we started looking into human brain, we realized like, when I say we, my current CTO and myself, we started back in Vienna University of Technology. It's called TUV in. And then, so I started my PhD, my co-founder was the master student there. And then what we wanted to do, we wanted to understand how we can bring in like continuous time math inside machine learning kind of system. And that's what we did. So we basically look into neurons, how neurons exchanging formation with each other.

• 1:18Speaker 01

  And then this was governed by very simplistic first order differential equations. And that's kind of how the nervous system dynamics are actually working at the level of nerve synapses. And then our first focus, we wanted to start with human brain, but the human brain has like a spiking neurons and it's very difficult to actually learn, like we still don't know anything about human brain, how competition actually happens. We know a lot more now about a fruit fly's brain, but we don't know that much about human brain. But I can tell you that we got to this animal because we wanted to do it first principles. We got to this animal's sea elegance worm, which has only 302 nerve cells in its body

• 2:01Speaker 01

  and it's like two, it's two millimeter long. And one of the nice thing about the worm was the fact that its neurons behave very similar to our vision neurons. They don't spike, you know, so that graded kind of behavior in these neurons of this animal, it's called in neuroscience, they call it electrotonic behavior of a neuron, you know, like the neurons do not spike. And that behavior was very, very interesting for us because if you build systems inspired by the brain of this worm, you would be able to build like very continuous time kind of systems and they're differentiable. So you can do backprop and all sort of other things on top of these things. So this was, this was the model organism that we were looking at. At that time, the most fascinating thing for us was that with 302 neurons,

• 2:46Speaker 01

  it could control 95 muscle cells better than any robotic systems that we had on the planet, you know. And this is 2016 and 2017. So then we thought that, okay, so, and also another thing, another interesting thing about the worm is the fact that 600 million years ago, we got a split, humans actually got split from this worm in the tree of evolution, you know. So it's actually like whatever, whatever neuroscientists are discovering about the brain of this worm or the cell structure of this worm is actually transferable like to humans as well to some extent. 78% similarity to human genome, you know, so it is actually a very, very important worm for us. And so far it has won four Nobel prizes for humanity, you know. So you can imagine like scientists are really lucky to study this animal.

• 3:29Speaker 01

  So it's a very, very popular animal, you know, and, and that's why like that's our research. We started even working with neuroscientists, you know, themselves that are working on this brain of these neurons. So, long story short, we mathematically modeled how two neurons exchanging formation based on the inspirations that we know and the things that we know about the brain of the worm. And then we got those mathematics and we built something called liquid neural networks, liquid time constant neural networks. Liquid was like basically liquid for flexibility because one of the things that we realized is that these are highly nonlinear systems governed by differential equations in a forward pass and they're differentiable, so you can actually run them backward pass and you can basically learn them like really nicely.

• 4:12Speaker 01

  Because of the multiple feedback structures that they had, they are really representation learners, you know, they learn representation from data really, really well. One of the things that we realized is that with a small kind of subset of these neural networks, you can control robots, like you know, very similar to how the worm itself was doing it. And when I say small subset, we're talking about tens to hundreds of neurons, you know, like we're talking about very, very small subset of neurons. And then we got very excited, so we started applying these things to controlling robots. On the robot, you don't have that much compute, so you bring like intelligence, the fact that you can actually pack that much intelligence to control autonomously a kind of behavior inside a robot, that was like a very attractive feature for us.

• 4:55Speaker 01

  So efficiency of this competition and bringing intelligence into like smallest kind of unit of compute, that has always became kind of a, a kind of research topic for us. And then 2017, Professor Daniela Ruus of MIT C cell, so she's, she's my, so she reached out to my professor in Vienna and then she's one of my co-founders as well. She's the mother of robotics, arguably, you know, she's like, she has done tremendous contribution to the field of robotics and distributed robots and soft robots. As a whole, she started learning about our research and then she was like blown away and she said, "Okay, you guys, would you guys consider coming to MIT and continue this research in real world, like really applying this technology into, let's say, cars and drones and many, many different kind of vehicles?" And we were like, both Matthias and I, we joined MIT.

• 5:40Speaker 01

  So since 2017, I've been like, working with Daniela, I met my fourth co-founder, Alexander Amini. So Alexander was also was a PhD student at that time at MIT. So Matthias and I were both there and the four of us since 2017 started working on these topics of, again, continuous time dynamical systems and continuous time learning systems. And then the scaling of all of these things, applying them into many, many different domains, predictive like sequence models, like sequence modeling became kind of our thing, you know, like we started like doing contributions. And then, you know, like in this field of continuous time, it became a little bit better because a lot of other researchers from different places, like from Yashua Bengio's lab, David Dubinand lab, like from, from University of Toronto.

• 6:25Speaker 01

  And then there was some people from Stanford. Everybody was contributing to this field of continuous time dynamical systems, you know. And this was like, imagine recurrent neural networks that are discretized kind of dynamical systems, you know, that are basically computing one step at a time computation, but the time difference between one step to the other step, it doesn't kind of change that much. But this, this type of continuous time dynamical systems allows you to have like a little bit of control also over the, when, when events are happening in a, in a discreet way, you also account for the time between the events that are happening. So sequences that are continuously evolving, you have like a lot more adaptability in the time dimension as well.

• 7:09Speaker 01

  You see, so that kind of aspect of recurrence and feedback kind of calculation, this was kind of the intuitions that we brought to machine learning. Again, not only us from the MIT side, but also like a lot of other kind of people were simultaneously working on this with the topics such as neural ODEs, state-space models, you know, like you can think about like all these other names that people started like putting on top of like these dynamical systems as a whole, you know. So, and then, what we tried to do then, then, then it was time in around 2020, everybody was talking about scalability, you know. At the time, everybody, like from the MIT side, we were, we were competing with the, with the OpenAI's of the world and Googles of the world, like from a research point of view, like we were putting algorithms that are efficient

• 7:53Speaker 01

  and then perform really, really well in, in new learning scenarios in the robotics environments. Because if you remember, between the 2017 and 2020, OpenAI was very active in reinforcement learning and also like doing a lot of robotics, like OpenAI Gym, like for example, like some of the things that they were putting out. So we were trying to like get creative around like controlling of robots with this type of continuous time algorithms and showcase how you can actually take learning to out of distribution, you know, like how, how can we have learning systems that are small, but at the same time they can take, they can do a lot better in out of distribution generalization, you know. So that was like the whole thesis that we were. But then we started thinking about a scale because everybody started thinking about a scale as well.

• 8:34Speaker 01

  Because we want to, we wanted to evolve from this worm-inspired kind of intelligence like to something that is like more like complex and get into human understandable signals, you know, like with audio, vision and text being like a corpus of data that we cared about. So as soon as we start scaling continuous time and recurrent kind of highly nonlinear loops, as soon as you start scaling them, you realize like how difficult it is to perform computation, you know, like it becomes really, really computationally not tractable. So it becomes very difficult to compute, so you need to get creative around how to scale this kind of systems. Recurrent neural networks have been like a, their major blocker has been sequential computation, you know.

• 9:17Speaker 01

  So you cannot like parallelizing recurrent operators when they are nonlinear, it's really difficult, especially on GPUs. This is one of the beauties of transformer architecture because they're unbiased, they are basically matrix multiplication and you can actually like scale them like it's the hardware lottery. That's the beauty of the transformer architecture for scale like and for unlocking like general purpose behavior. But the recurrence itself is like so attractive because the computation doesn't explode. Because if you think about like attention, when, when you, when you consume more and more data, you are exponentially kind of the quadratic cost of attention is something that people have paid attention to. So we've always realized that recurrence is actually bringing this computation lower and they make it sub-quadratic or linear, you know, closer to linear kind of computation.

• 10:02Speaker 01

  As you consume more data, so these algorithms are more efficient. But this is only theoretical, so if you can actually, if you cannot really scale and do the operations in parallel, you cannot scale these systems, you know. So one of the things that we tried was that one, one way to solve for this kind of nonlinear systems to become linear is to linearize them. When I say linear, dynamically, you reduce all the nonlinearities of the system. So you can apply linear algebra tricks on the system. So then the system actually becomes computable. You can perform linear algebraic operations in parallel. You know, so that's, that's something that, for example, state space models are linear version of recurrent neural networks.

• 10:47Speaker 01

  And they lost kind of that non-linearity, nonlinear relationship between the parameters. The reason behind why do you want to do that? Because you want to make a scalable computation. So SSMs are becoming very popular version of a dynamical system or a learning algorithm. Because you can now parallelize some other form of operations as well. You know, like, not just attention, but you can have like parallel scan. And you can apply linear algebra tricks, as I said, to scale another format of operations like very much. So linearizing this dynamical system allowed us to scale, right? So if you think about the SSM view, but the problem again, what arises is that like there is no free launch in computer science, right?

• 11:28Speaker 01

  So when you linearize a complicated dynamics, you're losing expressivity. So the system is not going to be as expressive as its nonlinear form. Then we realized like when we, when we apply, let's say, these state space models like to solving language problems. They are having like troubles to learning like dependencies at long contexts. They can compute really efficiently, but they cannot express very efficiently. So the expressive power of these neural networks always falls short of transformers. So with all the nonlinear kind of mathematics that we learned from liquid neural networks. We learned from dynamical systems, we learned from SSMs, we learned from convolution neural networks.

• 12:08Speaker 01

  You know, there's many different variations of operators that you can bring in together to build a general purpose computer. With everything that we learned, we thought that, okay, so you know what? Let's not put ourselves into a biased way to go and say, all right, so our architecture is like the holy grail. And everybody has to start like building an alternative architecture and then put a bet on a single architecture. What we thought day one at Liquid, this is like about three years ago. What we did at Liquid, we, we started bringing Avengers of alternative architectures like all in one place. You know, like we had, like, as I told you, like people from Yashua Benjus Lab, from Stanford, from MIT. Like they're all inventors of various types of technology.

• 12:47Speaker 01

  There was hyena hierarchy, there was a state space models like Jimmy Smith of Stanford and ourselves like on liquid neural networks. So we brought all of these teams into one place and we said, let's, let's build an unbiased way. Let's build a meta AI system that searches through the operation space. So we build a meta AI system that searches through operators of interest, given the deployment environment. Imagine you want to deploy these solutions on a GPU or NPU. So we applied basically this massive search algorithm on top of building hybrid architectures. That allows you to not, allowed you to optimize four things. Number one, no sacrifice on quality.

• 13:29Speaker 01

  So there are a bunch of quality metrics like reasoning capability, language, understanding, you know, knowledge. There's so many different kind of things that we want the model to do. Like there are a hundred different criteria. We thought that no sacrificing quality against the pure transformer model. Minimizing the memory consumption, minimizing latency and maximizing the speed of computation. These are like the four criteria that we optimize a search based algorithm. The algorithm, like, there's a paper that we published on the, just the science of this thing is called the STAR. You know, it's called the synthetic architecture of, synthetic tailored architecture, like synthetic design of tailored hybrid architectures. So I think if you, if you look at this, just say STAR Liquid AI, you can, you can actually find it.

• 14:12Speaker 01

  So I think that's kind of the science behind like how systematically we are designing kind of architecture. So that framework, that meta AI algorithm is kind of a recursive self improving kind of system. If you think about it like three and a half years ago, we applied it, we applied it to really have like a search algorithm to go through various hybrid architectures for a given, given chip. And then this system became kind of the computational graph of different generations of liquid foundation models. Liquid foundation model version number one was the first instance that came out of this architectural search. It had some elements of convolution, some elements of attention, some elements of liquid, kind of original liquid neural.

• 14:56Speaker 01

  It has like a, it was a mess of everything, you know, so, and that was like the first generation. The second generation, we optimized it specifically for running really, really well on CPUs. The LFM two instances of the models, they are, if you look at them, they have a very elegant and simple architecture. So the architecture is basically eighty percent double gated convolutions, you know, 1D convolutions. Plus some group query attention, basically twenty percent of work. So the architecture is just a very simplified, very fast and, and high quality kind of foundation model. That you can run them on a CPU really, really well. Now, the beauty of this, you know, architecture search is that our, now imagine a new innovation

• 15:40Speaker 01

  or a new invention in the architecture space or fundamental building blocks of intelligence happens. We add them to this massive meta AI search. So every generation of liquid foundation model that we put out, it might have, it would definitely have like an improvement over the previous version. But the operators are from operators of that we choose. And, and the target device, the target device implies basically what should be that hybrid architecture. So that's how we started designing this liquid foundation models. So what you see today, we are at the LFM two point five instances of the models. We are a foundation model lab, we are building everything from scratch ourselves, as you know, like we train, pre-train these models. We run the scaling laws of these models.

• 16:21Speaker 01

  We scaled these models from tens of millions of parameters now to seventy billion parameters. Like that's kind of the range that we have usually like running like scaling laws on top of models. And then we so far released the smaller instances of the LFM two and LFM two point five instances. And the next generation would be LFM three, which is like a little bit, a slightly modified kind of version of these things. So that we can have like added more quality, added more stability, added more kind of better latency control on set of deployment frameworks that we care about.

• 16:52Speaker 00

  You just gave us a PhD, you know, the abstract of what, what Liquid has. Okay, I have so much to dive into, you know, you know, neuroarchitecture search, you know, gradient free evolution, all these things. But just for the people also listening on the business side, let's also give an update, you know, I, I, so. You know, the last time we covered Liquid on the podcast was with Mikhail from Shopify, who's a big fan of you guys. I'm sure there's a lot more, so I think only you would know and you would know. And, and you sort of bring, bring people up to speed on, this is not just a research project, right? Like this is actually like, you're not like, people are actually effectively using this.

• 17:33Speaker 00

  The benchmarks are not just, you know, goal seeked, like they're, they're actually reflective of the world.

• 17:40Speaker 01

  Absolutely, absolutely. So, yeah, so I have to say like Mikhail is one of my mentors now, like Mikhail is like one of the biggest people like in the industry. He's like one of those no BS kind of type of people. I really like, like how he gives his advice very, very directly, you know? And then, so one of the, he told us is this. You guys, you know, like we have like a very competent team at Shopify and we're building models, right? So if you're, if you're telling me like you have, you have like models, I have to, I have to see.

• 18:08Speaker 00

  Yeah, he's a mercenary. He just cares about the input output.

• 18:11Speaker 01

  That's it. You, you bring in the technology, we will, we will test it and we will tell you, is it good or bad? Then they tested it first time and they were positively surprised like of the quality and also like the efficiencies of our models, you know? So I think in the space of alternative architectures, like we have really, really good models. So back, back to the business side of things. When we talk about this range of models that we are operating and I told you we are optimizing for CPUs too. We are looking into outside, bringing intelligence outside of data centers. We started working with enterprises that are caring about data that goes, the deployments that are outside of data centers as well. So we started working with car manufacturers, with robot manufacturers, with like, let's say laptop producers.

• 18:56Speaker 01

  We work with mobile companies. So they have like deployments of compute somewhere outside of data centers. So we go in there and our foundation models are really good and really like you can bring like the best of intelligence on top of like these devices for various types of applications. Examples of partners of ours would be like AMD on the AI PC side. Three hundred million AI PCs are getting generated on, on the planet like every year. Like these are laptops, you know, three hundred million, three hundred to four hundred million laptops. And I think from there, about hundred million of it could be AMD kind of powered. Basically like the core of the building blocks is AMD. So we are working towards bringing our liquid foundation models directly running on the laptop.

• 19:39Speaker 01

  Enabling some sort of local deployment of intelligence, you know? And they can solve multiple different problems. This could be for privacy filters, you know, they can be in a smart way, guardrail, like what wants to get out of your computer. Imagine your, your type, you, you want to send something to the chat GPT terminal or to your Claude code kind of agent. Those kind of bulk of sensitive data that you want to send, you can pass them first through a liquid foundation model. That, you know, like, like, you know, basically shields out like sensitive information in a smart way. And then you can share those information. That's like one thing. The other thing that you can do is you can have the small models to be a proactive agents directly running on your, on your device. So imagine if you have like an

• 20:22Speaker 01

  Agent that is having access to all sort of information that is right now on your laptop and then performing some sort of analysis and gives you that analysis to yourself. Like everything that you can achieve in the cloud, you you want to bring it like as an experience also directly running on a on a laptop. And that that would give you like also an offline capabilities, also like air gapped capabilities. So you can imagine when you're going into manufacture. Now let's go from the laptop to mobile phones. On mobile phones, like we are working with the giants in the world. So on that one, like what we are what we are trying to do, we are trying to power the on-device portion of intelligence. Today, when you think about device intelligence, there is a hybrid solutions out there. There is one model that is actually in the cloud and there are models that are running on the device, right? So and there is always like a collaboration of these two.

• 21:07Speaker 01

  Usually, right now, if you think about like the most used phones, like from Apple Intelligence, from Galaxy AIs, like, you know, like of all of these things, ninety percent of the calls are going to the cloud, right? So these are ninety percent goes to the cloud, ten percent on the device. The reason is because the device models are not powerful enough or reliable enough to really to really take more off of those cloud kind of use cases. Why do companies want to run these AIs on the device? The first reason of this thing would be cost. It's just sheer cost. If you have four hundred million phones or let's say billions of clients that you have to support and give them a free intelligence access and while you have to serve that intelligence somewhere in the cloud,

• 21:51Speaker 01

  that's going to become like really, really prohibitive kind of cost, right? So what we do, we we try to like really, and these companies are really trying really hard to bring the AI directly workload of AI directly on the device. Because in that way, like you're just creating margins for yourself, right? So that's like that's like the first and simplest kind of solution. And then apart from that, you would have active to, you know, as I mentioned, offline features like privacy sensitive kind of topics. And then many, many different advantages that you can bring like on the device. But it is very difficult. Building a model for the device and getting it to production quality is extremely difficult. I can tell you like it's not like just downloading a model from open source and then trying to put it on the device and then now you're successful. It goes a lot more than that. You have to do thermal control.

• 22:36Speaker 01

  The behavior of a model in the long context is actually changes. You know, like you really have to have like degree of reliability to add that. It's really difficult to actually get to that point, you know. So and and we have gone through like over the last two and a half years, like we've gone through like many cycles of production. And I can tell you like it is it is really difficult. Production grade AI is like way more than just benchmarking the some quality benchmarks on the device. You know, it goes beyond that. And then another example of these things would be car manufacturers. Like Mercedes Benz is one of our clients. What we do in the car kind of space, we bring in-car intelligence. So imagine we announce a partnership like last April that it actually happened really fast in the space of automotive.

• 23:18Speaker 01

  It's a significant kind of partnership that we thought that inside the car intelligence. So I want to give you like some some insights. So imagine you're sitting in your in your car, you can talk to your car. You can have like all sort of hey Mercedes features, you know, like you are communicating with your car and stuff. Those are all happening with a foundation model that goes directly inside the car. And this system, like the the the model that we are power, it's a multimodal solution that can get connected to the to the cameras and it can it can actually do with a voice interchange, you know. And also like it has like a reasoning powering through the text based kind of models. And what it so this model itself is about six hundred megabyte. So imagine like if you have like a six hundred megabyte intelligence that goes inside every car,

• 24:01Speaker 01

  you can do an over-the-air update or OTA. You can do over-the-air update of every car on the on on the planet. The first deployments that we have on Mercedes, which is going to happen this year, it's going to be on all North America Mercedes Benz cars that are generation three. We are going to have like the first deployment at scale of AI in like actually inside the cars, you know. And this is running on a on a chip that is probably a hundred bucks, you know, like in in cost. So what I want to say is that like it's absolutely insane to really think about bringing a really high quality kind of things, production grade AI. It feels really, really good to actually work towards this efficiency at the levels that we talk about and then taking these things and bringing them into production in the car kind of space.

• 24:46Speaker 01

  And then I can tell you about millions of things that we've done with Shopify. You know, Shopify has been an unbelievable partner for us. They are not doing on-device AI, but they care about latency. They care about latency and quality. So what they do, they have private deployments that they have to serve like through the shop app, the merchant facing and also like client facing products of Shopify. Across many different kind of things that Shopify does, we have a lot of liquid foundation models deployed in production. One of the examples would be if you go to the shop app today and type in something in the shop app, you would actually send the signal to a liquid foundation model. So there is actually liquid foundation model there that is serving across the stack, which is which is very nice.

• 25:27Speaker 01

  And and we are passing, we are passing through like some amazing statistics, exponential statistics as well in terms of use. It seems like clients are really enjoying those features. And now we are exponentially at over a billion requests per month. You know, so that's like the amount of requests that goes through like the liquid foundation models like across like that shop app kind of just just that kind of metric.

• 25:52Speaker 00

  Just Shopify, got it?

• 25:54Speaker 01

  Yeah, it's just that. And then in terms of popularity of the models themselves, as you know, we open weight, we open the we open, we we deliver kind of the liquid foundation models like to the to the community and we we really open source them. And I think the models are very popular right now. They have an overall over forty millions of downloads and we have like one point five, about one point five million downloads per week. So the models are getting downloaded really much. A lot of developers are taking these things and putting them in production. The range of parameters of liquid foundation models that so far we released are between hundred million parameters to twenty four billion parameters. And these are a mixture of expert kind of architectures, the the the bigger ones. And this range of models, they're also like they're adding multimodal capabilities.

• 26:37Speaker 01

  As I mentioned, like the models could be like both processing audio, vision and and and text at the same time as an input and then generating audio and text as an output. So that's kind of the the portfolio of things that we do.

• 26:49Speaker 02

  I was wondering about robotics, because that was kind of one of your inspirations when you started. And I noticed you've got like you've got multimodal models. You've got like a vision language model and an audio or models and audio models. So are those being used for, you know, robotics purposes? And can you maybe talk about some of those experiments?

• 27:09Speaker 01

  Absolutely, yes. So when you have like the vision language models, like you can you can actually solve vision action kind of problems. You know, like you can turn into robots seeing the world and then you take an action based on like what robot sees, you know. And you can also use them as monitors. So imagine like in factories, there are like cameras that are monitoring like the entire operations of a factory. And then we we usually are thinking about industrial operations of robots, you know, like not not mostly humanoids, but also like just industrial robots. We worked with a company called Robotech. I think you should you should find, if you just say Robotech, AMD and Liquid AI, you would find like there's a demo of like what we've done there. There is another company called G Bionics. I think it's in Italy. And this company is also like they are having like modular robots

• 27:54Speaker 01

  and they are using our vision language model for some action space for the body of their robot. You know, there's a camera actually like on the body. And they are using these things for, let's say, instruction following kind of capabilities, like basically vision action models. So use cases mostly goes inside inside factories. And then there are like obviously, like if you think about like multimodal intelligence, as you as you mentioned, the smaller instance of the models, they for low latency applications in robotics, they're really, really popular. There's a lot of new engagements that are coming our way. But robotics has been something that we deliberately kind of put it on pause day one. The reason behind it was that because the go-to-market motion around robotic and also like really building a business around robotics is very difficult. And and it is it is delayed. Because if you think about production grade robotics,

• 28:39Speaker 01

  you need to go down the rabbit hole of verification and validation of the technology. And then you get into like regulatory kind of aspect of its sort of deployment, right? So we always like because we are coming back from a robotics background ourselves, we know how how it is like to really productize some robotics kind of vertical. That's why we wanted to build like a horizontal layer of intelligence on top of like, you know, places where we can have like OEMs first and then entering into the action world where where verification and validation becomes more more sensitive as a later step of the company. But I think now it is like a good time, a lot of, because our models are matured a little bit better. Now we can, we we have a lot more control over like data and and and the post-training and customization process of these foundation models for certain type of applications.

• 29:22Speaker 01

  I think it's a very good time for us to really also like start the robotic vertical like much more seriously than before.

• 29:30Speaker 00

  Just on on this, because I wasn't gonna, I wasn't expecting to spend that much time on robotics. But since you mentioned it and you said it's a good time, do you basically use the same LFM family? I think a lot of people make a lot of a big deal about how different, you know, action models are or world models are for robotics. I don't know how much of that is just marketing versus fundamentally it is actually the same thing.

• 29:54Speaker 01

  That's a great point. So I will tell you something. So I'll tell you like innovations that happens in the space of foundation models, you know, that are getting used in any vertical. Like I would categorize them in three different classes, you know. But there is model architecture of intelligence. Then there is algorithmic aspect of intelligence, like algorithms that are innovative. And then the last thing would be data processing kind of mechanisms. Like what what kind of data work do you want to do? So innovation across this tree. What we've been focusing on like from our research point of view, it was like what we so far talked about was about model architecture innovation, right? Like it was just about the architecture, like efficiency of a model and the ability to become a general purpose computer, basically. So that's got the model kind of perspective.

• 30:40Speaker 01

  Then there are algorithms. The way you train a model, you can train a model with an autoregressive kind of loss. You can train a model as a, you know, like as a in diffusion processes. You can train a model in a word modeling context. So you can take that architecture and bring them into. So I would say word modeling, it's like an unsupervised way of learning. Systems, you know, like it's it to really build like a representation in between. So that from that representation, you have a worldview of the world, like you have like an understanding of the world. So of course, like I would attribute kind of word modeling on the algorithmic side of intelligence, you know, rather than on the model architecture, architectural change into the. So you could technically take some of these LFM architectures and use them in a word modeling pipeline

• 31:25Speaker 01

  and train them like to have like a better understanding of the physics of the world. You see, I would always argue that various algorithms allow you to learn the some physical structure of the world with different ways. Word modeling allows you to extract like a different set of primitives from the physical world compared to next token prediction. You wouldn't say next token prediction would create, um, would would not have a physical. They have a physical understanding, but it's very different than the human's way of understanding the physical world. So I would just say like that physical, and there is still a debate between like what learning algorithms are going to unlock,

• 32:07Speaker 01

  what type of, what set of capability this is needed for having the chat GPT moments for robots. That's like a big, big question mark on the algorithmic side. On the architecture side, I would say you could use any general purpose computer, like any general purpose kind of algorithm

• 32:24Speaker 01

  architecture algorithm to be used in those kind of frameworks that I mentioned.

• 32:28Speaker 00

  Just so I understand the business model side before we go back to a bit more research. I assume these are licensing agreements because obviously it's on, it's, you know, air gapped, it's on device, it's all these things. How much time do you have to spend to customize a model? Let's say I I come to you with like a very special set of compute constraints or whatever. Is it you got your guys doing it? Is it a collaboration? How does this work?

• 32:54Speaker 01

  Fantastic question. So this is actually our early engagements is usually with design partners, you know, like these design partners are the ones that we we would walk, we associate kind of an applied ML team to work with the client. And we charge also for that duration of kind of engagement, right? And then, once the solution is built, then we deploy the solution and then we charge licenses on a yearly basis, on a device basis, you know, like depending on the, depending on the, where where this thing is getting deployed, you know. So and we are charging that kind of for models as a recurrent kind of cost. Then, after we build a solution in a vertical, then the time of service reduces for the next customer in that vertical. So imagine if Mercedes Benz is our design partner and we build a solution around like our our products.

• 33:39Speaker 01

  Then the next customers that are coming in the automotive vertical, the amount of service that we provide is like reduces like from our side and we, it becomes just the solution deployment. So we make solutions in every vertical and those solutions are the ones that are actually like allowing us to sell faster. That being said, because as a foundation model company, we've gone through like the process of building a model from scratch and deploying these models in production. We've gone through this process many, many times. We are building now a platform for model development and that platform is kind of self-served. So you can actually get access to this platform and then using this platform, you would be able to automatically kind of do that job. And this is something that we are building.

• 34:21Speaker 00

  Wow.

• 34:22Speaker 01

  As of now, it's a very difficult problem, you know, and you you can close the loop also with like recursive loops and everything. But we want to give these capabilities of model building with the quality that Liquid builds models to the clients, basically to the enterprises themselves.

• 34:35Speaker 00

  Is this platform fully rolled out? This is in beta? I I this is my first time hearing about this.

• 34:40Speaker 01

  No, it's in beta right now. It's it is in beta, like we are, we are still like testing it out. We are very excited about it. It is a very interesting kind of problem. If you think about it like this customization and continual learning, it's a very, very important next step for the deployment of AI. Right now, like ninety percent of the market is around inference tokens, like Frontier Labs, like if you think about Frontier Labs and even inference providers right now, like fireworks of the world. What are they doing? They they host the models and then they basically they charge the inference, right? Like the inference tokens are the ones that are making money. But I think the next wave of companies that you have, you you you should know, I can name a couple of them, Reverb AI. There's like Core Automation, there's like Thinking Machines, you think about, um, uh, what else do we have?

• 35:24Speaker 01

  Trajectory, you know, like there's like many different companies like in this kind of category of customization. So a lot of people are turning into customization tokens. Because now there is like a lot of good base open source models as well. So people, enterprises could actually get, we are talking about enterprise AI, by the way. So you know, like in this space of enterprise AI, monetization of customization tokens is something that I think people are going to capitalize on. And it's going to be something that is very, very, um, interesting and it's going to be the next, uh, next wave of, um, Decacorns that are going to emerge from these kind of category, which some of them are already Decacorns like at at at the like Jeff Dean just started like a lab again, like on the same top of topics, you know.

• 36:06Speaker 01

  So you can imagine like customization is going to become like a very, and when I say customization, don't only think about post-training a model. Think about the depths of customization from pre-training to inference. That's customization. You can do pre-training, you can do mid-training, you can do post-training, you can do data gen, you can do RL. All of these things fall into this category of customization of a model and then allowing agents to do that. And then looping the whole thing so that you can actually have like a recursive loop that allows the customers to own their own intelligence. That's kind of something that we are planning to do and we are providing like this platform to customers so that they can, they can really build liquid foundation models at the quality that we can build, like that's number one.

• 36:48Speaker 01

  And then, once they put them in production, they can keep them always up to date and always kind of to the level that they, to the the level that they want them to perform.

• 36:56Speaker 02

  So just, um, before we hit the research, um, can I ask a little bit about your developer platform, uh, Leap? How are developers using this platform at the moment and what kind, what kind of process do they use to, um, deploy your products?

• 37:09Speaker 01

  So if you think about, um, uh, uh, Leap is like a, a library of, we have a cookbook, you know, like at the QD AI, this is like an open source kind of place, like you can go through our cookbook.

• 37:18Speaker 00

  Yeah, you got a lot of attention.

• 37:20Speaker 01

  There is so much going on in there, you know, so we literally like provide like even a lot of insights, you know, like the this is manual, this is built for humans, right? Leap is also built for humans. So you're a developer, you want to use liquid foundation model, you want to fine tune them. You go there, you can take like some of the like Leap fine tune, for example, is one of those places where you can fine tune one of our models. Then there are like deployments, you can, Leap allows you to also extract kind of a bundle that you can deploy like an inference kind of ready, GGUF kind of format, like a Lama CPP kind of ready, kind of system that can go directly on the device. If you want to have like a, let's say, CPU deployment, you can go to our Leap platform and it allows you, allows you to extract one of these bundles that are ready to be deployed, you see? So that's what what what Leap does.

• 38:03Speaker 01

  And again, it it, think about our current, uh, uh, uh, platform that we are building right now. It's literally the automated version of like what we have been doing like with Leap, right? Like there are some functionalities that Leap has and some functionalities that it doesn't. So the ones that we want to, we want to complete the Leap platform as tools and then be able to also provide enterprises with the capability to do that in the way that we we use those tools. But those are kind of individual kind of segmented tools that you can use or give your Claude agent to really like take them and then use them to fine tune an LFM. That being said, there are also like support of liquid foundation models in the, you know, uh, very popular kind of places like onslaught and, you know, talking face, kind of they have like the TRL kind of, uh, fine tuning kind of.

• 38:49Speaker 01

  You can use any of those kind of open source, uh, packages also, because we, we now try to like be really like connected to the open source community. We love the Hugging Face guys, we love the SG lang guys, you know, like we work very, very closely with them. And, um, and and and really bringing like support and Lama CPP project itself is an amazing effort to really deploy intelligence outside of data centers.

• 39:10Speaker 00

  Yeah, I I haven't heard TRL in a, in a while. It's, uh, you know, there's all these, every year or two, there's like a new hotness, like right now it's onslaught, but like, you know, we've also covered Axolotl in the past as well. One thing. My eyebrows just went up when you said self-serve. I think I had expected that you would do a lot of services. Maybe these days it's called forward deployed. I don't know whatever the hot term is. But is there a maturity or hierarchy of things, right? Like, so basically, the way I think about it is like, OK, everyone's talking about continual learning, but probably most people are not even ready. You release, for example, quantization aware distillation, right? And that I can do. That seems easy. That seems like self-serve, like, OK, you know, you give me some guardrails on how to do this.

• 39:54Speaker 00

  Like, I can distill big model into small model, right? Is there a stage of like, OK, you start here and then you work your way up?

• 40:00Speaker 01

  Absolutely, yes. You touched on a very, very important problem. I'll tell you a little bit a deeper perspective of you work. We've worked probably with 200 of the Fortune 500 companies like so far. Like, we really are like in touch with all the enterprises, you know? And what enterprises are exhausted. They went through the process of building their own intelligence for a long time. They tried to have like applied ML teams themselves. You know, like we're talking about mature enterprises, right? We're talking these mature enterprises, they tried out like intelligence, downloading models from open source, building their own models from scratch, you know, like for their own kind of topics and stuff. And this did not result in actual production kind of lines. And they always defaulted back to the cloud solutions, you know? So they never got to the production quality that they want.

• 40:44Speaker 01

  Some of these products are out there. But I can tell you the ratio is like 80% of the production grade kind of stuff doesn't, the POCs, 80% of them do not hit a production kind of level. So that's one of the exhaustive kind of things that I see is this. Second thing that I see is that there is massive demand for developers right now that they want to have access to the highest form of intelligence for development. They want to wipe code stuff, like everybody wants to token max. You know, like there was like these kind of things like it's, it's such a virus that is in the head of like developers, you know? And everybody is, and again, when you do the value attributing value, how much value was generated off of this wipe coded repositories

• 41:28Speaker 01

  and stuff that is happening at organizations, mature organizations? You will again end up with 90% of these tokens are useless tokens. So when you actually have enterprises see this and they want to also control the cost that they have, there is a massive demand now for them, for them to have a self-serve platform that allows them to really get to the point that they want to get as soon as possible to a value that they want to receive, you know, while satisfying the demand of their own developers. So what we try to do, we try to build the platform not as another UI, another new type of set or something. So our platform is literally gonna sit inside a client like very similar to how you interface with your cloud code,

• 42:10Speaker 01

  very similar that you take your, you know, your agentic harness and you just give it the tools of Liquid AI. And these tools would be able, like your own favorite kind of agentic harness would be able to, in a step-by-step manner, walk you through developing and building and deploying a liquid foundation model, production quality level, the way that we would do it. But this process is an interactive process with the developer. At the moment, human in the loop aspect of it is very important to the point that you mentioned. You mentioned like, are they ready to do this? You gotta guide the customers, right? Like, you gotta guide them, the developers, to really get to that point. So that is the automated kind of stuff. You can also put it on a steroids, you know, like we have like,

• 42:54Speaker 01

  you can put it on auto mode, like the whole platform and say one click and then just deliver this is I don't want to see any intermediary stuff. But obviously, the quality of the resulting model is not going to be as good as when you're having an interactive kind of developer actually interactively build something with it, you know? So we try to get ourselves closer.

• 43:12Speaker 00

  Let me push you.

• 43:13Speaker 01

  Yes.

• 43:14Speaker 00

  Let me push back on you, right? If I have good evals, why do I care what the process is, right? It should just auto research itself. Why do I have to look at the logs?

• 43:22Speaker 01

  For the first time, I fully agree with you. For the first deployment, right? So imagine, imagine if you, if you put a model in production based on the evals that you had. Now you've put the model in production, then new data is gonna come in. You know, like new data is gonna, the model is in production, in actual production environment. OK, now imagine new data is coming in, new requests, like completely, like all of a sudden, like the whole distribute, there's a massive distribution shift of, let's say, community wanting different things, you know, from your models. How do you handle that? You know, you gotta be able to handle these things like we, because your evals are gonna become obsolete. Static evals are going to become obsolete. So you need to have some sort of a dynamism like there. And this would only happen if you have a continuously evolving system.

• 44:06Speaker 01

  You see, evals cannot stay static. That's the thing. For the first iteration, I fully agree with you, you know? But if you go to, you know, production, then the evals are not gonna stay the same. You know, you always have to change the criteria.

• 44:18Speaker 00

  I definitely agree on that. I think it's why we're moving towards environments and, you know, harbor style tasks and all these things. What other trends are you seeing in like sort of post-training, RRL environments? Any, you know, like, there's a lot there, which obviously Maxime has talked about. But effectively, are we customers, let's call it, doing more and more of the post-training and you are doing the architectural work? Is that the divisional labor?

• 44:48Speaker 01

  Something like that, yes. I would say not just post, as I said, like, don't forget about just post-training. I talked about the whole model development from scratch. We want to be able to.

• 44:58Speaker 00

  Oh, everything. Well, including the architecture search, yeah.

• 45:02Speaker 01

  If you're as sophisticated at, let's say Qualcomm, that wants to, let's say, design their own, let's say, like for the, so the next generation of their chips, they want to have like a dedicated computational graph that works the best on top of Qualcomm chips. You're on an NPU that they're designing, you know? Or there are like also some companies that are etching kind of weights of a neural network directly inside the chips. Like, you know, like these, these things are happening.

• 45:24Speaker 00

  We've covered Talos, yeah, Talos and Etched before.

• 45:27Speaker 01

  So as you know, like, there's, there's so much in the silicon kind of world that is also, those guys would be interested in our architecture search more than anybody. So silicon is actually extremely, staying close with silicon partners is actually very important for us because our architecture search is hardware aware. So why? Because they want, they're putting their hardware or maybe next generation of the hardware in the loop of finding out what is the best architecture for the criteria that I care about. So imagine you launch a process like that. Silicon companies, they'd really like this automated foundation design kind of

• 45:59Speaker 01

  platform that we are putting in front of them. So that technology. And then, as you mentioned, I agree, customization wants to get more and more to enterprises. You know, like enterprises would be able to own their own model development and deployment. You see, these are the two things that you, like, this is kind of the true meaning of sovereign AI. You know, like, everybody would be able to own their own intelligence. And then hopefully, one step further would be all the customers, like every single person on the planet would be able to own their own intelligence as well. So that's kind of the next level personal intelligence, you know? Hopefully, we can unlock that.

• 46:29Speaker 00

  And I just wanted to quickly touch on things because like you left so much in there. It's, there's, there's actually a lot to dive into. In terms of neuroarchitects, you're the first, I think maybe you're like second or third company. You know, we recently covered poolside where they talked about their model factory. I'm sure you've talked a little bit about that. And then also a bit back, I was also very excited by the tooling for, for search, right? Apple had this paper on Talaria, which is their sort of LLM ops, like layer by layer inspection and ablation tool. What have you found in your neuroarchitecture search or model factory type things that you thought would work, doesn't work, or is like, you know, surprisingly effective?

• 47:14Speaker 00

  Like, anything you're gonna highlight from your process or research?

• 47:17Speaker 01

  When we do architecture search on, in the space of foundation models, we run a scaling loss. Like, imagine for every architecture candidate, we try to run a certain type, up to a certain number of tokens, we run a scaling loss. OK, so like this being like more, throwing more data at a certain fixed size of the model or increasing the size of the model, you know, like, and see like how well it can actually train, given that candidate architectures. The things that we learned is that the larger the neural network you are making, the architecture, it should be, if you want to build general purpose intelligence, it should be unbiased. So you have to unbiased your system. So, for example, if you look at LFM2 architecture that came out,

• 48:00Speaker 01

  this doesn't have any of these gated delta nets and all these things. It's like a very unstructured, it's like 1D convolutions. Literally like 1D convolution. This is like the simplest thing that you can actually do. It's like an unstructured way. Attention itself is like a very unstructured, as I said. When architectures are not biased, like attention, like transformers architecture, you would, you would, you would be able to scale them. So at scale, I would say a lot less bias is needed inside the architecture. You don't need to add, add features, add loops, add some sort of like, you know? So you, you gotta simplify the architecture the larger you make them. On the other side of the spectrum, imagine if you want to solve problems at a very, very

• 48:42Speaker 01

  small scale and you cannot afford to have like the, you know, like you have like memory limit, let's say like eight gigabytes of memory limit, you know, there you can get more creative on the architecture side, you can make the models to have multiple feedback loops, you know, you can add like a lot more complexity to the system. In the small model regime, the more biases you add to the system, the more expressive dynamics you get. In the larger scale, you remove those biases. This is what we learn. Second insights that I would tell you is data modality. There are some data modalities like audio. In audio, you could use a continuous time dynamical systems or SSMs, for example, you know, like they're very effective in audio, but on text, they suck, you know, like SSMs, like they

• 49:26Speaker 01

  don't work like really well on text. And when you, when you think about like, let's say if you just want to have an audio model, drawing in a recurrent neural network at the, at the scale that is needed for humanly like, like having like a really good quality kind of audio output, this is kind of the place where I would say your recurrent neural network would be very much shining, you know, again, at small scale, you see, like when we talk about like this, this category of models that we search, you know, below, let's say 20 billion parameters, you know, recurrent neural networks can be like really, really effective there, you see. And but I would call them a more biased architectures, you know, like they are, they have like more feedback, you know, like there's a structural gate, you know, like they have like a certain, certain architecture, architecture, archetypes

• 50:08Speaker 01

  like on top of them. And then you remove those things as you scale them out. That's kind of the, the general kind of understanding that we have.

• 50:16Speaker 00

  You know, no, no surprise that, I think Jason Wei had this post where like at small scale, you add in biases and then eventually you have to take them out. I also, I, I think layer looping or what, I don't know if this is called layer looping. Do you call it layer looping or whatever the?

• 50:33Speaker 01

  Yeah, yeah, yeah, yeah, looped, yeah.

• 50:35Speaker 00

  That one is a bit unintuitive, why it works at small scale, but not at larger scale. It seems like it would work. It's mostly like a efficiency thing, I guess. So maybe that's just my, that's my explanation. I don't know.

• 50:50Speaker 01

  But I would say still, you know, looping, it's again a bias. Like any, any format of looping, it is a bias that you're adding to your system, you know, like you, you make it complicated and then you, you gotta, you gotta find out an infrastructure for it. By the way, so we don't have the correct infrastructure, the current GPU infrastructure that we have is optimized for this type of workload, you know, parallel matrix multiplication, or everything that you can parallelize, like in a parallelizer, do, imagine if you have like photonics, you know, like I think this space of photonics would be like an amazing kind of a space. Like if you think about it, if you can actually communicate with the speed of light, now compute in a serial way. What is the speed of serial computation when you're moving with the speed of light? It's like humanly not perceivable, like, you

• 51:34Speaker 01

  know? So what I'm saying is that like, I think photonic could be like an unbelievable way for us like to move. If you think about like medium change, you see? So that, that could also unlock like nonlinear operators. I know companies that are working on like photonics and then the quantum computers, like, you know, everybody's working in this, in this space of alternative mediums for intelligence or, or compute in general. Those are kind of the, another opportunities that I see like opening up. And I, I'm really excited about photonics because I think, I think there's a lot, a lot to, to be like, because sequential computations, this is something that from a linear algebra point of view, we learned it for nonlinear systems, because we cannot really parallelize nonlinear systems. So but imagine if you have like a speed of light computational kind of speed for, for, for nonlinear operators, you know, then you can actually build the next generation

• 52:18Speaker 01

  of architectures, because then you're not limited by the, by the substrate you're hosting intelligence on. So one, I'll give you one quote from, I think it was from Christopher Koch, who was writing like some book on neuroscience. I think I read it like about 14 years ago that he's not, it wasn't a quote. He was at a conference and he was saying, intelligence is a property of carbon. So because the substrate of human, human is a property of carbon. So you'll see, he was saying like, okay, so natural life is happening on a different substrate and we're building intelligence right now on silicon, right? So the base of intelligence is very different. Like, I think substrate is going to play a role into like even the next generation of architectures, you know, if you really think about it, because

• 52:59Speaker 01

  it would unlock various formats of competition apart from like just parallel competition through GPUs. That's also another space that I would just bring in.

• 53:07Speaker 00

  Okay, we don't have a ton of time to mention this, but you're, you brought this up, so I have to ask. I would say actually a surprising number of people in my circles in SF, not that excited by quantum computing. If we did have at scale like a bunch of qubits, do you actually think that it would change anything? There's some people that disagree, so I want to hear your view.

• 53:28Speaker 01

  I think the only thing that it changes like massively would be search, because I think search is one of those algorithms that matters so much. And imagine if you can have like more parallel search, you know? And, and what is optimization, you know, at the, at the end of it, like we, we are, you know, we're submitting ourselves to gradient descent, you know, right now, because we don't know any better way to perform massive search at scale, you know? It is a search in the parameter space, you know? And, and if you think about quantum computers, like I think search is going to evolve massively.

• 54:01Speaker 00

  That one, hopefully that is not controversial, but yeah, it's, it's very interesting that there are many other bottlenecks that matter to us that actually is not solved by quantum.

• 54:09Speaker 01

  Absolutely. But I don't think we are close, because I think 2017, I had, I was at one of these NURIPS conferences and, and I was talking to, I think it was Microsoft or IBM, like they were, oh no, it was IBM, that they were, they were bringing, they were showing off their quantum bit, like kind of the, the thing that is like hanging from the ceiling, you know, and they were showing it at the, at the conference. And then I was talking to the guy and I said, okay, so how can, when can everybody actually use something like this? At that time, 2017, he told me 2035. Okay?

• 54:40Speaker 00

  Okay.

• 54:41Speaker 01

  So, I mean, I think it's still too early to say anything.

• 54:47Speaker 00

  Yes, yes. Okay, coming back closer, actually for listeners, we've done a bunch of related podcasts. We, RWKV, you know, is, is another like pretty widely adopted non-transformers architecture that's like sort of loosely inspired by RNNs. So we've done a couple of episodes on those things. We've actually had Tree Dao on, we haven't had Cartesian on yet, but like when you mentioned how SSMs are very attuned to voice. At first, when I, when Tree talked about it to me and Karan talked about it to me, I didn't understand like why voice. To me, actually, the, the simple explanation is there's just not that much state. Voice is like continuously inferencing anyway. So like, it is actually pretty ideal.

• 55:25Speaker 01

  Yeah, it's a, it's a continuous on sequence, you know, and sequences that are time series. You can say, if you throw a dynamical system that is designed to handle time series data, that's like the best version of thing that you can actually model time series, right? So that's why, that's why SSMs are pretty effective. SSMs and RNNs and continuous time systems, they are very effective in, in, in basically sequential kind of data, you see?

• 55:48Speaker 00

  You brought up time series. The other thing about my background that I bring into this is I used to be an options trader, right? Where we actually would do, have to do stochastic calculus and solve, solve those kinds of abilities, which is another kind of continuous time finance, which is what, what we, we talked about there. Yeah, I, I just, I just think like, it's weird that, that this hasn't been a priority till now. And like now it seems like, once you explain it, it's like relatively obvious that we, this is a thing that we should exploit. It's like a dimension that we should exploit. Are there any others in your mind that, you know, obviously you're scaling out this, you're maybe like, let's call it 10 years into this journey, but like, okay, if we're wrong about continuous time, what else?

• 56:26Speaker 01

  That's a good question. So I think there is a lot of, you know, like I, when, when we started like liquid as, as an idea, so we were thinking about efficiency of intelligence and adaptability of intelligence. I think we've, we haven't explored too much on the adaptability of intelligence, like forward pass for us is like basically forward pass to a, to like a model, like, you know, a lot of, a lot of forward looking kind of competition is like literally like just a forward pass competition. So what if like adaptability on the go? A lot of people would come, would tell you that, okay, so in context learning is solving that problem, because in context, like you, systems are actually like when you bring more data at the test time, they would somehow kind of solving

• 57:08Speaker 01

  for some sort of a back prob, like, you know, like a pseudo, very, very similar kind of to back, back problem situation. It's like a least, least square algorithm, like in, as, as in context, kind of learning is happening on the, on the test time compute. But in general, like I think, I think approaches where we can combine forward pass and backward pass at the same time, this would be a new generation of architectures that I think there will be an evolution of this thing. I know that I've heard that SSI is also working very on the similar architectures to the original ideas of liquid neural networks, you know, so that's what I heard. But we will see, we'll see.

• 57:45Speaker 00

  Have they not reached out? Like, come on, like, what's going on?

• 57:48Speaker 01

  Yeah, but, but in general, like, you know, like me, I know that all the founder, you should also, also already assume that all the foundation model labs, you know, Entropics and OpenAI's and everybody else is like massively invested. They have teams that are working on alternative architectures. Another thing that I want to say is that, again, why do you want an alternative architecture, right? This is the question you have to ask. The first thing for us would be efficiency and everybody else is now seems to be like focusing on post hoc optimizations or really like a speculative decoding, for example. Like it's like one of the spaces, like people try to make the quantization, you know, like it's another, the quantization of our training, you're like, these are, these are the kind of places where people try to pack information and you can do a lot in there, you know, because these are kernel science and, and I think we, we did, we did with better AIs,

• 58:32Speaker 01

  you know, I'm, I'm hoping for the next generation of AIs to get like a lot better at what they do today. They can, people can really optimize kind of the current architectures to get efficiency, you know, out of the models. Now, if you want to unlock the next generation of capabilities, that's a different story, you know, that could be a meta algorithm. So think about it also like right now, how does our workflows of coding today look like? There are like, there is an orchestrator and there are like a lot of agents that are, so in the future, multi, multi agentic kind of architectures, like that's another thing that I can think about. Like, we can think about at the atom level or we can think about at a system level, right? So you can also build like new systems that are composed of smaller kind of units of compute, like, like, let's say like a unit of compute could be

• 59:16Speaker 01

  neurons and synapses, or it could be an LFM, you know, or it could be a small model itself. And then you can use and bundle and it can build basically systems of systems, right? So that's kind of another direction that architecture, and I would call them cognitive architectures, you know, where you have like multiple kind of agents that you can actually bundle them together to build like a whole full blown system as a whole. So I think a bunch of directions I, I pointed out that I think could be an exciting future step.

• 59:43Speaker 00

  If I were to paraphrase, maybe, you know, agents should be able to train models casually as part of their job. And obviously, LFM would be quite adaptable. And it's not actually the LFM. It would be LEAP, because you need the full platform as well in order to adapt to whatever the situation you're deploying in.

• 1:00:00Speaker 01

  Definitely.

• 1:00:00Speaker 00

  All these named directions, probably you also have like sort of more near-term research directions. What would you say is the current research agenda for Liquid?

• 1:00:11Speaker 01

  We are very much focused on massively multimodal systems. You know, like we are thinking about like training every, all the data modalities at the same time. And at the same time, like longer horizon kind of reasoning, of course, like this is one of the directions that we want to do. Imagine if you want to do reliable on-device intelligence kind of services, like a system, you know, like if what, all of those things require like higher degrees of reliability for them, for the models. So we work a lot on that direction. We work a lot, we have a department for efficiency, you know, like we work on massive like layers of efficiency, you know, like you can think about like kernel design, it goes to inference, this goes to infrastructure, RL infrastructure that is like much faster, you know, like making like all of our processes that are happening, like from pre-training to delivering a model

• 1:00:56Speaker 01

  to production, how to make this, all the infrastructure like more efficient, you know, that's like the focus of the company. Then from a research point of view, we are looking also non-human understandable data. For example, think about signals, you know, you can think about biology as well. We have some biomodels as well, like for example, we are building a DNA foundation models, like your DNA, with your eyes, you kind of like really read it, they are long context data, you know, like they're like not dense as, like let's say in the vocabulary, there are like four vocab's like that you can actually like letters that you're dealing with, but I mean, from the sequences, lengths point of view, like they're very large. There are a lot of applications that we are excited about in that kind of realm. I think our type of technology is like naturally pushed us into like looking into this alternative type of data as well.

• 1:01:41Speaker 01

  As I told you, like our architectures, we have talked about dynamical systems and longer sequences of data. So DNA data could be like one of those places where we can bring value. And then, as I mentioned, multimodal research is extremely hot right now at Liquid.

• 1:01:56Speaker 00

  Yeah, yeah, yeah. You said video, audio in and video, audio out.

• 1:02:00Speaker 01

  Video, audio, text in, audio and, audio and.

• 1:02:05Speaker 00

  Text is assumed, text is.

• 1:02:06Speaker 01

  Yeah, text is assumed.

• 1:02:08Speaker 00

  The in is fine, the out is new, right?

• 1:02:11Speaker 01

  Correct.

• 1:02:11Speaker 00

  That is like, typically, I would never expect someone like you to care. Is it realistic?

• 1:02:18Speaker 01

  It is.

• 1:02:19Speaker 00

  To have, you know, image out, video, audio out, video out from a 7B model that like is usable.

• 1:02:27Speaker 01

  I don't think, I don't think it is visible to get a video out of that, of that, like a proper, like very, very high-quality, high-fidelity kind of videos. But, but directionally, if you want to solve physical problems, you know, like, and if you want to have like a vague understanding of the world, imagine you are, you're training these LFM's of, let's say, in the range of 7B below. In this kind of range, you want to train like a, with a word modeling kind of context, like you want to train a liquid foundation model. You would be able to encode some sort of physical knowledge inside the model, but, but the model would not be able to output something very high fidelity, you know, but it would add to its own understanding, but it would not be able to output that, you know, it can basically like have like a little bit of a better representation internal, you see?

• 1:03:11Speaker 01

  Like if you were to train them like with a word modeling, especially for video, applications where video generation matters, you know, like not just for entertainment, but we are talking about like grounded industry work of a robot that is like operating in a factory, you see?

• 1:03:26Speaker 00

  Yeah, video understanding is actually so underrated. There's still not enough, mostly it's just interleaved images, but like, we want more.

• 1:03:35Speaker 01

  I'll tell you, in the space of multimodal intelligence, it's really, really a lot to be learned like so far, because it's really not that many highly multimodal systems, you know, like this is, this is like a very, very open research topic, like for foundation models as a whole. It's going to enable like something completely different.

• 1:03:53Speaker 00

  One thing I wanted to double check is the, something that's adjacent to your space, which is modular. They just got bought, went to Qualcomm, they, they just released the, you know, 1.0 open source. You guys famously switched from Jax to PyTorch. Did you explore, you know, Mojo? It is supposed to be for you guys, like that you are doing custom kernel work, you are Python-centric. If it didn't work, why?

• 1:04:24Speaker 01

  I mean, modular is basically operating on a space of kernel, like they're, they're one, one layer below the foundation models, you know, and they are, they, they try to be also like us, like Switzerland, working with all the hardware providers, but they now got locked in into Qualcomm. I was on a panel with Chris, actually, like some weeks ago, actually, like on an AMD panel, like, and, and, and I can tell you, like, I have so, I mean, he's one of my heroes, you know, like Chris, Chris Lattner, like he's, he's unbelievable.

• 1:04:52Speaker 00

  Yeah, he's probably listening.

• 1:04:55Speaker 01

  Yes, but what I, what I can tell you is that, one thing that from first principles, we tried to do at Liquid, we don't want to reinvent the wheel, you know? If, if the fundamentals is actually like, I'll tell you, for example, it's so hard, there's a lot of inference companies that are trying to replace Llama CPP or building wrappers around Llama CPP for on-device AI. Almost every single of these things that are not locked in into like one ecosystem, and they try to be like a general-purpose language for all hardware that is actually out there, all of them are worse, if you dig in, they're worse than the original Llama CPP. So we don't want to change base, you know, so that's what I want to say. There are some successful ones that are locking into one ecosystem, and I feel like Mojo is going to go in this similar direction.

• 1:05:40Speaker 01

  You know, now that they are getting locked in into like the Qualcomm ecosystem, I think there, there's going to be like a lot more success, but, but the platform itself is extremely, makes things easier to bring like intelligence like at that layer, you know, for, for hosting intelligence, you know, like so, so what they provide is actually like useful, but we haven't, we haven't been using those for, for our, for our own deployments yet.

• 1:06:03Speaker 00

  You say Llama CPP is, you know, better as a, as a baseline, but like, is it just production hardening, or is this something fundamental about the approach? I just want to try to figure out.

• 1:06:12Speaker 01

  It's a compiler, so that you can. It's like very optimized C++ kernels, you know? That's what it is, I mean.

• 1:06:17Speaker 00

  Yes, but, but Chris Lattner knows C++ like this is not a, this is not an issue.

• 1:06:23Speaker 01

  As I said, you could, it's extremely difficult to unify it. I don't, I don't believe, for example, in the uniformity of the Mojo kind of platform, you need to do custom work, and what, this is the work that the modular team actually put into. So I think the version that, of Mojo that runs on, let's say AMD, you have to design like certain kernels for Mojo, so that they are AMD-specific, or let's say Qualcomm-specific, and all those kind of work that they do, that underlying work is like a very solid work. And if anyone can pull, pull it off, it's like basically Chris's team, you know, like that's what I would say. But I would consider those, like, of course, like what Chris does, what Jonathan Roth did, like with TPUs, and, and, and all those kind of things that they're doing. So I would tell you, like, these are kind of the very fundamental works that they do,

• 1:07:06Speaker 01

  but I would tell you, like, we would stick to the, the most native kind of language, the most native platform that allows us to touch the computer like as, as close as possible, you see?

• 1:07:16Speaker 00

  Ecosystem always wins, you know, it's, it's really brutal answer. It's not, it's not fair to new entrants, it's not fair.

• 1:07:23Speaker 01

  Correct. It's, it's just a hard problem, and I would, and I would also encourage like people, like, you know, a lot of entrepreneurs these days, like, you know, like I'm now four years in, into my, our entrepreneurial journey, a lot of people, my friends and, and people are coming and asking, hey, you know, how's, how's the journey? I would just say, like, if a problem is hard to solve, it's actually worth, it makes it more attractive, you know, it should make it a lot more attractive for you, you know, because easy problems, everybody else is going to go and solve it, you know, so.

• 1:07:49Speaker 00

  Maybe the, the question I was going to end the episode with is mostly, you know, it's going to be the 10-year anniversary of the transformer next year. Is attention all you need?

• 1:08:00Speaker 01

  Attention is definitely one element that you need, you know? And I think, I think it will be a long time that we will be living with attention-based systems. So attention is definitely going to be insight like any of the, it's like, you know, how, how do you see, you know? How do you see, how, how's your eyes are working, you know, but you, you need your body is.

• 1:08:17Speaker 00

  I randomly noticed, like, actually, you're not seeing the whole view, your eyes are just bouncing around all the time, and your brain is compositing, right?

• 1:08:25Speaker 01

  Yes, you, you see the difference between two frames, by the way. This is what you see. So it's basically, you see the differential.

• 1:08:31Speaker 00

  Oh, left and right eye?

• 1:08:32Speaker 01

  No, no, like you're, you're basically, the, the way that humans see, you don't see one frame, you see two, the difference of two frames.

• 1:08:40Speaker 00

  I need to go look at.

• 1:08:41Speaker 01

  Look this one up, look this one up.

• 1:08:43Speaker 00

  So, so attention is, is one component.

• 1:08:46Speaker 01

  Attention is definitely one component, but then again.

• 1:08:48Speaker 00

  Hardware awareness, customization, yeah.

• 1:08:50Speaker 01

  Embodiment, like there's another, there are other aspects, also, I would say, like, intelligence that wants to get deployed in the society outside of data centers, you've got to have like some sort of embodiment. Agents that are actually inside the virtual world, they have also an understanding of their virtual physical kind of environments that they are operating in, you know, so embodiment is also true for virtual agents, by the way. So, you know, so you gotta, you gotta have like those kind of aspects in there. So embodied intelligence and embodied reasoning is like some, is very important. And then adaptive intelligence is another, another level that we, we really want to add.

• 1:09:24Speaker 00

  Is that routing related? Or what do you mean by adaptive?

• 1:09:28Speaker 01

  Oh, adaptive, I, I mean, like systems that can always evolve, like continual, continuously kind of improving systems, yeah.

• 1:09:34Speaker 00

  Okay, well, that was a really good deep dive. Is there anything that we should have asked you that we didn't cover?

• 1:09:40Speaker 01

  No, I think that was a great, great session. Thank you so much.