State Space Models and Real-time Intelligence with Karan Goel and Albert Gu from Cartesia

This week on No Priors, Sarah Guo and Elad Gil sit down with Karan Goel and Albert Gu from Cartesia. Karan and Albert first met as Stanford AI Lab PhDs, where their lab invented Space Models or SSMs, a fundamental new primitive for training large-scale foundation models. In 2023, they Founded Cartesia to build real-time intelligence for every device. One year later, Cartesia released Sonic which generates high quality and lifelike speech with a model latency of 135ms—the fastest for a model of this class. Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @krandiash | @_albertgu Show Notes: (0:00) Introduction (0:28) Use Cases for Cartesia and Sonic (1:32) Karan Goel & Albert Gu’s professional backgrounds (5:06) State Space Models (SSMs) versus Transformer Based Architectures (11:51) Domain Applications for Hybrid Approaches (13:10) Text to Speech and Voice (17:29) Data, Size of Models and Efficiency (20:34) Recent Launch of Text to Speech Product (25:01) Multimodality & Building Blocks (25:54) What’s Next at Cartesia? (28:28) Latency in Text to Speech (29:30) Choosing Research Problems Based on Aesthetic (31:23) Product Demo (32:48) Cartesia Team & Hiring

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Speaker 00

  Welcome back to No Priors. We're excited to talk to Karen Gole and Albert Gu, the co-founders of Cartesian, and authors behind such revolutionary models as S4 and Mamba. They're leading a rebellion against the dominant architecture of Transformers. So we're excited to talk to them about that and their company today. Welcome, Karen, Albert.

• 0:25Speaker 01

  Thank you. Nice to be here. And Kate, tell us a little bit more about Cartesian, the product, what people can do with it today, some of the use cases.

• 0:31Speaker 02

  Yeah, definitely. We launched Sonic. Sonic is a really fast text-to-speech engine. So some of the places I think that we've seen people be really excited about using Sonic is where they want to do interactive, low-latency voice generation. So I think the two places we've really had a lot of excitement is, one, in gaming, where folks are really just interested in powering characters and roles and NPCs. The dream is to have a game where you have millions of players and they're able to just interact with these models and get back responses on the fly. And I think that's where we've seen a lot of excitement and uptake. And then the other end is voice agents and being able to power them. And again, low-latency there matters.

• 1:16Speaker 02

  And even with what we've done with Sonic, we're already shaving off 150 milliseconds off of what they typically use. And so the roadmap is, let's get to the next 600 milliseconds and try to shave those off over the course of the year. That's been the place where it's been pretty exciting.

• 1:32Speaker 00

  Love to talk a little bit just about backgrounds and how you ended up starting Cartesian. Maybe you can start with the research journey and what kinds of problems you were both working on.

• 1:41Speaker 03

  Karen and I both came from the same PhD group at Stanford. I did a pretty long PhD, and I worked on a bunch of problems. But I ended up working on a bunch of problems around sequence modeling. It came out of these problems I started working on at DeepMind during internship. And then I started working on sequence modeling around the same time, actually, that Transformers got popular. I actually, instead of working on them, I got really interested in these alternate recurrent models, which I thought were really elegant for other reasons. And it felt fundamental in a sense. And so I was just really interested in them, and I worked on them for a few years. A couple of years ago, me and Karen worked together on this model called S4, which got popular for showing that some form of recurrent model, called a state space model, was really effective in some applications. And I've continued to be pushing on that direction. Recently...

• 2:26Speaker 03

  Proposed a model called Mamba, which was kind of brought to selling which modeling and showed really good results there. And so people have been really interested. We've been using them for applications and other sorts of domains and so on. So yeah, it's really exciting. Personally, I also I just started as a professor at CMU this year. My research lab there is kind of working on the academic side of these questions while at Cortejo, we're kind of putting them into production.

• 2:51Speaker 02

  Yeah, I guess my story was that I grew up in India. So I was came from an engineering family. You know, all my ancestors were engineers. So I actually was trying to be a doctor in high school, but my aptitude for biology was very low. So I abandoned it and instead became an engineer. So, you know, I kind of took a fairly typical path, went to IIT, came to grad school and then ended up at Stanford. Actually, started out working on reinforcement learning back in 2017, 18. And then, once I got into Stanford, I started working with Chris, who was somewhat skeptical about reinforcement learning as a field. And so

• 3:30Speaker 03

  This is Chris Ray.

• 3:31Speaker 02

  Yes, Chris Ray was our PhD advisor. So I had a very interesting sort of transition period where I started a PhD because I had no idea what I was working on. And so I was just exploring. And then ended up, actually, we did our first project together, too.

• 3:46Speaker 03

  Oh, yeah.

• 3:47Speaker 02

  Actually, we knew each other before that. And I think then started working together on that first project. And we would hang out socially and then start working together. The only memory I have of that project was we I kept I kept filling up this disk on G-Cloud and expanding it by one terabyte every time. And then it would keep filling up and I would insist on only adding a terabyte to it, which would which he was very mad about for a while, so.

• 4:12Speaker 03

  Well, by the end of the project, it was like running a bunch of experiments and the logs would get filled up faster than.

• 4:18Speaker 02

  Yeah.

• 4:19Speaker 03

  Basically, like I would be there. Like tracking the experiments and currently be there deleting logs like in real time so that our runs didn't crash. It was a really interesting way to get start working together.

• 4:29Speaker 02

  Yeah, so we started working together then and then, you know, I eventually like started working with Albert on the S4 push when he was pushing from NERIPS. And I think he was working on it alone and then needed help. I got recruited in to help out because I was just not doing anything for that NERIPS deadline. So ended up spending about three weeks on that, two or three weeks, something like that. And then we really pushed hard. And that's kind of how I got interested in it because, you know, he had been working on this stuff for a while and, you know, nobody really knew what he was doing. To be honest, in the lab, it was just like over in the corner scribbling away, talking to himself. We don't really know what's going on.

• 5:05Speaker 01

  Could you actually tell us more about SSMs and, you know, how's it different from transformer-based architectures and what are some of the main areas that people are applying them right now? Because I think it's really interesting is sort of another approach.

• 5:15Speaker 03

  They really kind of got started from work on RNNs or current neural networks that I was working on before as an intern in 2019. It kind of felt like the right thing to do for sequential modeling because the basic premise of this is that if you want to model a sequence of data, you want to kind of process the sequence one at a time. If you think about the way that you will kind of process information, you're taking sequentially and kind of encoding it into like your representation of the information that you know, right? And then you get new information and you update your belief or your state or whatever with the information that new information that you have. You can basically say almost any model actually is doing this. And then there were some connections to other like dynamical systems and other things that I found really interesting mathematically.

• 5:58Speaker 03

  And I just thought this kind of felt like a fundamental way to do this. It just felt right in some ways. You can kind of think of these models as doing something. There's like some loose inspiration from the brain even where you kind of think of the model as encoding all the information it's seen into a compressed state. It could be kind of fuzzy compression, but that's actually powerful in some ways because it's a way of kind of stripping out unnecessary information and just trying to focus on the things that matter, encode those and process those and then and then work with that. We can get more than technical details, but kind of like at a high level, it's just this thing. It's just representing this idea of this fuzzy compression and fast updating. So you're just keeping this state in memory that's just always updating as you see new information.

• 6:41Speaker 00

  Is there better architecture for certain types of data? Or did you have, you know, applications in mind?

• 6:47Speaker 03

  Yes.

• 6:48Speaker 00

  Besides the sort of general architectural concept?

• 6:50Speaker 03

  Yeah, so it really can be applied to pretty much everything. So just like kind of transformers, these are applied to everything. So can these sort of models. Over the course of research over a few years, we kind of realized that there are different advantages for different types of data. And lots of different variants of these models are better at different types of data or others. So the first type of model we worked on were really good at modeling kind of perceptual signals. So you can think of text data as kind of a representation that's already been really compressed and like tokenized, right? Recooked. Yeah, sure. And it's kind of like very dense. Like every every token in text already has a meaning. It's it's kind of just dense information. Now, if you look at like a video or an audio signal, it's highly compressible.

• 7:34Speaker 03

  It's, for example, if you sample at a really high rate, it's basically like it's very continuous. And so that means it's compressible. And it turns out that, yeah, different types of models just have different inductive biases or like strengths that modeling these things. The first types of models we were looking at were really good, actually, at modeling kind of these raw waveforms, raw pixels, things like that, but not as good at modeling text. And transformers are way better there. Newer versions of these models like Mamba, which was the most recent one that's been out for a few months. That's a lot better at modeling the same types of data as transformers. Even there, there's subtler kind of trade offs. But yeah, so one thing we kind of learned is that in general, there's no free lunch there. So people think that like you can throw a transformer at like anything and it just works.

• 8:19Speaker 03

  Actually, it doesn't really like if you try to throw it at like the raw pixel level or the raw sample level in in audio waveforms. I think it doesn't work nearly as well. And so you have to be a little more deliberate about this. They really evolved hand in hand with the whole ecosystem of the whole training pipeline. So it's like the places that people use transformers, the data has already kind of been processed in a way that helps the model. For example, people have been talking a lot about tokenization and how it's both extremely important, but also like very counterintuitive, unnatural, and has its own issues. That's an example of something that's kind of developed hand in hand with the transformer architecture. And then when you kind of break away from these assumptions, then some of your modeling assumptions no longer hold.

• 9:03Speaker 03

  And then some of these other models actually work better.

• 9:06Speaker 00

  Do you think of the advantage as like natural fit that translates to quality for certain data types? At least if we think about like, let's say perceptual data or, I don't know, rich or raw, pre-cooked, not pre-cooked data, or like, you know, how do you think about efficiency or the other dimensions of like comparing the architectures?

• 9:25Speaker 03

  Yeah, so I guess so far we talked kind of about the inductive bias for the fit for the data. Now, the other reason why we really cared about these is because of efficiency. So, yeah, maybe we should have led with that even. So, people have yelled for a long time about this like quadratic scaling of transformers. One of the big advantages of these alternatives is the linear scaling. So it just means that basically the time it takes to process any new token is basically constant time for a current model. But for a transformer, it scales with the history that you've seen. This is obviously a huge advantage when you're really scaling like lots of data. But it is actually something that's sort of a little bit of like a no free lunch thing. The fact that the transformer is processing, is taking longer to process things, also means that there are things that it's better at modeling.

• 10:09Speaker 03

  So this is kind of what I was talking about, there's some like subtleties when you're talking about the trade-offs there. One way that we've sort of been thinking about it more is kind of thinking of, so as I mentioned in the beginning, we think of these state space models as kind of being fuzzy compressors. And I think maybe kind of like the bulk of the processing should be done there. But at the same time, it benefits from having some sort of like exact retrieval or some cache. And that's exactly what a transformer is. So one way to think about the transformer is that you're processing all this data and it's just memorizing every single token it's seen, basically. I mean, some kind of, you know, representation of it, but it's literally remembering every single thing you've seen. And you're allowed to look back over all of it. So that's why it's a lot slower, but that could be useful. But probably that shouldn't be what the bulk of your model is doing.

• 10:54Speaker 03

  Kind of the same way that like, I mean, again, using like very rough and probably not accurate analogies, but like the way a human brain is probably, most of the intelligence is in this, you know, it's this statefulness, this real-time processing unit. But it is helpful to augment it with some sort of scratch pad or like lookup ability retrieval, right? And so these Ideas are actually quite synergistic and what people have recently been doing is finding that combining them into hybrid models tends to work really, really well. Seems like it's better than either of them individually. So, and interestingly, kind of also maybe in line with that intuition, people have found that the optimal ratio tends to be mostly SSM layers with a little bit of attention.

• 11:39Speaker 03

  So maybe a ratio of like 10 to 1. I know of at least probably like five groups that have independently verified that this is kind of the optimal ratio of things. So, yeah, I think it makes intuitive sense.

• 11:50Speaker 01

  Are there specific domains that you're seeing initial applications of these hybrid approaches?

• 11:54Speaker 03

  People are mostly using this on text because that's what everyone cares about. I think they've been investigated a bit on some other things. So I actually just heard from some collaborators today that they applied a MAMA-based model on DNA modeling. They're basically bringing this idea of foundation models to DNA, which is kind of this new idea.

• 12:14Speaker 01

  Sure, I'm just wondering like, because DNA is just a, do you mean translation of DNA into proteins? Is it a protein folding model?

• 12:23Speaker 03

  No, so what you do is you can like pre-train a model on long DNA sequences and then fine tune it or use it downstream on things such as even just like-

• 12:30Speaker 01

  But DNA itself just encodes proteins and RNA that fold into certain molecular shapes. So that's why I was wondering what the problem set is.

• 12:37Speaker 03

  Yeah, there is a bunch of them and I'm honestly not familiar with every single, like, a lot of the exact details.

• 12:43Speaker 01

  Yeah, I used to be a biologist, that's what's interesting.

• 12:45Speaker 03

  Yeah, yeah, that's right.

• 12:46Speaker 01

  That's my background.

• 12:47Speaker 03

  Well, just like current, my brain can't handle biology.

• 12:51Speaker 01

  I think I get it. I was just curious, like, what the specific application area was.

• 12:55Speaker 03

  Oh, this is not protein folding per se, but probably more like classification tasks. Like, one thing that people are interested in is like detecting whether like point mutations in DNA, like what downstream effects that can have and stuff. But I'm not sure the exact classification setting, yeah.

• 13:09Speaker 01

  And then one of the areas that you folks really started focusing on from a company perspective is tech to speech and voice. How did the research read into that domain?

• 13:17Speaker 03

  We were kind of interested in showing the versatility and the actual use case of these models. Previously, it was done mostly in academic context. And at CMU, my students are still carrying that fundamental research forward. But we were pretty sure this would just work in a lot of places that are interesting. And so we think it will really work on all sorts of data, but audio seemed like a pretty natural fit at first for some of the benefits, like what we talked about is much faster inference and so on. And so doing streaming settings and so on are a natural fit. We just thought this would be a cool first application. Maybe Karin can say more about it.

• 13:54Speaker 02

  Yeah, I mean, there's so many applications that are interesting for these models because they're so generically useful. And I think part of the challenge is picking the ones that are most interesting and impactful long term. Obviously, DNA is an interesting one, but it doesn't personally motivate us as much, or we would have worked on DNA. But I think, to me, the things that are interesting about multimodal data are really the places where SSMs have the most advantage, which is that you have data that's very, very information sparse. Compression is actually an advantage because you can stream data through the system really fast and then process it very quickly.

• 14:37Speaker 02

  So you update this sort of memory. And so being able to handle very large contexts is something that you want by design. The other thing that I think is really interesting about audio is that I think commercially, there's a lot of very interesting applications where audio is starting to be important. I think both on the voice agent side and being able to interact with your system in a more natural way like you would with a human is something that a lot of people want to be able to do, because there's a lot of places where you don't want to type into your computer, you actually want to talk to a human. Even on things like gaming and stuff, I think it's really interesting to think about how in the future you will be essentially replacing... Graphics and rendering with essentially models that are outputting streams of data in

• 15:20Speaker 02

  real time. So I think the real time aspect of it is sort of really core to signals and sensor data and of which audio and video are both very important. And I think audio in particular for us felt like a very natural place to start because of, well, I did do some work on audio in my PhD, which was also something that was helpful. And I think there's just so many applications in audio that are really emerging right now that require these types of capabilities to exist. So I think that's particularly exciting. The other piece I think that's really interesting about SSNs and that we're very excited about and, you know, are trying to do is the fact that like the model is so efficient that you can hope to put it on smaller

• 16:00Speaker 02

  hardware and actually push inference closer to on-device and to the edge. And I think that so far the theme in a lot of the models that people use has been data center, very big model, lots of compute, lots of GPUs being burned. I think that in the long run, like what we would hope is that you're pushing this closer and closer to the edge, you're actually using much less compute to do inference and you're actually able to basically reproduce a capability that maybe, you know, today costs a million dollars in the data center in $10 on a commodity GPU or accelerator at the edge. So I think that will be a very powerful shift because essentially what

• 16:41Speaker 02

  it means is that instead of running batch oriented workloads in the cloud, you're basically pushing the processing and the intelligence to closer to where the data is being acquired and where the sensors are. And that's kind of what you want because, you know, if you think about like security cameras or any kind of like really sensor that's deployed, you really do want to be able to kind of sift through the information very quickly, discard what's not useful because most of it isn't, and then really kind of remember all the stuff that is and use that to do prediction and generation and understanding problems. So I think that's the theme, I think in general for what we're trying to build is is sort of the the infrastructure to be able to train these models, make them run fast, and then bring them closer and closer to kind of

• 17:25Speaker 02

  Be very edge oriented rather than cloud oriented.

• 17:28Speaker 01

  That's super interesting. I guess as part of the recent Apple announcement, they mentioned that a lot of the models that they're running on device for three billion parameters.

• 17:34Speaker 02

  Yes.

• 17:34Speaker 01

  So in size and so you really have to focus on.

• 17:36Speaker 02

  Yeah, so it is partly like in my head, like it's always like two waves, right? Like there's like the first wave of companies that came was sort of really about like, how do we figure out if we can do something interesting, right? You know, nobody knew that scaling to this amount of data and compute would be interesting. So somebody took a bet there and did that. And that was great because now we have these all these great models. I think the second wave is always about efficiency and that's been the case in computing as well. We're like, you know, now we have phones that can do so much, you know, powerful work. And similarly, I think for models, what you would want is the smartest model ever, but run it real cheap. So you can run it repeatedly at scale.

• 18:16Speaker 02

  You can like run it a hundred times where you might run it once today. So a lot of that needs to, needs to happen. So I think that's what's interesting. So the three D models are interesting, I think, but they're like, they're still small and not very capable. I think the question is like, how do you make the capabilities just very, very good, but then have that low footprint on device, all of that. So I think that's where like the technology that, you know, we've been kind of playing with for the last few years and then building with now has really huge potential to actually be kind of the default to run these workloads because I think that part of the challenge with transformers is the fact that, you know, if you try to take a L L M seven B and try to run it on your Mac and you open your profiler, you will notice that the, you know, tokens per second goes down and the memory goes up.

• 18:59Speaker 02

  So I think that's, you know, obviously not great and power and all these things aren't something that people have like, even I think in the data center, people are talking about it now in the last year or so, but now, you know, you'll start to see more of that conversation shift. And so I think that's kind of where we want to be, which is like, you know, the future will be more intelligence everywhere. And how do you kind of enable that piece? I think is kind of what we're excited about.

• 19:20Speaker 00

  Yeah, I think we get really different applications if people start making that assumption, right? As you see, as you said, like, we see it in the data center first, where even as an investor betting on applications or, you know, full-stack companies that say like, it costs a great deal to do 1,000 calls per query right now, but we're just going to assume we can make it cheaper and we can focus on quality first. I think when you assume that you can run the model, or if you make it possible to run the model on hardware everybody has already, then you just get very different applications continuously.

• 19:53Speaker 02

  Yeah.

• 19:53Speaker 00

  With that quality, without that cost and ongoing compute being a problem.

• 19:56Speaker 02

  Yeah, just the set of things you would want to be able to do will change because, you know, in the same dollars, you will be able to just do way more intelligent computation. And so I think that's cool. I think the way that, you know, you run games on your computer and the games are like very power, like very, very rich and interesting, like, how do you kind of bring models to that place where, you know, if you, if you think about on-device, like, I shouldn't be able to have a music model on-device and that should be my personal musician that I can like talk to and get it to play whatever I want and I don't need to, you know, go to the cloud to do it. So I think these are all things that should be possible and just require like this type of infrastructure and work.

• 20:33Speaker 01

  Yeah. And Cartesian recently launched its initial sort of text-to-speech product and it's really impressive in terms of performance and how fast you've gotten to ship something really is that performing. Could you tell us a little bit more about that launch and that product?

• 20:46Speaker 02

  Yeah, I think it was sort of a natural, you know, transition for us to kind of now start thinking about how to put the technology to work because, you know, there was a lot of pre-work that happened and Albert continues to do the pre-work for the next set of things. But I think it's sort of like, how do you kind of build an efficient system that will allow you to do, for example, in this case, voice and audio generation. So I think the way we're thinking about it is we're building these fairly general models inside the company that allow us to kind of do fairly generic tasks very efficiently. So in this case, it's audio generation and then being able to condition on things like text transcripts. The philosophy is like, oh, audio generation is a problem, needs to be very efficient, needs to be very real-time. And so we need to kind of work on the ground.

• 21:30Speaker 02

  Work there to build the sort of the model stack. And then we need to have great training stack so that we can actually train a model that's high quality that people want to use and and that actually has a really great experience. So when we were putting together the sonic demo, it was sort of like we wanted to show that the the tech that we were using really can kind of give you something that's really interesting. And text to speech is very interesting to me because, you know, people have been building text to speech systems for the last, you know, probably thirty, forty years. There's constantly improvements happening and yet we're not at ceiling, right? Like, there's still so much more you can do in the in this area.

• 22:02Speaker 00

  Can you actually talk about that? Because I I think a lot of people would say like, that feels a lot more solved in the last year. Just text audio generation.

• 22:09Speaker 02

  I feel like.

• 22:10Speaker 00

  Like, what's left between here and the ceiling in terms of thinking about the application experience?

• 22:14Speaker 02

  Yeah, I think like the way I think about it is like, would I want to talk to this thing for more than thirty seconds? And if the answer is no, then it's not solved. And if the answer is yes, then it is solved.

• 22:23Speaker 00

  Okay.

• 22:23Speaker 02

  And I think most text to speech systems.

• 22:25Speaker 00

  Karan's audio touring test, yeah.

• 22:28Speaker 02

  Are not that interesting yet.

• 22:29Speaker 00

  Yeah.

• 22:30Speaker 02

  You don't feel as engaged as you do when you're talking to a human. I I know there's other, obviously, other reasons you talk to humans, which is, you know, sorry, I don't want to come across as a as crazy here, but yeah, there's a society that we live in, so so we want to talk to people for that reason, obviously. But but I do think the engagement that you have with these systems is not that high. When you're trying to build these things, you really kind of get so into the weeds on like, oh, I can't say this thing this way. And it's like so boring when it says it that way. And how do I control this part of it to say it like this, you know, the intonation?

• 23:01Speaker 01

  Are there specific dimensions that you look at from an eval perspective that are that you think are most important in terms of how you think about?

• 23:06Speaker 02

  Yeah, evals for, you know, generation are generally challenging because they're qualitative and based on sort of, you know, the general perception of someone who looks at something and says, this is more interesting than this. And so there is some dimension to that. But I think for speech, like, you know, emotion is something that matters a lot because you want to be able to kind of control, you know, the way in which things are said. And I think the other piece that's really interesting is the how speech is used to embody kind of the roles people play in society. So like, different people speak in different ways because they have, you know, different jobs or work in different, you know, areas or live in different parts of the world. And that's sort of the nuance that I don't think any models really capture well, which

• 23:49Speaker 02

  is like, you know, if you're a nurse, you need to talk in a different way than if you're a lawyer or if you're a judge or if you're a venture capitalist, you know, very different forms of speech.

• 23:58Speaker 00

  The highest form of voice.

• 24:00Speaker 02

  So those are all very challenging, I would say. So it's not solved, is my claim.

• 24:04Speaker 03

  There's also an interesting point, which is kind of like even just for your basic evaluations of like, can your ASR system like recognize these words? Or can your generation, can your TTS system say this word? Even that is actually not quite a local problem. And for a lot of hard things, you actually need to really have the language understanding in order to process and figure out what is the right way of pronouncing this and so on. And so actually to really get like perfect, even just TTS or like speech to speech, you actually really need to have like a model that has more understanding, like at least of the language. But kind of like, it's not really an isolated component anymore. And so you have to start getting into these multimodal models just to even do one modality well. And so that's kind of like somewhere that we were kind of eyeing from the beginning as well.

• 24:50Speaker 03

  And we were kind of using this as an entry point into building out the stack toward all of that. And hopefully that's all gonna, you know, it's gonna help the audio as well, but also start getting other modalities into that.

• 25:01Speaker 01

  That's really cool. I mean, I guess you've done so much pioneering key work on the SSM side. How has multimodality or speech really impacted how you've thought about the broader problem? Or has it, and it's more just the generic solutions that ones that make sense?

• 25:13Speaker 03

  I don't think multimodality by itself has been kind of a driving motivation for this work, because I kind of think of these, these models I've been working on as like basic generic building blocks that can be used anywhere. So they certainly can be used in multimodal systems to good effect, I think. Different modalities have presented different challenges, which has influenced the design of these. But I always look for kind of like the most general purpose, fundamental kind of like building block that can be used everywhere. And so that's, like multimodality is more of like a sort of a different set of challenges in terms of like, how are you applying the building blocks to that? But like you still use kind of the same, the same techniques and they mostly work.

• 25:53Speaker 00

  Given that like versatility of model architecture, generality of the building block, like what's, what do you do next for Cartesian? You focus on like the headroom for sonic and audio? You work on other modalities?

• 26:05Speaker 02

  You know, I'll take that one.

• 26:08Speaker 02

  You know, we're obviously really excited about the Sonic work because I think it kind of shows the first example of something that we're excited about, which is it's a real-time model. You can run it really, really fast at low latencies and it's capturing this idea that you want to generate a signal of some kind. So we're going to continue to obviously improve that piece. Also, you know, just generally things that folks want out of speech systems that need to get built that are orthogonal to the technology piece, which is, you know, being able to support lots of languages and just generally providing more controls. Orthogonal access that's really important for generative models, which is, you know, how do you kind of add more controllability in general to the system?

• 26:48Speaker 02

  So you can kind of get the desired output that you want. So that's obviously one focus for us is how to kind of put that piece in. Few things that we're doing that I think in the short term are really interesting. One is bringing Sonic more on device. You know, you can run the model real time in the cloud. Wouldn't it be cool if you could run it on your MacBook and it ran real time? And it, you know, was just, just as good. I actually have a demo I can show there that I think is super cool. Over time, what we want to do is what Albert said, which is that, you know, audio benefits from text reasoning and, and, you know, the ability to kind of converse with these models and actually have them understand what you're saying beyond just, you know, superficial understanding is very important. So what we want to do is enable that piece next, which is, you know, you, you should be able to have a conversation with this thing

• 27:32Speaker 02

  and actually be able to have it respond to you intelligently and reason over data and context in order to do that. And so Sonic, I think of as sort of the output piece of that in some sense, which is like, what does the response for that model look like? And then there's the input piece, which is ingesting audio natively into these models and doing that kind of thing.

• 27:49Speaker 01

  Is the intention then to train a large-scale multimodal language model on your side as well?

• 27:54Speaker 02

  Yes, but you know, we have our own sort of set of techniques that we're developing in order to be able to do that effectively. I think that I will maybe leave for a, for another podcast. But I think, yeah, I think that is the intention at the end of the day is build a great multimodal model, but then make it really, really easy to run on device and make it really cheap to run. And, and really focus on kind of the audio piece and making that as good as possible. Because I think that's sort of where, you know, the fidelity and the quality that you get from SSMs is just very different than what you've been able to see.

• 28:27Speaker 01

  That's pretty amazing because it seems like a lot of the limitations right now in terms of different application areas or use cases where text to speech is basically the Extra latency around trip associated with pinging a language model in the middle?

• 28:38Speaker 02

  Yeah.

• 28:38Speaker 01

  As you go from speech to text, the text and then out. And so if you do have multimodality, then obviously that shrinks the time on the inference side dramatically and that has a huge impact in terms of the...

• 28:47Speaker 02

  Yeah, I think the latency is going to be a big thing, theme there because it's obviously like quite painful to orchestrate multiple models to do this piece. And then I think also just the orchestration itself adds so much overhead. It turns what is, in my mind, something that the model should do into an engineering problem that requires so much orchestration and just engineering work to...

• 29:11Speaker 01

  It feels almost inelegant.

• 29:12Speaker 02

  Yeah.

• 29:13Speaker 01

  From a computer science perspective.

• 29:14Speaker 02

  Yeah, maybe that's, you know, some of the, thematically, the general bias here, which is, you know, the inelegant things we're trying to chip away at.

• 29:24Speaker 00

  In the end, all the systems go away and it's just one model.

• 29:26Speaker 02

  And then we also go away, apparently.

• 29:30Speaker 03

  People ask me like, how do I treat my research problems? And I can't explain, my answer is just aesthetic. It's just like, there's something that I find elegant and aesthetically pleasing about things and to me, that's almost the most important thing. And that's kind of driven a lot of these things too. So like I said, like for how did SSM come about in the first place? It's just like, I just felt like there was something like really like nice about it, like elegant about it and I just want to keep working on it. And I'm continuing to try to like do that, like find like the simple, nice solutions to hard problems, but it's not always possible. So our criteria, we of course need to solve the actual like the engineering challenges and there's always going to be hairy things. But as much as I can, I'm always trying to strive to kind of like make everything simple,

• 30:15Speaker 03

  unified as possible.

• 30:16Speaker 01

  That's great. Yeah, I remember, I can't remember, is it Erdos or somebody used to talk about certain theorems coming out of like God's book.

• 30:23Speaker 03

  Oh, yeah.

• 30:24Speaker 01

  Or so elegant.

• 30:25Speaker 03

  Yeah, I very much adhere to that idea. So it's called proofs from the book, is what he would say.

• 30:32Speaker 01

  Yeah.

• 30:32Speaker 03

  And that's actually kind of thing that kind of guides a lot of... The way that I like picking trees and problems. And what you're referring to is, of course, in like in pure math. Sometimes you see like proofs or ideas that just feel like this is obviously just the right way of doing things. It's so elegant. It's so correct. Things are not in machine learning world, things are often not nearly that clean. But you still can have still the same kind of concept, just, you know, maybe a different level of abstraction. But sometimes certain approaches or something just seems like the right way of doing things. Unfortunately, this thing is also kind of like, it can be subjective. Yeah, sometimes I tell people this is this, this is just the right way of doing it.

• 31:15Speaker 03

  And I can't explain why, but you know, maybe we should kind of have like one of our pillars should be about the book. So I can start saying this.

• 31:23Speaker 00

  Let's see the demo.

• 31:24Speaker 02

  Yeah, I'd love to show you.

• 31:28Speaker 02

  Cool. Yeah, I have a, you know, our model running on our standard issue Mac here. Basically, this is, you know, our text to speech model. Sonic on our playground is running in the cloud. And so, you know, part of what I talked about earlier was how do you kind of bring this closer to on-device and edge? And I think the first place to start is your laptop and then hopefully bring it, shrink it down and bring it closer and closer to smaller footprint. So let me try running this.

• 31:55Speaker 04

  It's great to be on the No Priors podcast today.

• 31:58Speaker 02

  You know, we have the same feature set that's in the cloud, but running on this and.

• 32:02Speaker 00

  Prove it's real time and not cooked. Say, you don't have to believe in God, but you have to believe in the book. I think that's the Ardoche quote.

• 32:08Speaker 03

  Was that the quote?

• 32:08Speaker 02

  Let me grab a interesting voice for this one.

• 32:12Speaker 00

  Ardoche's, where's Ardoche from?

• 32:14Speaker 03

  Hungary.

• 32:15Speaker 00

  Hungary.

• 32:16Speaker 01

  I mean, that's a default guess for any mathematician.

• 32:18Speaker 00

  Oh, yeah, sure. He's just a student.

• 32:20Speaker 02

  All right, I'm going to press enter.

• 32:22Speaker 04

  You don't have to believe in God, you have to believe in the book.

• 32:27Speaker 00

  That's pretty good.

• 32:28Speaker 03

  Lancy is pretty, pretty good, so.

• 32:30Speaker 02

  Yeah, it works really fast. And I think that's, that's part of what I think gets me really excited, which is like, you know, it streams out audio instantly. So, yeah.

• 32:38Speaker 00

  I would talk to Ardoche on my laptop.

• 32:41Speaker 01

  Yeah, me too.

• 32:42Speaker 03

  That'd be a great way to get inspired every morning.

• 32:45Speaker 02

  Yeah, I know.

• 32:46Speaker 00

  Yeah.

• 32:46Speaker 02

  Get on and.

• 32:47Speaker 01

  Yeah, that'd be great. Your team is now how many people?

• 32:49Speaker 02

  We are 15 people now.

• 32:51Speaker 00

  And eight interns.

• 32:54Speaker 02

  Sarah always gives me shit for this. It's a big intern class.

• 32:56Speaker 03

  Yeah, we have a lot of interns. I really like interns.

• 32:59Speaker 02

  They're great. You know, they're excited. They want to do cool things, so.

• 33:04Speaker 03

  And are there specific roles that you're currently hiring for or adding up?

• 33:07Speaker 02

  Yeah, we are hiring for, you know, model roles specifically. We're hiring across the engineering stack, but really want to kind of build out our modeling team deeper. So always looking for, you know, great folks to come to Team SSM and help us build the future. The rebellion. Yeah, the rebellion. Yeah, we used to actually call it.

• 33:27Speaker 03

  Yeah, it's, what do we call it? Overthrowing the empire?

• 33:30Speaker 02

  Yeah, yeah, yeah. That was the theme during our PhDs. And yeah, I would love to continue to, you know, have folks inbound us and chat with us if they're excited about this technology and the use cases.

• 33:42Speaker 03

  A lot of exciting work to do, both research and bringing it to people.

• 33:47Speaker 02

  Yep.

• 33:49Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.