Pod Insights Eye On A.I. #232 Sepp Hochreiter: How LSTMs Power Modern AI System's About this episode In this special episode of the Eye on AI podcast, Sepp Hochreiter, the inventor of Long Short-Term Memory (LSTM) networks, joins Craig Smith to discuss the profound impact of LSTMs on artificial intelligence, from language models to real-time robotics.
Sepp reflects on the early days of LSTM development, sharing insights into his collaboration with Jürgen Schmidhuber and the challenges they faced in gaining recognition for their groundbreaking work.
He explains how LSTMs became the foundation for technologies used by giants like Amazon, Apple, and Google, and how they paved the way for modern advancements like transformers. Topics include:
- The origin story of LSTMs and their unique architecture.
- Why LSTMs were crucial for sequence data like speech and text.
- The rise of transformers and how they compare to LSTMs.
- Real-time robotics: using LSTMs to build energy-efficient, autonomous systems.
The next big challenges for AI and robotics in the era of generative AI. Sepp also shares his optimistic vision for the future of AI, emphasizing the importance of efficient, scalable models and their potential to revolutionize industries from healthcare to autonomous vehicles.
Don't miss this deep dive into the history and future of AI, featuring one of its most influential pioneers.
(00:00) Introduction: Meet Sepp Hochreiter
(01:10) The Origins of LSTMs
(02:26) Understanding the Vanishing Gradient Problem
(05:12) Memory Cells and LSTM Architecture
(06:35) Early Applications of LSTMs in Technology
(09:38) How Transformers Differ from LSTMs
(13:38) Exploring XLSTM for Industrial Applications
(15:17) AI for Robotics and Real-Time Systems
(18:55) Expanding LSTM Memory with Hopfield Networks
(21:18) The Road to XLSTM Development
(23:17) Industrial Use Cases of XLSTM
(27:49) AI in Simulation: A New Frontier
(32:26) The Future of LSTMs and Scalability
(35:48) Inference Efficiency and Potential Applications
(39:53) Continuous Learning and Adaptability in AI
(42:59) Training Robots with XLSTM Technology
(44:47) NXAI: Advancing AI in Industry
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Speaker 00 Now you have a task to solve. Perhaps was it a man or woman crossing the street? And man was at the beginning of the sentence. Now you go one time step back and look up was it man or woman? And then you go the next time step back now. The contribution becomes smaller by a constant factor. Perhaps the first time you go back it's 0.9 times the thing. Then it's again 0.9. 0.9 times 0.9 times 0.9 goes to zero. Then you say it's a encounter. That's a word man. The contribution is zero. You cannot see. You do not track a class element. Meaning if you go back in time, the contribution gets less and less and less. And that was the problem. Ideally, the contribution would be the same.
0:46 Speaker 00 No matter what is the word is the beginning of the sequence or the end of the sequence, it might contribute equal to the answer. And that was the idea first I discovered the vanishing gradient. One sensor idea came of the long short term memory. Here, say, here was can I build something? Construct some architecture? If we go back in time, it's scaling with factor of one. My name is the Pochreiter. I'm leading the Institute for Machine Learning in Linz. I'm a university professor and also a founded company called NXCI, which is dedicated to industrial AI. I'm very well known in this community
1:29 Speaker 00 because I invented LSTM, which stands for long shot memory. And I did this in my diploma thesis. Where my supervisor, who was Jürgen Schmidt Huber, said, yes, we can only networks. Do not work, so I cannot store something. I can memorize the last words. If you say a man in a red shirt crosses the street. Perhaps you only memorize crosses the street. But if you say the street is crossed by a man in red shirt, then you know it's the man.
2:14 Speaker 00 In speech, you can permut the words and it's the same meaning. Therefore, memorizing is important. It's also important for time series and so on. At this time, it was 1991. Recurrent network couldn't memorize. And then I was digging into the problem. And what I discovered was a problem which is now known as a vanishing gradient. So when I think gradient was also a reason why deep neural networks were not possible. Because what I discovered, the gradient, which you need for doing learning,
2:58 Speaker 00 vanishes if you go back in time and therefore you cannot see what happens at the beginning of a sequence. The same if you go through layers of a neural network. Also here's a gradient vanishes and you have the same problem. Server nicer recurrent network were possible, not deep networks were possible. And this was the problem of the vanishing gradient.
3:21 Craig S. Smith? Can you explain vanishing gradient a little more for listeners who are not mathematicians or statisticians?
3:31 Speaker 00 Yeah, it's it's it's a kind of credit assignment. Now you have a task to solve.
3:42 Speaker 00 Perhaps in my previous example, was it a man or woman crossing the street? And man was at the beginning of the sentence. Now you go one time step back and look up. Was it man or woman? And then you go the next time step back now. So contribution becomes smaller by a constant factor. Perhaps the first time you go back, it's 0.9 times the thing. Then it's again 0.9, 0.9 times, 0.9 times, 0.9 goes to zero. Then you say it's if you encounter the word man.
4:21 Speaker 00 So contribution is zero. You cannot see it. You do not recognize the moment. Meaning if you go back in time, so contribution gets less and less and less. And that was the problem. Ideally, so contribution would be the same. No matter what is the word, that's the beginning of the sequence or the end of the sequence, it might contribute equal to the answer. And that was idea first I discovered the vanishing gradient when sense idea came of the long short term memory. Here, say, there was, can I build something? Construct some architecture. If you go back in time, it's scaling this factor of one.
5:05 Speaker 00 Meaning it's equal important. If you go one step back, it's equal important. Go another time, back, it's equal important. And this was LSTM architecture is called the memory cell. The memory cell was a construction. Where you can go back over time and the credit assignment. How much something contributes to the final answer remains the same. And I build an architecture around this idea around the memory cell, which is called LSTM. And now it was possible to store information of hundreds or thousands of time step. It was not possible before. And then LSTM was used in many different applications.
5:48 Speaker 00 It was used at some point in every cellphone, in every Android cellphone, in every Apple cellphone. It was was used in Alex's side. It was used by By2 and Alibaba. Every where LSTM is for language, for speech and language. And so first large language models, yeah, LSTM models. Today we heard a talk, a test of time award. And this was Ilya Sutskefer, one of the founders of OpenAI. Got it, where he used an LSDM for sequence to sequence learning, which is actually translating.
6:28 Speaker 00 LSDM were everywhere and Google was at this point known as an LSDM company.
6:35 Craig S. Smith? Yeah, can I just ask on vanishing gradient and I'll edit this, so don't worry. Because I'm a journalist, not a machine learning guy. Is that related to gradient descent?
6:50 Speaker 00 Yes.
6:50 Craig S. Smith? So gradient descent, you're, you're, you're working step by step to find an optimal, right?
6:59 Speaker 00 Exactly, but it's gradient descent, it's the gradient. You want to compute the gradient to do gradient descent. But what turns out is that the gradient is super tiny. And if you have a tiny gradient, gradient descent doesn't work anymore because the direction cannot be determined because it's so tiny number. Your gradient is too small to do gradient descent, so gradient descent does not work anymore.
7:25 Craig S. Smith? Yeah, I see. And yeah, so LSDM was really the original
7:34 Craig S. Smith? working neural network, right?
7:36 Speaker 00 It was especially for sequences, for speech, language. It was only LSDM, everybody used LSDM, all big companies, Amazon, Microsoft, Apple. Everybody used LSTMs in every cell phone and LSDM was.
7:55 Craig S. Smith? Yeah, and what fascinated me about your, your talk is that you've continued working on LSDM. And, and in AI, there are these ideas that, that are the primary focus for a period of time. And then another strategy, another algorithm, another architecture comes along and that becomes the focus. But it doesn't mean, I mean, that's what I. I talked to Isabelle Gil about it doesn't mean the support vector machines are useless. There there and and there's still things that can research it can be done on them the same with uh
8:40 Craig S. Smith? with uh well Boltzmann machines ultimately led to uh deep nerd networks but can you talk about continuing that research and how frustrating is it to have the community then turn their attention elsewhere?
8:57 Speaker 00 Uh yes, uh this is uh uh two thousand seventeen. Uh uh there was a paper called Attention is All You Need. Yes. Say dear of attention. Was the attention was obvious to guess was LSD apps. The attention mechanism and LSD. It's ever this paper. And say we can do only attention looking back in in time with out and LSDM. And doing this, I could paralyze the whole thing much better. So I can push much more data into the learning algorithm. So I can use larger models and then uh uh this architecture of attention is called transformer. The transformer technology replaced LSDM.
9:41 Speaker 00 But LSDM were still uh used for example, uh OpenAI had this Dota uh five uh uh agent and it was a big LSDM network. Uh uh deep mind has the Starcraft uh AlphaStar uh uh network. It was an LSDM network. And this year uh is a uh in the nature paper, uh method come out to predict floodings or droughts. Uh and this is based on an LSDM network and it's done by Google. Google uh uh can have a Google app and you can look up is your home
10:24 Speaker 00 uh in danger of floodings, do have uh do take precautions uh or is everything okay? And even so, yes, government and the Canadian government the official method now is this LSDM networks to predict floodings or droughts. So the LSDM network still worked, but where's the where not used was in language was for the large language models. And we saw LSDM is a very, very. Good technology it excels everywhere but not in language. What's the main reason was the data you could not push enough data. And for x L S D M.
11:06 Speaker 00 Set revival of L S D M in language. We looked at can we do the same as with the transformer technology. Can we scale it up, can we build a very big network? And what does some limitations of L S D M. And we identified three limitations. One was it could not revise storage decisions if I store something. I cannot later say, oh, no, it was not correct. I want to store something else. This is what we did, this is a technique which we call exponential gating. The second thing is we build a huge memory. L S D M had only very tiny memory, only one number was stored.
11:50 Speaker 00 And we extend the memory. And also funny thing comes, the memory is a hopfield network. Perhaps you were, there was this thing which is called Nobel Prize. And John Hopfield got the Nobel Prize for his classical hopfield networks. And we use in Toronto, L S D M, a classical hopfield network. The coaches, it's very fast and very efficient. But equipped with L S D M, things like input gate, output gate. We merged hopfield network and L S D Ms to get an excellent system. And now it has much more memory. And the third thing was to make it parallelizable like the transformer.
12:34 Speaker 00 With this, the ingredients exponential gating, having a good efficient memory. And making it parallelizable, we saw we can be as good. S-Transformer in volksam ist er seven billion
12:52 Speaker 00 Model regime but wir have a big big advantage or advantages wir much cheaper cheap in terms of energy or in compute and we are faster because we don't have to compute so much wie also faster wir faster and more energy efficient both in training but especially in inference if you see the for user is using we can accommodate more users at the same time or we can
13:24 Speaker 00 have one user but much with much less compute with lot much less energy es ist es a huge advantage of this new technology compared to the existing technology which is a transformer but what I also have to say for XL is the language is not our prime target our prime target is using this technology because it's so powerful to go into industrial applications because language is not at the core of many companies it's nice for customer relations or PR or stuff like this but with XL is the which is so fast we can go in robotics
14:07 Speaker 00 transformer cannot go in robotics or so tried but we are
14:13 Speaker 00 not fast enough we have two advantages we are fast enough to go in robotics and we can fix the memory because we have a fixed memory compared in contrast to transformers and this is fixed memory we can adapted to the hardware which is available which is a smaller hardware and and robots and server we can go to robotics to embedded system like drones to embedded system like autonomous production systems and here we know it is the core of some industries here in the core at some businesses of of companies and here we see a huge huge advantage because we
14:58 Speaker 00 A priori, no, how large a memory is and we can go to the embedded system on the machine and we are fast enough to do it real time, that we can really control a robot or autonomous thing in real time.
15:16 Craig S. Smith? Is it, is it a light enough?
15:20 Craig S. Smith? In X L S T M model, could it be light enough to have on the edge?
15:25 Speaker 00 Assets where we want to go, the guys from Apple asked us, and so we have to look into it, what are the requirements? I think yes, if something can do it, then it should be X L S T M because it's memory efficient. Also the energy efficiency, if you have it on your cellphone or on the edge, you don't have these huge batteries with you and if it's energy, more energy efficient, it's also super helpful to use it. And therefore I think that would be my main target for this kind of models. Because also look into self driving cars, you cannot have these big batteries.
16:06 Speaker 00 We are in contact with some car companies, that would be an ideal system for us to have it in the car. Because we don't have so much energy in the car because we have only a battery. And we should work in real time, we should react very fast, both we can accommodate, both we can do. And I'm super optimistic that we have a super cool thing here.
16:30 Craig S. Smith? Yeah. Going back to attention, so attention existed.
16:38 Craig S. Smith? And forgive me because I don't know this history, where did attention come from? Where, who, who developed that?
16:46 Speaker 00 So transformer was 2007, attention is all you need, but the tension idea, it was always in combination with, with,
16:58 Speaker 00 it was in combination with L S D Ms, so we are different developments. So we are something like, let's have an external memory. Value access. Also Alex Graef had an idea some which was called Sanjure Turing Machine on LSDM. Which Turing Machine and from sister action. Some sense is harder tension. I store something and I really get an address to what I should attend to. That was Jason Weston. As a memory networks, I was memory networks. It was always can I store some information?
17:41 Speaker 00 LSDM has something in its memory, but it's time to memory was very small. And I wanted to have a bigger memory. Now this excellent is the only hop field network. What's used separate memory mechanism in addition to LSDM. To store more information from the past and several different ideas out. And then several additional tensions. There were multiplikative tensions, several different attention mechanisms. It was always can be integrated into LSDM. More powerful memory. But I didn't replace a memory cell. I started with the memory cell, it was constant.
18:25 Speaker 00 Not mentioned gradient, constant gradient going back. So didn't touch it because I don't want to destroy that. But you can build a memory directly into LSDM. This is what we did. See, I did a separate memory. The attention came and since I saw the separate memories powerful enough. And we don't need LSDM anymore. This was 2017. Attention is all you need. But in 2015, the first attention ideas were already out.
18:55 Craig S. Smith? I see. And so the the X LSTM, where you have an LSDM and a hop field network to expand the memory.
19:05 Craig S. Smith? And it's faster. How is it trained? When you say a seven billion parameter model, is it trained in the same way that a large language model is trained?
19:17 Speaker 00 It's exactly the same. Uh, we use the same backbone. And where's our tension mechanism was located? We now replaced this with an LSDM. But we're saying LSDM is a huge memory, uh, exponential gating and with the possibility to parallelize it. And we did the same. We looked what this transformer guys are doing. How do they train their networks? We did the same. We have to do some adjustments because we have a specific architecture. But the major thing came from the attention backbone, which is a resnet. It's we use, we only replaced in the big network.
20:00 Speaker 00 It's one item which is quadratic in context length. Also in this linear and we are much faster. By the blazing, this compute intensive part by a less compute intensive part.
20:14 Craig S. Smith? When did you do this work? Is this very recent?
20:20 Speaker 00 It's very recent. As attention came up, as I said, I said, oh, probably we can do the same with LSTMs. What's the development with transformer attention was so fast. Said we never had the chance uh, to do something. And since the problem was so, uh, models became so large. Said we, at the university, did not have the compute. Then I looked at some developments and sit down, computed everything through this, this HOP field networks with this idea. And then I went to the press and said, I found you're here. Excel SDM, I need money to try it out.
21:06 Speaker 00 And then after a while, there was a local private money, which gave us money and we founded this company, this NXY company was founded to build the Excel SDM. And the first month, we invested 10 million euros on the compute to see whether Excel SDM works. I cannot do this at the university, I needed private money. And it worked very well. Then we said, let's build a larger model, let's go to the billion model. And it worked better than I dreamed, it's so fantastic. And I'm so proud. At the first, all the big companies have used it,
21:51 Speaker 00 but I never saw any sense, nothing. Now I have the second chance, this Excel SDM, but in the company we have the IP rights and stuff like this. We try to use it on our own and bring it to industry. But it is so fantastic. It's really working like a dream.
22:14 Craig S. Smith? Yeah, that's amazing. And you say it's not particularly, it's not built for language, but can it handle language?
22:23 Speaker 00 Yes, it's as good as this language models, like you know Lama, Lama 2, Lama 3, it's better. Yeah, but it's also much faster. It's my energy efficient. But in the meantime, there are better models than Lama. It's in the middle field of this language models. But language, there are so many models, so many things. It's a big chance to go in industrialization. To have a very fast, real-time method, energy efficient, to go to the core business. Because this language, you often not as a core business, you don't make so big money.
23:06 Craig S. Smith? Yeah, yeah. And yeah, so give me a use case, an industrial use case for Excel SDM.
23:17 Speaker 00 What was already done, not by me, by others,
23:23 Speaker 00 some implemented Excel SDM in controlling drones.
23:30 Speaker 00 End it works fantastic. It works better since a transformer it works better since states by a model it works for them very good. It was not my idea, it's right it out. Asus used it for time series prediction. It's was better or even financial time series. And reported in a paper, it's published it. It's much better than transformer. Much better than classical time series method. And it's was another. I'm very proud, another reinforcement of. That's the method really works well.
24:10 Speaker 00 And a couple of papers, it's a trial it out and it's always good news. They have new good results. And now I hope it's getting more and more popular. More people are using it in see the power of this new method. I'm so excited.
24:31 Craig S. Smith? Yeah, I can imagine it's particularly since you had this huge success. And then transformers kind of eclipsed it. And now you're you're coming back. The is it open source or how? How are you approaching that?
24:51 Speaker 00 We made everything open source. Meaning the model is open source.
24:58 Speaker 00 So inference code is open source that you can use the model. But also the training code is open source. Everything is open source and now the fun is in cars and not the fun is in also we are very proud. We are fast as a flash attention. I don't know where's the flash attention is, so where the fast transformer technique. And it turned out that we are faster. And being faster is because we have this we can't know network here from LSDM. What we did is we have chunks of flash attention. And if you can decide how large the chunks are. You can better make use of the GPU of the graphic processing unit of the chip. If you have to do everything in one big flash attention thing.
25:41 Speaker 00 You have to squeeze it in, you can do this optimizing. But we have more smaller chunks and can optimize the chunks for the GPU. And now we are faster than flash attention. Because everybody thought I told before transformer become popular because they could be paralyzed and they are faster. Hey, and now we are faster, we are faster, sensor fast is transfer. It's unbelievable, never thought that this is possible. But we achieved it, we really achieved it.
26:12 Craig S. Smith? On the drone, excuse me, on the drone example.
26:19 Craig S. Smith? Is this, is this model on the drone or is it communicating with the drone remotely? And, and what exactly is the model doing? Is it guiding the drone? Is it guiding it through obstacles or can you talk about that application?
26:44 Speaker 00 I'm not a part of the team, it's somebody else. I was only told it's fine, but it can be wrong. It's our threat, our device, our throne, and it's guiding, controlling the throne. It's out, external, I have probably remote for safety reasons. But the idea is the whole system is on the throne. It's a different lab. I did not publish it until now. Sony told me, I will be very successful and that's Sony information have.
27:22 Craig S. Smith? Yeah, and so NXAI, is that how you're pronouncing it or do you call it NXAI?
27:29 Speaker 00 Yeah, we are not sure, until now we call it NXAI.
27:36 Speaker 00 Elon Musk has something which is called XAI. Have to shoot quality next XAI or a new XAI, I don't know. But right now it's NXAI. It's a company dedicated to industrialization of AI.
27:58 Speaker 00 It started with this Excel SDM, we need money to show, it's a cool technique. It's a toll that really worked out more than I ever dreamed. But we have a second pillar, we have a second thing. And set AI for simulation. It's also in the company, AI for simulation is here's the numerical simulations. Here many particles or your for mesh over your car. We're say as flowing. And it's the numerical simulations are restricted if the number of particles goes into millions or hundreds of millions. So the matrix becomes too complicated. You have too many computations.
28:43 Speaker 00 But AI can be faster, can be one thousand times faster, can be hundred thousand times faster. Because AI can see meta structures. I give you an example. You know, the moon is circling the earth. So moon has many, many particles and so on. But we describe the moon by its location, by its impulse, perhaps its mass only few parameters. And we have a very good prediction where the moon is tomorrow in an hour. And this is a very high abstraction. And many physical systems
29:23 Speaker 00 have, let's say, a set of particles which form one structure like quills
29:34 Speaker 00 or other structures. And I recognize this structure or only think you throw a snowball. You don't have to simulate every snowflake, but the ball. And if they recognize, hey, this is a snowball. Let's simulate the whole ball at S1.
29:56 Speaker 00 Senior much faster because one hundred thousand particles, one million particles are now one thing and say here in simulation is AI recognizes macro structures, simulates of macro structures, and it's much faster, but it's almost the same precision, it's in America the simulation which goes down to peeps atoms. And it's a simulation is super important, you can do it for the food industry, if you store something like in a hopper in some compartments, how it's flowing out, you can do it if you have some
30:37 Speaker 00 particles and you blow air into it to mix different kind of particles, you can see how the mixing, as it makes good, if you have the particles have different sizes, different structure, and all kinds of things, you need the simulations in Linz, we have big steel in industry, what is big opens of the steel industry, the simulations, the American simulations are not sufficient because there are so many particles, there are so many atoms to simulate, that doesn't work anymore, but AI can do, and here we see, this is simulation, a huge
31:21 Speaker 00 field in industry, in other companies, where we can do simulations, where the American simulations, structure, where the American simulations are not fast enough anymore. Set the second thing, Excel is the M, it's a new technology, it can be used for everything, and simulation using AI is the two main pillars we have in some new company, and XCI, dedicated to industry.
31:59 Craig S. Smith? Uh, the transformer scaled very well. But uh, it could have been anything that scaled well. He, he was saying that it's, they, they worked with the transformer because uh, it, it, it did what they needed to do, but he felt that it could have been a different uh, core algorithm.
32:23 Speaker 00 Yeah.
32:24 Craig S. Smith? Is, is that what you're finding with LSTN?
32:27 Speaker 00 Exactly. At the beginning, I had the same feeling, this is the attention at the core. One LSTM or something else, it's the scaling up, building a very huge network, storing all the information in the uh, uh, base, in the connections, as in, sets the important thing to build up a big network. Say just transformer, so I need this at once, this now, so transformer is everything else would perhaps also have worked, it's quadratic in the context length. Say just a little bit more complex thing and it's complex in time, but it's simple in the interaction because you have a query and you have keys, every query looks at one
33:11 Speaker 00 key, you have only per weihs interactions. This LSTM you have everything in a memory and a new query index with the whole memory. So you can have also more complex, more abstract uh, interactions, but I decided for this, I think also it's also go, but now if you look uh, in detail, perhaps it's better to have something more compute efficient, something which uh, does not need so many resources in terms of computational cycles.
33:43 Craig S. Smith? Yeah, uh, and, and so you've scaled this so far to seven billion.
33:49 Speaker 00 Exactly.
33:50 Craig S. Smith? And, and but there's no reason if it's faster and more computer efficient.
33:56 Craig S. Smith? Presumably could you scale it beyond what uh what they've done with uh L L M's?
34:05 Speaker 00 Yes, uh I think so, that we can do it. Uh we have to see whether we have enough money, enough compute to do that because uh we are looking uh for investors and so on, we we need money to do that. But I think we're in a good place because we are now faster, we can do even uh larger models. But the question is, how large should the model be? Also, Ilya today said, perhaps it has an answer scaling. Yeah, also my talk said, there was the scaling. As there is no more data available, we used all uh the language data up and uh using uh this existing uh methods, a new language data.
34:46 Speaker 00 Also, that's not work because it's not new information, it's old information, all language only trans uh transformed. Also, we don't have new data. And here I think uh we are saturating because we uh we uh used up all language data we have also from uh Chinese, uh Japanese, uh across the world, it was all translated. Uh if we want to build larger models, but we need also more data. But the data is, I think, is a limitation, uh we don't have more data. And I think this is an answer scaling. Now we have to think more in inference. Uh and uh OpenAI has this strawberry model.
35:27 Craig S. Smith? Yeah.
35:27 Speaker 00 Where uh you should think more or what I think is more. There's information in the model, but you have to get access to, you have to uh pick uh tweaks a model a little bit to get out what you want to have. And that's in inference. And now, so cool thing is, XLST is fast in inference. Exactly here, we have an advantage. We can do much more, uh if the transformer can uh uh do one hundred tokens, we can do uh ten thousand tokens in the same time. And here for inference, also we would have an advantage. And therefore, very hopeful perhaps still being in language. I said, we want to go to industrial application, but language is interesting after the one, after the strawberry of OpenAI.
36:13 Speaker 00 I'm super, super uh uh uh hopeful, said we uh have a system which can think much faster.
36:23 Speaker 00 And also systems are direction, I want to look into it.
36:26 Craig S. Smith? Do you think that uh that XLSTM has promised to to challenge
36:37 Craig S. Smith? the large language model paradigm? Where in in a few years, we'll be talking about XLSTM models instead of
36:48 Craig S. Smith? large language models?
36:50 Speaker 00 Depends. I don't know how everything is work because this
36:55 Speaker 00 existing technology is very good integrated in the system, in the hardware. Nobody wants to change a working system. But if I say you save a lot of money because you need less compute. I don't know whether we are too late to change because everybody bought already the language models, the
37:20 Speaker 00 product on the market. I don't know, but that seems very certain feels we have an advantage. But we can change things which are already fixed. I don't know, but in principle, it would be possible. You have a model. It's a GPT, it's a transformer model. Now you replace it by an XLSTM. And you have the same performance, but pay less energy costs. But I don't know, it's it's the marketing and sure.
37:53 Craig S. Smith? On on the training data. You know, I I heard Yann Le Cun speak yesterday. And he was talking about this problem. And how
38:08 Craig S. Smith? you know, we need to develop methods to collect data from the real world.
38:15 Speaker 00 Yes.
38:15 Craig S. Smith? Not filtered through language, through various sensors, through real time video, through haptic sensors, through.
38:28 Craig S. Smith? I don't know what else there is, but audio, I suppose.
38:32 Speaker 00 Yeah.
38:32 Craig S. Smith? Is, could XLSTM handle different modalities?
38:37 Speaker 00 Yes, absolutely. It can. And sets also with simulations, where we go. Simulations, we also need data. Where comes the data from numerical simulations, which simulate nature very precisely. Then we can generate more and more data. Very slowly, because AI methods are faster, but if we had enough, then we have enough data to build them. And also from sensors. If we want to use XLSTM in different applications, we rely on this sensor data. Exactly like Ian said. I think that's the future. That's where we get new data. We don't get new data in English,
39:21 Speaker 00 but we get new data in many different applications, because we can build more and more sensors, which just record what's happening in the world. For machine, how a machine is in the world. And then we can do the same as we do with languages, use like an XLSTM to build a good model. A new model, which can control a machine, can make decisions, so it's a flex. I think that's the future.
39:53 Craig S. Smith? And the learning is this? Are these models, as they?
40:00 Craig S. Smith? Do they continuously learn after the training phase? Or are they then fixed?
40:07 Speaker 00 That's your decision. You can do both. But normally, what you do, you fix them. Because if the user is training these models.
40:18 Speaker 00 You don't know what's happening. It could be very risky if you can also train a car to stop at green and drive at red because if you are very bad teacher to a car and for some safety reasons you don't want them to learn. But on the other hand, there could be machines, where the machine dynamics change over time because some parts worn out or not precise anymore. But since the model can adjust to the machine or adjust to the better parts in summer and then it becomes winter.
40:59 Speaker 00 It could have advantages, but one have to look into application. You can let it learn, but you have to see is it dangerous that the user is in control in learning something.
41:13 Craig S. Smith? Yeah, I wasn't thinking about consumer facing obligations, but for example industrial robots.
41:22 Speaker 00 Yes.
41:23 Craig S. Smith? You know, again, CoVarient, this company founded by Peter Beal, their dream is to have, and I think they already have them, network these industrial robots deployed around the world and as they encounter new situations and and learn to deal with new situations, that learning then is sent back to a central model and then deployed to all of the robots. So it's, it's, you know, you know this learning. Is that something that could happen with with X T L S M?
42:03 Speaker 00 Yes, Excel is then can do that, but it can do it even better. Right now with robotics, you cannot learn on the robot because systems are too big to climb on. But with Excel system, you can do it embedded, you can build it on the robot and an individual robot can even learn. You can use the data, pull it and learn a new big system, but.
42:28 Speaker 00 With Excel would be possible that each robot learns, adapts to its own environment, to its own task. And that's the thing what we have in mind for example is
42:43 Speaker 00 if we have industrial robots here of over 100,000 millions of items to build.
42:53 Speaker 00 But for many smaller companies you have only perhaps 500 products or 700. Now it's not, you don't program a robot because it doesn't pay off to program a robot. If we have a robot which can learn, perhaps you show a robot, you have to do this screen here, do this, do this. You should do a tour to the examples. It's a robot that's the 100 examples. Here Excel is then would be good. It's our robot. It's the foundation model, it knows how to screen and do stuff. But now you have to say what is item you have to do exactly. Several 100 of them and it's doing this.
43:38 Speaker 00 And that's possible right now because in that robot can do it. But the program it costs more than doing, let human, human doing. And here with Excel system we can put it on the robot. And server it can learn more in context because you would do some demonstrations from human into the context. And now it's in context it will do a smaller piece, a smaller number of pieces. And here is the huge potential.
44:16 Craig S. Smith? Yeah, have you, is this deployed already on industrial robots?
44:22 Speaker 00 No, no, because we come out with a language model. We first wanted to show our new technology is powerful enough. Our new technology is as powerful as existing technology which is called transformer. Now if we have shown this and everybody believes us, we will go in this different applications.
44:44 Craig S. Smith? NXAI, uh, when was it founded?
44:48 Speaker 00 It was founded end of last year, uh, uh, December last year.
44:53 Craig S. Smith? Yeah, and how much money have you raised so far?
44:57 Speaker 00 Uh, we have internally, I cannot reveal everything, but uh, uh, couple of tens of million euros, uh, but uh, now we wanted to start uh, second founding around. We didn't start uh, uh, around to get new investors in.
45:21 Speaker 00 In our first, put out a seven B model and uh, uh, seeing how it's going. And now we hope money comes in to go the next step. Because now we have proven we can do it. And I'm really, really optimistic that it's really gross to the roof.
45:43 Craig S. Smith? Yeah, are you, are you satisfied with the attention you're getting from the AI community? On this LXLSTM, I mean, I was really impressed by the talk that you gave. Uh, but I have no way of judging, uh, yeah.
46:02 Speaker 00 Pepsi for better ways than me, I don't know, because I also, I'm in the bubble. Uh, uh, everybody which uh, who's coming to me, say, hey, a cool thing. And also I used Excel, it's really good. Uh, but uh, I don't see the big picture. And we have to see how many downloads we have, how many publications come out, how many other success stories are there? If there are more and more and more success stories, it works. But uh, we have to see, I'm not sure how uh, was it really gross or was it, it's, yeah, I don't know.
46:38 Craig S. Smith? Sure, uh, do you think that there are other? Algorithms out there yet to be discovered that can scale and are maybe even more efficient.
46:51 Speaker 00 I think it's a field is open again. There is also states based models like Mamba, compared to Mamba, we have advantages to Mamba. It's a nice thing is Mamba 2, the newest version of Mamba is an XLSDM without exponential gating. We can watch to the same architecture even, we have a little bit more performant architecture. But this was also funny because they came from sister direction, we came from another direction. But we discovered the same things which work, but there could be other things out there. And also building on top of XLSDM is quite new.
47:34 Speaker 00 A handful of researchers have developed it. We don't know what is potential, in what direction can go. Everything is possible here. Mr. Transformer was the same, it was discovered, published. And so many, many, many ideas from the researchers come in. And here I think also there's a huge potential. For example, you can modify some memory. Here for the transformer, there was something like called prompt engineering. Here you can modify some memory. If you say now you are friendly in the past, now you are, we are the technical in the past. We can directly modify memory. I don't know what's in there.
48:16 Speaker 00 I know it for biological application, what's it did, so learned. Sort of proteins. Put it into memory and took some memory to generate something else. I can text this memory, this memory or combination of memories. I scan over, store information in my memory. And then I pull it out. It's a complete new ideas possible here.
48:44 Craig S. Smith? Is there anything I haven't asked that you think I should ask?
48:49 Speaker 00 Oh, that's, that's, I don't know what, what the audience is interested in. I, the main message, message came over, I hope, said there's a new technique. Everybody can use it because it's open source. Everybody should try it out, play around with it and you will have very rewarding time to check it. I don't know what's missing.
49:17 Craig S. Smith? Yeah, yeah. And on the reasoning, you mentioned strawberry or O1.
49:25 Craig S. Smith? You, you will, will, will you develop a reasoning capability? The sort of long thinking capability that OpenAI is working on?
49:39 Speaker 00 Yes, we also think going in this direction, I hope I did not reveal what's I really doing. But also from DeepMind, from Asa Research Groups, some ideas out how to do so, reasoning, how to do the thinking. We will also try out different things, which are out there because we see an advantage. That we are so much faster in inference and it should help us. But we will try out many things. And if OpenAI wants to reveal what's I have done, that's we also build it.
50:13 Craig S. Smith? Yeah, on the, on your site, is there a public facing demo or?
50:21 Speaker 00 Not now, I think it's because we built a seven B model, but didn't have time to find you need to find your chat or whatever. That's not what we had time right now. We are super proud that we got this cool model.
50:38 Craig S. Smith? Yeah, okay, so we, we've got to watch you develop.
50:41 Speaker 00 Yeah.
50:42 Craig S. Smith? NXAI, what's your website, NXAI?
50:46 Speaker 00 It's, oh, I, I have, I don't know it out of the mind. NXAI, no, NX-AI dot com.
50:58 Craig S. Smith? Yeah, NX-AI dot com.
51:00 Speaker 00 Yes.
51:01 Craig S. Smith? Yeah, well, I'll follow it. Okay, this is fantastic. I'm really glad that I got you up here.
Previous #231 Paras Jain: The Future of AI Video Generation with Genmo Next #233 Matt Price: How Crescendo is Disrupting Customer Service with Gen AI Page updated November 5, 2026 · transcript and summary are machine-generated
