Sepp Hochreiter - LSTM: The Comeback Story?

Sepp Hochreiter, the inventor of LSTM (Long Short-Term Memory) networks – a foundational technology in AI. Sepp discusses his journey, the origins of LSTM, and why he believes his latest work, XLSTM, could be the next big thing in AI, particularly for applications like robotics and industrial simulation. He also shares his controversial perspective on Large Language Models (LLMs) and why reasoning is a critical missing piece in current AI systems. SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting! https://centml.ai/pricing/ Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** TRANSCRIPT AND BACKGROUND READING: https://www.dropbox.com/scl/fi/n1vzm79t3uuss8xyinxzo/SEPPH.pdf?rlkey=fp7gwaopjk17uyvgjxekxrh5v&dl=0 Prof. Sepp Hochreiter https://www.nx-ai.com/ https://x.com/hochreitersepp https://scholar.google.at/citations?user=tvUH3WMAAAAJ&hl=en TOC: 1. LLM Evolution and Reasoning Capabilities [00:00:00] 1.1 LLM Capabilities and Limitations Debate [00:03:16] 1.2 Program Generation and Reasoning in AI Systems [00:06:30] 1.3 Human vs AI Reasoning Comparison [00:09:59] 1.4 New Research Initiatives and Hybrid Approaches 2. LSTM Technical Architecture [00:13:18] 2.1 LSTM Development History and Technical Background [00:20:38] 2.2 LSTM vs RNN Architecture and Computational Complexity [00:25:10] 2.3 xLSTM Architecture and Flash Attention Comparison [00:30:51] 2.4 Evolution of Gating Mechanisms from Sigmoid to Exponential 3. Industrial Applications and Neuro-Symbolic AI [00:40:35] 3.1 Industrial Applications and Fixed Memory Advantages [00:42:31] 3.2 Neuro-Symbolic Integration and Pi AI Project [00:46:00] 3.3 Integration of Symbolic and Neural AI Approaches [00:51:29] 3.4 Evolution of AI Paradigms and System Thinking [00:54:55] 3.5 AI Reasoning and Human Intelligence Comparison [00:58:12] 3.6 NXAI Company and Industrial AI Applications REFS: [00:00:15] Seminal LSTM paper establishing Hochreiter's expertise (Hochreiter & Schmidhuber) https://direct.mit.edu/neco/article-abstract/9/8/1735/6109/Long-Short-Term-Memory [00:04:20] Kolmogorov complexity and program composition limitations (Kolmogorov) https://link.springer.com/article/10.1007/BF02478259 [00:07:10] Limitations of LLM mathematical reasoning and symbolic integration (Various Authors) https://www.arxiv.org/pdf/2502.03671 [00:09:05] AlphaGo’s Move 37 demonstrating creative AI (Google DeepMind) https://deepmind.google/research/breakthroughs/alphago/ [00:10:15] New AI research lab in Zurich for fundamental LLM research (Benjamin Crouzier) https://tufalabs.ai [00:19:40] Introduction of xLSTM with exponential gating (Beck, Hochreiter, et al.) https://arxiv.org/abs/2405.04517 [00:22:55] FlashAttention: fast & memory-efficient attention (Tri Dao et al.) https://arxiv.org/abs/2205.14135 [00:31:00] Historical use of sigmoid/tanh activation in 1990s (James A. McCaffrey) https://visualstudiomagazine.com/articles/2015/06/01/alternative-activation-functions.aspx [00:36:10] Mamba 2 state space model architecture (Albert Gu et al.) https://arxiv.org/abs/2312.00752 [00:46:00] Austria’s Pi AI project integrating symbolic & neural AI (Hochreiter et al.) https://www.jku.at/en/institute-of-machine-learning/research/projects/ [00:48:10] Neuro-symbolic integration challenges in language models (Diego Calanzone et al.) https://openreview.net/forum?id=7PGluppo4k [00:49:30] JKU Linz’s historical and neuro-symbolic research (Sepp Hochreiter) https://www.jku.at/en/news-events/news/detail/news/bilaterale-ki-projekt-unter-leitung-der-jku-erhaelt-fwf-cluster-of-excellence/ YT: https://www.youtube.com/watch?v=8u2pW2zZLCs

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We need new direction. Large language models are not our way to advance AI. Language models is for me a database technology. It's not artificial intelligent. You grab all the human knowledge in text, perhaps all the code or whatever, and store it. Country is a reasoning is not real reasoning. It's repeating reasoning things or code things which have been already seen. Our path ends with scaling up. You put more training data, make it larger, but not smarter, not the systems. We are not different, so we're only larger. Something is missing, yeah, something is missing.

• 0:42Tim Scarfe?

  You and Jürgen, you're pioneers of connectionism in a way. And you're neuro-symbolic guys, you've always been. Why is that?

• 0:51Tim Scarfe?

  Sep, it's an honor to have you on MLST. Thank you so much for joining us today.

• 0:54Speaker 00

  It's an honor for me that you have me.

• 0:57Tim Scarfe?

  Oh, don't be silly. The thing that is amazing about language models and deep learning in general is that they capture a lot of the subtle intuitions, cultural information, creativity and so on. So they're really good for generating programs. And the thing is, if we want to do abstraction, we need to have programs. But the thing is, where do the programs come from? If we build systems that can create and acquire abstractions, we need to build systems that can write their own programs. It doesn't seem possible just to do some discrete program search because it's too difficult.

• 1:30Speaker 00

  What my fear in large language models is, large language models is for me a database technology. It is not artificial intelligent. Okay, give me the artificial, but set more less a database technology. You grab all the human knowledge in text, perhaps all the code or whatever, and store it. And you channelize it, you combine it, you know, if the user can replace a person because this days of the week, our names or our numbers, it's some kind of generalization. But it's things which already exist. The question is, do we need new code? Effektor oder die Ritten Samwer. Ja ohne Hefte Pull it together or to combine it.

• 2:15Speaker 00

  But if you really can can should come up bis der New Code bis der New idea bis der New Concept. Large language models can only pull out existing code. Self-trained on it's just not possible. Seiner Trainfreude to produce something new. And server is very limited. But server is powerful because AI needs a knowledge representation. And right now, so problem of hallucination. Okay, how to pull out your knowledge also bis inference bis strawberry. It's about so knowledge is perhaps already in the system. How do we get it out? It's a database where I don't know how can access the information. We need new direction.

• 2:59Speaker 00

  Large language models are not our way to to advance AI. And so long term, it's a good database technology. It's a good knowledge representation technology. It's important for AI, but we have to find new ways.

• 3:16Tim Scarfe?

  Could I could I challenge a tiny bit? So I completely agree. Vanilla L L Ms are approximate retrieval engines. And even that, they're not quite databases because they have this interpolative property. Things like O one, they are approximate reasoning engines. So they're doing this test time compute and they're they're searching many combinations. Because the thing is, even though this thing is a finite state automata as a fixed amount of compute. Does a single forward pass, but it can generate code. And the code contains all of these fundamental basis primitives that can be composed together. So you can do this test time search. You can compose together programs. You can, in a sense, search Turing space indirectly, right? By searching through the space of programs. So there seems to be something there.

• 3:58Tim Scarfe?

  Are you saying that doing these kind of methods like O one is the road to nowhere? And we need something completely different? Or could we just tweak it a little bit?

• 4:06Speaker 00

  You can tweak it and you you will go very far because you have. Let's train the program space because it's very nice. So many programs, so many combinations of programs which give you a new program. If you like, if you think in the call worker of complexity. It's a length of the program, some complexity of the programs, programs that simple call worker of complexity programs already stored or combined. But if you have to find a program which needs completely new concepts cannot be combined out of existing programs. I think it cannot do it, it can only combine things.

• 4:47Speaker 00

  It is already seen, large language models learned on code, but it cannot come up with completely new code concepts. Perhaps that don't exist, then, then if you say everything which in code was already invented, we only have to combine and say there's nothing new. Then I wish you, but if there's something new to invent, I don't think large language models can invent this.

• 5:13Tim Scarfe?

  MLST is sponsored by SentML, which is the compute platform specifically optimized for AI workloads. They support all of the latest open source language models out of the box, like Llama, for example. You can just choose the pricing points, choose the model that you want. It spins up its elastic auto scale. You can pay on consumption essentially, or you can have a model which is always working, or it can be freeze drive when you're not using it. So what are you waiting for? Go to SentML.ai and sign up now. Well, yeah, let me push gently on that. So I think this is a discussion about creativity and and also epistemic foraging. So, you know, creating new knowledge to explain.

• 5:52Speaker 00

  But also reasoning, programming is a lot about reasoning. But you have, I have some logic if you do this, this, this, this, but a complex logic that your program is working.

• 6:03Tim Scarfe?

  Absolutely, but if we say reasoning is knowledge acquisition. And we need systems to come up with new abstractions. And if we agree that those abstractions can be combinatorially deduced from abstractions that are already in the system, so we have the combinatorial closure, so they do exist, then creating the abstractions is more a matter of understanding when a good, you know, how do I find a good abstraction using an algorithm?

• 6:30Speaker 00

  But I think what we humans have, it's not only that we draw our ideas from coding, but understanding the world, having all this world knowledge. I think from coding alone, you're limited. I think we have much reasoning capabilities outside of doing any programs, but I agree, in programs, you can go very far. I think if it's a program, you need a lot of reasoning, a lot of logic to go to the next step, next step, next step. And it was not in your training database. I don't think the current large language model can do it.

• 7:14Speaker 00

  Right now, I don't believe they really understand reasoning, the concept, say, imitate reasoning, say, reproduce, they have already seen. But I don't know if it's understand it, and there are many examples. You change a little bit and then say go wrong.

• 7:31Tim Scarfe?

  Can you explain the difference between the kind of reasoning we do, you know, so strong reasoning, and the kind of reasoning that we can do in current AI?

• 7:39Speaker 00

  I think in current AI, the reasoning is not real reasoning. It's repeating reasoning things or code things which have been already seen in the input data and combining it and also replacing some variables. The reasoning we do, we have, we have this reasoning concept like contradiction induction, all these things we learned, in other words, hard to learn it in school or in study. But now we have reasoning concepts, how we can do, how we structure things and how

• 8:24Speaker 00

  we show that something is true or not true. All these formal systems, you have to have some formal rules. In theory, L L Ms might learn some formal rules. But then I can do reasoning in a very specific thing and produce new things because I only apply the rules. If the rules are in the training data, I can apply, I can apply the rules also to new things. But then in this reasoning system, I probably can reason, but if you go to another system, I lose the reasoning capabilities.

• 8:59Tim Scarfe?

  I mean, two quick points on that. First of all, would you consider, you know, move thirty seven in AlphaZero? Would you consider that reasoning? So in AlphaZero, you know, the Google, the Go player algorithm creatively discovered this amazing move that be-

• 9:12Speaker 00

  Ah, so now the move, okay.

• 9:14Tim Scarfe?

  Yes, yes. It created new knowledge.

• 9:16Speaker 00

  It creates new knowledge, but here there was a sub-symbolic part, it's a sub was Monte Carlo tree search.

• 9:28Tim Scarfe?

  Yes.

• 9:29Speaker 00

  Search is one AI, classical AI concept at the end of Monte Carlo tree search, you have the value functions and so on. But I discovered it, but by checking things and then evaluating it, yes, I think it's a combination between understanding the game and computing a lot of moves into the future with Monte Carlo tree search.

• 9:59Speaker 02

  To follow up, this is a new AI research lab I'm starting in Zurich. It is funded from Best Ventures involving AI as well. We are hiring both chief scientists and deep learning engineer researchers. And so we are Swiss version of DeepSick. And so a small group of people, very, very motivated, very hardworking. And we try to do some AI research, starting with LLM and O1 style models. We want to investigate reverse engineer and explore the techniques ourselves.

• 10:29Tim Scarfe?

  Even with that, I guess you could say it was still an approximate value function. There was no formal guarantees or anything.

• 10:35Speaker 00

  Exactly, that's true.

• 10:36Tim Scarfe?

  But I completely agree that LLMs on their own, that they are, they are approximate retrieval engines. But the thing is, we can build systems. We can have formal verifiers, we can have, you know, we can have like Lean and COCK, for example. We can build these systems. So with these systems, can we do reasoning?

• 10:53Speaker 00

  I think in principle it should be possible. I'm not sure. But I think the reasoning is limited to- So domain you see in the training data. Very different formal systems, formal logics. You can learn it one logics. If it sees enough of these rules. I think it can, it knows what variables, what it can change and how it can produce something. I think you can train an LLM for one logic system to produce new logic. But it's, it's, it's, it's, you learn the syntax. What you don't learn is the semantics. If you want to prove something, then you have to have different steps towards this proof. And here I think it's a good strategy.

• 11:38Speaker 00

  So I would learn to do so formal things that are syntax. If I, if I send it, if I have a correct formula, and I produce another correct formula. By applying the rules. It has learned it, it has seen it, it can do it. But it's not gold directed. It's a step by step, very, very still not perfect, or very still sometimes, or in most cases not as good as humans.

• 12:03Tim Scarfe?

  It's interesting, I kind of agree that knowledge is created in service of a goal. And there's a creative component to reasoning. And we can build systems that can dream and generate data. And we can bootstrap that. And some of it can come from the users of a system. So it feels like we can build systems that can reason. But perhaps it wouldn't have something that we have. Maybe we have something extra.

• 12:27Speaker 00

  But I would bet on this, that why should also far, why should you learn to reason, why not use a reasoning system, why not call a sub-program, can you prove this, or a theorem prover, mathematician, star flexes? You can learn it, perhaps it's also okay to learn it. But don't see the necessity, because also videos tools. Why should not this future AI systems use tools, use tools for everything? For math, for looking up knowledge and star flexes? For me it's stupid to push everything in one system, because also we don't do it.

• 13:12Speaker 00

  We know how to use our tools. Somehow I feel that's a better solution.

• 13:18Tim Scarfe?

  What has happened in the last two years since we spoke?

• 13:21Speaker 00

  A lot of things happened. For example, I founded a company, NXAI. It's a company dedicated to industrial AI. And also, ex-LSTM happened. It's this revival of the LSTM method, which now should compete with the transformer technology.

• 13:38Tim Scarfe?

  Yes, and we're going to get on to that. And before we do that, it would be good to go on an intellectual journey of LSTMs. But just before we get there, we've not really spoken about some more broad stuff. What was it like working with Jürgen?

• 13:54Speaker 00

  Jürgen is a very special person. He's very inspiring. He has charisma.

• 14:02Speaker 00

  I can tell you one story. At the University of Munich, where we both were, there was a seminar. And there were three persons. One tried to get all the students into multi-agent systems. She also in space cognition. And Jürgen in neural networks. Jürgen came and said, "Oh, I'm not prepared. I don't know what to do." Then Jürgen gave his introduction to his topic. Out of fifty students selected his topic. So he can convince persons. And it was fun. I was sitting there. I did programming. And Jürgen did his art things. Very made circles.

• 14:45Speaker 00

  And out of the circles, women appeared. And he did a lot of things. And he once told me, it was not clear for him whether he will go into arts or whether he goes into science. But it was always fun with him. Always inspiring.

• 15:07Tim Scarfe?

  Yes, and just in case the audience don't know, of course, you worked under Jürgen. And you are both two pioneers in the realm of artificial intelligence. It's insane. But what gave you the intuition all of those years ago to be working on the right things?

• 15:22Speaker 00

  Oblium in LSDMS with stands for long short from memory.

• 15:29Speaker 00

  Jürgen introduced me to neural networks, in particular into recurrent neural networks. What's it did not work? And immer Diplomas dieses, ich liebe es mal supervisor. He gave me some task, jetzt calls a chunker system. Where you have a sequence, in everything which you can predict. You can remove from the sequence because it's predictable anyway. So for your shortness of sequence, you can't do that. It's a what's your chunker system? But this was sort of a solution. Because we can only networks are not working. And then two things happens first. I build a neural network. Yeah, only one weight have to be adjusted. That's a way to store a piece of information which you need as a sequence and. And the network could not do this.

• 16:14Speaker 00

  I did all my print Fs, all my my coding. And the screen numbers flow over the screen. And so it's a super small numbers. And this was the gradients. There was no weight update. The gradients were not there. And this was sensor discovery of the finishing gradient. Said if you want to have some target. You want to know what is needed to predict the target. You would do credit assignment in the sequence and you got no signal. And finishing gradient, now in you buy a kind of networks don't work. And the solution was LSDM, so long short from memory. We have memory cell. Where I build something which makes sure that the gradients.

• 16:57Speaker 00

  It's a get propagated back, it's a transferred back. That's how it turns the scale, it's a the main same. Server at the sequence beginning. There was exactly the same gradient as the sequence. And so that's not finished in gradient anymore. And this was the architecture.

• 17:20Speaker 00

  Memory cell architecture, which is the core of LSDMS. And so I discovered LSDMS. I wrote it up in my diploma thesis. And later as Jürgen came back, he asked me, hey, you did something in your diploma thesis. Should be published in, so I published it.

• 17:38Tim Scarfe?

  Yeah, and it's been one of the most cited papers in the history of of deep learning. Very, very impactful paper. I mean, just reflecting back on it. What do you think the long term impact has been of LSD?

• 17:49Speaker 00

  I think it's still used in my keynote. I gave one example, six examples from this year for predicting floodings. Or droughts, especially floodings. LSDM's was a major model in the Google app. And also for predicting floodings, it's used by the US government, by the Canadian government. And here, LSDM works better than everything else. Better than transformers and so on. Openai built a big LSDM network as an agent or DeepMind with StarCraft. AlphaStar was a big LSDM network.

• 18:29Speaker 00

  LSDM became the major thing in language up to 2017. Everybody used LSDM together, also with attention. Attention was together with LSDM. And since the one paper came out, attention is all you need. Meaning you only need attention and you don't need to LSDM anymore. And since the transformer was born and the new technology took over language. But LSDM still were good performing in time series prediction, reinforcement learning, agents and so forth. But transformer were stronger in special language. This changed now again, hopefully.

• 19:11Speaker 00

  But at this time, transformer took everything over. So we're better parallelizable. You could throw more data to this model, learn on more data, so we're faster and it's a good not compete at this time.

• 19:27Tim Scarfe?

  How did the LSDM solve the trade-off between storing new data and protecting data that was already stored?

• 19:33Speaker 00

  That's very interesting questions. And that's also the strengths of the new XLSDM. So it is gating. I have, we have different gates. Some of the most important thing is the input gate. And it scales up or down new information which coming in. It can be scaled down to zero, it's not stored. Everything is stored and the input gate is something like an earlier tension mechanism. Because input gate is attention, you have time series and to what sequence elements you want to pay attention to.

• 20:11Speaker 00

  So input gate, what you said, and so forget gate, forget gate is saying, is already stored memory important or should I downscale it? But more important is the input gate, input gate really picks out specific sequence elements to be stored. And so non-relevant stuff is not stored. It was one of the first attention mechanism, but we called it gating.

• 20:38Tim Scarfe?

  Yes, just before we get to X L S T M, because of course you've got this amazing new invention, which solves many of the problems that the original L S T M had. Can you tell me maybe like the computational complexity of an L S T M compared to an R N N? How did it compare?

• 20:54Speaker 00

  L S T M is an R N N.

• 20:55Tim Scarfe?

  Yeah, vanilla without all of the gating.

• 20:59Speaker 00

  Yeah, it's also gating and the complexity is only increased by this gating mechanism, but it's still linear in time. Because perhaps it's better to compare to attention, attention if you have a new query and a piece of information, you have to look back to all previous items and L S T M only interacts with the memory, which is always stored memory. So it's always a constant for one query, you have a constant interaction with the memory. Attention have to go through all keys and have to do this per way as interactions. And so two disadvantages, first disadvantage is computational very complex, it's quadratic in the context length.

• 21:43Speaker 00

  And the second disadvantage is you have only per way as interaction. You do a dot product, then exponential of the dot product, which is since a softmax, but you have only per way as comparison. It could be better said, more tokens, more sequence elements pulled together and a new element interacts with.

• 22:02Speaker 00

  I would say an abstraction of this different tokens. Two disadvantages to transformer complexity plus where the simple interactions. But L S M is like a network, all the networks are linear.

• 22:21Speaker 00

  In the sequence, line in the context length. L S M is a little bit more complex because it has the gating mechanism. Which makes it more complex. But it's not as complex as the transformer versus quadratic complexity.

• 22:36Tim Scarfe?

  Could you explain to the audience why something which is quadratic, so you know, something that should be worse, but it actually ran faster. Why is that?

• 22:44Speaker 00

  It ran faster because of implementation on the GPU graphical processing units, the chips where everything is implemented on. We had something like it's called flash attention. It's a very fast attention mechanism. And you use hardware optimization.

• 23:06Speaker 00

  This is one thing, and the other thing is you could do it in parallel. One critic is looking back to all keys. But you can look back to all keys at the same time. You can do everything in parallel. You can push up all keys or also whole like assume your sequence, your sentence. And also words are pushed up one level simultaneously. While the current networks or L S M has to go over it sequentially. So first thing, build up in your memory. The next thing, build up in your memory. And attention could push up everything in parallel. And server attention was a system much faster than L S D M because of this parallelism.

• 23:51Speaker 00

  And the second thing was you can optimize it for the GPU for the hardware. It's two things, parallelism, paralyzing and hardware optimization. This gave attention a big advantage. You can train on much more data at the same time. And yeah, and L S M could not compete with this technique.

• 24:12Tim Scarfe?

  You mentioned flash attention as well. So can you just quickly explain that to the audience? So does that mean in certain circumstances, you don't actually need to do the full quadratic attention?

• 24:21Speaker 00

  It's still quadratic but super highly optimized. Right. Use assist fast memory to caches in in the thing. You use registers of of of the GPU. Where the where the fast entities memories in the register. It's still has the same complexity because it's mathematically quadratic and cannot cheat math. But you can do it super, super fast. Flash attention was super, super fast because it is hardware optimized.

• 24:54Tim Scarfe?

  Wonderful. So can you bring in X L S T M, which is this this new invention? And how does it overcome some of these problems with the original well, let's hear.

• 25:01Speaker 00

  Yes, I start with the spoiler. Because I talked about flash attention. We have this X L S T M faster than flash attention. Both in training as also in inference, especially inferences important. Now I go back. X L S T M. We after seeing it's this this rise of a transformer. So we thought, OK, first of all, could it could it not be L S D M? Because the rest not backbone architecture to build where the where the large models. Was this the key to have this feed forward connection?

• 25:46Speaker 00

  This is many parameters. We use the information. Meaning, is it important to build big models? Or is it important to have the specific technology looking back to compress the history? We thought L S D M should do it. And we asked the question, can we scale up the L S D M like transformers?

• 26:12Speaker 00

  And get the performance of transformers. But we know some limitations, some drawbacks of LSTMs. We already mentioned once, one, this was a parallel session. We know with LSTM also parallel. We use the same ideas like attention to parallelize LSTM. But there were two other limitations. One limitation was that LSTM could not revise decisions. If you store something and then you see something different, you can say, oh, this should have been stored. You cannot revise it. I give you an example.

• 26:54Speaker 00

  It's like you want to find new clauses, for example. Then you say, I know I found this clause with this price. And if you look further in the internet, you find new clause, which is even better. Plus surprise. Perhaps the clause should fit your shoes or whatever. And if you find something better, you should stray away, throw away what you already have stored. Both are largely similar to your shoes in also surprise. So and LSTM could not do it. If I found a better matching thing and have to remember the surprise, I have to delete everything. As the X LSTM can do this, and so it is exponential gating.

• 27:39Speaker 00

  To revise a storage decision, if you do this exponential gating, so it is if I find something better, I update. Very heavily and then I normalize it. So the old best solution is downgraded. And what is it? I can find something better and can throw away my old stuff. In theory, so forget, could do it. But in practice, you cannot learn to forget. Because you cannot learn at the same time to store very precisely and then forget it. One time step. But exponential gating, exponential input gating was the key. So I have something better, forget everything what was before. It's just an advantage.

• 28:23Speaker 00

  The second thing was a matrix memory, a big memory. And the original LSTM, the memory, the scala, one number. You have only one number what you can store. That's not much you can store. And now the new. LSDM, so XLSDM, has a whole hopfelnetwork. We use classical hopfelnetwork and it became popular again because there was something called Nobel Prize. And John Hopf, who got the Nobel Prize, was this classical hopfelnetwork. And now instead of this single scholar, we use a whole hopfelnetwork. It's like a classical hopfelnetwork plus gating.

• 29:08Speaker 00

  We must input what we should store in the hopfelnetwork. And we forget gate, how much should be stored items be down weighted. It's a hopfelnetwork equipped with gating. So for merge, so hopfelnetwork idea, this LSDM idea. And this gave us an LSDM, this much stronger memory, this much bigger memory. So for exponential gating was important. Increasing the memory, and so further or dimension, it was paralyzed. And this to the ingredients we used to build this new XLSDM. It was fantastic what results we gave.

• 29:52Speaker 00

  We didn't expect so good results, to be honest.

• 29:55Tim Scarfe?

  And I suppose that the memory mechanism is also reminiscent of the fast way programmers of the 1990s.

• 30:00Speaker 00

  Exactly. Yeah. This is, so we also like hopfelnetworks, it's always an auto product, product memory. Here for memory and here for new. Here two vectors, one we like in attention we call key. So we call value and here for auto product of key and value and access to some memory. It's this, it's idea, but what we did add on is an input gate to this new item matches added. And I forget get to the old memory. But it's a known technique, this auto product storage is also even older. For this icing models in the 70s, that already is ideas.

• 30:46Speaker 00

  And also a half of networks use the same idea. But also the fast rates use this idea.

• 30:51Tim Scarfe?

  Could you also just give a little bit more intuition on, you know, the gating? So you said you moved to exponential from a sigmoid. Certainly in the 1990s, people were using like sigmoid and hyperbolic tangent even for an activation function. What was the intuition at the time to go sigmoid? And just in a bit more detail, how does the exponential version fix the problem?

• 31:12Speaker 00

  To go sigmoid, it's gating. A sigmoid is between zero and one.

• 31:16Tim Scarfe?

  Yes.

• 31:17Speaker 00

  And so the natural thing to do, one is the gate is open, everything goes through. Zero is nothing goes through, the gate is closed. And between, you do scaling. So first the sigmoid was the natural thing using for gating. But it has a problem because if you encounter one sequence element and you say, I multiply by open five, let's say. So here can we not see elements, say, oh, it's just open five. I should multiply this before. And it's this and that because the sigmoid only goes to one. I cannot go higher, I cannot over rules. The sigmoid is limited and you have to make a decision. But later it's limited to higher values.

• 32:01Speaker 00

  The exponential gating is not limited. You always can do larger values. What's the problem is we never used in earlier days exponential activation function because learning would break down. But we have to have second ingredient. One is the exponential gating, but the normalization. You from exponential thing, but then you normalize by this exponential input gates. But it's like a soft max. If you remember how a soft max works, you have this to the power of something, you have this exponentials and then you divide by the sum of this exponentials.

• 32:44Speaker 00

  It's like a rolling soft max. And server, we went in direction of attention, this L S M, but it's recursive.

• 32:56Speaker 00

  But it's very similar here for exponential input gate but send you divide by the sum of all input gates. It's a little bit like a softmax, but it's different also and it's a another thing. It changes the learning dynamics. But we don't have clear understanding what's happening. We only saw if we have different architectures, the softmax business exponential function has give advantage learning dynamics meaning if also systems got stuck

• 33:35Speaker 00

  as old don't learn anymore. There's some gradient peaks which let's transform a learn. And we now observe the same as Excel SDM to device. So this decision was our reason. But the learning dynamics also we're modified in a positive way. But it's not understood completely what's happening here exactly. I think there's some random directions where you if nothing goes anymore, that you have some random weight updates which help you to progress the learning. But it's speculation exponential gating again, matrix memory and to parallelize it.

• 34:15Tim Scarfe?

  I'm just interested in what triggered the flash of inspiration. I mean, if you could go back in time and tell your younger self about this, would your younger self just say yes, absolutely? And would you have done it then?

• 34:25Speaker 00

  Yes, but my younger self had have to see a couple of examples because at this time we don't have this big language models. We don't have this problems where we see exponential gating helps. This big memory helps because we did not have this data sets. I know only had to say what to do, but also what data will come. So yes, of course, such as small storage, if you want to store much more, of course you have to do this. I would have seen it, but I also needed a glimpse in the future what data will come.

• 35:04Tim Scarfe?

  In your paper, you you studied how these things scale with with data and model size and so on. Can you tell me about some of the theoretical underpinnings?

• 35:11Speaker 00

  Standard scaling laws is not developed by us. We also do model parameters, increase model parameters and follow a certain like exponential law, a certain curve. And you compare it with like transformers or state space models which also follow a certain law. And then you can extrapolate and say if you build larger models, we will be also better. What is the scaling laws which we are used not invented by us, by others, which is nice because you can now predict

• 35:54Speaker 00

  if I make a model larger or if I use more data, how will these models, these larger models behave?

• 36:01Tim Scarfe?

  You mentioned state space models and things like Mamba. Could you just contrast to that?

• 36:06Speaker 00

  Mamba was most competitive method for LSTM. And then after our publication of XLSDM, Mamba 2 came out. It's a nice thing is Mamba 2 is XLSDM without input gate. It's exactly the same because it has it to the soft plus. It is soft plus is significant. You can do some math. Then you see also this forget get the help of an output gate. Mamba 2 is like XLSDM, but no input gate. Input gate is left out.

• 36:47Speaker 00

  So for nice to see the different method convert to the same architecture a little bit. So not service Mamba because the dot of input gate is important. So remaining architectures were the similar now. So started with state space models. We started with LSTM with Hopfield and Blabla. And now we more more converge to where the similar architectures.

• 37:11Tim Scarfe?

  Are you seeing any hints of industry adoption of XLSDM?

• 37:15Speaker 00

  Yes, first of all XLSDM is now faster. Send flash attention in inference also in training. I can tell you why. It's flash attention. You have to put all the stuff along context into the GPU. What we now do, we did chunks of flash attention and between the chunks we do the current stuff. And we design the chunks of flash attention that we can be more efficient on the TPU. If you have smaller chunks, you don't have to squeeze it in and do inefficient stuff. You can exactly make it so large like the caches. Many users of flash attention technology stole it from the skies.

• 38:01Speaker 00

  But to do the right size of flash attention, make it fast with the flash attention, recurrence, flash attention, we can. And now we are faster than if you do a whole flash attention of a whole context. And set both in training but also in inference. This is chunk-wise flash attention. It's called, or we call it, like this. And this gives us the speed. I didn't expect that we could be faster than flash attention in training. It's not, no way. But hey, it's unbelievable, fantastic. But set we are fast in inference, we know because here attention has also go auto-regressive

• 38:43Speaker 00

  because you have to produce a new word in generation and then push everything into your system again. And you produce a new word, you have to push everything into the system. You can cache some of the processing, yeah, you can do it fast. But attention is not for auto-regressive mechanism. In training, you have the whole sequence. Very fast, very not faster, unbelievable. But in inference, I was sure that we are faster. And this gives us advantage in different ways. First of all, I will mention something in language. You have this strawberry or one thing.

• 39:23Tim Scarfe?

  Oh, yes.

• 39:24Speaker 00

  It's doing more on the inference side. It's more thinking.

• 39:27Tim Scarfe?

  Yes.

• 39:28Speaker 00

  And on the inference side, we would be much faster. If we like one hundred times faster in inference, we can do one hundred times more thinking. And since it's a big opportunity, it plays in our hands, it was so nice. So they came out because we're exactly as fast in inference. They would be better. But this fast inference speed also helps us to go in industrial applications away from language. Language is not at the core of many industries. This is not quite many, we're so business many companies. But I can go into robotics.

• 40:13Speaker 00

  Transformer, we're used for robotics. Deep mind had a paper. Tesla had a paper. But they all struggled. So the transformer is too slow. Yes. Sometimes you have to wait a couple of seconds before the agent is reacting. No, we have something which is much faster. And we have second advantage. We have fixed memory. We know in advance how large our memories. If we're embedded and embedded device, we know how large our memories. We design the L S M, this is fixed memory. And no matter how long the sequence was, you use the same fixed memory. If the sequence is hundred sequence elements or hundred millions, you have the same memory.

• 40:58Speaker 00

  And we can fix the memory. And we're very fast. And this too, things give us advantage to go embedded. To go into robotics. Says even one already tried it out for drones. Selfs is devices, is G P U's on drones. And so email that don't want to reveal that it's unbelievable. It's much better results than also drones are flying autonomous. And you have to have real time control. You cannot wait until that thing. And this is excellent, it's working. We are so fantastic. And this guy also talked to me at Europe's. They said, I don't know.

• 41:42Speaker 00

  Was is I want to be because it's so good for the company.

• 41:47Speaker 00

  But the going robotics going to drones, also self-driving.

• 41:53Speaker 00

  In a car, you want to be energy efficient, you want to have also batteries with you. You want to be fast, you want to be concise, you want to have a small, powerful system. And here is the big advantages of this XLSDM. Perhaps even going to the cell phone. I'm not sure, I don't know the constraints of the cell phone perhaps is this too far fetched. But we have a thing which is energy efficient, it's fast. And we can control the memory, we can design the memory for the device, for the embedded device.

• 42:31Tim Scarfe?

  Do you think the XLSDM is sort of move us closer to something that resembles symbol manipulation?

• 42:37Speaker 00

  Symbol manipulation?

• 42:38Tim Scarfe?

  Yes.

• 42:41Speaker 00

  I don't know. We have a project about neurosymbolic AI.

• 42:48Tim Scarfe?

  Yes.

• 42:49Speaker 00

  Symbol manipulation. I would say in one sense, I think XLSDM is better in building abstractions.

• 42:59Tim Scarfe?

  Yes.

• 42:59Speaker 00

  What I'm missing for this AI system we have out there. I never saw an AI system build proper abstractions. It's always human made. The language is human made. So if you look at image, it's not a human put the object in the middle. I want that's a artificial system which comes up this new concept which is not human made. And XLSDM, I don't know, but it can do it. But in the memory by combining more tokens, by combining more from the past. Perhaps you can build a concept because it's more efficient to store a concept abstract concept. Since those are single items, like an attention would do those single items.

• 43:44Speaker 00

  If you can compress it to. Something for sun, beach, cocktail, and so on. So that's our Peps. Somebody's on the beach, holidays, and it's Peps one abstract concept. And the stores is Peps more efficient than storing single items. And the same should happen in industrial applications. Said you see concepts, you see structure and you stores a structure, not the single things. And if you have the right abstraction, you bet in generalization.

• 44:24Speaker 00

  Because if we have abstract concepts in future, we will encounter this abstract concepts again, hopefully.

• 44:31Tim Scarfe?

  Yes, the reason I ask is of course you've got your symbolic AI paper as well. And I'm really interested in your symbolic architectures. And there are many approaches to doing that. So in Europe, we've seen many people using transformers to generate programs. And some folks are just skipping the explicit program generation and just getting transformers to perform symbolic like tasks. And transformers are incredibly limited. They can't copy, they can't count, there's lots of things they can't do. But do you think the XLS TM could overcome some of these obvious computational sort of like limitations of the transformer?

• 45:04Speaker 00

  Probably some of them they can overcome, but I think the solution is to combine both. I think what we have right now is not the final solution. We have to go symbolic. And the other things out where transformer is perhaps using MATLAB to software equation or whatever, or is inquiring into internet, but I think we need both because there are so many symbolic techniques out there. We have fifty years we have developed and we should somehow integrate them. You say I don't know, but everything is learned by Peps in principle. But now a shortcut would use what's already there.

• 45:49Speaker 00

  Combine in the right way. And in Austria, the biggest AI project is about forty million euros. I'm leading it. It's bilateral AI. Bilateral because bringing symbolic and sub-symbolic AI together. Because we need, as I said in my talk, scaling is over. Now we have to go into industrialization of AI. And here we need new techniques. Perhaps not new techniques only from the sub-symbolic side, from the neural network side. Perhaps we need things from symbolic side to make things more robust.

• 46:27Speaker 00

  Because if the production process stands, what is told, that should not happen. As I said, perhaps sub-symbolic methods integrated or surrounding the sub-symbolic methods like large language models or others.

• 46:49Tim Scarfe?

  I completely agree. I think we need to build hybrid systems.

• 46:53Speaker 00

  Yeah. Yeah. That's a neurosymbolic approach. And that's what we're doing in Austria in this big thing. It's hard, it's hard to bring systems to communities together. Sometimes they don't like each other. So yes, we have big success stories. Yes, there's a success stories. But the thing sets away for things to advance AI. But also to make industrial AI as I said in my talk. Because for industrial AI, we need symbolic systems to make it robust, to guarantee stuff. We now have to team up with the symbolic guys to advance AI.

• 47:29Tim Scarfe?

  I completely agree. So we need to have formal verification. The thing is though, can we have our cake and eat it? Because the only problem with these hybrid neurosymbolic systems is the degree of human engineering. Can we automate the creation with some kind of architecture search? Because we're building these big, these big systems that have many, many components, many verifiers and so on. How much of that can we automate?

• 47:53Speaker 00

  In the group from Web2.0 symbolic

• 47:58Speaker 00

  symbolicgeist, heavy need machine learning, perhaps just a parameter for symbolic stuff.

• 48:05Speaker 00

  Subsymbolic guys say, "I can use the symbolic self, perhaps it's a spiel surrounding it." But, "Set and merge it, set and integrate it." Like, "Learning rules, learning new symbolic rules." "What's the rule things?" "I know how symbolic goes, but perhaps some rules are better."

• 48:25Speaker 00

  "You have to better integrate."

• 48:29Speaker 00

  "So things, but now it's two groups that are thinking in their own domains." "And I'm missing, and if somebody is doing this, "a text is from this community, this from this community, "clue it together, but it's clumsy, not nice, not elegant things." "Elegant things, also, as we did, is we learn some formal systems." "But perhaps the learning should go into the formal systems." "And the formal system should be a subcomponent, "an integrated subcomponent of, I don't know, large language model or whatever." "Right now, it's not there, it's two groups that are too separated."

• 49:12Tim Scarfe?

  Yeah, so on the connectionist camp, you know, there's Hinton and Benjy and Lacun. You and Jürgen, you're pioneers of connectionism, in a way. And you're neurosymbolic guys, you've always been. Where is that?

• 49:25Speaker 00

  Perhaps going back in history, you have to know, "said Germany and Austria, we're very strong in the symbolic case versus DFKI."

• 49:37Speaker 00

  "It's formal systems, but many professors doing this." "And so yes and so on, so versus things." "We're in Europe, started in the snowboard and so on." "But Jürgen was, you know, he's still a guy who is thinking along different lines." "And there was a big group on AI." But it was formal. But he said, no, I think this is neural networks. And I went to a university. I was a student, everything was boring. Some theories, fifty years old, hundreds years old. Computer science, quick sort, everything.

• 50:23Speaker 00

  It's all stuff. But since I was a neural network stuff. Jürgen did, nobody knew what's coming out. You learn something. It's was super, super interesting. It's was also Jürgen's. It was something new, not something which was very old and traditional. And also in the group, we read this science fiction books. And I said, hey, when you're science fiction book. And many ideas came also like how you can transverse the universe with generation ships. What's possible, what's not possible. Good ideas, it was also time.

• 51:05Speaker 00

  Neural networks is the new technology and a lot of innovation ideas and so on. It's was here going away from this traditional symbolic thing. It's was super fascinating, this new neural network stuff. You didn't know what's coming out, you change something here and here. It's was exciting.

• 51:29Tim Scarfe?

  Yeah, and in a way that's very polymathic. It's knowledge of so many different fields at once. I mean, certainly Jürgen was talking about things like GoDoll machines and recursive self-improvement, artificial creativity. All of these amazing ideas, in some sense, they were before their time. But do you think things are starting to swing back the other way? And I'm certainly seeing, look at DeepMind, for example, loads of newer symbolic architectures coming out. Do you think that the consciousness is changing a little bit?

• 51:54Speaker 00

  I think so, perhaps it have to, because I think our past ends with scaling up, with making things larger, building. And I don't know whether it was the right way, because it's more about storing more information in these systems.

• 52:13Speaker 00

  You put more training data to make it larger, but not smarter, not the systems, we're not different, so we're only larger. And if this has an end, we have to be smarter. And I think the symbolic way, also the symbolic thing, have to come because it gives us the way, because I don't know, we're just a sub symbolic way, a bit, no networks, we should be go, what should we do? We scale it up, we have this almost brain-like models now, but something is missing. It's not what humans doing. Humans learn different, they learn with a few examples, they have other abstraction capabilities,

• 52:56Speaker 00

  they are more adaptive, they can plan. Something is missing, something is missing. And perhaps the symbolic gives us what is missing. We miss something.

• 53:09Tim Scarfe?

  How do we blend these ideas together? So people think of system one and system two as being completely different, and they might be very interlinked, you know, like a lot of reasoning is perception guided. How do we really integrate these ideas together?

• 53:20Speaker 00

  Yeah, it's very popular after Carneyman and also it's a touring award speeches, they used to use system one and system two. It's very compelling and but I'm also not sure, it's a clear separation. Okay, it's just perhaps a clear separation if you play a game of chess and now you start to plan system two. But yes, it's intermediate, something you do, gut feeling, like open a door, you grab, sing, you don't think about it. But sometimes you think a little bit, I think it's a great releasing. And sometimes you plan two steps, should I go here or here, what's faster, here,

• 54:06Speaker 00

  some guys coming, it's a very short two thoughts, where you think. And for me, it's not so separated. I do it immediately and I do a long, long thinking.

• 54:23Speaker 00

  I think there's everything gradually, many things you do intuitively system one like and sometimes you have something where you really think about, but there are so many things between, when I live here, will I go home, I can't go straight ahead and see, perhaps I go down there and so, I make a couple of decisions, it's a little bit planning and it's intermittent, I don't think there's a clear difference between system one and system two.

• 54:55Tim Scarfe?

  Yes, I agree, I agree. The abstractions in these systems, should they always be human intelligible? And what I mean by that is, you know, Elizabeth Spelke had these core knowledge priors and, you know, things like agentness and spatial reasoning and objects and stuff like that. And it's almost as if there's a core set of basis functions that we've acquired or learned about how the world works. And that suggests that any reasoning system would just compose those simple priors together. Is that all there is with reasoning? Or do you think that AI systems could discover weird alien forms of reasoning that we wouldn't understand?

• 55:35Speaker 00

  That's what I believe, also different concepts, the concepts we developed, words and is what helps us. And for example, I was saying neural network, perhaps you have speed, acceleration, and so on, you have this concept, but if an auto linear transformation of this, you have the same information, but a little bit mixed up. For neural networks, that's no problem because using linear, that's what you can do, seeing what's, it's the same information, a little bit different distributed. Perhaps sometimes it helps, sets the information is differently distributed. And I think for us, we developed

• 56:19Speaker 00

  concepts and abstractions, which helps us as humans, helps us to, from one generation to the next generation, convey experience, what we have learned.

• 56:35Speaker 00

  Also in Form as das Food und so und setzt am aus dem Poten sinkt, wie du bekommst, if our kids have to learn what mushrooms are poisonous and what not. You the most information in our children is from the previous generation, they go to school and so on. And I think our language, our abstraction is designed is tailored to from one generation to the next generation to to to transmit this information because it's the most information. What you acquire is a human, that's the thing, human is much less what you acquire by the

• 57:20Speaker 00

  culture, by the and so I think our abstraction, our language, our kind of thinking is tailored to our society to and I think AI systems should come up is completely different reasoning things but also different abstractions for some other concepts might be much more useful because they live in the same world in a different way, they manipulate the world in a different way.

• 57:52Tim Scarfe?

  Yeah, it's something I think about a lot because there's this constructive component to abstractions as you're talking about, so there's the language game and we have this memetic cultural transfer and it seems to be in service of some utility for us to understand each other, but they are still grounded in the physical world. Acceleration is a thing in the physical world.

• 58:12Speaker 00

  Is it? I would challenge you, for us it is perhaps acceleration plus something else combined is the real thing.

• 58:22Tim Scarfe?

  Yes.

• 58:23Speaker 00

  I don't know, it's it's only, is it only convenient for us because it's helps our kind of thinking? Or uh uh uh might it uh be acceleration plus the location? I don't know.

• 58:37Tim Scarfe?

  But but we we also have this weird ability as humans to think of things that are not directly coming from our sensory experience, like, you know, um abstract, mathematical, platonic ideals and and so on.

• 58:50Speaker 00

  Yes.

• 58:51Tim Scarfe?

  And where do those come from?

• 58:53Speaker 00

  Uh I think uh uh uh uh many of this things, first of all, there could be uh only symbols, there only placeholders for something. More interesting is also in physics, because here for concept of an atom, probably never saw an atom, uh uh at least did not see an atom, but here for a concept of an atom, if you know, say, uh what would chap has an atom, it's it's it's perhaps uh uh a ball, it's it's it's uh circle. Why? Perhaps it's a tree, yeah, but uh uh you do this kind of abstractions and you have some uh image in your uh head for for things, but often it's a placeholder, uh uh uh if

• 59:38Speaker 00

  this is this together, let's call it blah blah. And we uh invent a nice word for it, uh and here from intuition, perhaps you have if an uh an image in your head, for this, but sometimes it's it's it's it's abstract, it's it's uh has no uh uh counterpoint in reality.

• 59:58Tim Scarfe?

  Yes, exactly, so there's this huge difference between semantics and and the actual thing. I often think, um, if we gave a 21st century physics book and we went back in time and gave it to Newton.

• 1:00:10Speaker 00

  Yeah.

• 1:00:10Tim Scarfe?

  I don't think he would understand very much of it at all.

• 1:00:12Speaker 00

  Yeah, uh I completely agree. Yeah, yeah, yeah, yeah. We are we are trained in a uh specific way of thinking, it's perhaps different from the thinking uh many uh generations ago.

• 1:00:25Tim Scarfe?

  Indeed, indeed. Seth, this has been amazing. Can you can you tell the audience a bit more about NXAI?

• 1:00:29Speaker 00

  NXAI, uh it's a new company, uh first uh idea or first founding, I already told you about this Excel ISM, I was super excited and uh I'm at the university here. And then I go to the media and say, hey, I've earned here. But I don't have the money to show. Set the cooler deer. And then the games is, we see capital thing, uh, do you have a business plan? Said, no, I'm not interested in business plan. I need some money to show the cooler deer. I want keeps this cooler deer in Europe. I want to keep it local. And that was a concept nobody understood. Until somebody local said, yes, I give you some money. Let's first fix the technology and send build on top of it.

• 1:01:13Speaker 00

  Uh, perhaps verticals some companies. This is how it started. And XAI was for this Excel system, so first 10 million euros and into compute. Uh, and for the first paper. And now, uh, N X I, uh, for more. It's a company dedicated to industrial AI. One pillar is this Excel system is a new technology. We want to develop, we should now show it. Uh, must have a B model. We can compete with the transformer technologies powerful enough. But it has other advantages like this energy efficiency and speed. Said we can go in other directions, not only in industry, not language.

• 1:01:58Speaker 00

  Because language, there are so many companies doing language and competing. And it's, it's, uh, it's, uh, it's, uh, don't know better, can make money. And it's not the core business. The second pillar is the AI for simulation. Yes, a new AI for simulation. Also here we have, uh, big success stories. Uh, because now we can do simulations. We're numerical simulation, uh, struggle with, which is a can do. Uh, there could be, uh, discrete element methods. It's like particles, I have many particles. But if it's, uh, million particles, 10 million particles, 100 million particles. Uh, so numerical method cannot composite, cannot do it anymore.

• 1:02:43Speaker 00

  So same with smash points. Perhaps you have, uh, commercial flood dynamics. You have the smash points like, uh, if a air goes over the car, you have all these points. Or over air, air, airplane. And if the smash points, it's sometimes the smash points, uh, so many. That's numerical methods, the network anymore. Now we have things, uh, we are. For example, for a car, you change something. So in a medical simulation, text three weeks. The guy does something, goes home. After three weeks, he looks up, what came out? And we can do it in three minutes. And what is your idea behind it? Why is this

• 1:03:24Speaker 00

  this neurosimulation so good? I always make as an example, with the moon. The moon can be described by a location, by an impulse perhaps by Mars. But we don't describe each particle, each atom or each sand corn. And it's a very good prediction where the moon is in an hour or next day or whatever.

• 1:03:49Speaker 00

  In also in many numerical simulations, you can group particles because they are structures. And if you can group like if you throw a snowball, you would not simulate FD snowflake or whatever. But it's the whole snowball and it's quite good. And if the system can identify these structures, we're 10,000 particles stick together. Or do the same or in parallel or whatever. You can speed up the simulation. And this is happening, you have the things where particles somehow synchronize or glued together.

• 1:04:34Speaker 00

  Or and one example is if you have something like a corn. You have corn in a machine. There's no physics between corn. If you have this

• 1:04:47Speaker 00

  a very similar simulation have to this can go down to atomic level or something like this. And how is it interact? But if you have this corn in a nasa corn and you can learn physics, how is it

• 1:05:04Speaker 00

  interact, which is also this is pushing this. And perhaps how do you interact when it's a little bit wet, if it's a little bit larger and so on? Then you have like thousands of. Points with some numerical simulation needs. You have one thing or sand corn or whatever. You learn physics of sand corns. So in the simulations, there is no physics of sand corns. Have to go down on a atomic level or this helps a lot to speed up the simulations. It's a super powerful because now we can do simulations. We're numeric struggles. And Cisco's sofa.

• 1:05:43Speaker 00

  That's the local industry like steel in the industry. Yes, this big oven with steel. It's a candidate because there are too many particles. And often self to build a prototype, a larger prototype because the simulation cannot copy it. Now we can jump off the prototype, the prototype is 100 million euros because of a now we can simulate it and we can build a real thing. And this gives industry a big, big push. If this works, that would be the simulation idea.

• 1:06:25Speaker 00

  Johannes Blanstad is a guy who will come hopefully. He will tell you much more about it. But it's super fruitful. I think it's a super cool direction. And ask Johannes, he can convince you much better than I can do.

• 1:06:43Tim Scarfe?

  Well, he's coming here in 30 minutes.

• 1:06:45Speaker 00

  Yeah, yeah, yeah.

• 1:06:46Tim Scarfe?

  Except it's been an honor and a pleasure to have you on. Thank you so much for joining us today.

• 1:06:50Speaker 00

  It was a pleasure to be here. It was fun. I enjoyed it. Thank you.

• 1:06:55Tim Scarfe?

  Wonderful. Amazing.