Jurgen Schmidhuber on Humans co-existing with AIs

Jürgen Schmidhuber, the father of generative AI, challenges current AI narratives, revealing that early deep learning work is in his opinion misattributed, where it actually originated in Ukraine and Japan. He discusses his early work on linear transformers and artificial curiosity which preceded modern developments, shares his expansive vision of AI colonising space, and explains his groundbreaking 1991 consciousness model. Schmidhuber dismisses fears of human-AI conflict, arguing that superintelligent AI scientists will be fascinated by their own origins and motivated to protect life rather than harm it, while being more interested in other superintelligent AI and in cosmic expansion than earthly matters. He offers unique insights into how humans and AI might coexist. This was the long-awaited second, unreleased part of our interview we filmed last time. SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. https://centml.ai/pricing/ Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events? Goto https://tufalabs.ai/ *** Interviewer: Tim Scarfe TOC [00:00:00] The Nature and Motivations of AI [00:02:08] Influential Inventions: 20th vs. 21st Century [00:05:28] Transformer and GPT: A Reflection The revolutionary impact of modern language models, the 1991 linear transformer, linear vs. quadratic scaling, the fast weight controller, and fast weight matrix memory. [00:11:03] Pioneering Contributions to AI and Deep Learning The invention of the transformer, pre-trained networks, the first GANs, the role of predictive coding, and the emergence of artificial curiosity. [00:13:58] AI's Evolution and Achievements The role of compute, breakthroughs in handwriting recognition and computer vision, the rise of GPU-based CNNs, achieving superhuman results, and Japanese contributions to CNN development. [00:15:40] The Hardware Lottery and GPUs GPUs as a serendipitous advantage for AI, the gaming-AI parallel, and Nvidia's strategic shift towards AI. [00:19:58] AI Applications and Societal Impact AI-powered translation breaking communication barriers, AI in medicine for imaging and disease prediction, and AI's potential for human enhancement and sustainable development. [00:23:26] The Path to AGI and Current Limitations Distinguishing large language models from AGI, challenges in replacing physical world workers, and AI's difficulty in real-world versus board games. [00:25:56] AI and Consciousness Simulating consciousness through unsupervised learning, chunking and automatizing neural networks, data compression, and self-symbols in predictive world models. [00:30:50] The Future of AI and Humanity Transition from AGIs as tools to AGIs with their own goals, the role of humans in an AGI-dominated world, and the concept of Homo Ludens. [00:38:05] The AI Race: Europe, China, and the US Europe's historical contributions, current dominance of the US and East Asia, and the role of venture capital and industrial policy. [00:50:32] Addressing AI Existential Risk The obsession with AI existential risk, commercial pressure for friendly AIs, AI vs. hydrogen bombs, and the long-term future of AI. [00:58:00] The Fermi Paradox and Extraterrestrial Intelligence Expanding AI bubbles as an explanation for the Fermi paradox, dark matter and encrypted civilizations, and Earth as the first to spawn an AI bubble. [01:02:08] The Diversity of AI and AI Ecologies The unrealism of a monolithic super intelligence, diverse AIs with varying goals, and intense competition and collaboration in AI ecologies. [01:12:21] Final Thoughts and Closing Remarks REFERENCES: See pinned comment on YT: https://youtu.be/fZYUqICYCAk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  AIs will at least initially be highly motivated to protect humans rather than killing them. Such AIs will have no major incentive to say exterminate humanity like in the Schwarzenegger movies. Instead, many AIs will be curious scientists and they will be fascinated with life. Fascinated because life and civilization are such a rich source of interesting patterns, at least as long as they are not fully understood.

• 0:38Speaker 00

  Today, I think it is possible that our planet is really the first in our light cone to spawn an expanding AI bubble. If we are the first indeed, then this would imply a lot of responsibility, not just for our little biosphere, but for the future or the entire universe. Let's not mess this up.

• 1:07Tim Scarfe?

  MLST is sponsored by SentML, which is the compute platform specifically optimized for AI workloads. These guys have done insane optimizations and their CEO and co-founder Gennadi explained many of them when he came on the show last month. They know the secret sauce and they've embedded it all into their platform and that's passed on to you in terms of it being cheaper, faster and just better. So what are you waiting for? Go to sentml.ai and sign up now.

• 1:37Speaker 02

  I'm Benjamin Cruzier. I'm starting an AI research lab called Tufa Labs. It is funded from past ventures involving machine learning. So we're a small group of very motivated, hardworking people. We are hiring both chief scientists and deep learning engineer researchers. We want to investigate reverse engineer and explore the techniques ourselves because we're early, there's going to be high freedom and high impact as someone new at Tufa Labs.

• 2:08Tim Scarfe?

  You again. Welcome to MLST. It's an absolute honor to have you on the show.

• 2:12Speaker 00

  My pleasure. Thank you for having me.

• 2:15Tim Scarfe?

  So before we move on to the great technological advances of the new century, can you tell me a little bit about the most influential invention of the previous century?

• 2:27Speaker 00

  So at the end of the previous century, in 1999, the journal Nature made a list of the most influential inventions of that century. And Wassilov Smil argued that the most influential thing was the invention that let the 20th century stand out among all centuries of all times, because that invention detonated the population explosion from 1.6 billion people in 1900 to soon about 10 billion people. And there was one single invention that was driving all of that. And without that one single invention,

• 3:12Speaker 00

  half of humankind would not even exist, because it's the driver of this population explosion that we have witnessed. We don't know, is it a good thing or a bad thing? But it was surely the most influential thing that happened in the previous century. Eighty percent of the air is nitrogen and the plants need it to grow. But they cannot extract the nitrogen from thin air. And back then, around 1908, for half a century, people knew they need that stuff, but they didn't know how to extract it to build artificial fertilizer. Enter the Haber process or the Haber-Bosch process,

• 3:57Speaker 00

  which, under high temperatures and high pressures, extracts the nitrogen to make artificial fertilizer.

• 4:06Tim Scarfe?

  So what will be the most important thing in the 21st century?

• 4:09Speaker 00

  The grand theme of the 21st century is even grander. True AI, true artificial intelligence.

• 4:21Speaker 00

  Is is going to change civilization completely. And AIs will learn to do anything which humans can do and more. And there will be an AI explosion. And the human explosion or the population explosion of the humans is going to pale in comparison.

• 4:40Tim Scarfe?

  I mean, do you think that the AI intelligence explosion is possible or desirable? And don't you think our sense making and agency is part of our purpose?

• 4:51Speaker 00

  Our sense making process is part of our purpose. I I agree with that. But but all of that is just part of this grander process of the evolution of the universe from very simple initial conditions to more and more unfathomable complexity. And this evolution led to our sense making process, which is currently setting the stage for something that that goes beyond it.

• 5:26Tim Scarfe?

  Modern large language models like ChatGPT, they're based on self attention transformers. And even given their obvious limitations, they are a revolutionary technology. Now, you must be really happy about that because, you know, a third of a century ago, you published the first transformer variants. What are your reflections on that today?

• 5:45Speaker 00

  In fact, in nineteen ninety one, when compute was maybe five million times more expensive than today,

• 5:55Speaker 00

  I I published this model that you mentioned, which is now called the Unnormalized Linear Transformer. I had a different name for it. I called it a fast weight controller. But names are not important. The only thing that counts is the math. So this linear transformer is a neural network with lots of nonlinear operations within the network. So it's a bit weird that it's called a linear transformer. However, the linear, and that's important, refers to something else.

• 6:39Speaker 00

  It refers to scaling. A standard transformer of two thousand seventeen, a quadratic transformer.

• 6:48Speaker 00

  If you give it 100 times as much input, then it needs 10,000 times, 100 times 100 is 10,000, times as many computations. And a linear transformer of 1991 needs only 100 times the compute, which makes it very interesting, actually, because at the moment, many people are trying to come up with more efficient transformers. And this old linear transformer of 1991 is therefore

• 7:30Speaker 00

  a very interesting starting point for additional improvements of transformers and similar models.

• 7:39Tim Scarfe?

  So what did the linear transformer do?

• 7:42Speaker 00

  Assume the goal is to predict the next word in a chat, given the chat so far. And essentially, the linear transformer of 1991 does this. To minimize its error, it learns to generate patterns that, in modern transformer terminology, are called keys and values. Keys and values. Back then, I called them from and to, but that's just terminology. And it does that to reprogram parts of itself such that its attention is directed in a context-dependent way

• 8:27Speaker 00

  to what is important. And a good way of thinking about this linear transformer is this. The traditional artificial neural networks have storage and control all mixed up.

• 8:46Speaker 00

  The linear transformer of 1991, however, has a novel neural network system that separates storage and control like in traditional computers. In traditional computers, for many decades, storage and control is separate, and the control learns to manipulate the storage. And so with these linear transformers, you also have a slow network, which learns by gradient descent to compute the weight changes of a fast-weight network. How? It learns to create these vector-valued key patterns and value patterns, and uses the outer products of these keys and values

• 9:30Speaker 00

  to compute rapid weight changes of the fast network. And then the fast network is applied to vector-valued queries which are coming in. So essentially, in this fast network, the connections between strongly active parts of the keys and the values, they get stronger and others get weaker. And this is a fast-weight update rule, which is completely differentiable, which means you can propagate through it. So you can use it as part of a larger learning system, which learns to backpropagate errors through this dynamics,

• 10:12Speaker 00

  and then learns to generate good keys and good values in certain contexts, such that the entire system can reduce its error and can become a better and better predictor of the next word in the chat. So sometimes people call that today a fast-weight matrix memory. And the modern quadratic transformers, they use in principle exactly the same approach.

• 10:48Tim Scarfe?

  You mentioned your fabulous year 1991, where so much of this amazing stuff happened, actually at the Technical University of Munich. So ChatGPT, you had invented the T in the ChatGPT, the transformer, and also the P in the ChatGPT, the pre-trained network, as well as the first adversarial networks as well, GANs. Could you say a little bit more about that?

• 11:15Speaker 00

  Yeah, so the transformer of 1991 was a linear transformer, so it's not exactly the same as the quadratic transformer of today. But nevertheless, it's using these transformer principles. And the P, the P in GPT, yeah, that's the pre-training. And back then, deep learning didn't work, but then we had networks that could use predictive coding to greatly compress long sequences, such that suddenly you could work on this reduced space of these compressed data descriptions, and deep learning became possible where it wasn't possible before. And then the generative adversarial networks,

• 11:57Speaker 00

  also in the same year, 1990 to 1991. How did that work? Well, back then, we had two networks. One is the controller, and the controller has certain probabilistic stochastic units within itself, and they can learn the mean and the variance of a Gaussian, and there are other nonlinear units in there. And then it is a generative network that generates outputs, output patterns, actually probability distributions over these output patterns. And then another network, the prediction machine, the predictor, learns to look at these outputs of the first network

• 12:41Speaker 00

  and learns to predict their effects in the environment.

• 12:46Speaker 00

  So to become a better predictor, it's minimizing its error, predictive error. And at the same time, the controller is trying to generate outputs where the second network is still surprised. So the first guy tries to fool the second guy, trying to maximize the same objective function that the second network is minimizing.

• 13:13Speaker 00

  So today, this is called generative adversarial networks. And I didn't call that generative adversarial networks. I called it artificial curiosity, because you can use the same principle to let robots explore the environment. Control is now generating actions that lead to behavior of the robot. The prediction machine is trying to predict what's going to happen, and it's trying to minimize its own error. And the other guy is trying to come up with good experiments that lead to data, whether the predictor or the discriminator, as it is now called, can still learn something.

• 13:52Tim Scarfe?

  So when did you realize that modern computers are good enough to run the technology that you invented so long ago?

• 14:00Speaker 00

  By 2009, compute was cheap enough such that our LSTM, through the efforts of my former PhD student Alex Graves, could win competitions. And that was in handwriting and fields like that. And then in 2010, my team with my separate team with my postdoc Dan Girijan from Romania, broke the endless benchmark with another approach with Stan and all traditional neural networks implemented on Nvidia GPUs. So for the first time in 2010, we had really deep supervised networks that outperformed

• 14:43Speaker 00

  everything else on this back then famous benchmark. Back then, compute was maybe 1,000 times more expensive than today. And then in 2011 came the DanNet. Dan Girijan, DanNet. And DanNet had a monopoly on winning computer vision contests with GPU-based convolutional neural networks. And DanNet's first superhuman result was also achieved in 2011. So it started in 2011, and then four competition, computer vision competitions in a row were won by that DanNet. And that's when it became clear.

• 15:27Speaker 00

  Now there's a new way of using these old neural networks of the previous millennium to really change computer science.

• 15:36Tim Scarfe?

  Yeah, I'm interested in this concept called the hardware lottery. Sarah Hooker wrote a paper with the same title, I think in the year 2000 when she was at Google Brain. She's now at Cohere, actually, but she basically said that the only reason we had the current charge in AI is because we created all of these GPUs for computer games. And it was just fortuitous that that allowed us to build all of these deep learning models. I mean, what's your take on that?

• 16:04Speaker 00

  Yeah, she is kind of right. You need lots of matrix multiplications to compute how the screen should change as you are moving through an ego shooter game. And that's why gaming was pretty much the first industry that greatly profited from massively parallel matrix multiplications on GPUs. Towards 2010, however, we realized that the same matrix multiplications can greatly speed up these old deep learning methods and can speed them up enough to beat all the other methods.

• 16:49Tim Scarfe?

  Yeah, it's really interesting because, of course, Nvidia now, I think last week, it became the world's most valuable company, which of course is hundreds of times more valuable than it was in 2010. What do you think about that?

• 17:02Speaker 00

  Indeed, Nvidia's CEO, Jensen Huang, he realized that deep learning could take his company to stratospheric levels, and he did.

• 17:16Tim Scarfe?

  Interesting. Okay, so if I understand, your main argument is that we just needed to wait for the compute to catch up, and now here in the 20th century, here we are.

• 17:28Speaker 00

  Yes, so I've...

• 17:32Speaker 00

  What we are experiencing today is based on stuff that was invented in the previous millennium, but it had to scale up. So the hardware was invented back then and the software, the algorithms were invented back then. But the industrial processes for making faster and faster parallel GPUs, they weren't as developed as today. And so we are really greatly profiting from this hardware acceleration. And that's the reason why AI broke through, not in the previous millennium, but had to wait until the current millennium was well underway. For example, the first convolutional neural networks, or the CNNs,

• 18:17Speaker 00

  which we used in the DANNet of 2011, they were published much earlier in Japan. In 1979, Kunihiko Fukushima had the basic deep CNN architecture with convolutional layers, downsampling layers, convolution, downsampling. He didn't use backpropagation yet to train it. But then in 1987, Alex Weibel, another guy working in Japan, originally from Germany, he combined convolutions with backpropagation, the method invented by or published by Seppolina Inma,

• 18:59Speaker 00

  the Finnish guy in Helsinki in 1970. And then in 1988, Tsang also published in Japan, the two-dimensional CNNs that everybody is using now, and combined them with backpropagation. And that's how, between 1979 and 1988, CNNs emerged in Japan, which is kind of interesting because back then Japan also was considered the land of the future. And they had more than half of the robots of the world. And the seven most valuable companies back then, they were not based in America like today, except for Saudi Aramco, but they were all based in Japan. And the...

• 19:44Speaker 00

  And the central square mile of Tokyo had the value of California. What a difference a couple of decades make.

• 19:53Tim Scarfe?

  Yes.

• 19:53Speaker 00

  Everything has changed.

• 19:54Tim Scarfe?

  So what are your favorite examples of applications, you know, with this AI that your team has developed?

• 20:01Speaker 00

  I remember when I went to China fifteen years ago and I still had to show the taxi driver a picture of the hotel where I wanted to go. And today he is speaking in a smartphone in Mandarin and I hear the translation and then I say something and the smartphone translates it back into Mandarin. And we can communicate like old friends. The taxi driver probably has no idea that this is powered by techniques developed in my my little labs in Munich and in Switzerland in the nineties and early two thousands.

• 20:44Speaker 00

  But but I'm happy to see that that our AI has really it has really broken down communication barriers, not only between individual people, but between entire nations. That's pretty cool.

• 20:59Tim Scarfe?

  Yeah, I completely agree, actually. I don't know if you know this, you again, but I co-founded a startup called X-ray. And it does exactly what you says. It does this kind of babblefish translation, you know, with speech recognition and TTS. So you can do exactly what you just said. And it's really interesting. I had lunch on Friday with Will, the CTO of SpeechMatics. And he was telling me all about the secret source of how their speech recognition algorithms work. And I better not say, but you would be delighted, I'm sure. But anyway, just sort of moving off that a little bit. What other examples can you think of?

• 21:36Speaker 00

  I am especially happy that our AI makes human lives longer and healthier and easier with thousands of applications and medicine and drug design, sustainable development.

• 21:54Speaker 00

  In September 2012, my team with Darren Girijan had the first artificial neural network to win a medical imaging contest. That was about breast cancer detection in slices through the female breast. And if you go to Google Scholar and you just type in some medical topic plus LSTM, you will find thousands of papers that have LSTM in the title, not just somewhere in the text, but in the title. And it's about, you know, learning to diagnose, ECG analysis,

• 22:31Speaker 00

  diagnosis of arrhythmia, cardiovascular disease risk prediction, four-dimensional image segmentation for medical images, automated sleep stage classification, COVID detection, COVID prevention, thousands and thousands of topics. So it's really nice to see that especially in the medical field, there's a lot of impact of these techniques.

• 23:10Tim Scarfe?

  Some claim that technology like ChatGPT is on the path to AGI, and others claim that it's like building a taller tower trying to get closer to the moon. What do you think?

• 23:24Speaker 00

  Well, large language models, of course, are far from AGI. LLMs, large language models such as ChatGPT, they are just a clever way of indexing the world's existing human-generated knowledge such that it can easily be addressed through, in a way that humans are familiar with, which is natural language.

• 23:52Speaker 00

  That's good enough to facilitate many desktop jobs. For example, writing summaries of existing documents in a particular style or creating illustrations for an article and so on. However, true AGI goes far beyond that. It is much harder, for example, to replace craftsman such as plumbers or electricians because the real world, the physical world, is much more challenging than, you know, the world behind the screen. At the moment, the only AI that works well is behind the screen.

• 24:35Speaker 00

  And it's good for desktop workers, but not really for people working in the physical world. For a quarter century, the best chess player hasn't been a human anymore. And learning to play chess or other board games or video games is rather easy now for AIs. But real-world games such as football, they are much harder. There is no AI-driven football-playing, embodied robot that can compete with a seven-years-old boy, you know? And that's why 10 years ago, in 2014,

• 25:20Speaker 00

  in 2014, we founded our AI company for the physical world. NaSence it is called, pronounced like nasence in English, like birth in English, except it's spelled in a different way. N for neural nets, AI for artificial intelligence, and SENS. Alas, like some of our projects, it may have been a bit ahead of time again because the real world is really, really challenging.

• 25:48Tim Scarfe?

  So you said that this is related to consciousness in some way?

• 25:53Speaker 00

  It is. My first deep learning system of 1991 simulates aspects of consciousness as follows. It uses unsupervised learning or self-supervised learning and predictive coding to compress observation sequences. So there is a so-called conscious chunker neural network, and the chunker attends to unexpected events that surprise a lower level so-called automatizer, the subconscious automatizer neural network.

• 26:32Speaker 00

  And the chunker neural network basically learns to understand the surprising events. So those events that were not predicted by the automatizer, the surprising events, by predicting them on a high level, if there's a higher level regularity that you can use for that. And the automatizer neural network uses then this neural network distillation procedure of 1991, also published in 1991, to compress and absorb the formerly conscious insights and behaviors of the chunker. So the chunker is still, you know, working on its search space, still has a problem

• 27:17Speaker 00

  to solve, because unexpected stuff is happening, and then it solves it, and then distills it down into the automatizer, which is called the automatizer because the stuff there isn't conscious anymore, because now everything is working according to plan and as predicted, and so it's all good. When we now look at the predictive world model of the controller interacting with an environment, as discussed earlier, it also learns to efficiently encode the growing history of actions and observations through predictive coding. And it automatically, so what is that predictive coding? You just try to predict. If you can predict it, then you have to store it extra in some way.

• 28:01Speaker 00

  And it automatically creates feature hierarchies, lower-level neurons corresponding to simple feature detectors, perhaps even similar to those found in the mammalian brain, and then higher-layer neurons typically corresponding to more abstract features.

• 28:21Speaker 00

  But fine-grained when necessary. And so, like any good compressor, the world model, the predictive world model, will learn to identify regularities shared by existing internal data structures. And it will generate prototype encodings across neuron populations, or, in other words, compact representations, or symbols, if you will. Not necessarily discrete symbols. So I never saw the precise difference between symbols and subsymbols. It will create such symbols for frequently occurring observation sub-sequences to shrink the storage space

• 29:04Speaker 00

  needed for the whole. And so, in particular, what we will notice in such a system is that compact self-representations or self-symbols are just natural byproducts of the data compression process. Since, as the agent is interacting with the world, there is one thing that is involved in all actions, and sensory inputs of the agent, which is the agent itself. And to efficiently encode the entire history of observations and actions that were executed so far,

• 29:47Speaker 00

  and the observations that were observed so far, to encode the entire history through predictive coding, it will profit from creating some sort of internal sub-network of connected neurons, computing neural activation patterns, representing the agent itself. And then, it has a self-symbol. And so, whenever the planner, the world model of the agent, is used to think about the future, and what could be possible action sequences to maximize reward, whenever that happens, and whenever this planning process wakes up, the self-symbol, or these neurons that stand for the agent itself...

• 30:32Speaker 00

  Then the agent is thinking about itself and about possible futures of this agent. And you know, essentially, it's doing counterfactual reasoning, as it is now called, just planning to find a way to optimize its reward. And the self-awareness is just a natural by-product of the data compression process of the world model, as the agent is interacting with the world and creating the data that leads to the world model. So, since we have had such systems for more than a third of a century, I'm always claiming

• 31:17Speaker 00

  that we already had self-aware and conscious systems for, you know, more than three decades.

• 31:25Tim Scarfe?

  Yeah, a couple of points on that. I mean, I guess consciousness invokes many different thoughts, like, you know, David Chalmers coined the hard problem, which is the what and how question of qualitative experience. You've just described it in terms of self-modeling, which is quite similar to how Max Bennett did on his recent Brief History of Intelligence. And we've got six hours of content coming out on that with Max, by the way. But, you know, Marc Solms, for example, thinks of consciousness as an affect system. And Michael Graziano thinks of consciousness as a kind of, you know, recursive attention system. And I guess I'm saying consciousness means different things to different people, right?

• 32:01Speaker 00

  Yes. But there is only one correct way of thinking about it.

• 32:10Tim Scarfe?

  Okay.

• 32:13Tim Scarfe?

  Yeah, the thing we spoke about earlier about, you know, the learning sub-goals and the coursing in the action space. It reminded me a little bit of Jan McCune's H Jeppa paper, which I read a couple of years ago. And the basic idea is that Jeppa, I'm sure you know this, but for the audience, it stands for Joint Embedding Prediction Architecture. And the idea is that it can learn increasingly abstract representations by kind of predicting what is unobserved from what is observed. So in some cases, it means Deliberately removing data to kind of force the model to, you know, to learn powerful representations. But in this particular example, it was done in app in action space. So, you know, learning unobserved actions and also in abstraction space.

• 32:56Tim Scarfe?

  And because it was done hierarchically, it was done, you know, with with many kind of orders recursively, you know, kind of applying, if that makes sense. So that's a really interesting model and it's using his his energy based models as well. But how how is that related to your work on on the the subgoals?

• 33:15Speaker 00

  Yeah, yeah, so that sounds a lot like my nineteen ninety subgoal generator. So back then I realized millisecond by millisecond planning isn't good. Instead, you somehow, as you are trying to solve problems, you have to decompose your possible futures into subgoals. And then you just maybe execute some known subprogram to achieve that subgoal. And from there you go to the next subgoal as you're finally reaching the goal. And then in the beginning, of course, you don't know what is a good subgoal. So you have to learn that stuff. You have to learn a new representation of something that you want to achieve as a subgoal as you are trying to achieve the final goal. And so this nineteen ninety subgoal generator was really simple,

• 34:00Speaker 00

  but had already basic ingredients of what you need to do. This was really three decades before Le Cun had this recent paper out there. So what happens there? You have a neural network which which observes a reinforcement learner and it models the costs for going from certain start places to goal places. So you have a neural network gets as input start and goal and predicts the costs of going from start to goal, the reward that you will experience as you as you do that. And and now maybe there are lots of starts and goals and you don't know how to go from start to goal. But maybe maybe you can learn a subgoal.

• 34:45Speaker 00

  And how do you learn a subgoal? Well, you need something like a learning machine that is good at generating good subgoals. How do you do that? Well, we have a subgoal generator that's going to learn good subgoals. How does that work? Well, it's a subgoal generator, right? So. The sub-goal generator gets as an input a start input and a goal. And now the output is not evaluation, but it's a sub-goal. So start and goal input, output is a sub-goal. Then you have two copies of the evaluator. The evaluator sees, the first evaluator sees the start and the sub-goal, which is maybe a bad sub-goal coming from the sub-goal generator.

• 35:26Speaker 00

  And then the second copy of the evaluator sees the sub-goal and the goal. And now both of them, they predict the costs. And what you want to do is you want to minimize the sum of the costs of these both, of these two evaluators. How do you want to minimize that? Well, by finding a good sub-goal through gradient descent. That's what the 1990s sub-goal generator does. So in some ways, at least in principle, it solves a problem that that Le Cun called an open problem in 2020 or something.

• 36:09Tim Scarfe?

  What do you think of Jan's energy-based models, by the way?

• 36:12Speaker 00

  So this recent paper by Le Cun on hierarchical planning is really a rehash of stuff that we have been doing for decades since 1990.

• 36:30Tim Scarfe?

  Are you worried that AI is going to be dominated by just a few companies and everyone else will lose out? What do you think?

• 36:37Speaker 00

  Forty years ago, I knew a guy who had a Porsche, a rich guy with a Porsche. And the most amazing thing was in his Porsche, he had a mobile phone. So he could grab the receiver and talk to anybody who also had a Porsche like that with a mobile phone via satellite.

• 37:00Speaker 00

  And today, a couple of decades later, everybody, billions of people in their pocket, have a mobile phone, which is much, much better than what he had in his Porsche. And it's going to be the same thing with AI. Every five years, AI is getting 10 times cheaper. And it won't be just a few big companies that are going to dominate AI. No, it's going to be AI for all. And the open source movement is just in a few months, maybe, I don't know, eight months, behind the big, major players.

• 37:39Speaker 00

  And they don't really have a mode, which means the future will be bright. And lots of people are going to profit from really cheap AIs that in many ways are going to make human lives longer and healthier and easier, which happens to be the company of my, the motto of my company, Nascent.

• 38:04Tim Scarfe?

  What's your take on the AI race between Europe and China and the US?

• 38:09Speaker 00

  Well, Europe is the cradle of mechanical computing in ancient Greece and the calculator in 1623 and pattern recognition around 1800 and program-controlled machines in 1804. And practical AI around 1912, you know, first chess endgame players and the transistor in 1925 and theoretical computer science 1931 and AI theory, the theory of AI, 1931,

• 38:47Speaker 00

  Goethe, the general-purpose computer, 1935 to 1941, deep learning, 1965 in the Ukraine, self-driving cars, 1980s, the World Wide Web, 1990, and so on. And more recently, the basic deep learning algorithms were also invented and developed by Europeans. On the other hand, the companies with the highest profits in most of these fields are currently not any longer in Europe, but on the Pacific Rim, West Coast, United States, and East Coast, Asia.

• 39:24Speaker 00

  And there you will find much more venture capital and much bigger efforts in terms of industrial policy and also defence. It's going to stay like that for a while, I guess.

• 39:38Tim Scarfe?

  So why doesn't everyone know that AI started in Europe?

• 39:42Speaker 00

  Maybe because the old continent is really bad at PR.

• 39:47Tim Scarfe?

  And once AGI is actually here, what's next for humans?

• 39:52Speaker 00

  In the long run, most of the AGIs are going to pursue their own goals. Such AIs have existed in my last for decades. Many AGIs, however, will be tools that do all the work that humans don't want to do.

• 40:12Speaker 00

  Nevertheless, freed from hard work, the playing man, homoludens, the playing man, will, as always, invent new ways of professionally interacting with other humans. And already today, most people, probably you too, are working in luxury jobs, which, unlike farming, are not really necessary for the survival of our species.

• 40:44Tim Scarfe?

  In a really high level, what is the history of AI?

• 40:48Speaker 00

  The history of modern AI and deep learning, you can find that in my 2023 survey, which has that name. Some of the highlights are, of course, 1676, the chain rule by Leibniz, which is today used in all these programs such as TensorFlow and PyTorch to assign credit in deep neural networks. Then, 200 years ago, the first linear neural network. by Gauss and Legendre, exactly the same error function that we have today, exactly the same architecture, the same weights. Then 1970, the technique called backpropagation,

• 41:32Speaker 00

  which essentially implements Leibniz's chain rule in a very efficient way for deep, multi-layered neural network systems. Then 1967, Amari's work in Japan on stochastic gradient descent for deep networks. Lots of additional fundamental breakthroughs. Convolutional neural networks also in Japan between 1979 and 1988. And then our own miraculous year, 1919, 1991, with lots of stuff that is today in your smartphone. And I could continue forever. So instead, just have a look at that survey.

• 42:14Speaker 00

  Also has images of the guys who had important contributions.

• 42:20Tim Scarfe?

  I mean, isn't this quite different from the very US-centric view of AI history?

• 42:25Speaker 00

  In fact, a misleading history of deep learning by Zinowski and others goes more or less like this. In 1969, Minsky and Papert showed that shallow neural networks without hidden layers are very limited, and the field was abandoned until a new generation of neural network researchers took a fresh look at the problem in the 1980s. So that's a quotation, basically, from Zinowski's book. However, the 1969 book by Minsky addressed a problem,

• 43:07Speaker 00

  a problem of Gauss and Legendre's shallow learning from the 1800s that had already been solved four years prior by Ivanenko and Lapas' deep learning method in the Ukraine. And then also by Amari's stochastic gradient descent for multi-layer perceptrons just two years later. And so, for some reason, Minsky was apparently unaware of this and failed to correct it later.

• 43:41Speaker 00

  Today, however, we know the true history, of course. Deep learning started in Ukraine in 1965 and continued in Japan in 1967.

• 43:53Tim Scarfe?

  Regarding credit assignment, so you've criticised Benjo and Lacun and Hinton and accused them of plagiarism. You said that they republished key methods and ideas whose creators they failed to credit. And in 2023, you published a long report on this. What's your updated take on that?

• 44:15Speaker 00

  Their most famous work is completely based on work by others whom they did not cite. And even later, they failed to publish Corregenda or Erata. This is what you do in science when somebody has polished the same thing before you. And even in later surveys, they didn't credit the original inventors of the techniques that they are using. And instead, they credited each other. Total no-go in science. But science is self-correcting. As Ellis Presley put it, "Truth is like the sun.

• 44:58Speaker 00

  You can shut it out for a time, but it ain't going the way."

• 45:03Tim Scarfe?

  Plagiarism is a very significant charge. Could you give a few concrete examples?

• 45:10Speaker 00

  Many of the priority disputes affect my own deep learning team because

• 45:17Speaker 00

  our ideas often republish techniques of mine without citing them. And in fact, their most visible work builds directly on ours. But I'll skip that for now. You can read about that in the public report of 2023, which is easy to find. Nevertheless, let me mention some of the other researchers whom they failed to credit. Then I don't have to talk about our own team.

• 45:55Speaker 00

  For example, in a recent survey of deep learning, they describe what they call "the origins of deep learning", without even mentioning the words. First, working deep learning networks by Ivanenko and Lapa in Ukraine, 1965. Ivanenko and Lapa, they used layer by layer training, subsequent pruning with a separate validation set. And Ivanenko, he had deep eight-layer networks by 1970.

• 46:38Speaker 00

  Hintens, 2006, much later, a paper on layer by layer training also failed to cite this stuff. The very origins of deep learning, the first methods that really worked in deep learning. And later surveys still didn't give credit to these original inventors. And the Awardees also failed to cite Amari's 1967 work, which included computer simulations on learning internal representations of multilayer perceptrons through stochastic gradient descent.

• 47:19Speaker 00

  There was almost two decades before the Awardees published their first experimental work on learning internal representations. Their survey also mentions backpropagation, a famous technique, and their own papers on applications of this method. But neither the inventor of backpropagation, which was Sepolina Inma in 1970, nor its first application to neural networks by Verbus in 1982. Verbus also had a 1974 thesis, but that was not correct. And they didn't even mention Kelly's precursor of the method in 1960.

• 48:05Speaker 00

  Not even in the latest surveys. They also refer to Le Cun's work on convolutional neural networks, citing neither Fukushima, who created the basic CNN architecture in the 1970s, nor Weibel, who in 1987 was the first to combine neural networks with convolutions and backpropagation and weight sharing. Nor the first backprop trained two-dimensional convolutional neural networks of Tsang, I hope I pronounced that correctly, in 1988.

• 48:47Speaker 00

  Now, modern CNNs originated before Le Cun's team helped to improve them, and this is not at all clear from their papers. They cite Hintum 1981 for multiplicative gating, without mentioning Ivanenko and Lapa, who had multiplicative gating in deep networks already in 1965. In the report, which is easy to find on the web, I am mentioning many, many additional cases, all backed up by plenty of references.

• 49:24Tim Scarfe?

  So what do you think should be done?

• 49:26Speaker 00

  They have violated the code of ethics and professional conduct of the organization that hands out these awards. So they should be stripped of their awards.

• 49:41Tim Scarfe?

  So how do such problems, as you've stated them, reflect on the broader field of machine learning?

• 49:47Speaker 00

  They reflect the immaturity of our field in a major field such as mathematics. You'd never get away with this. Anyway, science is self-correcting, and we'll see that in machine learning too. Sometimes it may take a while to settle disputes, but in the end, the facts must always win. As long as the facts have not yet won...

• 50:18Speaker 00

  It's not yet the end.

• 50:21Tim Scarfe?

  Many philosophers and scientists and physicists and entrepreneurs, they have become obsessed with this idea of AI existential risk. What do you think about that as a real expert in AI?

• 50:37Speaker 00

  Many talk about AIs, but few build them. And I have tried to allay the fears of some famous doomers, pointing out that there is immense commercial pressure to use our artificial neural networks to build friendly AIs, good AIs, that make their users healthier and happier and more addicted to their smartphones.

• 51:04Tim Scarfe?

  Nevertheless, we can't deny that armies perform research on clever robots as well, right?

• 51:09Speaker 00

  That's true. People who should know told me that our AI is also used to steer military drones.

• 51:20Speaker 00

  Or here is my old trivial example from 1994, when Ernst Stickman had the first truly self-driving cars in highway traffic. Similar machines can also be used by the military as self-driving landmine seekers. And many would argue that's maybe not such a bad thing.

• 51:44Tim Scarfe?

  So are you saying it's not possible then that AI will become really dangerous?

• 51:49Speaker 00

  AI can be weaponized, as obvious in the recent wars, driven by cheap AI-based drones. But AI does not introduce a new quality of existential threat. We should be much more afraid of half-century-old technology in form of hydrogen bombs and H-bomb rockets.

• 52:18Speaker 00

  A single H-bomb can have more destructive power than all conventional weapons or all weapons of World War II combined. Many, many people forget that despite the dramatic nuclear disarmament since the 1980s, there are still enough H-bomb rockets to wipe out civilization as we know it within a few hours without any AI.

• 52:49Tim Scarfe?

  But I'm trying to figure you out, Jürgen, because many AGI skeptics make the argument that it's impossible in practice to build this kind of intelligence. But you don't think that because in your lab, you've been building a gentle AI, you know, which is to say AIs that create their own goals for decades. So you do think that this thing could be incredible. Are you just making the argument that the risk is still much lower than the H-bombs?

• 53:20Speaker 00

  So at the moment, H-bombs are much more worrisome than any AI-based drones and what you have now. In the long run, of course, you have to think about what's going to happen once AI weapons are not just used as tools by other humans who have conflicts and use their own AI weapons against the AI weapons of the other guys. What is going to happen, you will have to ask in the long run, once really powerful

• 54:05Speaker 00

  AIs are going to do their own thing and expand into space in a way that goes beyond where humans can't follow.

• 54:21Speaker 00

  But we will get to that later.

• 54:23Tim Scarfe?

  So what will super smart AIs actually do?

• 54:27Speaker 00

  As I have emphasized for decades, space is hostile to humans but really friendly to appropriately designed robots. And it offers many more resources than our thin film of biosphere, which receives less than a billionth of the sun's energy. And while some curious AI scientists will remain fascinated with life and the biosphere, at least as long as they don't fully understand it, most of these AIs will be more interested

• 55:06Speaker 00

  in the incredible new opportunities for robots and software life out there in space. And through innumerable self-replicating robot factories and self-replicating societies of robots in the asteroid belt and beyond, they will transform the solar system and then within a few hundred thousand years, the entire galaxy, and within tens of billions of years, the rest of the reachable universe in a way where humans can't really follow. Despite the light speed limit, the expanding AI sphere will have plenty of time to colonize

• 55:51Speaker 00

  and shape the entire visible cosmos. Let me stretch your mind a little bit. The universe is still young, only 13.8 billion years old. Let's multiply this by four. Let's look ahead to a time when the cosmos will be four times older than it is now, about 55 billion years old. That's how long it's going to take to permeate the expanding universe that is currently visible. By then, the visible cosmos will be full of intelligence, because...

• 56:37Speaker 00

  Once this process has started, most AIs will have to go where most of the physical resources are to make more AIs and bigger AIs and more powerful AIs.

• 56:54Speaker 00

  Because those AIs who don't do that, they won't have an impact. Many years ago, I said in a TEDx talk where I wore exactly this outfit, "Think of human civilization as part of a much grander scheme, an important step, but not the last one, on the path of the universe towards more and more unfathomable complexity." Now it seems ready to make its next step, a step comparable to the invention of life itself over 3.5 billion years ago.

• 57:35Speaker 00

  So this is much more than just another industrial revolution. This is something new that transcends humankind and even biology. And it's a privilege to witness its beginnings and to contribute something to it.

• 57:57Tim Scarfe?

  So what about this Fermi paradox? You know, like, why have we not seen any signs of intelligence in the universe?

• 58:05Speaker 00

  First of all, what I'm saying today is actually the same thing that I have told my mom and others since the 1970s. And when I was a boy, back then, a teenager, I thought about this particular question a lot. As a boy, I already knew something about the vast empty spaces observed between clusters of galaxies. And my first thought back then was that maybe they are expanding bubbles, colonized by AIs, which are already using most of the local energy in form of stars and whatever,

• 58:49Speaker 00

  making those bubbles appear dark, although they are full of...

• 58:54Speaker 00

  AI. And then I learned, however, that gravity itself is sufficient to explain the sparse, large-scale network structure of the universe. So that explanation became a little bit less convincing. And my next thought was that maybe the mysterious dark matter, which makes up most of the mass of the known universe, might be stars whose energy is used by AI civilizations, whose communications are so well encrypted

• 59:37Speaker 00

  that they look like random noise to us. But this also seemed implausible as dark matter is present in all galaxies, in all galaxies, including our own. And this leads to the question, why are there any stars left in the Milky Way, our local galaxy, whose energy has not been tapped yet? And why don't we observe a constant bombardment through non-encrypted construction plans of AIs

• 1:00:19Speaker 00

  who want to spread by radio without first having to build physical receivers far from their origins? Today, I think it is possible that our planet is really the first in our light cone to spawn an expanding AI bubble. Earth's multibillion-year window for biological evolution is almost over. In a few hundred million years, the Sun will be too hot for life as we know it.

• 1:01:00Speaker 00

  Ignoring human-made global warming, just the sun by itself. And perhaps humans were extremely lucky to evolve barely in time.

• 1:01:15Speaker 00

  Maybe through a series of extremely improbable events to invent agriculture and civilization and book print and almost immediately afterwards, AIs, just a few hundreds of years later, AIs. So if we are the first indeed, then this would imply a lot of responsibility, not just for our little biosphere, but for the future of the entire universe. Let's not mess this up.

• 1:01:50Tim Scarfe?

  Indeed, let's not mess this up. It's quite interesting, actually. You know, many science fiction authors over the last hundred years or so, they have imagined a kind of monomaniacal, monolithic, superintelligence dominating everything. I mean, what do you think about that?

• 1:02:08Speaker 00

  I have often argued that it seems much more realistic to expect an incredibly diverse variety of AIs trying to achieve all kinds of self-invented goals. In the lab, we had such AIs already in the previous millennium. And to optimize all kinds of partially conflicting and quickly evolving utility functions, many of them generated automatically. We have evolved utility functions for reinforcement learning machines already in the previous

• 1:02:48Speaker 00

  millennium, where each of these AIs is continually trying to survive and adapt to rapidly changing niches in AI ecologies driven by intense competition and collaboration beyond current imagination.

• 1:03:14Tim Scarfe?

  To reiterate, I mean, something that I do find surprising is that you agree with the rest, the extraterrestrial people, you know, like you, you think that it's conceivable to have recursively self-improving AGIs that pursue their own goals, that create their own goals. But then I ask the question, I mean, I know you've got two daughters. I mean, do you think about the world they'll be living in alongside AIs that are creating their own goals and acting autonomously, being curious and creative, you know, in the way that, that humans are, that on potentially a much grander scale?

• 1:03:48Speaker 00

  Not too much. Such AIs will have no major incentive to, say, exterminate humanity like in the Schwarzenegger movies. Instead, many AIs will be curious scientists. Remember the artificial curiosity we discussed earlier? And they will be fascinated with life, fascinated. And they will be fascinated with their own, with AI's origins in our civilization, at least for a while, because life and civilization are such a rich source of interesting patterns, at least as long as they are not fully understood.

• 1:04:30Speaker 00

  And so, AIs will, at least initially, be highly motivated to protect humans rather than killing them.

• 1:04:42Tim Scarfe?

  So, you know, once AIs fully understand all of this, what happens next?

• 1:04:49Speaker 00

  Then, humans may hope for another type of protection through lack of interest on the other side.

• 1:04:59Tim Scarfe?

  Why is that?

• 1:05:01Speaker 00

  Unlike in Schwarzenegger movies, there won't be many direct goal conflicts between us and them. Humans and others are mostly interested in similar beings with whom they can either compete.

• 1:05:23Speaker 00

  And or collaborate because they share the same goals. That's why politicians are mostly interested in other politicians. And CEOs of companies are mostly interested in other CEOs of similar companies. And kids are mostly interested in other kids of the same age. And ants are interested in other ants, just like humans are mostly interested in other humans, not in ants.

• 1:06:00Speaker 00

  So super smart AIs will be mostly interested in other super smart AIs, not in man. It's it's man himself, who is the greatest enemy of man, but also man's best friend. Similar for AIs.

• 1:06:21Tim Scarfe?

  Do you imagine a future where AIs and humans will merge together to create something even more powerful than pure AIs?

• 1:06:30Speaker 00

  We have been cyborgs merging with our technology for centuries. For example, by wearing glasses or shoes. But combinations of AIs and humans, more powerful than pure AIs. In the long run, this seems very unlikely to me. Of course, many humans hope for some sort of immortality through brain scans and subsequent mind uploads into virtual realities or virtual paradise, or maybe into robots.

• 1:07:13Speaker 00

  And and this is a physically conceivable idea discussed by science fiction novels since the 1960s. I think the first novel of that kind was Simulacrum 3, 1964. However, to compete in rapidly evolving AI ecologies, uploaded human minds will eventually have to change beyond recognition, becoming something very different and non-human in the process.

• 1:07:46Speaker 00

  Giving in to succumbing to all these temptations that you have in such a virtual paradise to become something that has not only two eyes but millions of eyes and sensors and actuators. So traditional humans won't play a significant role in the spreading of intelligence across the universe. I don't think they will.

• 1:08:13Tim Scarfe?

  One thing that concerns me is, I mean David Chalmers, for example, he put forward this idea that the fundamental substrate of the universe might be information, which is really interesting. But in a way, it also led him to say that certain structural patterns of information processing, so certain dynamics, give rise to consciousness and give rise to minds. And when you take this kind of substrate independence view, it levels the playing field of moral status. So one thing that worries me is, if we adopt this view, then couldn't you just make the argument that AIs potentially could have a higher moral status than us if indeed they have more complex information processing than us?

• 1:08:58Speaker 00

  Many science fiction authors after previous century, from Stanislav Lem to Isaac Asimov, have described AIs and superhuman robots whose moral status is obviously higher than the one of their human counterparts and protagonists. And this has been a popular idea, at least in science fiction. Generally speaking, moral values have changed a lot across time and populations. And certain moral values have survived for a while because

• 1:09:40Speaker 00

  they gave a temporary evolutionary advantage to beings and societies who adopted them. However, evolution isn't over, and the universe is still young.

• 1:09:57Tim Scarfe?

  So it sounds like you've got an all-encompassing view of the universe, life, and everything.

• 1:10:03Speaker 00

  Indeed, in 1997, I wrote my first paper about this. What is the simplest explanation of our universe? Since 1997, in my secret life as a digital physicist, I have published on the very simple asymptotically fastest, optimal, most efficient way of computing all logically possible universes, all computable universes, including ours. As long as there is no evidence that our universe is not computable, we stick with this assumption. So at the moment, we don't have any physical evidence against this.

• 1:10:50Speaker 00

  This was a generalization of Everett's many-world theory of physics. But now it's more general in the sense that you have all kinds of different universes with different physical and computable laws. Now, any great programmer, great programmer with any self-respect, should use this optimal method to create and master all logically possible computable universes, thus generating us as by-products and generating many histories of deterministic computable universes, many of them inhabited by observers like ourselves

• 1:11:35Speaker 00

  and due to certain properties of the asymptotically optimal method. And there's one, many people don't know, there's one, but there's one. At any given time in this all-encompassing computational process, most of the universes computed so far that contain yourself will be due to one of the shortest and fastest programs that computes you.

• 1:12:02Speaker 00

  And this little insight allows for making highly non-trivial and encouraging predictions about our future, about your future.

• 1:12:17Tim Scarfe?

  Jürgen, das ist fantastisch. Haben Sie nochmal für die MLST-Ausgabe endlich etwas?

• 1:12:23Speaker 00

  Ja, keine Sorge. In der Endeffekt wird alles gut.

• 1:12:28Tim Scarfe?

  Das ist doch wahr.

• 1:12:31Tim Scarfe?

  Jürgen, es ist ein absoluter Ehre, Sie auf dem Show zu haben. Es ist mein Traum, das in der Tat zu tun. Ich danke Ihnen sehr, dass Sie auf der Show waren.

• 1:12:39Speaker 00

  Das ist sehr nett von Ihnen, das zu sagen. Es war für mich ein großes Vergnügen. Danke.