Pedro Domingos: Tensor Logic Unifies AI Paradigms

Pedro Domingos, author of the bestselling book "The Master Algorithm," introduces his latest work: Tensor Logic - a new programming language he believes could become the fundamental language for artificial intelligence. Think of it like this: Physics found its language in calculus. Circuit design found its language in Boolean logic. Pedro argues that AI has been missing its language - until now. **SPONSOR MESSAGES START** — Build your ideas with AI Studio from Google - http://ai.studio/build — Prolific - Quality data. From real people. For faster breakthroughs. https://www.prolific.com/?utm_source=mlst — cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst Submit investment deck: https://cyber.fund/contact?utm_source=mlst — **END** Current AI is split between two worlds that don't play well together: Deep Learning (neural networks, transformers, ChatGPT) - great at learning from data, terrible at logical reasoning Symbolic AI (logic programming, expert systems) - great at logical reasoning, terrible at learning from messy real-world data Tensor Logic unifies both. It's a single language where you can: Write logical rules that the system can actually learn and modify Do transparent, verifiable reasoning (no hallucinations) Mix "fuzzy" analogical thinking with rock-solid deduction INTERACTIVE TRANSCRIPT: https://app.rescript.info/public/share/NP4vZQ-GTETeN_roB2vg64vbEcN7isjJtz4C86WSOhw TOC: 00:00:00 - Introduction 00:04:41 - What is Tensor Logic? 00:09:59 - Tensor Logic vs PyTorch & Einsum 00:17:50 - The Master Algorithm Connection 00:20:41 - Predicate Invention & Learning New Concepts 00:31:22 - Symmetries in AI & Physics 00:35:30 - Computational Reducibility & The Universe 00:43:34 - Technical Details: RNN Implementation 00:45:35 - Turing Completeness Debate 00:56:45 - Transformers vs Turing Machines 01:02:32 - Reasoning in Embedding Space 01:11:46 - Solving Hallucination with Deductive Modes 01:16:17 - Adoption Strategy & Migration Path 01:21:50 - AI Education & Abstraction 01:24:50 - The Trillion-Dollar Waste REFS Tensor Logic: The Language of AI [Pedro Domingos] https://arxiv.org/abs/2510.12269 The Master Algorithm [Pedro Domingos] https://www.amazon.co.uk/Master-Algorithm-Ultimate-Learning-Machine/dp/0241004543 Einsum is All you Need (TIM ROCKTÄSCHEL) https://rockt.ai/2018/04/30/einsum https://www.youtube.com/watch?v=6DrCq8Ry2cw Autoregressive Large Language Models are Computationally Universal (Dale Schuurmans et al - GDM) https://arxiv.org/abs/2410.03170 Memory Augmented Large Language Models are Computationally Universal [Dale Schuurmans] https://arxiv.org/pdf/2301.04589 On the computational power of NNs [95/Siegelmann] https://binds.cs.umass.edu/papers/1995_Siegelmann_JComSysSci.pdf Sebastian Bubeck https://www.reddit.com/r/OpenAI/comments/1oacp38/openai_researcher_sebastian_bubeck_falsely_claims/ I am a strange loop - Hofstadter https://www.amazon.co.uk/Am-Strange-Loop-Douglas-Hofstadter/dp/0465030793 Stephen Wolfram https://www.youtube.com/watch?v=dkpDjd2nHgo The Complex World: An Introduction to the Foundations of Complexity Science [David C. Krakauer] https://www.amazon.co.uk/Complex-World-Introduction-Foundations-Complexity/dp/1947864629 Geometric Deep Learning https://www.youtube.com/watch?v=bIZB1hIJ4u8 Andrew Wilson (NYU) https://www.youtube.com/watch?v=M-jTeBCEGHc Yi Ma https://www.patreon.com/posts/yi-ma-scientific-141953348 Roger Penrose - road to reality https://www.amazon.co.uk/Road-Reality-Complete-Guide-Universe/dp/0099440687 Artificial Intelligence: A Modern Approach [Russel and Norvig] https://www.amazon.co.uk/Artificial-Intelligence-Modern-Approach-Global/dp/1292153962

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed sponsored jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed sponsored jobs. Get a $75 sponsored job credit at indeed.com/podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed sponsored jobs.

• 0:30Speaker 01

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid and risked 350,000 Virginians healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.

• 0:59Speaker 02

  Tensor Logic unifies not just symbolic AI and deep learning. It also unifies things like kernel machines and graphical models. I'm Peter Domingos. I'm a professor of computer science at the University of Washington in Seattle and a longtime machine learning researcher. And my dream from my PhD onwards has always been to unify all the different paradigms of AI into a single one. My PhD unified two of them. My best non-research unifies a couple others of them. I wrote this book that turned into a big bestseller surprisingly called The Master Algorithm that is precisely about this goal and where we are towards that goal. And my latest work, which this podcast will talk about,

• 1:42Speaker 02

  is a new language called Tensor Logic that I would say for the first time brings this dream of a unified representation, a unified solution to AI within reach. So if you want to find out, you know, how are we going to do that? Watch this podcast.

• 1:57Speaker 03

  You know, I can set the temperature to GPT to zero and it still hallucinates. And I can, I can have a, I can have a poor deductive system that hallucinates all kinds of things. So to me, like, those are separate, separate problems.

• 2:10Speaker 02

  Good, so precisely the problem or one of the problems with GPT is that it it hallucinates even when you set the temperature to zero. What the hell, right? I want to have a mode. Yeah, yeah, yeah. Right? Not I, but like every Fortune 500 company, if it's going to use AI, needs to have a mode where the the logic of the business is just a bait. The security isn't violated, the the the customer doesn't get light to it, etc., etc. We gotta have that oriat in that they will not take off, right? And transformers can't do that. Tensor logic can do that precisely because in this, you know, reasoning and embedding space, you know, mode that I just described. If you set the temperature to zero, it does purely deductive reasoning. And by the way, the temperature can be different for each rule.

• 2:55Speaker 02

  Tensor logic is just based on this, to me, gobsmacking observation that an ein sum and the rule in in logic programming are the same thing.

• 3:07Speaker 02

  There is this thing called predicate invention, which is discovering new predicates, discovering new relations that are not in the data, but they explain it better. I would say that, you know, in some sense, discovering representation like that is the key problem in AI, is the Holy Grail. What was Turing's achievement that we now take for granted? Turing's achievement was, for which he is deservedly famous, right? Is to postulate this notion of a universal machine. The amazing thing about computers is that the universal machine, which in his time was a was a completely counterintuitive notion. What do you mean a machine that can do everything? The typewriter can type, you know, like the sewing machine can sew.

• 3:50Speaker 02

  You're telling me there's a machine that can type with one hand and sew with the other? What are you talking about? So like, this is the genius, right? So first step, you want to have this property of having a machine that can do anything. What we're missing to be able to do what the universe does and evolution does is universal induction. What is the Turing machine equivalent for induction for learning? That's what I'm after.

• 4:15Tim Scarfe?

  MLST is supported by CyberFund. The link in the description.

• 4:20Speaker 05

  Hey folks, I'm Omar, product and design lead at Google DeepMind. We just launched a revamped vibe coding experience in AI Studio that lets you mix and match AI capabilities to turn your ideas into reality faster than ever. Just describe your app and Gemini will automatically wire up the right models and APIs for you. And if you need a spark, hit I'm feeling lucky and we'll help you get started. Head to ai.studio/build to create your first app.

• 4:47Speaker 06

  The idea of having to traffic in squishy people in order to make our systems go is not immediately appealing, let's put it that way.

• 4:55Tim Scarfe?

  This episode is sponsored by Prolific.

• 4:58Speaker 07

  Let's get few quality examples in. Let's get the right humans in to get the right quality of human feedback in. So we're trying to make human data or human feedback, we treat it as an infrastructure problem. We try to make it accessible, we make it cheaper, we effectively democratize access to this data.

• 5:15Speaker 02

  I'm a longtime fan of Machine Learning Street Talk. In fact, I was a fan of it before it was big. Just like I was doing deep learning before it was big, so very close analogy. So you should definitely watch Machine Learning Street Talk. It's one of the best ways to not only learn about machine learning, but to find out about what's going on at a deeper level than you see everywhere. And that is very important, so you should definitely subscribe to Machine Learning Street Talk.

• 5:41Tim Scarfe?

  Professor Pedro Domingos, it's amazing to have you back on MLST. I've lost count of how many times we've had you on the show now, so it's amazing to have you back. The main reason that we've invited you today is you've just released a brand new paper, a very exciting paper called Tensor Logic: The Language of AI. And fields, you said, take off when they find their language. So you gave the example of calculus in physics and Boolean logic when designing circuits. What's the idea behind this paper?

• 6:11Speaker 02

  Well, tensor logic in many ways is the goal that I've been working towards my entire professional life. Because I really do strongly believe that a field cannot take off until it has really found its language. And tensor logic, I believe, is the first language that really has all the key properties that you need in the language of AI. For example, it has automated reasoning right out of the box, like, for example, Prolog has, right? The classic AI languages had a number of things that we just took for granted. The transparent and reliable reasoning, you didn't even have to worry about. It was just already available, right? At the same that you don't have that in PyTorch at all, right? You have all these hacks to try and do reasoning on top of it.

• 6:54Speaker 02

  At the same time, the Lisp and the Prolog, they never had the auto-differentiation, the ability to learn, right? One of the beauties of the current moment in many ways is that you barely have, you look at most papers, people barely talk about the learning because it's already implemented under the hood. So you want that as well, right? And you want the scalability on GPUs is the other thing that things like, you know, PyTorch and TensorFlow and whatnot give you. There was no language before they had all of these, and there's a number of others, but these maybe are some of the key ones. So tensor logic is basically a language which, as the name implies, is a marriage, a very deep unification, not just some superficial combination, of the tensor algebra that deep

• 7:37Speaker 02

  networks are all built out of and the logic programming that symbolic AI is built out of. There's only one construct in tensor logic, and it's the tensor equation. You can do everything with tensor equations.

• 7:49Speaker 03

  Are you saying that there's only one language of AI? Because certainly in some fields, like physics, you gave the example of calculus. I mean, yeah, like, you know, almost all of calculus, I mean, almost all of physics, you know, involves quite a bit of calculus. There are other fields where actually there are kind of multiple, multiple languages that play, you know, almost equal roles. I'm wondering if you think that... Is tensor logic gonna be eighty-five, ninety-plus percent of the way that we should be talking about and thinking about AI? Or will it be kind of a mixture of different languages?

• 8:26Speaker 02

  That is a very good question. And in fact, we know very well in computer science that there's no one programming language that is better for everything. There's just people who think it is. Right? Everybody has their favorite language that they believe, you know, is the universal solvent, but it never really is. So we, and we know also for fundamental reasons, like, you know, going back to, you know, Shannon and whatnot, that there is no language that is the most, you know, pithy for anything you might want to say. Right? Having said that, right? Physics is a good example because calculus is so fundamental, you know? Feynman famously said that he thought in calculus. Right? And the thing that I found with tensor logic is that, you know, I don't know how much of

• 9:09Speaker 02

  AI it's going to be, or how much it should be. But what I have found, in many ways to my surprise, is that in some ways tensor logic is more than just a programming language. It really, I think, captures the fundamentals of what you need in AI in a way that going in, I didn't even think was possible. All of tensor algebra can be reduced to this operation, which, you know, going back to physics, is called the Einstein summation. Right? Einstein summation was something that was introduced by Einstein when he was working on relativity and got tired of writing summation signs. It was all about tensors, right? General relativity is all about tensors. And he jokingly called it his great contribution to mathematics. But the bottom line, and you know, there's this great paper by Tim Rocktassel or, you

• 9:52Speaker 02

  know, blog post saying, "Einsum is all you need." And truly, you can do all of deep learning with just einsum. All of the matrix multiplications and, you know, tensor products, all of that are instances of einsum, on the one hand. On the other hand, in symbolic AI, it's all about rules. Right? And tensor logic is just based on this, to me, gobsmacking observation that an einsum and the rule in logic programming are the same thing. They are actually the same thing. The only difference is that one is operating on real numbers and the other one is operating on booleans. But you know, that's just a different, you know, atomic data type. Right? And then on top of that.

• 10:36Tim Scarfe?

  Yeah.

• 10:36Speaker 02

  So, so, so to summarize, at this point, I think that it would be. I look at all different things that I and others have done in AI. And I think it would be crazy to not do these things with tensor logic. There may be other better things coming, you know, after. But at this point, I would say tensor logic probably better for what people are doing across the board. But hey, that's me, I may be a little biased.

• 10:58Tim Scarfe?

  First of all, shout out to Tim Rocktash. Or I read that blog post from 2018 earlier that you were referring to. But I suppose the thought occurs that if it is mostly about Ainsum and, you know, you might make the argument, why do we need an abstraction when we already have a great abstraction in Ainsum? So folks now can use PyTorch and, you know, JAX. What exactly does your abstraction allow them to do that they can't do with PyTorch?

• 11:30Speaker 02

  No, very good. It does several things. So first of all, and this is going to be an increasing order of importance. The syntax of Ainsum in this language, there's also this package called AINOTES, is incredibly clunky. So at a very basic level, tensor logic is just a much pithier, more compact, easier to write and understand way to write Ainsums. And you know, physics and mathematicians famous like to say that a good notation is half the battle. So this might not seem like a big deal, but my experience is that you can just think better and faster. Once you have this notation, they're like this funky procedure call with these indices and these arrows and these arguments. It's a nightmare. And you know, the syntax of tensor logic is like you write a Ainsum like you would

• 12:15Speaker 02

  write a rule. There's a tensor equation with a tensor on the left hand side and, and, and this join of tensors on, on, on, on, on the right, on the right hand side. Right? So this is one aspect. Another very important aspect, and one that I think could prove decisive is that people don't use Ainsum much.

• 12:34Speaker 02

  Because it's not very efficient. Under the hood, it's not as efficient as your typical. You know, sometime, you know, this could be done so much better, right? But I've done some of programming of this and that wound up, even I wound up not using Ainsom because it's so slow and clunky. And all of that can be fixed. Once you have this one abstraction of the tensor equation and you implement it on CUDA, for example, you can optimize the heck out of it. And you'll just be able to, you know, Ainsom will finally be able to reach its potential, right? But actually, none of these things is actually the most important part. The most important part is that the Ainsom as we know it is only good for tensor algebra. Tensor logic is a language where the same construct

• 13:17Speaker 02

  does all the symbolic and all the numeric parts and any mix and variation between them, including learning the symbolic part and whatnot. These are all things that in Ainsom world just didn't exist, right? You talk about the people who knew Ainsom, whether in AI or mathematics or physics, and they just had no idea that any of this had anything to do with reasoning. You look at all the ways that people are trying to do reasoning today and just want to pull out your hair.

• 13:41Speaker 03

  Let me ask a very concrete, you know, in some sense, I'm a simple, simple man. I need like a very concrete example because I completely agree with you, which is that the symbols we use, the language we use are just... simplicity is so fundamental to our ability to like reason at higher and higher levels. So let's take one example, you know, from your paper, which is a logical or, logical or of a bunch of values is equivalent to an Ainsom within a Heaviside, you know, function applied to it. Like you give this example, right?

• 14:14Speaker 02

  The or, just to, so to be precise, what I did in, you know, maybe this is an important piece of context. So if you look at, so the simplest form of logic programming is data log, right? Which is the foundation of databases, right? You know, most SQL queries are variations on data log rules. And data log rules are composed of two things, joins and projections. This is like databases 101. And what I have done is I have generalized join and projection to numeric values. There's this thing which I define called the tensor join and the tensor projection, which when the tensors are Boolean becomes the regular symbolic database one. But now the numeric version has all these things as special cases. And by the way, it's also more general than the Ainsum, right?

• 14:58Speaker 02

  So another benefit of this is that it actually goes beyond the Ainsum. Now, an or, right, the way you get an or is by having more, is it just as in, so, how do you get an or in prolog or data log? Is by having, you know, multiple rules with the same head. And then those rules, then they implicitly being disjointed, right? So if I have A if B C and A if D E, then that means A if B C or D E. And the same thing happens here. And you could also, of course, just put them all in the same equation because like, you know, it might be more convenient to say like, well, A B plus C D, right? So doing an or is a completely, you know, straightforward thing, but it's really not where the main action is. It's in the tensor joins and tensor projections.

• 15:41Speaker 03

  Thank you, like, for laying all that out. Completely agreed. Makes sense. I just wanted, the very example I was giving is that an Ainsum over a particular index of Boolean values, then with a Heaviside function applied to it, which is just a, you know, zero if it's less than zero or one if it's greater than zero, is equivalent to a logical or of the, over that same index.

• 16:03Speaker 02

  Oh, yeah, sorry. I see. I understand your question. So again, it's more than that. It's a DNF, right? It's not just, so a DNF is a disjunction of conjunctions.

• 16:15Speaker 03

  Sure, it's disjunctive normal form.

• 16:16Speaker 02

  Yeah, exactly. And so what happens is that like, you know, like the Ainsum is, so in numeric land, right, a dot, think of a dot product, right? A dot product, right, is just the sum of products, right? Which in Boolean land will be a disjunction of conjunctions. If more than one is true, you get an number that is greater than one, which is you need to pass it through a step function to reduce all the values greater than one back to one. Yes.

• 16:37Speaker 03

  But my question was more like, okay, I have these two different representations of the same, the same operation, at least at the element level. Like, an Ainsum over an index followed by a Heaviside on that element is equivalent to an or over all the same Boolean values of that, of that index. And I guess, and my question, or per element. So my question to you is, I give up one thing, which is instead of having a single symbol, which is kind of like an or, I've now got two operations, you know, einsam, hevicide. And there are many examples of that, right? Like, I can build every single circuit out of NAND gates. I think we discussed this like

• 17:14Speaker 02

  Yeah, exactly.

• 17:15Speaker 03

  Once, actually. Or, or I can have like other kinds of gates. And it's useful to have other kinds of gates. So in your, in your language, do you foresee people not having syntactic sugar, like an or operator, which under the hood is einsam, hevicide? Or would they still retain those? It's just that the fundamental, you know, the most basic, you know, constructs of the language are tensor logic.

• 17:40Speaker 02

  We can do everything with NAND, so why do we need high-level programming languages at all? Right? The, the point, so there's two things that you want a language to be. First of all, you want it to be universal, so you can, for some things you don't, but in general, right, for AI, surely you want a universal language. You want something to incomplete and to, and tensor logic is that. But then, this is actually the most important and most difficult part. You want something that is at the right level of abstraction for the things that you want to do. And NAND definitely is not. And I can show with a lot of examples that I have in the paper, for example, you can, you know, code the transformer in a dozen tensor equations, as opposed to a vast mass of code, right?

• 18:22Speaker 02

  And then what happens when people have a language that suits their needs, is that then they just get used to that. They often wind up using it for, even for things that it wasn't the perfect thing for, but at their point, it's what they're comfortable with. So my guess is at the end of the day, people are just going to do everything in tensor logic. And, and, and they know, you know, in the back of their heads, that yes, there are ors going on here, and you can think of them as ors, but they just think of them as joins and projects in tensor equations.

• 18:49Tim Scarfe?

  Very good. And by the way, you can implement transformers and anything else in tensor logic. It's so easy, in fact, that I fed your paper into Claude Code and I got it to implement the whole lot this afternoon. And maybe I'll publish that on GitHub if folks want to have a look, but it's quite straightforward. But just to get the trajectory a little bit here, Pedro, you're famous for writing this master algorithm book. And in that book, you spoke about all of these different tribes in machine learning, you know, like Bayesian folks and logic folks and kernel methods and neural networks and all of this. And I guess, do you see this as a step towards unifying these things together? Because now in tensor logic, you can actually create a composition of different modalities of AI.

• 19:32Tim Scarfe?

  And it just works. But this might seem a bit weird to people. I mean, can you explain what that might actually look like?

• 19:39Speaker 02

  Absolutely. So in a way, the master algorithm was laying out my agenda, right? Was asking the question, what is the master algorithm? I did say at the outset, I'm not going to give you the master algorithm in this book. I'm just going to tell you where we are and why I think this is the central goal of AI. I would say that tensor logic is that answer. Tensor logic, we haven't talked about that yet, but in tensor logic unifies not just symbolic AI and deep learning. It also unifies things like kernel machines and graphical models. You know, the things that graphical models, for example, are built out of, and then you can compute probabilities with them, are they are a direct, I didn't do this on purpose, but it just fell out.

• 20:21Speaker 02

  You know, the factors that graphical models are made of, those are just tensors. And then the marginalization and summation, and sorry, the marginalization and the pointwise products that are what probability difference is made of, they are just tensor joins and projections on those tensors that represent potentials, in the case of Bayesian networks, conditional distributions. So at this point, we do have this very simple language where you can do the entire gamut of AI, which honestly, I didn't think this was going to be possible going in. I thought the answer would be much more complicated. Now, is this the master algorithm? Tensor logic per se is not the master algorithm because it's just a language.

• 21:04Speaker 02

  I would say that it's the scaffolding on top of which you can build a master algorithm. Now, tensor logic is not just a language, it's also the learning and reasoning facilities under the hood. So for example, one of the best things about tensor logic is that the autograph is incredibly simple. Because there's just one construct is the tensor equation. And the gradient of a tensor logic program is just another tensor logic program. So this is all there. So the learning and the reasoning are all there. However, you know, what I would say is that this is not the master algorithm per se. But it's what we need to producing and I intend to produce it on short order.

• 21:40Tim Scarfe?

  You've described a language and certainly if components of that language are turing complete, that's a big vexed issue. We'll come back to that a little bit later. But because of computational equivalence, we can, you know, from an expressibility point of view, we can describe anything in the universe. So we've got this framework. But to me, the challenge in AI is structure learning, right? So as well as being able to express stuff, it's being able to adapt to novelty and create perhaps from building blocks that we already have, a new structure to allow us to do something useful in that domain. And I can't quite make that leap with with your technology yet.

• 22:20Tim Scarfe?

  So how do we do the meta thing where we actually build the tensor logic constructions to represent the kind of world that we're seeing?

• 22:30Speaker 02

  Oh, very good. So I actually go into that in the paper, you know, but briefly, the paper is just, you know, an informal introduction to these ideas. Inductive logic programming, right, is the field that deals with discovering rules from data. But it does this by things like greedy search or beam search and it's a very large, you know, search space and it's extremely inefficient, right? Which is actually one of the things that killed it, even though it could do all these things that people know in deep learning are just painfully rediscovering. In tensor logic, this is one of the best parts of it. The structured learning falls out of the gradient descent. The gradient descent actually does structure learning. And then on top of that, this is actually the best part as far as the learning is concerned. There is this thing called predicate invention, which is discovering new predicates,

• 23:15Speaker 02

  discovering new relations that are not in the data, but they explain it better. I would say that, you know, in some sense, discovering representations like that is the key problem in AI, is the holy grail. Everything that we know, you know, like when you look at the world, right? You don't see pixels. You don't see photons hitting your retina, right? You see objects. The objects are invented predicates. All the way up to in science, right? The most like Newton's genius was to introduce a new quantity, which is force and energy and entropy and all these, et cetera, et cetera. Right? So in tensor logic, that also just happens by gradient descent. Right? It's, you know, it's hard to believe,

• 23:56Speaker 02

  but let me just give you a hint as to why this is the case. There's this other thing that is folded into tensor logic, which is tensor decompositions, right? And tensor decompositions are a generalization of matrix decompositions. And if you think about matrix decompositions, to take that simple case, what a matrix decomposition does is it takes a matrix and decomposes into two new matrices that together are more compact, but essentially reproduce the same data, right? And there's a generalization of that to tensors called the Tucker decomposition. There's others, but the Tucker one is the most relevant one here. And so if you write in tensor, to answer your questions very directly, if you write in tensor logic, a rule schema, you know, including, you know, a data tensor on the left-hand side.

• 24:42Speaker 02

  And by the way, your entire data can just be reduced to one tensor, embedded as one tensor. We can touch on that later. But you write a rule expressing that as a function of a few other tensors, and the gradient descent, just as in matrix factorization, will discover the best values for those. And then if you want to, for example, then discretize it, say like, I'm going to threshold this and make it boolean again, you will see what is the concept that that learned, or you can leave it in numeric form. So the learning is actually extraordinarily powerful. I've always thought, and you know, I think, you know, a lot of people in deep learning really believe this, that, you know, gradient descent can do amazing things, provided you give it the right architecture to operate on.

• 25:24Speaker 02

  And in a way, what all these million papers are about is about finding the right architecture for gradient descent to operate on. And of course, transformers are a great leap forward. But I think, I think, you know, tensor logic is an even greater leap forward.

• 25:37Speaker 03

  How so? Because, for example, I can picture, suppose we, we, we want to get rid of Python. So like, I'm over here in, you know, PyTorch, and I've described all my, my layers, kind of in the clunky syntax. And instead, I'm like, no, now I have, you know, the Tensor Logic GitHub programming language. Let me go do there. I'm still going to construct my layers, right? So, because like, for example, you do, of course, you allow for the, the, you know, nonlinearities, right? So after every insum, I can apply whatever kind of nonlinear function I want, to ReLU, or sigmoid, or whatever else, right? That's still going to be described in my program.

• 26:18Speaker 03

  It's like, I'm going to have this shape, insum, followed by this linearity feeds into this shape, followed by, so I'm still going to have to do that, kind of like, you know, structuring of the network, if you will, except now in Tensor Logic. And in my opinion, that's one of the biggest limitations right now, is these are all just divined incantation structures that people have come up with. Like, let's put in a dropout layer here, and this kind of layer there, and there. We don't actually allow the machines to learn the overall topological structure. We only allow them to, to find weights within that structure.

• 26:54Speaker 02

  No, but, okay, I understand your question. But Tensor Logic does allow that. You know, step one.

• 27:03Speaker 02

  You can encode a multi-layer perceptron, the entire multi-layer perceptron, and I do that in the paper, with a single tensor equation. All the layers, provided that they all use the same linearity, can be encoded in one equation. Okay? Number one. You can also have different equations for different layers, or typically sets of layers, you know, however well you please. But from the point of view of structured discovery, the thing to realize is that if you create, if you set up one of these very general equations that you can in tensor logic, that in some sense, it's a very broad classes of architecture. Then what the learning does is it discovers the architecture within that space. Right? Which, if you think about it at some level, is what a neural network, when you compare

• 27:47Speaker 02

  an ordinary multi-layer perceptron with a set of rules. Right? A multi-layer perceptron is, you can take, in fact, there was a system called K-Ban in the early days that did this, very clever. It initialized a multi-layer perceptron with a set of rules. Because each neuron is a rule. Right? But it's also more flexible because now you can have weights. Right? But, you know, a neuron, a single neuron can represent a conjunction, and therefore a layer can represent an conjunction, and so forth. So when you're learning weights in an ordinary neural network, you can actually see it as learning the structure of a set of rules. What tensor logic is doing, this is at a more powerful level. Like, that was just propositional, and now this is at the full level of generality of first order logic.

• 28:31Speaker 02

  But you can learn the structure, and then, of course, then there's more than one way to do that. And you can also decide how black and white you want the structure to be, what you want to leave as weights, what you want to discretize. But the structure itself can be learned by taking a tensor equation. A tensor equation is a very general thing. Right? When you learn the weights of those tensors, that materializes to a specific network structure.

• 28:55Speaker 03

  Yeah, so I understand that. Let me bring this back to like, to the folks who are familiar with, you know, PyTorch or traditional techniques. What you described is, yeah, I can just create a fully connected, you know, network with however many layers I want, and then let SGD, you know, find all the weights. That doesn't work. Like, it doesn't work in practice, and it's not going to work with tensor logic. You know, it's just a different representation of the same fundamental problem, which is there's too many degrees of freedom. It's not going to learn anything useful. This is why so much alchemy goes into structuring, you know, constrained networks to have certain, you know, built-in, you know, inductive bias, right?

• 29:34Speaker 02

  No, absolutely. So to take another example, you can also do an entire ConvNet in just one tensor equation. And, and you know, like the quintessential example of like, yes, complete connections don't work is a multi-layer perceptron for, for vision, right? Which you replace with a ConvNet that actually has a local structure. That is also a tensor logic equation. Now we are saying, well, how do you choose between the ConvNet and then MLP, right? Very good question. And now there's a range of things you can do. You can actually, these days, start out with a very general structure. Because, I mean, GPUs and, and large, you know, server farms are an amazing amount of power for something like this, right? So you can almost, I would say, brute force that search, provided you have the data. I'm not actually recommending you do that, right?

• 30:18Speaker 02

  You can also, however, and more interestingly, you can, and this is actually one of the key benefits of tensor logic, is that you can write down what you believe are properties of the, say, right now, what happens in, in, in, you know, when you, for example, program a network in Python is like, you have to commit, you say like, here's the structure. And now, you know, the, the learning of, the only thing that happens is the learning of the weights. In tensor logic, you don't have to do that. You can, you can just, you can set up one of these very general structures and then you say, let me give you a bunch of equations that are things that I believe to be true about the structure, but do not completely determine it. And those just work like priors, and indeed like soft priors, right? Those can, and then again, you can turn up the temperature on this or down

• 31:02Speaker 02

  and say like, you got to obey this equation and that one, you know, sure, you can override. And then, and then this, in my experience, this is actually what is important, is that the gradient descent, instead of starting from a tabular razor, have this kind of soft knowledge. And then, most importantly, like you, the developer, you, the AI researcher, you, you get to, uh, this is really the essence, is that it's not, this, you, every, every, you know, deep learning researcher or data scientist know this. It's like, you don't do the same priority and then push the button and hope for the best, right? It's like, it's, there's a iterative loop. Of you set up the structure and then you learn, you get the results and then you refine the structure. And what this does is it makes that more efficient, much more, because you just have

• 31:45Speaker 02

  to, in your interpreter, you write one more equation or you modify an existing equation. And also, the entire stack is, of what you learn, is much more interpretable than it was before. It's actually, in some ways, one of the most important properties of tensor logic, is that you can understand what's going on much better than you could in two ways. One is that the code is much more transparent than the whole pile of things that you have sitting under a bunch of, you know, PyTorch procedure calls. But also, the result of learning, at least if you do it in certain ways that I discuss in the paper, the result of learning is transparent in a way that a transformer just, you know, can't hope to be.

• 32:22Tim Scarfe?

  So we've covered some interesting topics on MLST before. And of course, there's geometric deep learning, which is this idea that symmetries are fundamental. We've spoken with Andrew Wilson from NYU recently about soft inductive priors, and I've just spoken with Yee-Ma about his great series of architectures. And I guess the prevalent idea here is almost platonistic, that there are real, natural patterns. And if we kind of bias the model, as you were just alluding to, that it will converge on really good representations that describe reality. Now, the alternative view is that reality is constructive and gnarly and that won't work. But you were talking about your Tucker decomposition earlier, and that's this idea that, you

• 33:06Tim Scarfe?

  know, we might have a large sparse matrix, we might want to densify it, we might want to factorize it. And the factorization will kind of pull out some of these natural orderings, you know, of the universe, perhaps. And I guess I was thinking, isn't it a bit like a Gzip algorithm? I mean, what if these factorizations are just semantically meaningless? You know, how do you know that you've got a good one?

• 33:29Speaker 02

  You know, great question, and you've touched on several things there. Let me start with, you know, the geometric deep learning, right? I'm a big fan of this. In fact, you know, I gave a keynote at the second ICL on something that I called symmetry-based learning, which is in some ways an ancestor of geometric deep learning. I really do think that the universe possesses these fundamental symmetri— actually, I don't think that. This is known, right? In physics, right? The standard model is basically a bunch of symmetries. And this is extraordinarily powerful, right? That such simple things could be such universal regularities that you then basically can build everything else out of, right? And if you think about it, in machine learning, the problem is like, what is the learning bias that you should start from? Right? Should you pull in a lot of knowledge?

• 34:13Speaker 02

  Should you have a very, you know, very vague architecture? The thing about machine— and there's the No free lunch theorem, right? That says, you know, if you don't assume anything, you can't ever learn anything. The thing that's amazing about machine learning is that with very weak biases, you can get very far, right? And I would submit that those weak biases fundamentally, at the end of the day, the most important ones are these symmetries. And tensor logic is precisely, you know, I think, the perfect language for expressing those symmetries, as the physicists will tell you, right? It's what they use in like, not the logical, you know, version, but the numeric version, right? So I think we can discover those regularities. I have some suspicions as to what they might be, but I think, you know, we're not quite there yet. But I think, once we have those regularities in some sense,

• 34:57Speaker 02

  you know, they will play in AI the role that the standard model plays in physics, right? Now, of course, as you say, you know, there are people who say like, oh, forget that, right? You know, going back to Marvin Minsky, right? There's like, there is no small set of AI laws or anything. It's just one damn thing after another, blah, blah, blah, blah, right? Like, you're dreaming, right? And I respect that point of view, right? And, you know, we will find out empirically. But if I had to guess how this is gonna play out, at the end of the day, it's gonna be like this. The stuff that I'm talking about gives you, you know, the 80/20. You know, it gets you 80% of the way. And then the other twenty percent away, you have to do a lot of these things, you have to do a lot of hacks, et cetera, et cetera.

• 35:39Speaker 02

  But something like tensor logic still makes it much easier and faster to do those hacks than if you didn't have it. So it actually gives you a benefit both in the eighty percent part and in the twenty percent part.

• 35:51Tim Scarfe?

  There are folks, you know, complexity science, there's this guy called David Krakauer. And in his book, on the first page, actually, the very first sentence, "The scientific and social implications of differences between A, closed reversible symmetry-dominated and predictable classical domains," I think that's what you're talking about, the kind of the Roger Penrose-type world, "and B, open self-organizing dissipative uncertain and adaptive domains." Now, I think the latter is where all the interesting stuff in the universe is, it's where life and intelligence and all the stuff we want to model. And could it be the case that those things are not reducible in the way that you're arguing they are?

• 36:30Speaker 02

  I'm glad you asked that question, because this really is the crux of the matter. Also, you're probably familiar, I know you're familiar, because we've talked about it before, Steve Wolfram's notion of computational reducibility, right?

• 36:40Tim Scarfe?

  Yes.

• 36:41Speaker 02

  And of course, the whole notion that we now understand very well that systems are, you know, many systems are chaotic and therefore inherently unpredictable, right? And, you know, complex systems and all of that. But so where does, and you know, the whole notion that, you know, more is different, right? Like, very famous, you know, notion in-

• 36:58Tim Scarfe?

  Yeah, Rosanderson.

• 36:59Speaker 02

  Convince me. Exactly, which I'm a very strong believer in. So doesn't that contradict what I just said? Actually, no, right? I would say the following is, you know, from physics all the way to AI with biology in the middle. The universe is basically composed of two things: symmetries and spontaneous symmetry breakings, right? God made the symmetries. The symmetries are the laws. As far as we can tell, none of these systems at any level violate the laws, right? Those symmetries are there. I mean, you can go into that, there's a lot to be said there, but essentially, you know, most people, the great majority of people may be accepting, you know, some, there are some exceptions, but they believe that the laws of physics apply to everything. Like, my brain obeys the laws of physics.

• 37:44Speaker 02

  Society of laws of physics. The problem is that the laws of physics are useless. At some point in understanding, you know, even biology, let alone psychology or sociology or AI. Why are they useless? Because we have inherited from the beginning of the universe. A series of spontaneous symmetry breakings, right? And my brain is doing spontaneous symmetry breakings one after another continuously. And those like those then some of them die out, right? Or become irrelevant, stay the same. But others balloon into very big things. And that's actually what evolution is, is one of these things after another. And once you have that, so. So the computational visibility problem is that at some level. It is true that although in principle, this is all predictable and reducible in process, it isn't. Right?

• 38:30Speaker 02

  But now here's here's the point is like, how do we handle that? Our brains know how to handle this in a way that AI doesn't. And the way they handle is this like you predict you computationally reduce everything you can to begin with. And I'm actually, I've talked with Steve, you know, at some length about this. And I'm actually more much more optimistic about how much is reducible than he is. And the things are like your overall. Universe is not reducible, but it's full of these irreducible of these reducible pieces. And in a way, evolution is a chemical, our brain is an accumulation of these reducible pieces. So you do that, you want the machine learning to discover it, you want the inference to exploit it. But then after that, you have to have no choice. But to just keep gathering data and using that to inform your predictions, right?

• 39:14Speaker 02

  In a way, the physics goal of like, I give you the initial conditions and then I just predict like the, you know, the. The Laplace's demon dream, it is a dream. But but I think that the problem that, you know, some of the complex systems people have not realized is that. We don't have to do that. Ask any, you know, engineer, any aerospace engineer using a Kalman filter. What you do is you predict just what's going, you know, or reinforcement learning, right? It's like you want to have a sense of where you're going, but at at every step of time. You you you recalibrate your predictions with a new data that comes in. So you actually only need to predict things well enough to control them to make them predictable. Right? We humans are always controlling the world to make it more predictable.

• 39:59Speaker 02

  And this is what robots need to do as well. And this is sort of like what I'm trying to, you know, support with a language like like tense logic.

• 40:06Speaker 03

  Increasingly more of a believer and kind of Hofstadter's, you know. Concepts, right, that there are multiple levels of description. And even within a level of description, there may be multiple languages, you know, to describe things at that level. And I think part of the lesson is not only do we observe, like not only do we kind of observe a particular level, and sure, we try to reduce things and come up with theories at finer grain levels, higher resolution theories or whatever. But we also observe a certain layer and we're able to, by whatever sort of miraculous mechanism, to almost pull out of thin air, to abduct a theory at this level. Like, here's thermodynamics. Somehow, we came up with that, right?

• 40:48Speaker 03

  And even if we learn theories at lower levels or higher resolution theories, actually, most of the time, you don't replace those older ones. It's like, within their domain of operation, you know, Newtonian mechanics is still extremely useful for lots of things that have to do with our scale, right? Our scale of activity. GR is useful to different scale. Quantum mechanics, a different scale. So, we retain all these languages. And I'm hearing that tensor logic is a great language for a certain, you know, layer of description and for activities of AI. But you're not arguing that it's the language to sort of replace all other layers, right? Like, you still buy into the idea that there are other languages at different.

• 41:32Speaker 02

  I'm glad you asked that question. I am absolutely arguing that tensor logic is the language to use in all these layers. And let me give you some evidence towards that. Express relativity in tensor logic, it's tensors and, you know, differentials of tensors and whatnot. That's, you know, that tensor logic does that out of the box. Do the same thing with quantum mechanics. Do the same thing with all these others, with all of the different pieces of AI that I know. And why is that possible and why does tensor logic do that? Again, I think this gets at a very deep fact about the universe, which, you know, complex systems, people and physicists have, you know, suspected as well, which is that the universe has this amazing property without which it would not be comprehensible, that you can have a lot of complexity at one level, that then organizes itself into a new level

• 42:16Speaker 02

  at which now a different set of, you know, laws applies, right? And, you know, in a way, what we do with computers is do that by design, right? But here's the key. What you want is a language in which to express this process. The whole process by which multiple levels get creative, by which multiple representations get created, including different representations at the same level. For example, going back to the assignment, many people, at least, and I have believed that the essence of human intelligence is your ability to switch between representations as the problem dictates. And as long as you pick one representation, you've stuck yourself in the box. But at that level, tensor logic is a meta-representation. It's a way to construct representations. And yet, a large language model,

• 43:02Speaker 02

  to take a very simple example, what has the transformer learned when it looks at all that text? Precisely, I would say, where a lot of its power comes from is that it has looked, it's like Seb Bubeck says, it's like it has learned this soup of algorithms. There's all these different pieces and different ways of doing things that it has gathered from different places, and it doesn't choose between them. It's the prompting and the fine-tuning and whatnot that then pull out the parts that are better for one thing or another. So we absolutely have to do this in AI. I think it also reflects a deeper truth about the universe. I think there are going to be laws of this. We're not then describing laws of the universe. And I think tensor logic, at least, is my best attempt at having a language in order to do both this AI and this type of scientific discovery.

• 43:46Speaker 02

  I also believe, and I discussed that briefly in the paper, that tensor logic is not going to be just a good language for AI. It's going to be a good language for science in general for several reasons. One of them is this, but the other one is that if you look at the difference between the equations on the page and the resulting program from implementing them, often there's a lot of complication. In tensor logic, it's almost, the tensor equation is an almost symbol-for-symbol translation of the equation on the page. So now you can just do science on a different level. Also, if you look at scientific computing, it's usually these tensor operations with some logic wrapped around it. Tensor logic does the tensor operations and the logic in one language, but more importantly, the logic now becomes learnable.

• 44:31Speaker 02

  You can now learn the logic as well.

• 44:33Speaker 03

  Let me just challenge you on this because, for example, like in your paper, when you got to the RNN section, right, like, you know, tensor logics can represent RNNs, but then you hacked in star T. You're like, oh, I need this little star T here. What's star T? Well, star T is a virtual index that doesn't create new memory. That's not tensor logic. You hacked in star T because you needed that in order to express RNNs.

• 45:00Speaker 02

  No, no, no, no, no, no, no. Look, great question. So there's two very important things to distinguish here. One is, which star T is not, but let me mention that first, the RNNs also illustrate that, is syntactic sugar, right? You always have syntactic sugar because, for example, in an RNN, you want to express x of t plus 1, right? And I could, you know, tensor logic is too incomplete, but I don't have the t plus 1. I can, it's a very simple syntactic, piece of syntactic sugar to add. Why wouldn't I do that? Right? Again, there's an 82 rule of like which of these contrasts you want to have, but the star T is actually completely different thing. The star T is there for

• 45:38Speaker 02

  computational efficiency purposes. Star T is a hint about how to implement that tensor that saves a ton of memory, right? And you know, you know this notion of a leaky abstraction, all abstractions are leaky famously in computer science. Tensor logic is no exception. For the most part, when you write tensor logic, you don't have to worry about what goes on under the hood, but sometimes you want to. And this is precisely one of those things. The idea of the star T is that some, is like, I don't, we don't have for loops anymore, right, which is great, forget all of that. But sometimes I don't want to be computing a new tensor or even just a new vector for every new thing that I do because that would be a waste of memory. The

• 46:19Speaker 02

  star T is just saying, you know, you have one vector and you reuse it at every iteration. So you have the initial x 0 and then x 1 is over, it overwrites that, right? So this, this is a piece of the language, right? You can do everything without it, but it would be silly to not use it.

• 46:34Speaker 03

  All right. So let me, let me push back on something because you mentioned it twice now, which is like the Turing completeness. So your paper relies on like Siegelman's, you know, 1995 sort of paper. She herself now like decades later has admitted that that thing is a total toy that has no practical relevance whatsoever. Okay. Because it requires like infinite precision, rational. Registers that encode in a fractal way, etc. And by the way, in her paper, all she demonstrated was that, under these infinite assumptions, that she could build a particular RNN that was a universal Turing machine. The problem with you using that for your tensor logic is two things.

• 47:18Speaker 03

  One, that restricts the field over which you can have your tensors. It must be one of these fields that has like infinite precision. So infinite precision rationals or whatever. I can't use any other fields, like no modular arithmetic, which is actually what runs on, you know, GPUs, for example. And secondly is, it would restrict the actual structure of the weights to her universal Turing machine. Therefore, it wouldn't be a general-purpose tensor logic.

• 47:44Speaker 02

  Well.

• 47:45Speaker 03

  Do you realize this problem?

• 47:46Speaker 02

  No, no, no. So actually, there is no problem there. Let me tell you exactly why, right? And let's do this in three steps. First of all, Turing completeness doesn't matter at all whatsoever. Because the only difference between a Turing machine and a finite-state machine is the infinite tape. And in the real world, there is no infinite tape. So if you can implement the finite.

• 48:06Speaker 03

  If it doesn't matter, why do you keep mentioning it?

• 48:07Speaker 02

  I, that's part two. That is part two, right? This is actually a very interesting set of questions. So let's, let's, let's, let me get a part of the like. So tensor, so Turing completeness doesn't matter. What matters is that you want to be able to express any computation that you might want. That's what matters, right? You might choose a specific language for specific purposes. For something like TensorFlow, you want that generality. You have that generality irrespective of Turing completeness. So this is part one, right? We can debate, but let's, let's set this aside for just a second now. But, you know, the way, I don't get to change the way computer science is. And Turing completeness is a shorthand for universality. I just want to show people that tensor logic is universal.

• 48:50Speaker 02

  And now, I have a proof that tensor logic is computationally universal that does not rely on the, on the Siegelman construct, right? I chose to not publish in this paper because it would take too long, right? The beauty of that is that in one paragraph, I can just say, "Look, the, the equation in the Siegelman paper, you can implement it here." And we're done, right? I can also, you know, there's, there's so many ways to prove that things are Turing complete. So I only, I completely agree with you and her that that construct is ridiculous, right? It's, it's silly, right? It's, it has no practical significance. But the reason I use is like, it's just my way of telling people in one sentence that and why, you know, tensor logic is Turing complete, right? But the real action is.

• 49:32Speaker 03

  Well, hey, I'd love for you to share. I'd love to see the other proof.

• 49:36Speaker 02

  Oh, I can, I can, I mean, the other, so, so actually,

• 49:41Speaker 02

  there's even more than one other type of proof that is possible. Let me, let me tell you what, what that one is and what's, so here's two, you know, not just, so three ways. There's the Siegelman way, right? Another one is you have a finite control with access to an infinite external tape. Right? That is a much more reasonable thing in my view, right? You have a memory.

• 50:00Speaker 03

  Yeah.

• 50:00Speaker 02

  The, the memory is infinite, but, but all that you have to do in, in the tensor logic is know how to access that memory. So like, it gets back, remember, a Turing machine is a finite control and an infinite tape, right? So if the, if the tensor logic can realize the finite control, which obviously it can, and you give it an infinite tape, then we're done, right? And then on, on that note, you can even just do it the following way, right? Which, for example, like, you know, Dale Schuurmans says a great paper about this is, there, you know, people have come up with various, very simple ways to set up a Turing universal, you know, computer. And one of them is there's a set of rules, right? That, you know, sets up that machine, right? Without going into details. And that set of rules, you know, you can just write in tensor logic without even, you know, having to wake up from your sleep. So there you go.

• 50:45Speaker 03

  I totally agree with you, like, and I, and I was, I often say to people, I'm like, a Turing machine is just, and I really hate to use the word just, because it just doesn't do justice to Turing, to Alan Turing and like the genius of his, you know, creation, the theory of computation, right? But it's just a finite control with an unbounded read, write, you know, external memory. Totally on board with that, absolutely, tensor logic is a finite control. But then you need to add to it these operations to, you know. Manipulate external memory, right? So it's kind of tensor logic plus some operations to deal with external read write memory, no?

• 51:22Speaker 02

  I mean, so those operations are just read, write, move left and move right. That's all there is.

• 51:28Speaker 03

  I know, but that's an extension of, I mean, at least in my view, I mean, I don't know if before you there was such a thing as tensor logic. I'm not sure. I know that a lot of people have talked about tensors for like a decade or more, but um, you know, it seems like some kind of an extension to the typical, it's certainly an extension to the way tensors are used in GR. There's no read write to external memory in that.

• 51:51Speaker 02

  Of course, but that is why tensor logic is more than tensor and tensors in mathematics, right? The tensors that people mathematics just don't do this, right? But tensor logic does because of the of the logic programming side, right? If tensor logic can do logic programming, then it can do everything that a computer can.

• 52:08Speaker 03

  Have you specified fully like all the operators in tensor logic somewhere? Like on a website or something?

• 52:13Speaker 02

  There's only, there's only two tensor project or three, right? There's tensor projection, right? There's tensor join and there's universal linearities. And the linearities are crucial, right? Tensor algebra is multilinear. Algebra is linear, tensor is multilinear, right?

• 52:31Speaker 03

  Totally agree. Where does the, where do the memory operations fit in there? Are they projections? Are they joins? Are they?

• 52:37Speaker 02

  Oh, no, I mean, like it's, they're not,

• 52:41Speaker 02

  they're not even projections or joins, right? I mean, think of a trivial projection where you're not summing things. You only have one, right? That's what a write is, right? Actually, let's not even worry about tensor joins and projections. Let's just think about, you know, propositional rules, which of course are what you, if you want to implement propositional rules in tensor logic, all that you need is tensors with no, with no indices, with zero indices, right? So all you're dealing with is scalars. And the write, right, is just, you know, you know, a rule that says, you know, the target of the writing is on the left hand side and what you, when write is on the right hand side. Now, to get, but to get very concluded to the issue of an infinite memory, right? What is an infinite memory? An infinite memory is just an infinite vector, right? Indexed by the memory address. That's all it is, right?

• 53:26Speaker 02

  And so how do you write this infinite memory in tensor logic? You just have the memory as your tensor on the left hand side. It's kind of so, you know, it's so simple, it almost, there's nothing to think about.

• 53:39Speaker 03

  I'll have to work through some examples.

• 53:41Speaker 02

  And sorry, just to finish that thought, how do you advance the tape? Well, you just increment the index. And how do you, you know, move it left? You decrement the index, right? It's like, it's duh.

• 53:53Tim Scarfe?

  Well, could we come up with a solid example? Because I don't think we sufficiently described the star T function. So roughly, as I understand it, rather than it becoming a dimension, it becomes a transition function. So we don't need to model the full trajectory. But just to give an example, if I wanted to compute, you know, let's say I want to write a function to compute the nth digit of pi or to approximate it. Would I not need to fix the size of the tensors before, right? So the way I understand it is these things have a fixed size. So how could it possibly solve unbounded problems?

• 54:30Speaker 02

  No, very good. So to clarify, star T is not a function. Star T is a notation about an index. So for example, if I have a vector, you know, like, you know, x of i, right? Or better example, a matrix, m of i j, right? This occupies, you know, if i and j are each, you know, 100, this occupies 10,000, you know, positions in memory, right? But if what I do is m i j star, right? On the left-hand side of my tensor equation, then this is just, you know, instead of being whatever, 100 by 100, it's just 100. Because what this is saying is like, if I put the star and the j, what I'm saying is like, run through the i, right?

• 55:11Speaker 02

  And for every j, you overwrite the result, right? You can do this in either, you know, in either dimension. But, you know, so pick one whichever one. It just says like, keep overriding the results, right? So you lose your old ones. So let me put it this way, m i j star is actually a vector.

• 55:34Speaker 02

  Is a vector where the only dimension is i. J is actually just an iterator for a for loop. You see what I'm saying? And concretely, for example, in an RNN, this is what you want. Because x, right? Xi is your vector. And the j, let's call it t, right? Xit. At every new step in time, when the state evolves, you don't want, I mean, you could, but in general, you just want to overwrite the old state with a new one, as in any state transition system. Okay? Now, you're, you know, does this make sense?

• 56:03Tim Scarfe?

  Well, it does, but you're describing an accumulator. And is, do you lose something by losing the history? So because if you think about it, you're overwriting what went before with new information and you're just, you know, unrolling in time. Do you lose anything doing that?

• 56:19Speaker 02

  Oh, of course you lose. So if you don't want to overwrite it, then, then, then don't put the star in. Right? But not to answer your question about pi, right? What, how would I, you know, compute all the digits of pi? Right? In, you know, infinite Turing machine land, right? I have a vector of the digits of pi that has a start, but not an end. Right? And what the computation in tensor logic does, it computes every successive. So like, we didn't talk about this, but there's, you know, how is inference done in tensor logic? Forward chaining or backward chaining? Forward chaining is a general, they are both generalizations of the corresponding operations in, in symbolic AI. If you applied forward chaining to a set of rules that computes the digits of pi, actually just one rule, because it's very simple. What it will do is in each iteration,

• 57:03Speaker 02

  it will fill in the next digit of pi. Right? Now, if your vector is infinite, this will go on forever, as it should. If your vector is finite, well, at some point you run out of memory and you're satisfied with the number of digits, which is what we do with any real computer in the real world.

• 57:18Speaker 03

  I don't want to, I always get us bogged down into Turing issues. So I think we should move, move on. But I think it'd be fun to talk about it, you know, more at another time or just to work through some examples. I think I'll probably work through some examples, but.

• 57:30Tim Scarfe?

  I think this was an interesting one. There's a, there's a strange attractor with Turing conversations. And normally it goes the, the Schmid-Hoover direction where, you know, the universe is finite. There's no difference between an FSA and a Turing. And I felt, I felt that we actually had some information gain in this conversation.

• 57:44Speaker 03

  Well, actually, you know, so, so on that point, and, and this is a bit of an aside. It doesn't actually have anything to do with tensor logic. So I hope you don't mind me asking. But since we have a computer science professor, like I want to just run something by with you, you know? So I always get this kind of pushback from people where I'll say, for example, you know, Auto-regressive transformers, and I mean classic auto-regression, not extended auto-regression, not generalized auto- just auto-regressive transformers are not Turing-complete. Like, DeepMind admits this, and they write a paper showing how you can extend them to become, you know, Turing-complete. So I'll say something like that, and somebody will be like, "Oh, yeah, but, you know, if I can't do 100-digit multiplication with this context size,

• 58:29Speaker 03

  all I gotta do is just have more context, and then I'll be able to do it." And I keep making the point, here's the crucial difference, right? Between, so, and you brought this up beautifully when you said, "Look, a Turing machine is a finite control with an unbounded rewrite memory." And here's the really cool thing about those Turing machines, is they can run in a way where they're churning, churning, churning, churning, and then they say, "Out of memory." And all you gotta do is just give them more memory and hit "Continue." You don't have to reprogram them, you don't have to retrain them when you've like, increased their context size, right? That's the whole difference, is that with a neural network, a traditional transformer, if you increase its context size, go back to the training board, you gotta retrain it, right?

• 59:14Speaker 03

  Because you've run out of memory. Is that a fair point that I'm making?

• 59:17Speaker 02

  So, this is actually extraordinarily simple, and it's to me incredibly frustrating that there's so much confusion about it, starting with computer science and theoretical computer science, and now playing out in AI and transformer land. And, you know, it just boils down to this, right? You said earlier, and I violently agree that, and correct me if I misinterpret it, but you said like, Turing completeness is not important, but that shouldn't cause us to underrate Turing's achievement. Absolutely. What was Turing's achievement that we now take for granted? Turing's achievement was, for which he is deservedly famous, right? Is to postulate this notion of a universal machine.

• 1:00:00Speaker 02

  The amazing thing about computers is that they're universal machine, which in his time was a completely counterintuitive notion. What do you mean a machine that can do everything? The typewriter can type, you know, like the sewing machine can sew. You're telling me there's a machine that can type with one hand and sew with the other? What are you talking about? So like, this is the genius, right? So first step, you want to have this property of having a machine that can do anything. This is the foundation of computer sciences, of computers, as a revolutionary technology, right? So point one. But point two, and getting to the transformer part, right? I don't know, and unfortunately, these confusions then they build on each other and never get, it's one of those symmetry breakings, right?

• 1:00:42Speaker 02

  We went down this road of defining things a certain way and worrying about infinity, and now we're stuck there, right? NP completeness is another example, but ignoring that. So the problem with transformers, so like, the real problem is the following. It's people say like, "Oh, but if you only have this many, you know, blocks, then you can only do so many computations." The thing, for example, that in that logic programming has and we want is that you can learn things from very small examples, like children do in elementary school. You learn to do addition on tiny examples, but then if needed, you can do addition on numbers of any length. Of course, your life is finite. You will never add infinite numbers, but that's not the point. Infinite is just a shorthand for something that's so large,

• 1:01:26Speaker 02

  it doesn't matter how large it is. And what I want in machine learning is to precisely be able to learn to do, to handle problems, graphs, structures, knowledge bases, inference problems, whatever, of any size, from very small ones. That's the limitation that a lot of these transformers have. And that's the one that you want to fix and can fix, and tense logic helps you do that.

• 1:01:49Speaker 00

  This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed sponsored jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed sponsored jobs. Get a $75 sponsored job credit at indeed.com/podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed sponsored jobs.

• 1:02:18Speaker 01

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com. Not authorized by any candidate or candidate's committee.

• 1:02:49Speaker 03

  And just to cap off the discussion about Alan Turing, because I think he deserves, you know, us mentioning this. You know, you mentioned that this was the real achievement, this universality. And I mean, it wasn't just a machine to do a typing, can't do this and that. It was even within computation, right? In his time, people didn't know this. They're like, well, what if I have a machine that just has a separate read tape and a separate write tape? I don't know. Well, how about if we add two write tapes? Does that make it more powerful? What if it's read-write? What if it's just a stack? What if it's lambda calculus? What if it's... There were so many myriad of, you know, lag systems, blah, blah, blah. All these different computational models, right? And nobody knew that they were all equivalent. And that was the real, you know, remarkable achievement.

• 1:03:34Speaker 02

  No, very good. And to be fair, Turing wasn't the only one doing things like this. And precisely, now we know that all these things are equivalent, and then extensions on that power. But here's actually a really important point, right? The question that has been on my mind for decades is this. A Turing machine is a model of deduction. It's universal deduction. What we're missing to be able to do what the universe does and evolution does is universal induction. What is the Turing machine equivalent for induction, for learning? That's what I'm after, right? That's what the master algorithm is. And I know it exists. And again, just as you can have a million different versions of Turing machines that are all equivalent, you can have a million different versions of the master algorithm that are all equivalent, and that's okay.

• 1:04:17Speaker 02

  The point is that first, we have to realize that there is one. We have to prove what it does. And then we can refine it with the syntactic sugars and whatnot, and that's all good. But the main point is having, you know, gotten the universal induction machine, which I think we are pretty close to.

• 1:04:31Speaker 03

  But Pedro, I know the answer. It's Bayesian tensor logic. No, just kidding.

• 1:04:36Speaker 02

  No, if you're Bayesian, it is Bayesian tensor logic.

• 1:04:39Tim Scarfe?

  This is a good segue because we are talking about reasoning and deduction. And transformers, they don't really reason, right? And I, I think of them as a kind of collection of fractured bits of knowledge, maybe with a little bit of understanding two levels down. But we understand many levels down. And when we do reasoning, what we're doing is we are respecting all of the constraints of this epistemic understanding phylogeny thing that we have. And that allows us to build new knowledge, right? Because you can build new knowledge, you can create new things when you respect all of the understanding that you already have. And transformers don't do that, but let's talk about how this works in tensor logic.

• 1:05:23Tim Scarfe?

  So you have this temperature parameter. So for example, you could do something akin to deduction, even in an embedding space, right? And, and certainly with an MLP. And this is where I was a bit confused because I, I can, I can appreciate that if we have a logical model, which is in the domain of certainty, we can do deduction, right? And then if we have something like an MLP and, and we learn the weights and we turn this temperature parameter up, right? So it's, it's actually introducing some degree of randomness. Why would that be anything like the kind of logical deductive reasoning we do? Would that not just do what neural networks do now, which is they just look for similarity in some embedding space?

• 1:06:08Tim Scarfe?

  And the type of reasoning it's doing isn't actually semantically meaningful at all?

• 1:06:13Speaker 02

  I would actually say that like of all things in the paper, this is the most exciting and important one is that you can do sound and transparent reasoning in embedding space with tensor logic. And how come, right? Why is that possible? And to, to just sort of like give the gist of it, here, here's, here's the key, right? Is, you know, think of kernel, let's go to kernel machines for just a second and like the Gram matrix, right? The similarity matrix, what is it, right? You're in feature space and it's for every pair of objects i, j, the dot product of their feature representations, right? And now if you embed all your objects, we already know. Who do that like, you know, you know, there's a matrix with the embedding vector for FJ object, whether it's a word or a token or anything else, right?

• 1:06:55Speaker 02

  Now, I can do the dot product of the embeddings of two objects, right? And let's suppose that all you need vectors to keep things simple, right? And now what happens is that if I, you know, and let's, let's, for the moment, let's say you're not even learning the embeddings yet, right? Let's say you just have random vectors, right? Your embeddings are random, right? That's actually already useful for a lot of things, but of course, it's not where the action is, right? And now there's the following very interesting property, which is the dot product of a vector with itself is 1. But the dot product of two random vectors in a high dimensional space is approximately 0. So your Gram matrix, your similarity matrix will be approximately the identity matrix. Okay? And now what happens like, if I have a tensor logical that operates in this way,

• 1:07:40Speaker 02

  and then it has something like a sigma nonlinearity, right? Then, then, then what's going to happen is that it's going to clean out that noise and it turns into the identity matrix, right? And now I have all these rules that are just operating in a purely logical mode, right? They're Boolean, it's Boolean tenses going in, meaning relations, right? And it's Boolean tenses going out, right? So that way, you can do pure deduction in embedding space with these embedding, random embedding vectors, right? That's right, it's something interesting. But now, let's say you learn the embeddings, which of course is the whole point, right? When you learn the embeddings, what's going to happen by trying to minimize the loss function is that the, the embedding vectors of, of objects about which you tend to make the same inferences will be, will get closer, right?

• 1:08:23Speaker 02

  Because if I'm saying something about one object and that one is similar, like this, like to, to, you know, the gradient descent to minimize the loss is going to, is going to make them, you know, is going to increase their dot product. So you're going to wind up with a similarity matrix that has high values for objects that are quite similar, right? In the limit 1, in the diagonal, and has low values for objects that are quite similar. And now, if, if you turn the temperature parameter, meaning the steepness of the sigma, right? At, at one extreme, at zero temperature, you have a step function. And, and, and, and, and the similarity matrix is discretized back to 0, 1. So, so at the zero temperature extreme, you have pure deduction. But this is very... You see where I'm going with this?

• 1:09:08Tim Scarfe?

  I do, but could I challenge it a tiny bit? So when we train neural networks, we think reasoning is good when we are building... You know, let's say we'll use the Lego analogy. So we're building these blocks, and the new understanding tree that we've created is a good one if it represents the world in an abstract causal way. So I can see how you've, you know, framed this as deduction in the sense that, you know, you've got this Boolean operation and you can build from it. But what if you're building on a sandcastle? What if the component, let's say it's an MLP component, what if it just doesn't represent the way the world works?

• 1:09:45Speaker 02

  No, no, so very good. So like, again, there's more than one thing you can do with Tensor Logic. One of them is you can just re-implement existing things like MLPs and transformers and whatnot. And if all that you did was re-implement them, it will have all their pros and cons, right? It's the same thing, just implemented much more elegantly, blah blah, right? What I'm talking about here and talk about in that section of the paper is doing something different. It's not an MLP, it's not a transformer. It's actually doing these things of like, you embed objects, you embed relations in a certain way that follows from the object, you embed the rules, you embed the reasoning, right? So this is a different process. What this different process allows you to do is that when you raise the temperature, you get to do analogical reasoning.

• 1:10:28Speaker 02

  You know, Douglas Osterstader came up before. Douglas Osterstader, I think, would like this because it's an analogical reason. Like, he has this whole 500-page book arguing that all of cognition is just analogy, right? And again, this is one of the schools of thought, like this is one of the tribes in the master algorithm is reasoning by analogy. You do reasoning by analogy because what happens is you generalize from one object to an object that has a high dot product with it. So now I get to borrow inferences from similar objects. And the higher the temperature, the looser the inferences, the more analogical inference can be. But for example, and again, Douglas goes into this in some of his books, and any mathematician, like Terence Todd the other day,

• 1:11:13Speaker 02

  I just heard him say this, right? It's like mathematicians reason by analogy. They notice similarities between things. But at the end of the day, you need to have a proof. In tensor logic, in this scheme, in this particular scheme of reasoning in embedding space, this is just simulated annealing. You start out with a high temperature being very analogical, and then you lower it. At the end of the day, you have a proof. It's a deductive proof that is guaranteed to be correct, but you couldn't have gotten to it because the search space is so large without the analogical part, right?

• 1:11:45Tim Scarfe?

  Okay, but I understand what you're saying. So you can generalize reasoning outside the domain of certainty. But the question I'm asking, the reason why we have metaphor and analogy, is there's this incredible process of evolution and intelligence, and it's led to the coarse graining of all of these concepts that we use in our language. And there's this rich, beautiful phylogeny that kind of represents the causal reality of what's happened. And why is statistical similarity the same thing as analogy?

• 1:12:19Speaker 02

  Oh, it's not so. Again, I skipped over some steps here. It isn't, right? So analogy, so the most powerful type of analogy, so kernel machines, in some sense, are the least powerful type of analogy. It's just, oh, here's a similarity or nearest neighbor, right? I have a distance function. That's not really where the action is. The action is in what is called structure mapping, right? Structure mapping was this thing proposed by Deidre Gentner, where you solve a problem by mapping its structure to the structure of problems that you know, right? And the canonical example is Niels Bohr's model of the atom, which he came up with by an analogy between an atom and the solar system. The nucleus is the sun, the planet's [unintelligible].

• 1:13:01Speaker 02

  Turns out to be a bad analogy, but it was crucial in the development of physics, right? And structure, and there's also this whole subfield of AI called case-based reasoning, where I'm a help desk, you come up with a problem, and I don't try to solve it from scratch because I don't need to, that would be wasteful. I go to my database of similar cases and I find one, and then I tweak it. So structure mapping is an extraordinarily powerful thing. But it's this combination of similarity and compositionality, which kernel machines per se don't have. But Tensor Logic does. The point in Tensor Logic is that you do have all the part of the kernel machines, but all the compositionality of the symbolic AI. So again, the structure mapping just, you know, just comes out of the box. You don't need to do anything more to have structure mapping. And all the power of analogical reasoning that comes with it.

• 1:13:45Speaker 03

  Yeah, could I suggest a good analogy is to ad lib? Do you think that's fair? It's like you've got the general structure there and you can plug in parts into the blank spaces and you get, you know, a solution, right?

• 1:13:55Speaker 02

  That's one mode in which things can function, right? You can also, yeah. The whole process of structure mapping or of case business can actually be very rich. I can combine, for example, two big pieces, but like, that's one example, yeah.

• 1:14:09Speaker 03

  Oh, yeah, yeah, no, that's fair. I mean, yeah, it has this nice nested structure, you know, property. Since, while we're on this topic, let me ask you about something that I was confused about in the paper. So I don't understand your connection between hallucination and deduction or determinism. Because in my mind, you know, I can set the temperature to GPT to zero and it still hallucinates. And I can, I can have a, I can have a poor deductive system that hallucinates all kinds of things. So to me, like, those are separate, separate problems. Like, what was, I just kind of misunderstood.

• 1:14:45Speaker 02

  Very good. So precisely the problem or one of the problems with GPT is that it hallucinates even when you set the temperature to zero. What the hell, right? I want to have a mode, right? Not I, but like every Fortune 500 company, if it's going to use AI, needs to have a mode. Where the logic of the business is just a bait. The security isn't violated, the customer doesn't get lied to, et cetera, et cetera. We gotta have that orient in that they will not take off, right? And transformers can't do that. Tensor Logic can do that precisely because in this, you know, reasoning in a betting space, you know, mode that I just described, if you set the temperature to zero, it does purely deductive reasoning. And by the way, the temperature can be different for each rule.

• 1:15:29Speaker 02

  And I think this is what almost all applications are going to have is like there are some rules that are the mathematical truths or logic that you must guarantee will not be violated. They are the laws, right? And those have infinite temperature. And then there's all these others that are more qualitative reasoning and like more accumulating evidence, maybe stuff that you mined from the web. And those, you know, those will have lower, those will have higher temperature. And that temperature parameter, you know, can be learned in some rules and not others, right? So now you have this whole spectrum between the deductive and the, and the more, you know, or even fantasizing, like truly hallucinating at the far end of the high temperature, right? But precisely the point that I'm making in the paper is that, you know, at with L L M's, the best that you can get at zero temperature is still a lot of hallucinations.

• 1:16:13Speaker 02

  Then there's things like RAG, but all they do is retrieve. And even then you still hallucinate, right? Compare, you know, tensor logic in this mode with RAG, right? And it doesn't just retrieve things. It computes the deductive closure of your knowledge, which is an exponentially more powerful thing to have, right? And with zero hallucinations.

• 1:16:32Tim Scarfe?

  Well, it, it, it is if, if the model represents the world. You know, because what, what does hallucination mean? Or actually, what does slop mean? My definition of slop is when a creative artifact is produced by something that doesn't understand. So if I'm, if, if I understand a domain deeply, that artifact looks incoherent to me because it was generated by a process that doesn't understand the world. And it, isn't it even the same with tensor logic? That, you know, deduction is great, but if the model isn't a good one, then wouldn't that just be a hallucination as well?

• 1:17:08Speaker 02

  Oh, absolutely. But let's, let's make some distinctions here, right? The only claim I'm making here, because that's the only one you can make, is that tensor logic at zero temperature in this mode will give you the soundness properties that logic has, right? Soundness in the technical sense of soundness. All that means is that you only reach conclusions that truly logically follow from the premises. You don't say anything about whether the premises are valid or not. If the premises were hallucinated, so will the conclusions be, right? There's like, there's no. There's no magic there, right? But that is a very important property to have. Again, if I give to a transformer a bunch of, you know, true facts, it still hallucinates. And that's what I can guarantee will not happen in tensor logic.

• 1:17:53Speaker 02

  Now, coming up with the true facts, well, that's a different part of the game. You can write them down, you can learn them, you can refine them. You never know for sure if you have the perfect model. And of course, that's more the machine learning and knowledge acquisition part, right? So I do, I think, have a very important guarantee here of non-hallucination. But it's not a guarantee that the, you know, that the model that you're working on, you know, came from the real world. That's a whole other, you know, neck of the woods.

• 1:18:16Tim Scarfe?

  Who's going to adopt this first? How are we going to bootstrap this as a community? How do you see this progressing?

• 1:18:22Speaker 02

  No, very good. So, you know, the last section in the paper is discussing adoption and what needs to happen and things like that. Let's suppose that everybody agrees tensor logic is a beautiful, perfect language and what we need for AI. Just for that reason, that would not be enough to make it take off, sadly, right? Because, you know, there's a very, you know, things get, you know, people are still using COBOL these days, right? I rest my case, right? So legacy, there's this irony in computer science or in the information technology, instead of like, it moves faster than anything else, but at the same time, you know, things never die, right? You can't kill them. You can't kill COBOL, right? And I really do believe, I know, I like Python, right? I program in Python. It's very nice in many ways, better than FORTRAN for some things, et cetera, right?

• 1:19:04Speaker 02

  Even though it was never, or NumPy, if you will, but you get the point, right? Like, it's like, for AI, it's just a terrible thing. But like, I'm a Python programmer, like, I, you know, general is like, like, okay, your tensor logic is nice. I'm not going to rewrite all my code, right? Forget that, right? So what is going to make it happen, right? But now, we can look at what has made this happen in the past, right? And it's several things. One is that, for example, look at how Java took off, right? Java took off at the time of the internet because it was the language of networking. Allegedly, you could debate that, but like, people wanted to do things that it was, you know, very hard to do with, you know, things like C and blah, blah, and C, right? And so Java took off, right? And we are in exactly...

• 1:19:49Tim Scarfe?

  It was the language of embedded programs and web browsers. Like, that was, that was the only option we had, right?

• 1:19:54Speaker 02

  Exactly, right? So, and you know... There's big arguments about this, but not relevant to us here. The point I'm trying to make here is, we are precisely at a, also very relevant. Why did languages like Listen, Progot, you know, fall out, right? Because they were better for AI than, you know, Fortran or C or whatever, right? Or Java, is that like, they were niche languages. And the network effects of the more widely used languages and all their aspects just completely overrode that, right? We understand that very well, now people didn't in the 80s. But now, we're in a different ballgame now. Now, the big technology, the center of everything, is AI, right? If you have a better language for AI, that is the one that is gonna, you know, have the biggest users.

• 1:20:36Speaker 02

  And moreover, if you have a language that solves the big pains, right? To adopt a new language or a new anything, right, you know, like a new app, right? It needs to solve some big pain, right? Is there a big pain that Tensor Logic solves? Well, hell yeah, it solves halluci, potentially, okay? It solves, all this is subject to empirical verification, but it potentially solves hallucination. It solves the opacity, right? Like, we're in this world right now where there's like multi-billion corporations and systems that are like, they're driven by this black box. And nobody, I know I've talked with CEOs of big tech companies that say like, you know, "I can't sleep at night, because I don't know what this thing is gonna do." And the people who trained it have left the company, and who knows, right?

• 1:21:19Speaker 02

  So, if we can make a dent in that, people will converge to it very, very quickly. Also, I think, when people have the experience of how easy it is to use Tensor Logic compared to the big pile of stuff that is, lies under PyTorch and whatnot, I think that they will actually be very, very motivated to migrate very quickly. And then, you know, like, and there's several things. There's like, developing the open source community and vendor competition and whatnot. But, but, but, you know, like, there's, there's a couple of other important things here, one of which is the following. Tensor Logic is ideally suited for AI education.

• 1:21:56Speaker 02

  It's one language in which it which has very little, you know, extraneous stuff. And you can and you can teach the entire gamut of AI very well and do the exercise. It'll be a language that the professors, the TAs, and the students will like, right? And and history shows, you know, going back to things like Unix, that if you have something like that that takes off in computer science education, then people go to industry and say like, I want to use this because it's what's good, it's what I like. And and a generation later, it's it's it's what everybody is using. And and one more thing is the following. There you the transition to tensor logic from from from Python doesn't have to happen all at once, right?

• 1:22:38Speaker 02

  You can have, for example, and I already have, actually, a another's have, like again, because it's very easy to do, right? You read the paper and you like and you do that in the next whatever 30 minutes. You can write a preprocessor that just converts tensor equations into Python. And again, all it does is a one-to-one mapping between the syntax of of tensor logic and einsum, right? Then making things efficiently, as we discussed, is another matter. But from this point of view of of developer, you know, uptake, right? All it and there's a long, long history of people doing this with different languages, right? It's like you have a preprocessor that that that lets you write into equations, but then

• 1:23:18Speaker 02

  it converts you, it converts those equations into into PyTorch or or Python or just NumPy, let's say Python, right? And then you do everything else in Python that you did before. You don't lose anything, you don't lose any of this encode. It's just that a set of things and in particular reasoning have now become much easier than they were before. And then, once you have this little ball people like, oh, but I can do this and I let me have that piece of syntactic sugar. And before you know it, people are like, well, I don't need all that, you know, Python stuff anymore. I'd just rather live in tensor logic world.

• 1:23:49Speaker 00

  This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed sponsored jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed sponsored jobs. Get a $75 sponsored job credit at indeed.com/podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed sponsored jobs.

• 1:24:18Speaker 01

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com. Not authorized by any candidate or candidate's committee.

• 1:24:48Tim Scarfe?

  So you said about AI for education and tensor logic. It's a declarative language, which means it's the what, not the how. It's this incredible coarse-graining that screens off a lot of unnecessary detail. But is it unnecessary? I guess is the question. Do you think that people learning about AI should know about how the underlying things work? And certainly, folks working at Google, they might need to do some domain-specific optimizations for certain components of the machine behind the scenes. Do you think that we can screen off all that detail?

• 1:25:20Speaker 02

  Great question. But actually, let me start by correcting something. Tensor logic, like Prolog and Datalog, has actually both declarative and procedural semantics. You can look at a tensor equation. That's actually the whole beauty of logic programming in some senses. You can look at a tensor logic equation as like it's an equation. It's like Einstein's equation. It's the statement about the world. But you can also look at it and treat it as a function call. The left-hand side is the call, and the right-hand side is the body, which is a bunch of other calls and a way to combine them. Right? So you can... And in fact, most of the time, in my experience, that I've used tensor logic so far, I tend to use it in procedural mode. It's a set of equations. It's a bunch of statements,

• 1:26:04Speaker 02

  like you would have in any imperative language. Right? So very important to bear that in mind. Now, but to the heart of your question, which I think is very important. When you're teaching people something... I mean, I would actually say this is the tragedy of computer science education. From high school to intro courses to the most advanced things is that you want to teach them the beauty of what you can do and the essence of the algorithms and so on. But then you, and particularly they, the students, they spend all their time bogged down in all this crap, all these details where you get the semicolon wrong and the program doesn't work anymore, and they hate it. And they decide that computer science is not for them.

• 1:26:47Speaker 02

  Or at best, they waste 10 times more time than they should, right? So precisely the whole point of having the right abstraction is to avoid that. Now, so I would say this is one of the best features of tensor logic is to do that for AI. Now, but you also say correctly that like, well, a lot of the time you need to go beyond that level of abstraction and for example, from a part of you of efficiency and so on and a lot of things, right? But I would say, and again, you know, like, we won't know until tensor logic is used widely and we see what happens. But tensor logic is a language that's some level. It's like C, right? It's very low level. The beauty in my mind, again, this gets back to multiple as a like, you can use it to say very high level things. You can also to express, you can also to express the lowest level possible computations, right?

• 1:27:33Speaker 02

  Like a tensor equation is something that you can map onto a GPU with almost no change, right? And then optimize the heck out of, right? Like, in fact, you know, I've joked like with folks at Nvidia that, you know, CUDA is a nice mode, but tensor logic could be the end of that mode.

• 1:27:49Speaker 03

  I sometimes feel like, and I'm not sure exactly how much money was spent on bigger and bigger transformers, you know, deeper and deeper, or you know, wider and more data and whatever transform, more parameter transformers. But it's got to be a lot, like a trillion dollars or something like that. And I feel like sometimes we've spent a trillion dollars to learn yet again, lessons that people could have learned if they'd have taken certain, you know, basic courses in computer science. Like, I'm wondering if you sometimes feel like that and what lessons, if any, you think people should have known before spending a trillion dollars?

• 1:28:24Speaker 02

  I violently agree with that. In fact, the paradox of the current moment in AI is that on the one hand, this is super exciting, right? This is what we've worked all our lives towards, right? It's like the dream is happening. I used to tell people, you know, when I went into grad school, they're like, one day machine learning is going to take over the world and be like, uh, what? And I'm like, see, it is taking over the world there. Take that. So on the one, and you know, on a more serious note, like transformers are a great leap forward and you know, anybody who's used a chatbot is like, wow, look at the things this can do. This is great. But at the same time. The sheer amount of like wastefulness and stupidity and ignorance going on is just unbelievable. It's like, why are you reinventing?

• 1:29:07Speaker 02

  Why, for example, I've talked with people at, for example, OpenAI that do the reasoning. And many of them are very good people, so I'm not trying to, you know, pick on anybody. But it's like, oh, what is reasoning? We need to figure that out and like, and then they say a bunch of stuff that is completely wrong. And I'm thinking to myself, why don't you spend an afternoon reading a couple of chapters of Russell and Norvig? And save a hundred billion dollars in wasted compute? Please, just do that, right? And in a way, you know, part of what I'm trying to do with Tensor Logic is make things go in that direction. Because the current direction is just too damn painful. And it's not just that it's painful, this is gonna end badly, right? People are spending, right? It's like, in a way, like, you know, spending all this money on data centers is not wasted.

• 1:29:52Speaker 02

  Because it's not like the fiber that went dark, right? We in AI have an appetite for unlimited compute, right? But they're spending all this money prematurely on stuff that isn't ready for that yet, right? The demand is probably not gonna be there. And we're gonna look back onto them and go like, wow, ninety nine point nine percent of that compute was completely wasted. Because of a lot of the reasons that we've been talking about, including like, you didn't know how to do reasoning. So you brute forced it, et cetera, et cetera. So like, you know, we gotta change the direction of this ship.

• 1:30:23Speaker 03

  It's like that well-known quote from, you know, Matt Damon and Good Will Hunting, right? Like, you know, to paraphrase it, you know, you've wasted a trillion dollars on an education you could have got for a buck fifty in late fees at the library.

• 1:30:37Speaker 02

  Exactly, exactly.

• 1:30:39Tim Scarfe?

  Well, Professor Pedro Dominguez, it's an absolute honor to have you on the show. Thank you so much for joining us.

• 1:30:43Speaker 02

  Thanks for having me.

• 1:30:44Speaker 03

  Thank you.

• 1:30:45Speaker 02

  Always a pleasure.