#340 Steffen Cruz: Training AI Without Data Centres

What if you could train a frontier AI model without building a single data centre? In this episode of Eye on AI, Craig Smith sits down with Steffen Cruz, co-founder and CTO of Macrocosmos, to explore a radical alternative to the way AI models are built today. Instead of billion-dollar GPU warehouses, Steffen is training large language models using idle compute from devices distributed around the world, coordinated through the Bittensor blockchain. Steffen breaks down why the centralised data centre model is heading toward a wall. Projects like Stargate and Colossus cost tens of billions of dollars, and as appetite for larger models grows, the economics simply stop making sense. He explains how distributed training flips this on its head, tapping into surplus energy, underutilised GPUs, and even consumer devices like Mac Minis to train models at a fraction of the cost. We also get into IOTA, Macrocosmos's flagship technology, an orchestration layer that takes compute nodes scattered across the globe and makes them act like a single supercomputer. No single device runs the full model. Instead, each one carries a small slice, a technique called model parallelism, and together they can train frontier-scale models that would otherwise be out of reach for startups, researchers, and enterprises. Finally, Steffen shares what he's building toward: 70 billion parameter models trained at 10 to 20 percent of centralised costs, a two-sided marketplace for compute, and a future where anyone with a spare GPU or Mac Mini can earn passive income while contributing to the democratisation of AI. Subscribe for more conversations with the people building the future of AI and emerging technology. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI Timestamp: (00:00) Introduction: The Problem With Blockchain AI Projects (06:39) Meet Steffen Cruz: From Subatomic Physics to Decentralised AI (09:16) What Is a Bittensor? The Blockchain Built for AI (11:53) How the Blockchain Actually Works: Registry, Clock, and Rewards (15:08) Why Data Centres Are Hitting a Wall (22:01) Distributed Training vs Federated Learning: What's the Difference? (27:47) Train at Home: Turning Your Mac Mini Into a Passive Income Machine (32:49) IOTA Explained: Building a Global Supercomputer From Spare Parts (39:43) How the Network Scales: From 256 Nodes to Limitless Compute (44:39) The Road Ahead: 70B Parameter Models and the Future of Affordable A

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  So my name is Stefan Kruse. I am the co-founder and the CTO of MacroCosmos. I hold a PhD in subatomic physics in University of British Columbia. So I was a physics researcher for the beginning of my career and then I pivoted to AI. When it became apparent to me that there's a lot of opportunity for scientists that are just about to graduate from their postgraduate studies. And there's just, there's an entirely new science sort of being born and developed in real time for us. And I think my decision is one that a lot of other people have made as well, where they want to go from being in what can feel like the sort of the old school machinery of traditional

• 0:42Speaker 00

  science in academia, where things move slow and your contributions can be very narrow and sparse, to becoming a pioneer of this entirely new exciting thing, which is not only abstract and theoretical, but it's being applied in new and interesting ways constantly. So when I saw that, I concluded my PhD and I decided to leap, leap, leapt out of academia and into sort of the application of AI and also physics in various private sector domains, such as manufacturing and systems optimization. Following that, I discovered BitTensor, which is a very interesting AI blockchain project. And ostensibly, the purpose of BitTensor is to allow AI to be developed in a way that

• 1:26Speaker 00

  is fully democratized and is sort of done in a global, a global way with incentives. And I thought that was such a fascinating proposal that I decided to join the network. I've been within the BitTensor ecosystem for around three years now, and I have been relentlessly experimenting and building within BitTensor, the kind of things that I think would make a lot of sense and would bring a lot of value to the world, using a blockchain as a prerequisite tool to enable AI research to be done in various ways and scales that would be very difficult otherwise.

• 2:00Craig S. Smith?

  Yeah, and specifically, well, first, let's talk about BitTensor. But before you began, we were talking about how there are a few of these blockchain ecosystems for either training or marketing AI models or AI applications. And I'm familiar with SingularityNet. BitTensor is much larger, is that right? Is it the largest of these or one of the largest? And how many are there?

• 2:37Speaker 00

  I believe so. So BitTensor is actually over 100 projects under a trench coat. And I think that's also why it feels so encompassing and so large. And this is what is very interesting about BitTensor is it doesn't seek to solve a specific narrow problem in the AI industry. It's not like trying to solve something like, how do we provide people with access to models? Or how do we train models? Or how do we do one of a plethora of different things? It actually has done a lot of foundational work on how do we make it possible to build a base layer for people to come and solve just about anything they can imagine in a way that it uses the blockchain as a reward mechanism, as a coordination layer. But then kind of BitTensor does a very good job of stepping out of the way and letting

• 3:21Speaker 00

  people's creativity drive them through to build what they want on top of it. So BitTensor feels very big and very sort of like it's everywhere because there are, today, there's 128 different teams that build on top of BitTensor. And they're all building something different. A lot of them choose to work within AI, but there's also some that work outside of AI as well. And so for that reason, I think that BitTensor punches well above its weight.

• 3:45Craig S. Smith?

  Yeah, and for those who are not familiar with the blockchain, and even me, the blockchain in effect is a registry of addresses, right? And it points toward the actual assets that are being registered. I mean, this is the thing about blockchain art that was such a craze for a while that you're not actually buying a piece of... I remember Trump was selling these...

• 4:20Craig S. Smith?

  I think playing cards or something like that on the blockchain and you're not actually buying anything. You're you're registering your ownership of something.

• 4:33Speaker 00

  Exactly.

• 4:35Craig S. Smith?

  But the actual asset is a digital asset that can be copied millions of times. It's up to you then if you want to enforce your ownership to track people down and and and that sort of thing. So so in in in the the projects that I've seen related to AI, they it's a registry that points to an AI project? Or is there something more happening on the blockchain?

• 5:06Speaker 00

  So in a project like Bitens, so the blockchain is used in a number of different ways. But fundamentally, it's it's an immutable database that is globally shared. And that in itself enables a lot of things that Bitcoin pioneered some fifteen years ago. What what becomes possible when you have this sort of shared globally distributed state? Is it you can create a store of value and then you can start building higher and higher order commodities on top of that. So Bitcoin sort of addressed the most principal version of this. But what was demonstrated as a result of that is you can actually aggregate a huge amount of compute. Or let's just call it effort.

• 5:47Speaker 00

  Once you have this interesting distributed sort of system architecture, people come to it and and because there's economic incentives involved, people are constantly innovating and optimizing and they're trying to find a way to sort of percolate to the top and to become the most competitive people that are doing whatever this mining task involves. In the case of Bitcoin, the mining task is just guessing random numbers faster than anyone else can. That's why you just throw a more powerful computer at it and you become a more competitive miner on the Bitcoin network. And other crypto projects have tried to orient that effort, that sort of collective coordinated effort of all the participants. They've tried to orient that towards different tasks, whether it's distributed.

• 6:29Speaker 00

  File storage or creating a global residential proxy network or solving data scraping from the web. And all of these things are really interesting. Whereas what BitTencer uses the blockchain for fundamentally, as you described, the registry component is very important. It's also sort of a shared synchronization clock. The notion of blocks and a blockchain gives everyone sort of a drumbeat that you can actually anchor work to, which is an integral part of a lot of our research on BitTencer. We have this sort of shared clock, which the blockchain regulates. But ultimately, you can do as much or as little on-chain, as they call it, or off-chain as you want. And I think some of the benefits of doing things on-chain is you have complete auditability.

• 7:13Speaker 00

  You can see precisely what the details were that went into a specific piece of work or transaction. So it gives you transparency. The problem with that, of course, is if you've got billions and billions of transactions being written to a database, your database gets very big very fast. And then it becomes kind of unyieldly for people to actually have a copy of, which can interfere with the true decentralization ethos of it. And you end up with just a few copies. So I think that there's a pragmatic middle ground where you store exactly what you need, but nothing more, to ensure the core parts of the work that you care about are captured and untransparent for everyone to audit, review and verify. Whereas the rest of it, you can store elsewhere. You can store offline, you can do a million other things with it.

• 7:57Speaker 00

  So I hope that adds a little bit of light to the question, Greg.

• 7:59Craig S. Smith?

  Yeah, in your case with MacroCosmos, you're providing compute for training models. Is that right? Explain precisely what MacroCosmos does.

• 8:12Speaker 00

  Happily. So today we operate three subnets in BitTencer. Each subnet is, you can think of it as effectively a different project, a different service. And our three different subnets, I would consider them to be three different. Three different axes or three different use cases for the blockchain that I think are very potent and powerful. So one of them is just the aggregation of raw compute at scale. And if, again, harking back to what I described about the Bitcoin network, I think one of the most important things of our time is set to be AI itself. And those who train the models and own the models will have an enormous advantage. And so a big part of

• 8:54Speaker 00

  the purpose of BitTensor is to provide an alternative way for us to create these models, to train these models, to deploy these models, such that they are not held in walled gardens and we're having to pay for the privilege of access to them. But we want to reinvent that in a way that's more democratic and also borderless, something that's censorship resistant, something that cannot be controlled. So our primary, our flagship project within BitTensor involves what's called distributed training of large language models. And that's quite a, quite a mouthful. Distributed training is quite different to how a lot of the frontier AI labs train models today. Typically, you have huge warehouses stacked from floor to

• 9:38Speaker 00

  ceiling with GPUs, which are specialized computers for training AI models and serving AI models. And you've got hundreds of thousands, hundreds of thousands of them in the largest data centers in the world. And they're all plugged up together with extremely high data transfer speeds. And as a result, you can basically use the whole thing as a single computer. And this is fantastic. And it's been demonstrated that you can just scale the amount of compute and the amount of data you throw at a model and they get predictably better and better and better. This is called a scaling law. Now, there's an alternative to this, which is called distributed training. Distributed training doesn't rely on having a single warehouse stuffed with computers, but actually, it's based on the premise that you can train an equivalent model using computers that are distributed around the

• 10:22Speaker 00

  world. And now this has a lot of very interesting outcomes. When you don't do everything in a single data center, one of the first thing is, one of the first things is that the, the capex of actually building this massive warehouse in the first place is massively alleviated. There's also the local energy that is required to run one of these massive data centers can also now be distributed around the world in a way that creates less impact on local communities, generally much, much less detrimental to the environment.

• 10:53Speaker 00

  And lastly, it allows you to do something that is actually fundamentally not available. Once you've built your data center and stuffed it with GPUs, sort of the cost of training models is already baked in to that initial build process. Whereas in the case of distributed training, which is what we care about, we can actually perform what is effectively cost arbitrage. So if there's, if there's a surplus of energy in Iceland, very, very cheap energy, we can actually use that pocket of energy, even if it's only for 12 hours of the day, and we can target a lot of that compute and use it in a very elastic way. And in fact, that's what we do. Today, we are training models, we're training multiple models all at once, using pockets of cheap energy, which translates into cheap compute.

• 11:35Speaker 00

  So it has all of these different economies of scale, which are certainly unusual, but are becoming more and more accepted as a viable alternative. And as our appetite for bigger and bigger models is only going to increase as the years go on, we're going to find that we need to start looking at alternatives before we hit a hard ceiling on a lot of this, because things like the Stargate project and the Colossus project, these are multi-billion dollar GPU build outs. And so we think that, just like the fundamental physics experiments, like the Large Hadron Collider, at some point, you require the budget of a nation's state. It's $20 billion to build a bigger ring to smash protons or electrons. You, you need to start thinking about different experiments, because it just becomes unpalatable eventually.

• 12:19Speaker 00

  So we're, we're trying to get ahead of this problem. I think that by 2028, we're actually, the world, the mainstream, the Overton window about training models is going to shift, and we are going to have to start thinking about how do we do this in a way that arbitrage is more efficient in both cost and energy. So we're, we're trying to do a lot of the early work right now on that. And BitTensor is a wonderful place for us to do this research, because we are able to use the blockchain to reward people that contribute their compute to our training experiments.

• 12:48Craig S. Smith?

  Let me ask a couple of questions. I had a guy on the program who was doing something interesting. He was, and I don't remember whether it was blockchain based, but, but he would aggregate spare compute from data centers, or, you know, whether on premise or, or.

• 13:12Craig S. Smith?

  Uh, you know, independent data centers around the world. And then, uh, offer that compute to AI projects, uh, at a, at a cheaper price than, than the hyperscalers or

• 13:29Craig S. Smith?

  uh, the big clouds, uh, could offer. Uh, and, and so is it, that's one question, is it like that? And you're just tying it to the blockchain to, to organize it, uh, and distribute it. And the other is, uh, you know, for a long time, people were talking about federated computing. Uh, so that you wouldn't have to lose, uh, control of your data. But you could make it available for training, uh, models.

• 14:06Craig S. Smith?

  And, uh, again, that, that was using the blockchain, uh, to register the, the data, I guess. Uh, so first of all, uh, i- is it related to either of those concepts? And, and then the, the deeper question and the one even talking to the people I've spoken to, I've never really gotten a hold of. The chain again is just a registry. There's no training data on the chain. There is no compute on the chain. The compute resides, as you said, uh, maybe in Iceland.

• 14:48Craig S. Smith?

  What is the link between the chain and the compute in Iceland?

• 14:53Speaker 00

  So starting with your first questions, the relationship between those two examples you gave and what we're doing. I, I would say that they're cousins, but they're not, uh, they're not siblings. So they're, they're a little bit different. For example, federated learning, uh, is built on the idea of anonymizing client data, but being able to continuously train models. We, we're working with a slightly different version of this, which is not so much the edge devices providing training data today, but the edge devices are providing a train compute. I suppose there'd be a way to think about that. And secondly, I think. Regarding what BitTensor actually does, or what the role of the chain is, is at its core,

• 15:37Speaker 00

  it's really not that revelationary at all. At its core, what the blockchain is effectively doing is it's providing a trustworthy, transparent record for everyone to audit, especially when there's code that's being run on the blockchain as well. People understand how this code is going to interact. It can't be tampered with, which means that it's predictable, and that's a proxy for safety and trustworthiness. So if someone had already deployed a smart contract, for instance, we're not actually building a smart contract at Microcosmos, but I guess just to illustrate the point, if someone has written something in the smart contract, then the creator of the smart contract can later change the terms and conditions on you.

• 16:21Speaker 00

  The point is, it's a modular block of code that will run under these conditional expressions, and that makes it something that you can effectively count on. So I believe I've given you a bit of an illustration of one of the projects that we're working on at BitTensor, which is distributed training. This project is called IOTA, and as I mentioned, I think that it's a precursor to some very powerful downstream technologies that we may find are an instrumental part of the way that we create AI in the next decade, let's say. Yeah. We really do believe that there's also a real change in the way that people are thinking about personal agents and personal compute.

• 17:04Speaker 00

  So I've seen personally in the last few months, a lot of people are starting to stockpile Mac minis. I'm not sure if you've seen this trend, but basically now that agents are starting to become kind of economically useful, a little bit, we're seeing the beginnings of it, I think, is a fair appraisal. What we're starting to see now is people want one of these things running for them privately around the clock, day in and day out, so they're buying a dedicated computer. And on that computer, your personal agent is running, and it's checking your emails, it's maybe doing some online shopping, maybe it's doing some work for you on the side or some hobbies. And this is a wonderful use case, and this is also... A really interesting place where our technology of IOTA developed by Microcosmos comes in because all of these computers that people have now built at home.

• 17:48Speaker 00

  Well, there's two ways to think about it. That's your personal agent that that is creating value for you personally. But wouldn't it be great if that compute could earn passive income? Almost like you own a property and you can Airbnb it out when you don't need it. And this is actually what we are offering as we develop IOTA is we've created something called Train at Home, which is people that have unused devices at home. Let's say MacBooks, MacMinis, or even consumer GPUs. They can plug those into our network and that means that they become part of this global supercomputer. So we can use them for training, training models, which we can commercialize and they can also basically create some passive income just from having that device sitting around that was otherwise underutilized.

• 18:31Speaker 00

  So I think there's something very parsimonious about that arrangement of things. And the more people that are choosing to buy personal devices to run these agents, you realistically don't need the agent running 24 hours a day. You need pockets of productivity. I want to make sure it, you know, plans my weekly agenda. I want to make sure it checks this, that and the other. Maybe there's only four hours of the day that you actually need this thing and you can actually get a return on your initial investment within, you know, 12 days, 30 days by plugging it into a network like this and also being part of something that I I believe has a great purpose, which is the democratization of AI itself. So it it's a really interesting intersection of timelines right now with with agents becoming demonstrably more useful and engagement increasing.

• 19:12Speaker 00

  And we basically can use all of that consumer compute. We can glue it together and we can train models at greater and greater scales and greater and greater utility. So, yeah.

• 19:23Craig S. Smith?

  Yeah, yeah. As a matter of fact, I'm running Mac Mini stockpiling. I think started with Open Claw. And uh

• 19:31Speaker 00

  Exactly.

• 19:32Craig S. Smith?

  I'm running Open Claw on this computer, which I know is a bad idea and I have an unused Mac Mini upstairs. So uh uh I'll probably go fix that after this call, but uh but the uh so so how does somebody use this? Uh well, first of all, yeah, back to the point of, so you've got compute

• 20:00Craig S. Smith?

  around the world that is plugged into the network. Do people who are using a macrocosmos, do they need to know what's how the what the link to the blockchain is?

• 20:19Speaker 00

  Absolutely not.

• 20:20Craig S. Smith?

  Yeah, and and the blockchain is just keeping track of where the compute is. And what's available at any point in time, is that right?

• 20:33Speaker 00

  Yep, and it's just responsible for rewarding participation. So it's making sure that any contributions made by anyone's devices are fairly fairly rewarded. That's really the way I think about this. The blockchain is just a really convenient way of taking care of compensating people for their contributions. The actual onboarding experience for people is we we keep this as low cognitive load as possible. Right now, it's a one click app store download. This thing will just run passive in your machine, adding some more progressively more useful user experiences. Things like telling it, oh, you can only use my computer when I go to sleep.

• 21:13Speaker 00

  So it's by default, it's disabled until 10 P M. And it's only enabled from 10 P M. Till 6 A M. So that will be a simple user control that will, you know, get it out of the way of when you're actually trying to be productive with your machine. That's assuming it's your primary machine. If it's a secondary machine that just sits at home, we're also creating a way for agents to decide, oh, I finished my work. I'm not going to need to do anything for four hours. So the agent just decides, hey, I'm just gonna go and make 20 bucks. And that would be nice to come home from work too, right? It'd be nice to come home from work and you're like, hey, what have you done today, Claude? And it's like, oh, hey, I finished all my work by 9:15 A M. And you were going to be out till 5 P M. So I thought, well, I thought I'd just shop around and see how I could make a little bit of side cash. And here you go.

• 21:55Speaker 00

  So I do think that as our computers become more autonomous, you know, thinking about our relationship with computation, it's going to change fundamentally. And our computers are not going to sit there waiting. They're going to be very proactive and they're going to be very resourceful, especially when you've got these clever clever little agents that are running the show inside of the machine. And as a result of that, I think we need to really rethink what your computer can do for you. You know, our relationship to that world. And I think bringing this back to the top of that BitTensor, you can think of BitTensor as it's just a bunch of places where you can use that machine. And you can go make some some income from it. Whether it's providing something like human intelligence,

• 22:36Speaker 00

  or whether it's training models, or whether it's detecting if an image is real or fake, you can almost think of it as a mechanical Turk agents and humans alike. Where you can go there, you can monetize your skills and your experience, or just your raw compute.

• 22:51Craig S. Smith?

  Yeah, and and for the training,

• 22:56Craig S. Smith?

  how does, let's say I have a large model that I want to fine tune. How do I do it with Microsoft Cosmos?

• 23:09Speaker 00

  So today, our work is focused on the first stage of training models specifically, which is called pre-training. So pre-training has historically been, by and large, the most computationally expensive part of creating a model. It's the part where they systematically inhale the entire internet, which usually takes months and tens of thousands of GPUs. Beyond that, fine-tuning is comparatively a much shorter computational task. So I think it's the essence of pre-training requiring a long-time horizon and workloads. That is actually why we're so interested in moving this from a centralized to a decentralized context.

• 23:52Speaker 00

  And there's a lot of pioneering research that needs to be done that we hope will be very useful, even outside of the Web3 world. We hope that this is really useful research to anyone out there that is interested in economically or capital-efficient ways of training models. We hope that this is a contribution to the field. That's really how I see this. Because what we're doing at the root of our work is we're taking a network of very unreliable compute. People are joining, people are leaving. It's just constantly churning over and churning over. How do we create something that is persistent and stable? Out of something that is fundamentally so noisy and unstable? And it's core, that's the problem we're trying to solve.

• 24:34Speaker 00

  And it just so happens that the use case for this persistent compute fabric we're creating is trading models, but there's nothing stopping that. From doing something else. Perhaps there's a world where universities could use a similar decentralized network for their traditional HPC cluster jobs where they're doing academic research. That's not necessarily AI related at all. It could be bioinformatics, it could be physics. The same thing, the same thing is still, uh, still remains to be true, which is if you need 10,000 nodes of compute for 12 hours, what's the cheapest way you can get them? And we're trying to build software that basically orchestrates compute around the world to act like

• 25:16Speaker 00

  it was all plugged into itself and it's a supercomputer. And I think it's a, it's so, it's, it's a fundamental thing that we're working on that we're applying towards training models, but I hope has a much broader value to the community.

• 25:27Craig S. Smith?

  Right. Let's go on pre-training. Uh, how does somebody use it?

• 25:32Speaker 00

  Absolutely. So we're sort of emerging from, from research mode right now. We spent about nine months on IOTA so far, and we've demonstrated that we're able to reproduce a lot of important baseline benchmark performance metrics using this, what we call it the wonky vegetables, sort of the ugly vegetables that you get in the vegetable aisle that no one necessarily wants to buy. We're going to turn that into a premium soup. And we're basically graduated from research mode right now while we're showing that the two are indistinguishable. If you treat them in the right way, if you use the right spices, tastes just as good. And what, what we're moving to next is thinking very deeply about what is the, what is the commercialization opportunity for a technology like this? Who are our target audiences? We think that, that the world is going to train more

• 26:17Speaker 00

  models in the next 10 years, not less. We think that researchers, small startups, especially cash strapped startups, academia, all of these people are interested in training more models, but perhaps for budget reasons, it's very hard for them to do so. So we're, we're sort of targeting the, the commercialization of IOTA as two things primarily. One is what we call supply side. So supply side would basically mean people that have surplus GPUs. So the NeoClouds, the hyperscalers, perhaps they have 10,000 GPUs in their inventory, but they can only rent out 9,000 of them at a given time, or 9,500 of them. Well, for us, them plugging in their compute into our network allows them to increase their overall utilization,

• 27:02Speaker 00

  which goes straight to their bottom line. So they'll be happy with that. And even better, that if you actually to look at the usage of all of those GPUs that a hyperscaler has, you might find that, well, it's rented out for four hours and then it's not rented for two hours and then it's rented out for four hours. And that little two-hour gap is exactly what we're trying to utilize. These short, interruptible bursts of compute, and we turn that into, again, a continuous stream of compute is exactly what excites us. So for the supply side, it's basically anyone that has GPUs. What they have to do right now is if they can't rent it out, they're sort of a buyer of last resort on the market. They'll rent out those GPUs at cents on the dollar for inference tokens.

• 27:45Speaker 00

  And we can basically say, well, hey, instead of doing that, how about you plug them into our network and we can basically give you a better margin? Because trading is a higher-order commodity than inferencing. And as a result of that, we believe we could pass on the benefits to them as suppliers. So that's the supply side. On the demand side, as I mentioned before, we have researchers, we have startups, we have all kinds of different profiles of users that we know right now, they're training models all the time. What we want to do to that, for those guys, is basically provide them with an interface to train models in a way that's recognizable to them. So there's already very popular libraries like PyTorch, like TensorFlow. These are very popular libraries that are basically used by everyone in industry that's training models.

• 28:29Speaker 00

  Well, we want to make it as simple for them in terms of abstractions as if they were just building models in the normal routine way. So with no additional cognitive overhead. You basically say, I want to train a model. You're not going to painfully specify, oh, I want to use a Nigerian GPU for 12 minutes. Of course, you're not, right? What you're going to do is you're going to say, this is my sort of executive level objective. This is what I want to get done. These are the parameters that I want to have precise control over for reproducibility purposes, for deterministic purposes. All of this section down here, feel free to go and arbitrage and get me the cheapest. And then we have this really now we have a two sided market, right? Now we have a sort of supply side and and and a demand side that are sort of dancing together where we're arbitraging,

• 29:13Speaker 00

  offering cheaper rates and and I order is a technology that basically is this infrastructure layer that is is the one that brings this compute to market, pairs it up with users that want it. And we believe that that's something that is phenomenally valuable.

• 29:28Craig S. Smith?

  Yeah, well, and then how does the the under the hood? How do you get the the the compute to the to the model? Cause they're not sending the model around all over. Uh, yeah.

• 29:46Speaker 00

  The way that you get the compute to the model is we it's it's effectively a orchestration deployment layer, um, something reminiscent of Kubernetes. But Kubernetes that deploys to heterogeneous compute nodes around the world. So it's it's a dynamic deployment layer that basically takes some containerized code that a user writes, sends it out to those machines and then very importantly ensures that those machines around the world have a tunnel to communicate with each other. And that tunnel is the part that makes them act like one big blob of compute instead of fifty different random pieces of compute. So it's it's effectively that it's it it deploys anyone's code as an interactive system or a network of compute that can in that is sort of geospatially aware.

• 30:33Craig S. Smith?

  Right. And and that orchestration layer um is where is that based?

• 30:42Speaker 00

  The orchestration layer is something that we maintain and operate. So you would effectively dispatch your code from your laptop or anything like that. Your code would then be sent securely to these backend nodes wherever they might be distributed around the world. And then you would effectively as a user, you would have the ability to monitor logs, monitor trace data, the usual sort of dashboarding, tooling, things like that as the user. But effectively what's happened is your code has now been sent out, cloned, replicated to all these different copies. Those copies are hosting a section of the model, which is what makes IOTA very, um, in my opinion, very novel.

• 31:24Speaker 00

  We, we don't actually have each of these compute nodes running a full copy of the model. They're actually running a small sliver of the model. It's called model parallelism. And in effect, what this means is you can train really large frontier sized models using very small building blocks. Like you can build a huge Lego tower out of small pieces. It's the same idea. And the way it works is in order to train the model in its entirety, you have to root information from all of the sections of the model and knit that together. And as a result of that, you can basically train this, this sort of distributed model architecture as if it was all stuck together in one place. And that's, uh, that's why it's taken us nine months to get out of research mode. To be honest, Greg, it's, it's quite a formidable task.

• 32:08Craig S. Smith?

  Yeah. And, and is the blockchain important for the synchronization? Because you need a very tight synchronization between nodes of, or between, um, uh, different compute, right?

• 32:26Speaker 00

  So the way that we utilize the blockchain today for IOTA's purposes is we need a registry of everyone that's in the network right now. So it exposes them as, hey, I'm an available worker. Here's how you can find me and here's my unique, uh, address that I know to identify you unambiguously. So we, we use that as sort of, uh, uh, an authorization layer, an identity layer. Also with that, we, we have an off-chain layer to IOTA, which is constantly tracking the contributions of that node. And then once we have computed the total work done by each node, that is sent back down to the blockchain layer where it triggers a payout loop.

• 33:10Speaker 00

  So everyone that contributed that compute is then rewarded in the IOTA token for their contributions of work. And, and this actually can be articulated in a very tokenized context, or even a US dollars context. And all you effectively need is building blocks here is, is, uh, you, you need to know exactly who it is that's assigned to this model training experiment. You know where you can find them, you know, how to glue them all together, and you know how much work they've done. And if you've got all of those things, you have an operational distributed trading network.

• 33:40Craig S. Smith?

  How large is the network and how do you, how do you grow the network of compute?

• 33:48Speaker 00

  Yeah, so when we began, the source of all of the compute for our distributed trading experiments was people that came to our project in BitTensor, the IOTA project, with, they have their compute, they know what we're here to do. Usually there's a philosophical alignment or sometimes they just see it as an opportunity to make some cash. But basically, people show up armed with usually pretty powerful hardware. So the actual, the supply side of computing BitTensor is very much not a concern. People know that BitTensor is there. People know the rules of the game. So we were almost always oversubscribed in terms of people that wanted to contribute to our experiments.

• 34:32Speaker 00

  However, in December, November, December last year, we wanted to go beyond the scale of the network that is supported natively by BitTensor. So the maximum number of people that could be participating in the IOTA experiments was 256, which, if you consider that to be a data center, it's tiny. It's a pebble. So we actually worked around that and we've now made it effectively a limitless-sized system by instead of having every unique entity as a unique address on the blockchain, we now keep track of it off-chain effectively. And what this allows us to do now is we can have an arbitrary-sized registry of participants.

• 35:11Speaker 00

  So we had 2,500 people download our macOS app in the first two weeks since it launched. And now all of those people are available at a moment's notice to join our network. And they're paid out identically to the ones that are strictly on-chain with a BitTensor. We have a transparent payout system that people understand. It's a predictable system. It rewards, it directly rewards contribution in proportion to basically hours of work done per day. But the nice thing about this is it's not limited by the amount of storage capacity for the blockchain or the number of slots that a blockchain can have as miners. We actually decided to just rewrite a lot of that using different rails so that it's not, it's not limited in that way.

• 35:53Speaker 00

  And now as a result, we have 500 or so today running. The amount of work that we have to give the miners to do is actually quite...

• 36:05Speaker 00

  It changes over time. There's times when we run lots and lots of experiments, we train lots of bottles. And there's also time when we only need a small amount of compute and we only pay for what we need. We're not just, the forster isn't just always running. We actually have experiments that are live or we have production runs where we're going to say, this is the model we're going to train. We're going to go find 500 devices around the world. Fortunately, again, the bit tensor piece has made it very easy to find 500 random people in the world with powerful computers that I could just plug into my network, which is the miracle of technology, really. And so we can effectively not worry at all about where that compute comes from. And then downstream, we basically pay everybody out for that work that they've done.

• 36:47Craig S. Smith?

  Yeah. And is there, do you have a sense of scale of comparative scale that that would give me and listeners an idea of of how this would compare to

• 37:04Craig S. Smith?

  to a major data center?

• 37:07Speaker 00

  Yeah.

• 37:08Craig S. Smith?

  Yeah.

• 37:09Speaker 00

  The the largest data centers in the world are measured that the ones that have started to come online are now hundreds of thousands of GPUs. So these are multi-billion dollar projects. They're absolutely enormous. We're not, we're not at that scale yet. But the nice thing is the technology that we're building scales in a very beautiful way. In fact, you know, to go from a day, to go from our network with 10,000 nodes to 20,000 nodes, there's nothing that we fundamentally need to change. We just need to expand the the distribution surface of what we're doing and our infrastructure layer scales to much, much higher capacity. So again, this is the nice thing about not being committed to a bricks and mortar

• 37:50Speaker 00

  build out in the way that the centralized compute paradigm is. In our case, if we want to double, triple or 10x the capacity of our compute network, we simply need to go out, increase the surface of the network. So what I think is in the books for us is by the end of this year, I want, well, by the middle of this year, I would like us to reach 5,000 compute nodes. I think this is a respectable sized cluster that you can train a model that will get people's attention and make them understand that there's a lot of utility in this technology. I think that's an important milestone for us. Something around the 70 billion parameter model size, I think, is when you go from, you know, you graduate from the sort of small models that are proof

• 38:34Speaker 00

  of concepts to things that are actually taken much more seriously and enterprise customers that have been waiting to train an in-house model, whether it's a legal specialist model or whether it's a medical model or something like that. Perhaps they've been waiting for a while, but they couldn't really afford it because it's a very high price tag that comes with training a model of that scale. Because of the discounts that we can offer using this energy and cost arbitrage approach, I think that we're going to start to see a lot more people train their own sovereign models, their own enterprise models. And so by mid this year, it's important to us that we demonstrate that that is possible with the technology that we're working on. And in a year, a year and a half from now, I'd like to go beyond 100 billion parameter models. So the parameter count of the model

• 39:17Speaker 00

  is very much the number that we think about when we consider what it takes to make this technology something people take seriously. I think we need to have a portfolio of models that we can point at and say, these are just as good as the ones that you would have gotten in a centralized context, but at 10% the cost, at 20% the cost. I think then people really start to take notice and it becomes an interesting proposition. And that's where we'd like to get to.

• 39:44Craig S. Smith?

  Yeah. And we were talking about people stockpiling Mac minis. The people that are joining or contributing compute to macro cosmos are GPU, people with GPUs, spare available GPUs, right? Not CPUs.

• 40:08Speaker 00

  It's yeah, we're not going to get anything done if we wait for CPUs to do the work, unfortunately. Perhaps there are workloads in the future. As I mentioned before, we can imagine this technology is something that allows you to do more than just model training. We're working on a more fundamental infrastructure problem, not just a model training problem in many respects. And there are a lot of scientific problems, computational expensive, scientific problems that are CPU limited. And in those cases, absolutely, it will be very interesting to apply this technology in those cases. But today, we support effectively CUDA devices and silicon devices, which are Mac Minis, the new MacBooks.

• 40:49Craig S. Smith?

  Is this online yet? I mean, can, are people training on this network yet? And if not, when will it be available? Do you have people lining up?

• 41:00Speaker 00

  Yeah, you can find us at iota.macrocosmos.ai. That's I O T A, which I should probably have introduced as an acronym for the incentivized orchestrated trading architecture.

• 41:13Craig S. Smith?

  Okay.

• 41:14Speaker 00

  So iota.macrocosmos.ai is where you can learn more about this project. Today, we are, we don't have any customers in yet. As I mentioned, we're coming out of research mode right now. We're trying to really calibrate the system and tighten this thing up. There's a lot of moving parts that we want to make sure we have really nailed down. But we have startups that are already ready and committed to working with us, training models towards the second half of this year, which is incredibly exciting. So we'll have some collaboration results and some early partnership results by the end of summer.

• 41:44Craig S. Smith?

  Okay, great. And in one other question, this is for compute, but we spoke about federated computing. Could you register data on the network or make data available? And then for pre-training and then have Macro Cosmos send the model around to different

• 42:15Craig S. Smith?

  blocks of data to train the model?

• 42:20Speaker 00

  It's a very interesting idea. One of our other projects in BitTensor is actually a web-scale data scraping service, where we decentralize the efforts of hundreds and hundreds of miners that scrape social media data for us. And we can use that for many reasons, anything from journalism to marketing, brand analysis, all the way to AI model training. So we actually have an entirely standalone project which is dedicated to data scraping, which we think is an important part of our... I call it sort of virtuous cycle. We have data, we have compute. We also have a lot of what we call innovation networks, which I didn't get much time to talk about today. But to your question directly about whether we could actually outsource client-side data

• 43:04Speaker 00

  and use that for model training, my answer is we already are. We actually just have it in a dedicated, highly scalable system called Data Universe.

• 43:14Craig S. Smith?

  Okay, okay. Well, this is fascinating. I'm gonna be paying attention to see how this scales. Do you think this model will be replicated by others? If you... Yeah.

• 43:31Speaker 00

  I think it's very important that it is. Do you mean the technology or the actual downstream model artifact?

• 43:40Craig S. Smith?

  The technology, the technology. Because, as you said, the appetite for compute, for pre-training compute is gonna grow. The number of models out there is only gonna grow. And compute is expensive and a lot of it's underutilized. So it just feels like there would be other people doing this. And another question is, are you working with any of the big cloud providers to maximize their GPU utilization?

• 44:29Speaker 00

  To your first question, I certainly, I see other teams right now that are working on very similar projects. Some of them are focused specifically on model training, or others are more oriented towards this orchestration technology. I think we're one of the very few that's combining those two in the way that we're doing it right now. So I do expect the world of devices, you can call it the Internet of Things, you can call whatever you want, but I think that there is a second law of thermodynamics at play here, which is things get more connected over time. And I think that especially when there is a lot of value added and a lot of potential to having multi, the orchestration of multiple devices to do higher order workloads.

• 45:13Speaker 00

  So in our case, there's simply not enough memory on one Mac Mini to train the models that we want, which is why we have to do all this work to glue them together. And there's many, many other problems where there's just not enough resources available on one device to do what it is that you need. So I think that there's a lot of other industries that are already thinking carefully about how can we get access to the kind of compute in the specific topology we need it. And I think that what we're building is going to be very interesting to them. Devices, as I said before, devices are only going to get more intelligent, more autonomous, and they're going to be, become much more useful if you can stick them together like Lego bricks and make these bigger pieces out of them, these different constellations. So I see a bright future for that effort right there. Whether we're the ones that really take this all the way to the top of the mountain,

• 45:57Speaker 00

  or we get halfway there, I'm proud of the work that we're doing. And I think that it's, again, I think it's very valuable for everybody that we work on this.

• 46:06Craig S. Smith?

  Okay, great. So iota dot macrocosmos dot.

• 46:15Speaker 00

  Dot AI. Yeah, you can find everything you need about us just at macrocosmos.ai. And you can find all of our projects and our research in decentralized AI.