Photonic computing for AI acceleration

There are a lot of people trying to innovate in the area of specialized AI hardware, but most of them are doing it with traditional transistors. Lightmatter is doing something totally different. They’re building photonic computers that are more power efficient and faster for AI inference. Nick Harris joins us in this episode to bring us up to speed on all the details. Sponsors: Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipLaunchDarkly / TrajectoryConf https://trajectoryconf.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=the-changelog – Software powers the world. LaunchDarkly empowers all teams to deliver and control their software. DevOps and feature management are reimagining how we build and release new products. On November 9th and 10th, LaunchDarkly is hosting Trajectory Conference 2021 — a two-day event for software innovators who want to break orbit, not systems. Trajectory is a fully-virtual conference that focuses on the technology, people, and processes that continuously deliver better user experiences and more powerful software. Register for free at trajectoryconf.com https://trajectoryconf.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=the-changelogLinode https://linode.com/changelog – Get $100 in free credit to get started on Linode – Linode is our cloud of choice and the home of Changelog.com. Head to linode.com/changelog https://linode.com/changelog OR text CHANGELOG to 474747 to get instant access to that $100 in free credit. Featuring: Nicholas Harris – Website https://nicholascharris.com, X https://x.com/theanalognickChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Lightmatter https://lightmatter.co/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  If you look at technologies for powering AI right now, they are all based on transistors. There's never been a computer that's not based on transistors. Like, this is the way that the world does computation. But we've run into this kind of fundamental challenge around shrinking how much energy the chips are using. And you can't really do that going forward. So if you look at the Department of Energy's estimate for energy consumption, 2030, 10% of the entire planet's energy consumption will be on compute and interconnect. So if we look at 2040, you're talking about most, like the overwhelming majority of the power being used on this. You always have to think in business about the use case that could motivate using that much of the planet's power, because that would cost a lot of money. You will start to see that progress in AI will slow.

• 0:47Speaker 01

  Big thanks to our partners, Leno, Fastly, and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 1:10Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at PracticalAI.

• 1:36Daniel Whitenack?

  Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International. And I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

• 1:50Chris Benson?

  Doing very well, Daniel. Excited about today's show. I think we have some pretty good stuff coming up here. I'm just kind of rearing to go.

• 1:57Daniel Whitenack?

  We've talked about a lot of computers on this show and accelerators and cards and GPUs and VPU's and all sorts of use, but we have yet. To talk about photonic computers. And that's going to be the topic of today's conversation. Today, we've got with us Nick Harris, who is CEO of LightMatter. And we're going to hear a lot more about that and photonic computers today. So, welcome, Nick. It's great to have you here.

• 2:29Speaker 00

  Thanks for having me, Daniel and Chris. And nice to meet you.

• 2:31Daniel Whitenack?

  So, a lot of people might have not heard of photonic computers, or even know where they sit in the sort of landscape or ecosystem of accelerators and types of processors and all of this stuff. So, maybe just give a little bit of a background on how you came to encounter this topic? And then also, like, maybe some of the motivations behind this whole topic of photonic computing?

• 2:56Speaker 00

  Yeah, so I'll start out with just talking about the AI field as it's growing. So, in around 2012, we had AlexNet come out. AlexNet was a neural net that was able to do image recognition tasks that were really incredible. And there was no other technique that could really keep up with it. And it made a great business case for companies like Microsoft, Google, Facebook, Amazon, and so on, to start building neural networks to do special kinds of problems, things that were very hard to write code for, but that neural networks could solve. And over the past 10 years now, we've had a really fast rate of progress in the amount of compute that's going into these state of the art neural networks.

• 3:38Speaker 00

  It's motivated a lot of companies to start building accelerator chips to try to speed up these computations and make it so that we can make bigger and bigger neural nets. And some people's objective on this bigger and bigger scale is really to try to build something that's on the scale of a human brain. So, that's like, kind of, the trajectory where things are going. There's huge capital expenditures that go into building these kinds of AI supercomputers that can do incredible tasks. And what we're doing at LightMatter is we're trying to take a crack at that market with a unique twist. We're using photonics to do computation. I can tell you how I got to that place.

• 4:17Daniel Whitenack?

  Please do.

• 4:18Speaker 00

  Yeah, I was an engineer at Micron, working on transistor device physics. So basically, I was getting a lot of exposure to the fab processes and some of the challenges that companies, not just Micron, are having in shrinking transistors and getting more performance out of them year over year. So I decided to go to graduate school, ended up at MIT, and studying quantum computing. That's a bit of a leap. What LightMatter's doing isn't quantum computing, but I did spend those years building photonic quantum computers, and that happened right at the advent of that AlexNet neural network kind of coming out. And what we realized in Dirk England's group at MIT was that you could

• 5:02Speaker 00

  actually use the same kind of processor we were doing for quantum computing to do computing on traditional neural networks using just lasers, and the benefits would be pretty massive.

• 5:14Chris Benson?

  Could you tell us like what that implies? What does that mean? How to use lasers for computing? Can you kind of bridge that gap between those two?

• 5:22Speaker 00

  Yeah, there's a little bit of history on this. So we're not the first people who have looked at photonic computing. People realized first, I think in the 80s, that you might be able to do neural network evaluation with photonics. And that field kind of like petered out for a while, but we came at it from a totally different approach. So we were looking at quantum computers, and we had a different motivation and a different set of like funding pitches for getting money to build these machines. We really took integrated photonics. That's a field that largely, the way that you experience integrated photonics is through silicon photonics transceivers. So these are devices in data centers that send your communications, your data, between different server racks. Maybe they're hosting

• 6:04Speaker 00

  a game you're playing, maybe they're mediating your iMessages, something like that. But we used that core technology to start doing computing. Now, how do you do it with lasers? In the quantum computing case, we weren't using lasers, we were using single photons, but that's a story for another day. Lasers are devices that generate light. You give them an electrical signal, and they're able to start generating light. And they're used in all sorts of traditional communications. The internet that we're talking over right now is over fiber optics, and the communication is sent using lasers to do that. So we're able to leverage that same infrastructure, lasers and silicon photonics, to actually do the core computations that happen in deep learning.

• 6:49Speaker 00

  And these computations are, I think, tractable really for anybody to understand. Deep learning as a field is really just relying on multiplication and addition to do evaluations. So the mathematics behind deep learning are relatively straightforward. Understanding how to come up with these things isn't straightforward. But when you actually run these programs, it's a lot of multiplication and addition. And indeed, you can use lasers and silicon photonics to do multiplies and addition. And I'm happy to go into how that works a bit later on.

• 7:23Daniel Whitenack?

  Yeah, I'm curious. Probably the main thing people think about maybe is like GPUs with computation on deep learning. And you're exactly right. I was just pulling up while you were talking the recent work from Microsoft and Nvidia, I think, on the Megatron model. So they just have a figure where it's like, I think most people maybe in our audience have heard of BERT, you know, heard that in the context of AI, which had 340 million parameters in the model. GPT-3, which I think, Chris, you just had a conversation with someone about on the show. Last week, yeah. Yeah, 175 billion parameters.

• 8:01Daniel Whitenack?

  And then this Megatron, which is the new natural language generation model, which is talked about just recently, 530 billion parameters. So just commenting and falling off on your conversation about these large models is pretty insane. I don't know, like, I think generally people have in their mind racks and racks of GPUs when they're thinking of this. So like with a photonic computer, and this is maybe just like a very simple question, but to have something in people's mind where like, so I've been near some like quantum computing labs where like they've got dilution fridges and it looks like this like whole like vacuum system and all that.

• 8:44Daniel Whitenack?

  And then I, I think about like laser labs with all these like, you know, lasers and you've got these lenses and stuff. And then that seems very different to me than like, oh, I'm going into the data center. I see these racks of computers just as a

• 8:58Chris Benson?

  Very simple question, like if you were to go to like, look at a set of, you know, photonic computers, what does that look like form factor wise and like connection wise, maybe as comparison to some of these other things?

• 9:12Speaker 00

  Well, I would say the biggest deployments of AI are in hyperscaler environments and in cloud environments. And to play in those spaces, you need to avoid doing anything weird. So a big dilution refrigerator or a vacuum, that's going to be a problem. And so if you look at photonic computers, like the things that we're developing, they just look like a normal silicon chip. They do have optical fibers that come out. You do need to get light into the processor so that it can do the calculations. But you know, it's really just a standard computer chip. It looks like a card sitting in a server, but there's a little addition of having a laser in there. And you know, just maybe on the larger picture here, there are so many different ways that people are looking at powering AI computation,

• 9:55Speaker 00

  because the market opportunity is so big and we're all trying to power these 500 billion weight neural networks. And it's going to take a lot of computing power and that power is the principal problem. Like computer chips are getting way too hot. That's really one of the fundamental things that we're interested in trying to help with.

• 10:14Chris Benson?

  When you said it was kind of one of the principal things, is that kind of the the primary value proposition? When you're saying, you know, go with photonics, you know, laser driven, it doesn't heat up. You don't have the massive, the massive problem that you have to deal with in a data center where you have a much. Is that, is that the primary thing? Are there other performance characteristics or non-performance characteristics that play into the field in general? In terms of the field itself having value proposition?

• 10:39Speaker 00

  Yeah, so I can say some really general things about computers. If you have one processor and you run a program on that processor, you will get one processor worth of performance. If you take two processors and try to scale it up, it'll be just less than two processors worth of performance. When you get to 1,000 nodes, you're going to be looking at something that is kind of doing the amount of computation that half the number of units will be doing.

• 11:03Chris Benson?

  Yeah.

• 11:03Speaker 00

  So what I'm getting at with that is as you try to power bigger and bigger neural networks. You need to scale up. And the reason that you need to scale up is that the individual computer chips that we build today, Intel's chips, AMD's chips, Nvidia's chips, they all consume a lot of power. And you kind of have to spread them out. And they're getting to be really big. And so this heat problem, in addition to what's known as Amdal's Law of Scaling, where every time you add a unit of compute, you do not get a unit of performance out. Those two things kind of work together. So really, power efficiency is tied to compute scaling. It's tied to compute per chip. There's really a maximum amount of heat that you can

• 11:44Speaker 00

  pull out of a processor. So it's just this whole story that's built around, you can't be trying to dissipate kilowatts of power in a single chip. You're not going to be able to cool it. And you're certainly not going to be able to continue a roadmap where you're pumping more power through chips. So it's performance, energy efficiency, and all that stuff is fundamentally linked. And they sort of hamper your ability to scale out.

• 12:08Daniel Whitenack?

  And I know that we've talked on the show a couple of times about the environmental impacts of what we're doing with these sort of large models. And I think, you know, there was that one study, I forget how old it is now, it's actually probably a couple of years old now, where it was talking about, like, one of these large language models is like running five cars into the ground for like their whole life period to train like once. And, you know, we're training them multiple times. So as you're talking about power, a lot of what I'm thinking is related to that sort of sustainability side of things. So like, in terms of that power requirement side of things, I mean, lasers still require power, I'm assuming. But so I don't really have a sense of like, what sort of scales

• 12:53Daniel Whitenack?

  are we looking at versus on like, a sort of GPU accelerated AI versus like photonic based?

• 13:00Speaker 00

  Yeah, maybe we can start with like a bit of the bigger picture around the energy scaling problem. So

• 13:06Daniel Whitenack?

  Sounds good.

• 13:07Speaker 00

  When you double the number of transistors on a computer chip, that's Moore's Law. It should happen every 18 months. Maybe we're a little bit behind schedule on that. But Moore's Law is mostly okay. Every time you double the number of transistors on the chip, they're going to need to use less energy in order for that chip to not get really hot. And since around 2005, when we've shrunk the transistor, the amount of energy that they use isn't shrinking commensurately. And so the chips are getting hotter and hotter and hotter. It sort of pushed us to a spot where you've heard a lot about system on chip and Apple's recent announcement of the M1 chips. You can see it. These are system on chip platforms. They have lots of different functionality, but it turns out that if they turned all

• 13:49Speaker 00

  of those functionalities on at the same time, you would really hit the thermal limits for the system. So this is a result of the fact that the energy scaling in transistors hasn't continued with their shrinking. That's called denard scaling. So that's kind of what's toast right now. So when you look at solutions from Nvidia, for example, they're really pushing the limits of what's possible to cool. I can tell you, we have a packaging team at LightMatter. Their job is to make sure that you can pull the heat out of your computer chips. And we're all very impressed with how much heat they're able to get out of the A100 processor. It's something like 450 watts. There's a new chip from Intel, Pontivecchio, and that chip is 600 watts. Really cool technology.

• 14:34Speaker 00

  It's awesome, but this power thing, it's a real challenge. It turns out that once you get to those kind of numbers, Pontivecchio is water cooled. So you really have to go from a heat sink and a fan, which is what you'll find in your computer at home unless you're an enthusiast and you love water cooling. You have to move towards water cooling and after that, you're seeing advertisements from Azure, Microsoft Azure, where they're doing immersion cooling. So they take computer chips and put them underneath apparently edible oil. Don't ask me how I know it's edible. From the diner? Yeah, I don't know if it's fry grease, but it's apparently edible. I don't know why you would ever eat it. Maybe it's like a reusable type of material. I don't know why they would advertise it as edible. It's an odd characteristic to note. It shows safety, I guess.

• 15:19Speaker 00

  If you're like, if you can eat it, it's probably like, you know, pretty safe. It sounds like a long chemical word, so edibility is probably a good sign. So the basic point here is that if you look at technologies for powering AI right now, they are all based on transistors. There's never been a computer that's not based on transistors. Like, this is the way that the world does computation. But we've run into this kind of fundamental challenge around shrinking how much energy the chips are using. And you can't really do that going forward. So if you look at the Department of Energy's estimate for energy consumption, 2030, so in about eight years. 10% of the entire planet's energy consumption will be on compute and interconnect.

• 16:01Speaker 00

  So if you know about compound annual growth rates, if we look at 2040, you're talking about most, like the overwhelming majority of the power being used on this. You always have to think in business about the use case that could motivate using that much of the planet's power, because that would cost a lot of money. And I think what happens is that you will start to see that progress in AI will slow because of the heat problem, because it puts a lot of financial pressure on data centers and the people who scale these things. To be clear, those neural networks that Daniel was talking about, those are trained by the biggest companies in the world, and they're hundreds of millions of dollars for the supercomputer, and running those models costs like $10 million. These are massive scale problems, and I would go out on a limb and say it's already

• 16:46Speaker 00

  probably an uncomfortable amount of money for those companies that they're spending on these things. So that's the state of things today. Everything's based on transistors, and you've really got denard scaling, which, by the way, is underpinned by quantum mechanics. It's not something that we can fix. TSMC doesn't have a solution for how they're going to get rid of this little pesky energy scaling problem. I wish they did, but that's kind of where photonic computing comes in. So we don't really have to worry about the quantum tunneling effects, because we're not using transistors. It's a totally different type of device, and so that part of the scaling doesn't matter so much. So hopefully that starts to give you a picture of like, we just sort of said, transistors, that's how people do computation.

• 17:30Speaker 00

  For deep learning, we think we can do it with optics. We think we'll use a lot less energy, and we'll get rid of that whole energy scaling problem. Just walk around it, because we're using different physics.

• 17:50Speaker 01

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes, or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 18:36Daniel Whitenack?

  So Nick, you just started getting to where I really was interested in discussing next switches around like, okay, we think we can do sort of these computations related to deep learning and AI with photonics. I'm curious, like, I guess the first question is, where are we at in that process? Is it like, this is being done and has been being done? And, you know, I suspect you've done certain things. Like, how far are we to like, doing training and our inference in a reasonable way with photonics?

• 19:14Speaker 00

  All right, so there's been a history here. I've been building these systems as part of my graduate work and now the company at LightMatter for about a decade. And over that time, we've demonstrated a number of applications, running language processing on these chips. That was done at MIT, along with a bunch of other cool applications of the processor technology. At LightMatter, we announced the Mars chip, and that was at Hot Chips 2020. Hot Chips is a conference on computer architecture, and that chip is capable of running state-of-the-art neural networks. And what we're up to right now at LightMatter is we are gearing up to start delivering our processors to customers.

• 19:55Speaker 00

  Very big companies that are interested in energy-intensive AI and trying to power their roadmap. They really care a lot about, how do you get to the trillion parameter neural network and beyond? So, we're very far along in this journey, like, quite confident that you're going to get to see it. And, you know, we've built lots of prototypes along the way, peer-reviewed stuff in academia. It's been published in Nature, Nature Photonics, Nature Physics, all the Natures.

• 20:24Speaker 00

  And so it's real stuff and we're going to be selling it.

• 20:28Daniel Whitenack?

  Yeah. And before we go into like the actual like what you'll be selling and some of like how it's related to AI and such. I'm curious, like, that's been a long road. I'm sure like it hasn't been all smooth sailing. Have there been points along that path where you're like, oh, this is like, I don't think we're going to make it. Or like, you know, what have been major like bumps in the road or like major achievements along the way that have happened? Entrepreneurial war stories here.

• 20:56Speaker 00

  Yeah, building companies is a lot of work, hiring people, hiring really good people. It takes a lot of time. I have a lot of entrepreneurial war stories. In terms of challenges along the way, I'd say that the first couple of years of the company, we were spending a lot of time trying to figure out exactly what the photonic compute architecture would look like. We knew it was silicon photonics. We had Global Foundries as our partner fab, but we had to figure out exactly what we were going to build. And there are a lot of ways to build these processors, like make no mistake, we've patented about 70 ways to do that, but there's only one that really works well from what we've seen. So narrowing that down was a lot of work. Building teams is super hard.

• 21:39Speaker 00

  And especially building teams where you've invented a new field. Silicon photonics is a technology that's been deployed for about 10 years now. So data centers have been using it long haul communications, like across the country. Type communications have been using silicon photonics, but people were not trained in using silicon photonics for building computers because it didn't really exist. So that's some of the the big stuff. The other pieces are, you know, you have to build a supply chain for these things. And it's, you know, it's not straightforward. There are a lot of people who touch the hardware from start to finish. And some of the steps are with companies that are, you know, smaller. And you have to kind of work with them to try to get things productized and stable and and all that sort of thing. You can think about lasers, for example.

• 22:24Chris Benson?

  So can you describe a little bit for those of us who are trying to kind of wrap our minds around what this kind of chip is like? Obviously, you can't describe in detail the internals, nor would you want to, but kind of give me a sense of what a photonic chip is like? What are some of the considerations that go into it? For those of us who are brand new to this field and just trying to, I know at the end of the day, you're producing a board that's going to go into my computer. But what's this, what's this little photonic magical thing that's inside that? You know, how does that work? What's different about it from the way I might already be thinking about these other architecture chips that we've talked about?

• 23:02Speaker 00

  So I can say something

• 23:03Chris Benson?

  Okay.

• 23:04Speaker 00

  sort of boring, which is, have you heard of the Google TPU? So GV, formerly Google Ventures, is an investor and our technology looks a lot like a TPU.

• 23:14Chris Benson?

  Okay, yeah.

• 23:16Speaker 00

  So it is a matrix processor with a lot of other things wrapped around it. It's not just doing matrix processing. There is a lot of things wrapped around it, emphasize on that. But it's basically doing linear algebra. So we have multiple processor cores. Our product's actually a quad-core computer, just like an Intel like quad-core chip. And each one of those cores is doing linear algebra. If you were to look at the chip and just follow the light, what you would see is values from an image, like let's say we have a pixel, it's red, green or blue. And there's an intensity for each pixel from zero to one. What you'd see is a brightness of light corresponding to that intensity number if we're doing image processing, kind of like being distributed around the chip.

• 24:01Speaker 00

  And then you'd see these photonic units in a two-dimensional array, receiving the signals and then doing multiplication and addition. That's basically it. So it would look to you a lot like a TPU. This like box, like a 2D array of multiply accumulate units, except you would see light being distributed to all of the individual components.

• 24:23Daniel Whitenack?

  You talked a lot about heat and its motivation in this. I imagine that there's a number of... You're thinking about the heat problem, but also with this chip, I imagine there's a lot of interference-type things. And we're thinking about light. Is there different considerations? Like, maybe you have some benefits in the heat side of things, but are there other shielding or interference or other challenges related to that in terms of building the physical unit that you have to take into consideration, that you wouldn't have to take into consideration with another type of computer?

• 25:02Speaker 00

  So, yes and no. We're using light, and so light signals can't interfere with each other. So that's an interesting property. However, whenever you build a metal wire, it ends up being an antenna. And so in chip design, there are these things called antenna rules, and digital chips have to worry about these, too. And so we have to observe these things so that we don't get unwanted signals from radio hosts or something getting into the chip and messing up the optical computation through the wires. But otherwise, I think that it really just looks like a mixed-signal chip. So there's analog circuits, there's digital circuits, and then the photonic circuits, which, for all practical purposes,

• 25:46Speaker 00

  are just analog-type circuits. So you have to be careful about coupling unwanted signals through antennas that you didn't intend to be there. But otherwise, no, not really.

• 25:58Daniel Whitenack?

  So I could still have all my RGB lighting in my computer case and have flashing, cool stuff?

• 26:06Speaker 00

  Yeah, if that's the question, absolutely. It turns out that it's very hard... We have these things, and I should have mentioned this. We have these things called waveguides. They're about 300 nanometers wide and about 200 nanometers tall. And they're optical wires that are on our chip. So they're really tiny, and we have hundreds of thousands of them on the chip. It's very hard to get light into those. And so if you shined a flashlight on our waveguides, you'd get nothing into the waveguide. You have to really want to get the light in there for that to work out, which is fortunate and unfortunate.

• 26:38Chris Benson?

  I'm kind of curious, is there, if you're looking at the wire world that we've been in for so long, and the physics, as it gets smaller and smaller, and they're running into all sorts of challenges, heat and otherwise, how does that translate? Do you, you just talked about how small that optical wire is, if you will, and the fact that because of that, you don't get the light into it. What are some of the, kind of in the science of this, what are some of the limitations as you are constantly trying to get to smaller and smaller architectures to scale out? What are the things you have to think about?

• 27:11Speaker 00

  Yeah, it's an interesting question. So at first glance, you would be like, alright, you've got a new kind of computer, we're going to want to shrink it every 18 months and have a photonic Moore's law or something, some kind of scaling law. You could go down that path, and the way you would do it is you'd start out, we work at 15, 15 nanometers wavelength, you could go down to 13, 10 nanometers, and then you could go down to 900 nanometers. By the way, your eye will just start to pick this up, and it will look like red, and so on down, and then maybe you get to ultraviolet, which is what gives you skin cancer. Okay? And then we keep going, we get to X-rays. And the whole time, these optical components are shrinking. They're going from 300 by 200 nanometers, eventually at the X-ray kind of scale to a few nanometers kind of dimensions.

• 27:56Speaker 00

  You could do that, it's possible, but it turns out that the light sources that you need to follow that path would be pretty tricky. And when you build optical wires, they care a lot about the quality that you build those wires with. The light wants those wires to be very smooth. So if you're not really, really careful about the quality of those wires as you shrink it, you're going to have a hard time. And luckily, it turns out, we don't need to shrink them at all.

• 28:21Chris Benson?

  Well, you just took the next question right out of my mouth on that one, so keep going, please.

• 28:26Speaker 00

  Yeah, so it turns out that in a traditional computer, if I said that your CPU is clocked at 3 gigahertz, you'd be like, yeah, that totally makes sense. If I say 20 gigahertz, you're like, eh, I've never heard of that, and it probably will never be a thing, because I remember, like, when I was in undergrad, like, 2005, I had a 3 gigahertz processor then, if I recall correctly. So like that frequency hasn't been scaling. With optics, the frequency you can operate at is very high. So we work at about fifteen fifty nanometers. That corresponds to a frequency of one hundred ninety three terahertz. That's a lot of bandwidth.

• 29:02Daniel Whitenack?

  Yeah.

• 29:03Speaker 00

  Turns out that you will never practically get anywhere near that kind of bandwidth. Because you have to talk to the thing. You know, like ultimately, these things have to talk to to electronic computers because that's sort of how the world works. And so you get limited. Maybe you can do twenty gigahertz, fifty gigahertz, something like this. But the point is that we're able to turn up the clock frequency to obviate the need for shrinking these things. Because we're going to give you more performance per unit area through the clock frequency. And there's one other thing. So yeah, we can go fast in clock. We can do something else special, which is we can use multiple colors at exactly the same time. So if you remember, we have this two D array that's doing matrix computation.

• 29:45Speaker 00

  Each one of the elements in the array can process different colors of light at the same time. So each each element could do three colors at the same time, sixteen colors at the same time, something like this. And what that means is that in the same area, that same array size, two colors of light. You've just doubled the compute in the unit area, three colors of light, three times the compute per unit area. And you see, like, there's, there's really no need in principle for this sort of shrinkage of the photonic devices.

• 30:15Daniel Whitenack?

  Well, as Chris knows, I'm always interested in the very practicalities of things. And I'm imagining, OK, we have this this processor and it works and can do some computations. You have it integrated into some system. But ultimately, like, I have TensorFlow. And I know how to run TensorFlow on a regular computer or on a GPU. Because there's underlying libraries, which eventually get mapped into some type of machine code. That, you know, is mapped onto like this processor. So maybe just describe like, OK, you you have this photonic computer.

• 30:54Daniel Whitenack?

  How do you even start to think about that process of integrating software into the system? And like, making sure you support like tooling that people want to use and that sort of thing.

• 31:07Speaker 00

  Yeah, that's a huge amount of work. So, building the photonic computer is really hard in terms of the physics and engineering. But building a software stack that integrates with PyTorch and TensorFlow is a ton of work, but it's something that we've bit off and we call it Idiom. So Idiom is LightMatters Software Development Kit. What we allow you to do is take neural networks that you've built in PyTorch or TensorFlow, import our libraries, and we have a compiler that can take the emissions from PyTorch and TensorFlow and build machine code that runs on InVise. InVise is our photonic computer. So, we have our own instruction set architecture and our compiler emits that ISA is what people call it, instruction set architecture. It's a ton of work. We have a pretty big software team here.

• 31:52Speaker 00

  I expect by the end of next year that we'll have significantly more software engineers than hardware engineers, even though we're building these crazy chips. And generally, that's the trend, certainly in the machine learning space, but in computer chips in general, because people, they don't really care about computer hardware. They just care that it's fast and it keeps getting better and that it doesn't have errors and it's not annoying to use. So, we're very focused on just delivering the same experience that you're used to with PyTorch and TensorFlow. And you can just use our alien technology and not worry about what's under the lid. But it's a very big effort.

• 32:29Chris Benson?

  So, if you're thinking about that, just to clarify for a moment, if we're going to PyTorch or TensorFlow and we have your chip in the system and we have Idiom there, it's basically taking that, converting it into something that works on your hardware in that context. But from our standpoint as data scientists or software developers or deep learning engineers, our workflow is more or less the same if I'm understanding you correctly? Is that accurate?

• 32:56Speaker 00

  That's right. So, right now, we're targeting inference. So, the way you'd imagine this working is you've got a neural network that you've trained in PyTorch or TensorFlow. And then, we have a bridge to Onyx and then Onyx to our compiler and you're good to go.

• 33:12Chris Benson?

  That makes sense.

• 33:13Speaker 00

  Yeah. And so, the chip can do training, but we're not focused on it right now. There are very, very different markets. With very different things that you optimize for. Happy to to talk about that, but inference and training are really drastically different beasts.

• 33:28Daniel Whitenack?

  Yeah, maybe you could touch on that a little bit because I think maybe the, at least for people getting into deep learning, a lot of the tutorials and whatever they see is is all about training, right? So it becomes this perception that like, training is the major problem to solve as a data scientist or an AI researcher at your day job. Could you maybe like, on some of your work with light matter, but also just working with clients and understanding their needs, like, maybe speak into that in terms of why inferencing is like such a big market, but also really important for, you know, practical use cases of of AI?

• 34:06Speaker 00

  Yeah, so training is R and D mode. You're building a tool that takes inputs and then tells you something about those inputs. If that's all we ever did in machine learning, no companies would would fund this stuff. Because the ultimate goal of building those models is that you want to use them for something. And if you're Google or Facebook, these neural networks get deployed at massive scale. And their job is to take a user query, run a neural network on the query, give you results. And sometime there are lots of neural networks in that chain. And that is where the overwhelming majority of the energy footprint of AI will be over time. Training is a really big deal. You gave the example, I think MIT Tech Review did that analysis that you were quoting,

• 34:51Speaker 00

  where it was like five cars over their whole lifetime is how much energy that the training was using. And I think the equivalence was in like carbon emissions. So yeah, inference is deployment, which is where most of the scale is. And and training is just doing the R and D that lets you do that thing that makes you money. Because the training stuff doesn't make you money.

• 35:10Daniel Whitenack?

  As you go, I actually want to go back for one second and touch on something to combine two things that you've said. So as we're doing this and we get to, you know, as practitioners, keep the same workflow more or less that we're doing, and we are still able to use the tools that we're already using to be productive. We've now been kind of supercharged by having

• 35:29Chris Benson?

  These photonic processors in there, I may not be using the wrong term, the right terminology on that. But and we've talked about scaling without having that limitation from the thermal limitation that we've talked about. Does to go back for just a second of a curiosity, is your limiting factor then the receptor of the light inside there? Being able to detect the number of different light wave frequencies, the number of different colors that are available. So that in theory, if so long as as you can build better and better receivers for different for more and more colors in there. They can distinguish.

• 36:03Speaker 00

  Yes.

• 36:04Chris Benson?

  Then you're essentially unbounded by that constraint until you hit whatever that practical limit may eventually be.

• 36:09Speaker 00

  Yeah, that's a great question. So what are the ultimate bounds on using color and frequency?

• 36:14Chris Benson?

  Yeah.

• 36:15Speaker 00

  Detectors are pretty broadband, so they can detect a large number of different colors at the same time. No problem there, but there's a device called a multiplexer or a demultiplexer. If you can think of the Pink Floyd album cover, you have a prism.

• 36:31Chris Benson?

  Yeah.

• 36:31Speaker 00

  And there's a white light that comes in and a rainbow that comes out the other side. That's identically what I'm talking about. So that specific kind of device that takes in a stream of multiple colors and spreads it out into the individual constituent colors. That device is what will limit you from getting to like 64 colors or 128 colors. Eventually it gets really hard to accurately separate those colors. And then when you're at this point, scale out becomes really important. By the way, that's a long ways away.

• 37:02Chris Benson?

  Okay.

• 37:03Speaker 00

  In terms of the roadmap for photonic computing. When we're at 64 colors, the amount of compute on the chip is something like hundreds of peta ops. It's it's crazy. So like decadal time scales. But then when you get there, what's really important is that you can efficiently take lots of force, lots of processor cores and connect them to each other. Earlier I was talking about Amdahl's law and the fact that you add another unit of compute. And unfortunately, because of communications, you don't get another unit of throughput. We've invented an interconnect technology that surprise, surprise, uses light. It's a wafer scale computer chip, eight inch by eight inch. So it's the size of your laptop screen practically. And it allows you to let your processors talk optically and they can dynamically configure how they're connected.

• 37:49Speaker 00

  You remember those old switch rooms where people are making phone calls and they need to connect to some other? So imagine you have someone doing that, but at the microsecond time scale, they're able to unplug the optical waveguides built into this computer chip and plug them into different locations at crazy high speeds. And that allows you to scale it. So whenever you run out of colors, scale out is going to be really important. And we've invented a technology for that and it's called Passage.

• 38:14Daniel Whitenack?

  That's really cool. I love that analogy and I love the fact that you're also thinking about like scale out, not in terms of like always making things smaller and pushing those boundaries, but doing it in other creative ways as well. So if I'm, if I'm understanding right, you kind of have the chip. So in terms of like what you're bringing to market and what you're doing, you have the sort of chip or processor, you have this passage technology and then the Idiom software integration. I'm wondering if you could just give a few or highlight a few sort of use cases or tests that you've done with these technologies in terms of maybe models that we might be aware of about like inference time or like what sort of differences are you seeing?

• 38:59Daniel Whitenack?

  And how is that impacting actual inferencing?

• 39:02Speaker 00

  Yeah. So comparing our technology to competitors.

• 39:05Daniel Whitenack?

  Right.

• 39:05Speaker 00

  So on the Idiom side, it's a software stack and the goal is really just to make sure that it doesn't slow down the hardware compared to what it's theoretically capable of. So we'll leave that on the table. On the Invis side, on our website, you can see a comparison between Invis server and Nvidia's DGX A100 server. We use a very small fraction of the power. So that chip's quoted at about 80 watts compared to Nvidia's 450 watts. And a pretty significant boost on ResNet 50, BERT. And then I believe we also looked at DLRM. I don't have the numbers off the top of my head, but in many cases it's multiple times faster. But what's extremely important from a scale out perspective is if you look at the power.

• 39:50Speaker 00

  And just like if you think about it this way, we can up the performance. We're not anywhere near that power envelope. So we can keep going up there. There is a point though, and this is why we invented Passage, where making Invis faster is great, but it's just sitting there waiting for work. It's sort of like this ultra capable person and you like hand them a paper once a day and they're just bored out of their mind. So we really had to invent Passage as a way to keep Invis busy. So that should give you an idea.

• 40:19Chris Benson?

  It does. I know this is day-to-day stuff for you, but this is quite remarkable for those of us not in the field. Where do you see this going? I mean, you know, is this what everything inevitably has to go to? Do you think that the field will stay with lots of different technologies that are competing? What does the future look like to you? How should the rest of us be kind of thinking about the years forward in our career versus where we're at now? How much change is coming?

• 40:44Speaker 00

  So what we're doing is targeted at AI. For general purpose computing, where you're running Windows and you're playing a video game, probably photonic computing will not be able to contribute meaningfully. There are some really hard challenges in running general programs. So to be Turing complete, you need to have these sort of nonlinear operations. If you're a programmer, you can think about branching, like this ability to have conditionality, if and then. Doing these nonlinear conditional-type behaviors with optics is super hard. And so you mentioned a future where potentially there are lots of competing technologies. I think that's exactly what will happen. We've had this incredible platform with transistors and Moore's law scaling.

• 41:28Speaker 00

  That's been so general purpose, we use it for everything. What you're going to see going forward is you'll have photonic computers for AI. And our plan is to help them dominate that field. Because we think that they're very well-suited to the mathematics that underlies deep learning. And I think you're going to see quantum computers. I love quantum computing. I did my PhD in quantum computing. But I'm not as bullish on the right now timeline. I think they've got quite a while left, but I'm rooting for everybody working on it. And in general, I just think it's the case that we've reached the limit of transistors being able to kind of do everything good enough. And you're going to see a bunch of different types of technology platforms, all competing. You'll see analog electronics at the edge.

• 42:10Speaker 00

  You'll see digital electronics for running your games. You'll see photonic compute units for doing your deep learning. Like my goal was to have all of Google run on light matters processors. So when you say, hey, Google, that goes through and buys. That's where I want to get to. So you can kind of see, this is how I see it. These technologies are all just suited to different kinds of problems.

• 42:32Daniel Whitenack?

  We're really looking forward to seeing how you bring this technology to market. It's incredibly impressive, sort of blown away by the amount of work that's gone into this and just the amazing thought process that goes along with it. We'll include links that we've talked about in our show notes for everyone listening out there. Definitely check out LightMatter, see what they're doing. There's links that we'll post there with some of these benchmarks and other things. But Nick, it's been a real pleasure to talk to you. It's just amazing stuff that you're doing and I'm really looking forward to maybe talking next year and seeing how far you got and how many, how many invised processors are running in Google.

• 43:14Speaker 04

  Awesome. Well, thanks for having me, guys.

• 43:19Speaker 01

  That's our show. Thanks for listening. For more like this, check out our master feed. It is all changelog podcasts in one easy to consume place. Let your podcast app snag everything we produce and then pick and choose which ones to listen to. Subscribe today at changelog.com/master or just search for Changelog Master in your podcast app of choice, you'll find it. Special thanks to BrakeMaster Cylinder for providing our music and to our longtime sponsors, Facely, Launch Darkly, and Linode. That's all for this week. We'll talk to you again next time.