⚡️Accelerators @ 3x NVIDIA H200 perf, Made in the USA - Thomas Sohmers + Mitesh Agrawal, Positron AI

Watch the full show on YouTube: https://www.youtube.com/watch?v=9bCICtQd9yY

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Alessio Fanelli?

  Hey everyone, welcome back to another Late Space Lightning Pod. This is Alessio, Partner and CTO at Decibel, and I'm joined by swyx, founder of SmallAI.

• 0:11Speaker 01

  Hello, hello. Today, we are joined by Mitesh and Thomas from Positron AI. Welcome.

• 0:17Speaker 02

  Thanks, Sean. Thanks for having us. Good to be here.

• 0:19Speaker 01

  And also special shout out to Maxime from DFJ for putting us in touch. So I don't know which of you you want to go first, but both of you have a background in some of the top, like, you know, chip companies in the world. I think I think mostly Lambda Labs, but also a little bit of Grok. How do you find your way to Positron? What's what's sort of like the founding thesis and the founding story?

• 0:40Speaker 02

  Yeah, I guess you're just starting my journey a little bit. I got my start in the semiconductor industry back in 2013. I started my first company called Rex Computing that was focused on the DSP for like a mobile base station workloads. And I started that after receiving the Thiel Fellowship. So I was, you know, seventeen, with the crazy idea of starting a semiconductor startup. And thankfully, was able to, you know, convince some folks to join me and we raised, you know, $2 million and ended up taping out, you know, first chip when I was 19 and had, you know, leaning for for that time, silicon for

• 1:24Speaker 02

  for that signal processing workload. Sadly, didn't really get commercial success with it, but was amazing experience to be able to actually produce silicon from scratch. You know, after that, I worked on a couple cryptocurrency ASICs before joining a friend startup called Lambda that I was principal hardware architect at and built out the first couple data centers for Lambda Cloud. And that's where I had the opportunity to work with Mitesh. I actually was the one that introduced him to Lambda originally that that had him join before I was an employee there. And Mitesh and I have been housemates and just very good friends for over a decade. After my stint at Lambda, I wanted to get back into actual semiconductors. And so I joined Grok, the company with a Q, and I was there as a director of technology strategy for

• 2:10Speaker 02

  For two years and then in early 2023 decided to leave and start Positron. And I originally tried to recruit Mitesh to be CEO of Positron back then. You know, he was growing Lambda crazily at that time. So I was actually, you know, CEO of Positron for the first 18 months or so. And finally was able to bring Mitesh over and he joined at the beginning of this year as CEO. CTO.

• 2:36Speaker 03

  Yeah, and kind of for me, a lot simpler. I think Thomas mentioned, he introduced me into Lambda Labs. You know, Steve and Michael really loved working with the Balaban Brothers, right? Really grew the company from zero dollars in revenue all the way to where it is now, well over half a billion in ARR, right? And, you know, worked there for almost a decade, did all types of roles. So, you know, just very early employee, I was employee number five there. So did engineering, sales, support, you name it. And was CEO and head of cloud. The main thing for me was just really focused on building out the cloud at Lambda, you know, all the way. And then was primarily focused like in one word, like on growth. So revenue, operations, data center, you know, megawatts, a vendor relationship and vendor procurements, engineering and engineering headcount.

• 3:18Speaker 03

  And primarily for me, I think, I mean, Lambda is still growing massively, right? So I don't want to imply that I left because, you know, anything, I still have great connections there. And, and really, really proud and very, you know, have huge admiration for what they're continuing to build. For me, I think, you know, the two reasons really why I left was one is really could see a little bit around the corner at Lambda. You know, we worked with frontier model labs, enterprise companies, you know, yeah, startups. Was seeing how the model architectures were growing, the requirements around memory architecture, memory requirements, capacity and bandwidth. And really wanted to get to contribute in an underlying technology stack. You know, Lambda is a great company built on cloud services layer. Really wanted to work on the underlying compute that will power kind of the generations of the future. And, and then second thing was, you know, my background is chemical engineering.

• 4:02Speaker 03

  So study fabrication, really wanted to get into, back into the silicon space and build that out. So joining, you know, this year itself, January 2025, I was, I was talking to Thomas in December. I had my first kid in November and then Thomas was like, well, you already had one major change. Why not make it two major changes in your life? And decided to join in. So that's that's a quick summary of the background.

• 4:22Speaker 01

  Yeah, pretty amazing. Also very unusual to have a, you know, it's like kind of like a handover of CEO that you've been trying to recruit for a while. That's that's a fun story. But we could, we could, we could go technical. Unless you, I think I.

• 4:33Alessio Fanelli?

  Yeah, I was curious if you guys thought about sticking to the software side at all? Or if you were busy looking at all the software improvements and you thought that was kind of like a logo maxima that was kind of capped. And you thought the hardware had to be rebuilt, or was it just a matter of?

• 4:50Speaker 03

  Yeah.

• 4:50Alessio Fanelli?

  You have experience in the space. We think we have a unique angle. We should just go do this.

• 4:55Speaker 02

  Yeah, I would say for me, I've always been a hardware guy. You know, I started tinkering with electronics when I was like five, six years old. While I programmed a bit, you know, as a kid and in school, I was always so much more interested in in physical things that could actually burn or blow up in my face. So I think the natural sort of gravity, but also just looking at the actual application space, et cetera. You know, being a huge believer in, you know, the capabilities and future growth of this industry. It's the biggest limiting factor for that being, you know, distributed at scale. You know, the the quote of, you know, the future is here. It's just not evenly distributed yet.

• 5:33Speaker 02

  I think that the limiting factor for that is, you know, the the cost and from both, you know, dollars and and power perspective for deploying these applications. And so starting the company was really focused, how can we get the best performance per dollar and performance per watt for these workloads? And with my my experience on, you know, going all the way back to my my DSP work to, you know, the time at Lambda, et cetera. Seeing that the kind of way that the whole industry had been going for, you know, the decade before ChatGPT was not really going in, you know, optimizing for the right things to what transformer inference really needed.

• 6:11Speaker 02

  Because that the the actual real underlying hardware or the underlying algorithms have very different hardware requirements than than what was needed for CNNs. And and, you know, the first wave of, you know, our modern machine learning explosion. So it was seeing that everyone else was focusing on the wrong things. They were just. Just trying to have more and more flops when memory bandwidth, memory capacity were the real, real bottlenecks. And then that, that realization that, hey, everyone that has been focusing on that is like completely missing. These sort of neglected parts of computer architecture and, and processor design that thankfully I, I got exposed to through, through my career.

• 6:50Speaker 03

  Yeah, for me, I'll keep it kind of one lighter, you know, the, the bitter lesson, right? You know, compute really has been the driving force for a lot of just amount of compute. And then the way you can make it efficient has been the driving force for a lot of model improvements. This is not to say, I mean, I personally don't do the software at, at local maximize. I think it's more around our backgrounds, you know, what we are good at. That's, that's obviously the driving factor there, but, and many people that are doing amazing work on model improvement sides, both on small and, and frontier edge model, we're seeing it today. But really like what the bitter lesson has taught us is like, the more amount of compute you can provide, the better it is for all the people that are working on software to drive in more utilization, to drive more intelligence from it. So yeah, that, that was for me, it's still the case.

• 7:32Speaker 03

  It's like, how can you just drive more compute in the world, both from flops, memory and deployment perspective?

• 7:38Alessio Fanelli?

  Cool. Um, yeah, no, that's great. And I know you touched on some of this, but maybe this is a good time to talk about what you guys are doing that is unique. And maybe contrapos that both to, you know, Nvidia and the kind of traditional ones as well as Grok. Obviously you worked there, Cerebras, and then there's kind of like the long tail of all the other GPU alternatives things. But just give people a lay of the land of like what's working, what isn't, and what you guys are doing differently.

• 8:03Speaker 02

  Yeah, um, I guess I'll actually share my screen just to cover a couple of things at a high level that,

• 8:14Speaker 02

  can set the stage a bit.

• 8:16Alessio Fanelli?

  Surprising how many clicks it takes to share your screen.

• 8:19Speaker 02

  Yeah, uh, it's amazing. We're on the verge of AGI and you know, we can't get presentations or calls to work properly.

• 8:28Speaker 02

  Yeah, so just to show here, you know, some, some of the viewers here may be familiar with the roofline plots, but this is, you know, a pretty abstracted way to be able to look at, you know, one of the key differences between, you know, really convolutional neural networks and transformers, more specifically trying to focus in on them at inference time. So this showing, you know, operational or arithmetic intensity as a fraction of, you know, the, the, over the performance. The, the real key thing here that this is showing is that in the top right corner here is example of one of the most compute bound problems you could have of doing your inner

• 9:13Speaker 02

  products, matrix, matrix multiplication for CNN, where depending on the exact parameters can be anywhere from 500 to 1,000 flops for every byte that actually needs to be moved. And on the other side of this chart, you have the case of a transformer where when you're actually, you know, doing attention, or really just any case where you're doing, you're fundamentally doing matrix vector multiplication rather than matrix, matrix multiplication. And in that case, you are, are now actually completely memory bound. For every step in the sequence that you're doing, you're now having a one to one ratio of flops to, to bytes that have to be moved.

• 9:57Speaker 02

  And so in that case, you know, the, the really big realization at the beginning of Positron was, hey, this looks a lot like when you're doing Quam 256 in a, you know, mobile base station for, for LTE, just in terms of what actually needs to be moved. And the system's built for doing that, you know, namely utilizing FPGAs and specialized DSP chips for that. Out extreme focus on the memory system and the ability to actually saturate.

• 10:31Speaker 02

  Whatever DSP compute elements you have and be able to feed through at that one to one memory to compute ratio. And if you look at sort of traditional big iron compute systems, the last like major compute platform that had that balance of memory to compute ratio was the Cray 2 supercomputer. So, you know, we're pushing, you know, 40 plus years of computing developments, you know, that has been focused on just providing more and more flops, just, just, you know, raw math compute capability while not balancing that by actually having memory scale in the same way.

• 11:11Speaker 03

  And capacity and memory.

• 11:12Speaker 02

  Yeah, yeah, I'm really mainly talking on the memory bandwidth side here.

• 11:19Speaker 02

  But yeah, that this, this was like the key thing that we saw like, okay, everyone had been optimizing for the CNN portion of the workload. And the other thing I should point out here is when you're doing training for, for transformers, you're also in this very compute bound case. In the case when you're doing training, that what is matrix vector in inference, because you're, you're generating one token at a time and you're just appending that to your KV cache. In training, you already have the full sequence of, you know, whatever you're, you're feeding in from a data set perspective. So that gets to be a matrix and it is compute bound. But it is this particular case of transformer inference that is just massively memory bound.

• 12:03Speaker 01

  Can I, can I also just spell out that this also applies to Grok and the Cerebras and other sort of big chip companies, right?

• 12:10Speaker 02

  Yeah, I mean, this is just fundamental to the algorithms.

• 12:13Speaker 01

  Right. I mean, just because most people are familiar with Nvidia H series. I just wanted to

• 12:18Speaker 02

  Yeah, yeah.

• 12:19Speaker 01

  clarify.

• 12:20Speaker 02

  Yeah, I mean, I, this really is kind of hardware agnostic from, from perspective of, of the operational intensity of the algorithm.

• 12:28Speaker 01

  Okay.

• 12:29Speaker 02

  You know, the, the way in which different people approach that. And I would say Grok, Cerebras and others have tried to take, have tried to say that, hey, we're providing more memory bandwidth. But typically that comes at a massive.

• 12:45Speaker 02

  Trade off of then not having the memory capacity. I'll get to the kind of the memory technology in a second, but the, you know, very quick thing I'll just throw up here is, you know, just trying to visualize the difference between matrix, matrix and matrix vector. And so when it comes to like the actual memory accesses that happen, you know, when you're doing a matrix, matrix multiply, the, the simple way to say this and doesn't take into full effect, the tiling and, and more complexities there. But, you know, the, the simple high school math version of this would be, you know, for every element in matrix B, you get to reuse the same element in matrix A when you're then making matrix C. While when you're doing matrix vector, you actually need to change the elements in both

• 13:27Speaker 02

  matrix A and vector B for every single operation that you do. So the, the second level of this is that matrix vector multiplication is basically uncashable. When you're doing transformer inference, matrix A is the weights of your model. And so if you're talking about model weights that are tens of gigabytes, hundreds of gigabytes, now going into terabytes of memory, you're not going to be fitting that in any on-chip cache. And so even with, if you had perfect cache prediction, it just won't work. And so a lot of the, the memory architecture decisions of the past few

• 14:06Speaker 02

  decades just go completely against what you need for, for this workload. So fundamentally kind of getting to, to posture on here, you know, the, what we're show, showing here with, you know, A100 and H100 is that sort of in Nvidia's absolute best case scenario for an H100, you know, for Llama 70B here, uh, you know, they're able to achieve about 29% of the theoretical memory bandwidth in advice. So you pay for 3.35 terabytes per second memory bandwidth for an H100. And in practice, you're only actually getting about a terabyte per second. And the A100 comparison here is interesting because in most of these cases, they're actually, they

• 14:52Speaker 02

  Percentage of theoretical memory bandwidth actually has gotten worse generation over generation. And we'll see exactly where Blackwell ends up. But all indications are, even though they, you know, more than doubled the theoretical memory bandwidth going from Hopper to Blackwell, the actual percentage of theoretical that you can achieve is again going to be less than than the previous generation. And really what Positron's whole mission is about is getting absolute maximum memory bandwidth utilization possible. And so our fundamental architecture is enabling us, you know, today with hardware that we're shipping right now to be achieving, you know,

• 15:32Speaker 02

  93% of the theoretical memory bandwidth of our device consistently across all use cases. So you can see that this degrades in the Nvidia case as you scale up the number of concurrent users, but you would see the same degradation as you scale the context length or scale the model size. You know, our architecture, the key things is that we have, you know, a, our core compute elements are designed to be optimized for one-to-one flops to memory bandwidth. So we can actually feed and completely saturate our device. And, you know, even when you then scale out and, you know, to external memory, you know, we're able to to fully utilize that.

• 16:17Speaker 02

  And basically that missing 7% that we've got is effectively just due to the the refresh on the DRAM, which can't really get around right now.

• 16:28Speaker 02

  So, you know, what that actually results in is like today, we're, you know, able to achieve about, you know, 70% higher performance than Nvidia with the the cards that we're shipping today. Significantly lower power and price point.

• 16:46Speaker 02

  Yeah, that's uh and just to reemphasize, you know, this we're actually shipping.

• 16:51Speaker 03

  Yeah, I think I think the question I was asked around like the difference between not that media but the alternatives like that's a big thing. It's like we got to market shipping within fifteen months of founding of the company with only to see ground raised. And and again, are now planning our next generation silicon within eighteen months of the first generation. So like we are not only matching Nvidia on the speed of performance and performance per dollar and performance per watt where we are really trying to outcompete there, but also coming up with the generations of the product, uh, because they're they're setting the pace, they're the trendsetter, right? So so we're keeping up with that, which is kind of classic going away from the case of a lot of silicon and AC companies of the past decade where they've taken three to five years to really get out a product and then taken another three to five years to get the second product in the market.

• 17:33Speaker 01

  But I want to talk about the technical achievement, but actually this is an operational achievement, right? The fact that you've been able to to ship. What's your secret?

• 17:41Speaker 03

  So I think when a small team, I would say, I think it's it's it's it's it's

• 17:45Speaker 01

  Tiny team?

• 17:46Speaker 03

  Like, I think the part about I wanted to say with small team is like with small team, you're forced to focus on applications that you want to ship out. And for us, from day zero, it was about, you know, accelerating the linear algebra of matrix vector map. And and second of all, look, from a marketing term, we call it like falling within the Nvidia ecosystem. But like from both of our past experiences, I can tell you that most of the the silicon providers, you know, whatever technology they might come up with, or whether it's good or bad, I think the biggest hurdle has been actually really convincing people to use it because there is always change of workflow of how they integrate into their existing inference work systems or engineering ecosystems, whether that's, you know, figuring out some compiler compiling the same model again, or some kind of backend system which is complicated, doesn't scale out.

• 18:28Speaker 03

  And for us, like we were from from early days, we really focused on getting a product out in customers' hands so that we can actually show proof of concept and show really our architectural, like memory architecture and the silicon architecture working and and showing the performance improvements. And then second was On the same point, making it easy for people to really work those systems without really having to change any of their engineering workflows. So, you know, basically what that means in actual world is when we say part of the Nvidia ecosystem, what it means is we basically take the raw binary weights output from when you train your model on an Nvidia GPU, you get a dot PD dot save tensor file that you upload on hugging face and that's your raw binary weights for your model.

• 19:08Speaker 03

  So we take those, that file into that our hardware and we can actually output an open AI compatible API that people can then send their tokens in and tokens out. So from a workflow perspective, you can imagine you have an Nvidia server running and you have a positron system running and you can have them side by side. And all you have to do is just to direct the tokens to whichever system you want to, and it'll have the same kind of workloads running without having to really change or without having to recompile the model for a positron system or so on.

• 19:35Speaker 02

  So like the really key thing there was, you know, from the get go, we knew that the, you know, having a single step that a user had to take was going to be a, the ultimate barrier to getting any sort of adoption. And, you know, be it a lot of the companies that you mentioned and others, I mean, even big guys like AMD have more or less, you know, flopped many times in their, their pathway to try to, to get people to use their, their, their hardware. The approaches that people have taken before have been just saying, well, we're going to build a PyTorch backend. And AMD had their own separate, uh, you know, non-mainline PyTorch, uh, distribution for years. That was always six to nine months behind any new PyTorch releases.

• 20:17Speaker 02

  But I would say AMD has done an amazing job catching up in the past, you know, one, two years. But still you have so many other libraries and functions in the PyTorch ecosystem, which just aren't built around being able to work with AMD. And it makes a lot of people's workflows basically fall flat. And that's before you get into the mess with drivers and everything else in the rock end side of things. So you have that on, on one hand, and then you have others like Intel was trying to push one API for Gaudi and, you know, all of their other stuff. A lot of the chip startups, like a big part of their story was that, hey, we made a super compiler. And our compiler will just automatically be able to take anything and get it to run. And there is different levels of that actually working or not.

• 21:01Speaker 02

  But I would say the the core premise there was flawed from the beginning. If you are requiring a user or having yourself as the company needing to actually recompile a workload, that's already one step too far, even if you assume it works perfectly. And so from the very get go of Positron, we said, hey, we're only really going to care about, you know, transform models. And we were betting on the hugging face transformer library and sort of the framework that that was created there. And we designed our hardware to directly ingest those raw binary model weights. So rather than having like, we don't have a compiler whatsoever. There's no compiler, there's no translator, no tooling that's involved in actually taking those, those and getting that to, to, you know, for your, your, uh, your, your common, you know,

• 21:46Speaker 02

  hugging face transform models to be able to then have that executed on our device. And so we truly made it a zero step process to take weights that you're already running on an Nvidia based platform and have that run on us. So like the bits, I, you know, as the CTO, I can, you know, I like to be technically precise in that. You know, the way that we say that we're in the CUDA ecosystem, it's by nature of the fact that we are ingesting the, the output of everything that exists in, in the CUDA infrastructure. So the fact that we are betting that people are going to continue to train on Nvidia for at least the foreseeable future, where, since we're able to, you know, and I'll say, I really hope others are able to be successful in, in providing competition against Nvidia.

• 22:30Speaker 02

  But given the realities of things, we're going to assume people are going to still be buying a whole lot of Nvidia hardware. We want to be able to have it be a zero step process for people to be able to take models trained and that they're using on Nvidia and then be able to deploy that on Positron and get the performance, get the, the performance per dollar and performance per what benefits from that.

• 22:51Speaker 01

  Yeah, I, I think that's a very reasonable, uh, business bet and, and, you know, thereby technical bet. One of the questions I was fed was just actually a little bit, you, you always mentioned, I think, the per, per watt uh, efficiency, you're like a third, uh, uh, of the, of the normal power consumption. And maybe that's underappreciated, but like how much actually power consumption is a part of inference cost?

• 23:14Speaker 03

  It's.

• 23:15Speaker 01

  Apparently, FPGA's are not, they're supposed to be more power hungry. Like, how do you, how do you get there?

• 23:22Speaker 03

  Yeah, I think so. Just one thing on the power consumption to your point about under appreciation. I think for, for us, like, we have run a cloud business, right? So when we look at the cost structure, it's always you have CapEx, so that's performance per dollar, like what you're paying for the chip. And then you have OpEx, which is performance per watt or energy that you're doing. And that's why, by the way, like, just wanted to very explicitly say why we focus on those two metrics is we wanted to drive both the CapEx and OpEx. Either efficiencies higher or like basically their cost structures lower, right? Like that, that's kind of how we really focused on. And with the FPGA, I mean, in general, that's true, but I think that's kind of where our own silicon design and memory architecture. And actually finding the right FPGA was the critical point to, to really drive that.

• 24:04Speaker 02

  Like, well, I would just say that it really depends on what application use case. If you're comparing an FPGA power to a cell phone, then yeah, the FPGA is a lot more power. But like our cards are only using 150 watts. So it's really, you know, how you're using it, where it's being used, et cetera. Like, I'll say, like, FPGA's traditionally have had like three or four main areas. One, you're using them, you know, internally, typically in a semiconductor company as a prototyping vehicle for, you know, whatever custom silicon you want. Two, they got very popular in the HFT space to be able to design algorithms. And, and maybe I should explain FPGA's for people that don't know, but, uh, you know, FPGA stands for Field Programmable Gatorade.

• 24:43Speaker 02

  It's basically the, the abstract way to think about it is it's a special type of chip that you can kind of think of as being a, a sea of transistors. And you can actually program in, you know, and, and reflash the device to have different wiring connections of it. That's not an actually technically factual statement, but it's the, the easy way to think about it. So FPGA's can enable you to take any chip design that you have and you can implement on the FPGA. And rather than having to tape out with great expense and time and everything, you can try out those designs on the FPGA at very low development cycle cost and, and time.

• 25:22Speaker 02

  That being said, that FPGA, in order to simulate like one gate that you would have in a normal piece of silicon, is using an order of magnitude more gates to simulate that. They are inefficient from that standpoint, but I would really be pointing back to what I was making before of that, you know, everyone else has been focused on that 500 to 1,000 flops per byte world, and they are that imbalanced that we can use an FPGA that was not designed for doing this workload, like that hardware is very general from that perspective, but implementing our unique architecture on that, we're able to get these huge efficiencies because we're focused on that specific transformer memory bound use case.

• 26:06Speaker 03

  Yeah, and it kind of reverting back to your question of like, how did we launch so quickly, like, I mean, relying on FPGA was the way that we got a product out so quickly. With our next generation, it's going to be a dedicated silicon where, you know, the inefficiencies of FPGA, such as just the brute force specs on both flops and memory capacity, memory bandwidth, where those are lacking compared to the latest GPUs or things like those, will all be addressed, will be improved upon, so that our next generation silicon will have the highest memory capacity by almost, you know, four to five times than any of the existing silicon at that point of time. So we are going to be coming out with our ASIC in like later 2026, it will have more

• 26:47Speaker 03

  memory capacity than any other silicon in late 2026 or in 2027, actually. And so that's kind of where we are showing that, hey, you know, with our architecture, we can run it even on an FPGA this fast and provide those per dollar and per watt. So imagine when dedicated for a system designed for that architecture, how much more we can perform there and ensure performance.

• 27:12Speaker 02

  Yeah, I think like the really key thing is like in terms of flops, we have about 20 teraflops of, you know, technically it's FP19, it calls it TF32, and I don't want to give them credit for having a very misleading. So Nvidia's TF32 number format is a 19 bit number format. They just call it 32 bit because it's closer to FP32 in precision than BF16, so they just said, why not just double the number? I'm exaggerating a little bit, but it's a 19 bit format to TF32.

• 27:50Speaker 02

  You know, we have that as our number format in our FPGA, which gives us better precision in our actual multiplication, which gives us, you know, a lot of benefits from a quality of result perspective. But, you know, our FPGA is literally, you know, 20 teraflops versus about a petaflop on the H1, and we're still providing about 70% faster token generation rate. So it kind of goes to show that the flops that Nvidia spends so much silicon area on and so much power and everything else is not well utilized.

• 28:24Speaker 03

  For, again, for transformer.

• 28:26Speaker 02

  For inference.

• 28:26Speaker 03

  Again, those flops are needed for the training side, which is why when you're using the same chip across the board, like that's kind of where we see our opportunity into the inference efficiencies that we're driving.

• 28:35Speaker 02

  But yeah, I will say for our second generation product, we're, you know, in the petaflops of compute range and everything else. It's, you know, to go back to your original question of how did we do it so fast? You know, small team, not much sleep, et cetera, et cetera. But, you know, the core, core element, like just philosophically different, is we decided, hey, we can actually use FPGAs for doing this because our architecture is so unique and targeted to this problem in a very different way than, than other approaches.

• 29:02Alessio Fanelli?

  What's the penalty, so to speak, to go from FPGA to ASIC? And then there's kind of the question of, well, maybe I should just get an ASIC for like the actual model in it, which I think obviously you shouldn't do because the models evolve and all of that. But what are you exactly burning on the ASIC? And then what are you giving up by moving away from a FPGA model?

• 29:23Speaker 02

  Yeah, so I would say another big difference between us and, you know, there's other startups out there that have, you know, said that they're burning or, you know, etching the transformer architecture into silicon, and that's their approach. We're very opposed to that kind of philosophy. Like fundamentally, what Posh from is built is a linear algebra accelerator, which is optimized for matrix vector math in particular. And I would say, you know, more importantly, the fact that, you know, we achieve this, this, you know, massive memory bandwidth to that pretty general compute architecture. So like, I think it's pretty foolish. For anyone in the industry right now to be saying that like transformers are going to be absolutely a hundred percent the thing that gets us to AGI.

• 30:07Speaker 02

  Or even if they do, that there isn't a better architecture. And, you know, doing any of that like hardening for specific model things, I don't think lasts more than, you know, two or three months at the rate that the industry moves at. And so, but the thing that I would, you know, be willing to bet, you know, uh, you know, good money, you know, the company on.

• 30:27Alessio Fanelli?

  Exactly.

• 30:27Speaker 02

  Um, is, is the fact that, uh, um, uh, you know, fundamentally linear algebra, uh, like building a good linear algebra processor. Matters just as much in, you know, nineteen fifty five as it does in twenty twenty five as it will in twenty fifty five. Having, you know, solving that as a core problem is what we've done and what we're focused on. And the, the software layers that enable you to get these models to run efficiently on that. You know, is, is a lot of our secret sauce to, you know, not, you know, harden the hardware in a way that makes it, um, uh,

• 31:05Speaker 02

  too optimized for just the thing that is hot right now, but really just build something that's good at the actual underlying math.

• 31:13Alessio Fanelli?

  Is there something that will make you change your mind? Something in maybe the model architectures or anything like that?

• 31:21Speaker 02

  I would say that the, like, I, I totally believe that there are application use cases that at our high enough volume, enough demand, et cetera, that, you know, a company can say, okay, we want to build this super optimized chip that uses single digit watts or something like something that is so optimized. Cause we know we're going to build a billion of these things and we want it at the lowest cost and lowest power for just this one thing. I believe that those applications can exist. I've not seen something that like would meet those criteria for me that would be worth the, the N R E and, and everything involved to, to do that. And if you're trying to build something that has general purpose to saying, okay, building a transformer specific chip that is hardening things in a way that are very, very specific to current transformer architectural decisions.

• 32:06Speaker 02

  I think pretty much everyone in the space should know that. This industry is moving astronomically fast and it's accelerating.

• 32:12Speaker 03

  There's, I mean, I don't have the example of what you asked, but there's counter example, actually, to what Thomas is saying. I mean, even the attention mechanism changing, when deep seek, well, not changing, but like if deep seek opened it to the world, which was the MHLA stuff. I think if you had burned or edged in the, the architecture of doing transformer the way before that, you know, you might not be able to actually gain all of those efficiencies that the attention mechanism that deep seek announced to the world, right? So, so that's, that's in my mind, a counter example, why that wouldn't, that is not the best approach, or at least we think that's not the best approach, right? So that's not what we are doing is, is burning to a particular architecture.

• 32:47Speaker 01

  Cool. Yeah. I mean, I, I, I think this is relevant to the, always the discussion of the other ASIC companies like etched. I don't know if you've seen the beef that George Hots has had with etched. Any comment there? What, what, is he, is he right? What, what is, what is he, what are they talking about?

• 33:04Speaker 02

  George loves having beef with everyone. George and I have known each other for probably 10, 12 years now. I, I, I don't actually remember specifically all of his beef or anything. My, my, my opinion in general is it's a remarkably difficult thing to build any new silicon. And I know I've met, you know, the etched guys as well. They're cool guys and, you know, I respect them tremendously for, for building a company and, and, uh, tackling a hard problem. I think that the, I would disagree with the idea that you should be hardening in that way. And I would say I'm skeptical of how much optimization you can even get with those things, with that wouldn't be putting you in a crazy restricted corner.

• 33:45Speaker 02

  But I'm, I'm looking forward to seeing what they develop. Like I'm a technologist. I, I, I want to see cool things in the world, you know? Even if I think, you know, that we're building something that's going to be better in a more pure sort of linear algebra direction. I can totally believe that we'll have success in what we're doing and others will have success in their areas. You know, I would say the same thing for any other company, like I would say, just in my talking with, um, you know, the, the founders and, and people in, in all of the different semiconductor, you know, startups trying to go after Nvidia. Success for any one of us is a success for all of us, like anyone that can like take. A little tiny chink in Nvidia's armor is a, you know, a great win for eventually Goliath not collapsing. I mean, the reality is Nvidia is going to have a very, very good decade ahead for them.

• 34:30Speaker 02

  And the market is growing so fast that all of us in the space trying to, to take them on can be very happy with, you know, very, very small wins in the space.

• 34:40Speaker 03

  Not, not a huge fan of the particular, I am not, but obviously that, that, that's just, I think the other way, the way I like to present that one is actually, and it's not even just trying to be nice or anything, it's just really like, look, the, the markets are growing so massively. And when CERN app inference applications will come out, they'll be at such a big scale that people will want to minimize costs, right? Because inference is what drives revenue, so that's actually driving true margins for companies. So they'll want to minimize costs. And if our architecture is suitable for certain industries, much more than, let's say Nvidia's or AMD's or any other silicon architectures, I think they would use us. So like, in a way, it's kind of like expanding the market for those companies to be much more efficient than what they would be on an Nvidia GPU

• 35:22Speaker 03

  or an AMD GPU. So I think, you know, from, from my perspective, it's not just like, hey, you're trying to capture, you know, small point one percent or one percent. I think the idea, the goal is from a technical point that Thomas and I always talk about is obviously going and supporting a hundred percent of the market and then really pruing that out and proof is in the pudding and then capturing as much of it. But you have to start somewhere and, and we're starting somewhere, which is, okay, we're focused on this really, you know, where, where we think today are the primary inefficiencies in AI inference workloads and really solving them and showing that for, for it. Thomas said that, you know, we are accelerating matrix vector. It doesn't mean that we can't do matrix, matrix, we can, and we can do it fairly well. It's just that the point becomes is like, are you really bidding Nvidia on it from a per, per dollar?

• 36:05Speaker 03

  And if you're not, then Nvidia is, is, is the existing thing. So people won't move on it. So today we won't be able to, but in, in the future, I think that's kind of, as the company matures and, and if we get the opportunity to grow revenue to that scale, that we can actually focus on these things, then, then we will try to.

• 36:20Speaker 02

  And I would also just say from like that competition scale, like, again, I'm very ho, like hopeful that many people come out to market with great pieces of silicon. I would say from a competition angle, the key thing that everyone should really be looking at to measure success fundamentally comes down to revenue. You can, you can announce the greatest piece of silicon, it could be a great, have all, all of the right specs, et cetera, but. I would, I'm a big believer in capitalism and you know, the invisible hand and that's like the companies that everyone is trying to sell to, the companies that are building the future grand applications that we're all going to be using, they have all either raised money or they've generated, you know, their own revenue and they have dollars and they're going to commit those dollars to the, the infrastructure providers that actually generate value to them.

• 37:04Speaker 02

  And so the, the biggest thing that everyone in the market, I think, forgets when they hear so and so company has raised a huge amount of money. Raising money is one thing, being able to deliver real value to the customers, to the end users, such that those customers are then, you know, paying you for, for that value is, is the only true measurement. I mean, I'm really happy that, you know, we had our first revenue, like we, we sold our first system, you know, 15 months in, you know, we've deployed now multiple rack scale deployments, we're announcing, so I think with, when this is being timed, we'll

• 37:43Speaker 03

  Can embargo this, yeah.

• 37:44Speaker 02

  Yeah, um, you know, we're, we're announcing two, two customers with our fundraising, but we have multiple others that we're hoping to announce in the coming, coming weeks and months.

• 37:53Speaker 03

  So we have customers in Cloudflare and Parasale, both using us. Cloudflare because of performance per what kind of a story, like, you know, they, they really love the fact, look, they have data centers at the edge in this, in metropolitan cities, right? So they can't really scale out the power that much. So they, they want to maximize performance for, for the given amount of power that they have. So that's kind of where they're really working with us. And then Parasale, which is, you know, much more focused on performance, raw performance, because they're an inference as service company, right? So, so that's kind of where they're targeting us. And then by the time this comes out, I think we'd have our fundraise announcement done, uh, you know, and that's, you know, we, we've raised in the history of the company so far, they've publicly announced roughly

• 38:32Speaker 03

  24 million. And then we just did a series A close with Valor, Atreides and DFJ growth for a roughly $51 million series A. So, um, and, and that money for us is really like focused on coming up with the next generation silicon, we get to tape out, we get to sampling, we get to production with that money. And again, we, we really want to prove it in a capital efficiency way. Look, I think that's what we've done so far. Ideally, we get the customers to pay for our, our production. So like, like, and We want to sell systems, like we want to sell systems to customers. Don't ideally like, you know, want to go into this cloud services where you're owning the the infrastructure and your balance sheet. That's kind of where a lot of the current AC companies have gone. You know, if you look at Cerberus and Grok and others, they've, they've really tried to do this,

• 39:18Speaker 03

  their cloud kind of portal. And, and for us, you know, it, it shows two things. One is the difficulty of software for a third party to, to implement that. That, that's why they're kind of abstracting away that layer and then providing us a cloud API. And then second is to convince a third party customer, you really have to show the economic viability of your system, right? Like that, hey, this, the return on investment on this system will be two years, three years, one year. Nvidia has shown that, like, you know, if you deploy Nvidia GPUs, hyperscalers, NeoClouds, customers have all seen that they can make that revenue back over a certain timeframe. And, and that's what we're really here. We want to sell systems and show people can make that ROI back on that system. So that's the way we are approaching and going to market.

• 39:59Speaker 02

  I think the, the foremost important letters are ROIC.

• 40:04Speaker 02

  You know, from, it's, it's not just from, as Mateusz was saying, that from a customer's perspective, if they're deploying hardware, they're going to be choosing between be it Nvidia, AMD, Positron, or anyone else. They're looking at it from an ROIC perspective to also in our, from, from our side, we don't want to go and raise hundreds of millions of dollars. Like, we, we had the opportunity to raise, you know, many, many times more than what we, we raised earlier. I was adamant, like, we raised originally a $12 million chunk and then about 10 million afterwards to, to get us to the point that we are today. And it was because I want to raise as little as possible to minimize dilution and because I feel that companies that just raise huge amounts of money typically don't spend it very well.

• 40:45Speaker 02

  And I have a like really key metric for our success is how efficiently we can actually deploy that capital to creating real value that we can then provide to customers.

• 40:54Speaker 01

  Yeah, amazing. I love the messaging. One thing I wanted to double check and then we can sort of wrap it is, is another way to phrase it is that a lot of the other folks are focusing on pre-fill optimization and your, because of the linear algebra and, and sort of general matrix math capability, you will just decode a lot faster.

• 41:12Speaker 02

  Yeah.

• 41:13Speaker 01

  So, okay, I see you guys nodding.

• 41:15Speaker 02

  Yeah, so like, I would say definitely our, our performance focus and where we're getting the best relative advantage over Nvidia is on that generation or decode side of it. So, you know, for, for any transformer, you know, you have a prompt that you're sending in. And that prompt, you know, of some amount of size.

• 41:40Speaker 02

  Computationally, that forward pass is exactly like the forward pass when you're doing training, because you've got the whole sequence of tokens that you, you want to process. All of that is there for you upfront. Then, when you switch into generation mode, then you're generating one token at a time. And, and that is, is the memory bound portion. So, yeah, I, I would say, you know, we're still fully capable and, and perform well with, with pre-fill. But I would say the thing that we believe, and, and I was so happy, you know, last year, when, when all of the new reasoning applications happening. Because if you, if you go back a year from, from today and, and July of, of last year, the ratios of like input to outputs for, for LLMs were very, very heavily on, on inputs,

• 42:20Speaker 02

  where you could be doing, you know, ten, ten, fifteen to one ratio of input to output. But that has completely flipped. And it's obviously use, use case dependent. But if you look at all of like, when people are running benchmarks on these new models, they're generating, you know, it's not just flipped. You could be generating like a hundred reasoning tokens for every token that you had as a input pre-fill. The other big thing, and like our big belief in having massive memory capacity, is the fact that, you know, if you're able to actually just cache the, the KV caches, if you're able to hold massive system prompts, massive data sets, et cetera, that are typically recomputed every time you, you send in a new request. If you are able to just store that, because you've got a massive amount of memory that you,

• 43:04Speaker 02

  you know, lying around, then that even makes it so that the true pre-fill compute that you're doing is significantly less.

• 43:10Speaker 03

  Yeah, and then I think Thomas touched upon like token generation. If you also look at like other modalities, like video generation, for example, right? And I think if you convert that into effective tokens or whatever, like then, and then if you look at the ratios there. That's that's also a completely flipped script. So like, I think that's other part of it where we will see a lot more integration into our models, reasoning models, multimodal models. And I think, but as Thomas mentioned, I think we are definitely like where our true advantage shines through is is is generation. So like time to last token, generation speeds. We are way off the charts compared to any other existing solution out there.

• 43:44Speaker 02

  Yeah. I'll just add there on like the video example. The other exciting thing to see over the past few months, year has been that a lot of things have actually been moving away from diffusion to being pure autoregressive transformers for image and and video generation. So like the latest, yeah, this V O three and and since image gen three on on Google's side have been pure autoregressive, moving away from diffusion. The latest.

• 44:08Speaker 03

  I'm not sure we knew that.

• 44:10Speaker 02

  Um, the Athena model from X A I on image generation side is pure autoregressive transformer. The chat G P T image gen, the the, you know, that's pure autoregressive transformer. And so all of those cases are now diffusion could be extremely compute bound because you're generating all of your pixels. You're doing compute on all of them simultaneously. But when you are in autoregressive mode, that is back to this one to one bytes to flops ratio, which is where we. The second part is like when it comes to video. Like literally right now, the the rumors, estimates I've I've heard is that like for the eight seconds of, you know, V O three image generation is, you know, around, you know, eight hundred thousand to a million tokens.

• 44:55Speaker 02

  So if you think, if if you're just ballpark of saying it's a hundred thousand tokens per second generated, the actual input prompt to that video model is hundreds of tokens, maybe if that. So that that's a that ratio is just absurd when you get into modalities beyond text. And that being said, I do think bringing in video and then saying, okay, I want a new version of it. Like the ratios will always adapt and it'll be different for different applications. But I think the key thing everyone needs to remember, you know, everyone calls this generative A I. I'm going to say it's maybe reductionistic, but that's a generative AI. Hey, generation is a lot more important than that pre fills.

• 45:33Alessio Fanelli?

  Awesome, guys. This was great. Any call to action? I'm assuming you're hiring, you just raised a new round. What type of people are you looking for?

• 45:41Speaker 03

  Yeah, I think that's, that's, that's a big call. I mean, we are a small team, 27 people right now at this moment. We are definitely heads down working at our next silicon, right? And as with any silicon, you know, looking into hiring great, you know, verification engineers, design engineers, silicon folks that have experience, you know, N5, N7, N3, obviously is, it's very relevant to us from Ross's notes perspectives. Really looking to get that team ready to be testing verification emulation. And then when we actually get the tape out in hand, really running quickly to, to system integration. So that's kind of the big part of it. Other part that I really do want to call out and not forget is the software engineering side of things. I think, no, we've, we've done well with the small team and the approach we've taken is really, really good.

• 46:26Speaker 03

  We bet on the Hugging Face Transformers. We do want to make sure that we are keeping up with the multimodalities of, of the different models that are coming out. Obviously, as I said, like, we are accelerating matrix vectors. So we're really good for transformer inference, but it doesn't stop us from supporting other model types as well. And so we really want to build a very rounded software engineering team as well. So looking forward to hiring that. Yeah, we, we focus around the assets.

• 46:52Speaker 02

  Yeah, so we, we are a remote distributed company. Uh, we do have, you know, a few, uh, two offices right now.

• 46:59Speaker 03

  Yeah.

• 46:59Speaker 02

  Uh, but I would say that like, really want to be able to bring people in that have no hardware experience. And especially those that have been absorbed in the machine learning space and really understand models and want to take it to the next level of seeing how do you actually get the, the most efficiency? And how better models can be created, given, you know, new hardware that comes out. We're, we're definitely, you know.

• 47:22Speaker 03

  He's talking about the next step after, like, you know, when you think about someone looking at our architecture and like, actually, you know what? I can make a model architecture to be a lot better because I have all this memory capacity or all this memory bandwidth available to me, uh, as well.

• 47:36Alessio Fanelli?

  Cool, awesome guys. Thanks for joining.

• 47:38Speaker 03

  Thank you so much. Thank you for having us, Sennett.