Software and hardware acceleration with Groq

How do you enable AI acceleration (at both the hardware and software layers) that stays ahead of rapid industry shifts? In this episode, Dhananjay Singh from Groq dives into the evolving landscape of AI inference and acceleration. We explore how Groq optimizes the serving layer, adapts to industry shifts, and supports emerging model architectures. Featuring: Dhananjay Singh – LinkedIn https://www.linkedin.com/in/djsh/, X https://x.com/djays%20Chris Benson – Website https://chrisbenson.com/, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaLinks: Groq https://groq.com/Sponsors: Augment Code https://www.augmentcode.com?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog - Developer AI that uses deep understanding of your large codebase and how you build software to deliver personalized code suggestions and insights. Augment provides relevant, contextualized code right in your IDE or Slack. It transforms scattered knowledge into code or answers, eliminating time spent searching docs or interrupting teammates.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Speaker 00

  Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, and accessible to all. If you like this show, you will love The Change Log. It's news on Mondays, deep, technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at fly.io. Launch your AI apps in five minutes or less. Learn how at fly.io.

• 0:44Daniel Whitenack?

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined, as always, by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris? Doing very well. How's it going today, Daniel? It's going great. Yeah, it's been a fun, productive week in the AI world over here at Prediction Guard, so no complaints. But this is a really, I'm excited about this episode because it's one I've been wanting to make happen for quite a while. Today, we'll be talking about both AI hardware and software with

• 1:27Daniel Whitenack?

  DJ Singh, who is a Staff Machine Learning Engineer at Groq. How are you doing, DJ?

• 1:34Speaker 02

  Hey, Daniel. Thanks for having me. It's been going well.

• 1:38Daniel Whitenack?

  Yeah, good, good. Yeah, and I guess we should specify for our audience, this is Groq, as in G-R-O-Q. I imagine maybe some people get confused these days with that. But yeah, this is one that I've been really excited about, DJ, because I've been observing what Groq has been doing for some time, and of course, innovating in a lot of different ways, like I mentioned, on the hardware side and on the software side. So could you maybe just set the stage for us a little bit in terms of the overall ecosystem as you see it in terms of kind of what may be a bloated term of like AI accelerator or hardware, and also the

• 2:21Daniel Whitenack?

  software that goes along with that, and kind of where Groq fits into that ecosystem?

• 2:27Speaker 02

  Right. So I think I'll first start and just quickly brief about Groq. So Groq is of course a company which provides fast AI inference solutions. So whether it's text, image or audio, we are delivering AI responses at blistering speeds and order of magnitude more than traditional providers, right? Now, you spoke of AI accelerators and traditionally training and inference has been done on GPUs. But I think in the last few years, we've seen all sorts of AI accelerators come in place. So there are those more mobile device oriented ones that phone companies like Samsung and Apple come up with, right?

• 3:12Speaker 02

  And then there are more, more stuff happening on the server side, part of which is what Groq is also leading towards.

• 3:22Daniel Whitenack?

  Yeah, that's great. And on the server or hardware side, am I correct, is sort of Groq does have their own sort of hardware that they've developed over time? Is that right? What's kind of been the progression of that and the current state?

• 3:38Speaker 02

  Absolutely. So Groq developed this technology which we call as Groq LPU. It's essentially a software and hardware platform which comes together to deliver that breakthrough performance of low latency and high throughput. But Groq's, how Groq got into it was first to develop that software. So we developed the software compiler first before moving on to the hardware side. Kind of a shift in how traditional development was being done previously.

• 4:11Daniel Whitenack?

  And did that, I mean, that does seem very unique to me. So the, what was the, I guess, the motivation or the, the thought process behind taking maybe that non-standard, standard approach kind of compiler first, then hardware?

• 4:25Speaker 02

  Yeah, no, absolutely. So traditionally, as I mentioned, a development is done and that a new accelerator is developed. So if somebody makes the hardware first and then the software has to deal with the inefficiencies of the hardware. Whereas when Grok decided, and this company is founded by Jonathan Ross, our CEO, who was a co-founder of Google's TPU program, the Tensor Processing Unit program. And based on his learnings from there, one of the key decisions was let's develop the software first, right? So we have developed the software compiler,

• 5:04Speaker 02

  which helps to convert these AI models into this code which runs on the Grok LPU. But specifically, the compiler is responsible for scheduling each and every operation of that AI model. So you can think of it like an AI model in terms of compute is made up of like additions and multiplications. And it kind of, the software compiler, it decides where and when to schedule something. And that goes into our, you know, various design principles, one of which, of course, I mentioned is to be software first, right?

• 5:47Speaker 02

  Now, you might ask, why do we do this, right? So, one, one key consideration is that not only does the software have to deal with hardware inefficiencies, right? But there are other aspects of the hardware which can add on delays, whereas Groq prefers to have a deterministic system in place. So determinism, I would say, is like deterministic compute and networking. So kind of have an understanding of where and when to schedule an operation. So to understand this, we can consider an analogy.

• 6:27Speaker 02

  Now, imagine a car driving around along the road with several stop signs. Stopping at every sign is essential for safety, but it does add some delays, right? Now, what if the world was perfectly scheduled and we knew where to start the car? And drive at maximum speeds so that there are no collisions, right? Uh, so there would be no need for these stop signs, no delays as such. And it also makes a more efficient use of the road since you can then have more cars and everybody's like going at maximum or near maximum speeds. So to reflect this analogy and back to the hardware space,

• 7:08Speaker 02

  Groq chose to remove components which can add delays. Uh, so it could be, let's say, networks, switches, or other even algorithmic delays, some sort of algorithms which control packet switching. These all things add non-determinism into the system.

• 7:28Daniel Whitenack?

  I did wanna, maybe some of our listeners out there, like you've been talking about this compiler level, which, you know, I think of a compiler, um, similar to what you said as, hey, I'm writing some higher level software code that's compiled to these instructions that run under the hood on the actual hardware components doing, as you said, additions or or whatever those sorts of numerical operations are. Um, but people might sort of be confused also in terms of the software stack. They may be familiar with something like CUDA, which helps, you know, uh, have, have drivers to run on certain hardware like Nvidia GPUs.

• 8:10Daniel Whitenack?

  Or I know, um, you know, we've, we've worked a little bit with, uh, Intel Gaudi processors and there's driver package synapse, which is similar in, in that sort of way, helps translate, um, kind of your higher level code to, to run on these, these hardware components. Could you help us kind of map out that software stack? Like where this compiler fits in? And are there other components like these, these drivers that would have a parallel in, uh, in the Groq world?

• 8:39Speaker 02

  Yeah, so traditionally, as you've mentioned, like on, let's say, Nvidia ecosystem, there are like tons of engineers who go and create these kernels, which are invoked when you have some sort of model operations. So there would be maybe even thousands of engineers in the company would work towards developing this very specialized kernels to go and execute things. However, due to the the structure of the GPU itself architecturally, this is not the best philosophy for design. You know, I'm sure the audience is familiar with GPUs.

• 9:23Speaker 02

  I remember playing games on them growing up and editing videos. And these grew up to be more powerful in the recent decades. But, you know, GPU started in the nineties and the design hasn't changed all that much. They've had an addition of high bandwidth memory and other hardware components to it. But all of it essentially is still the original design, you know, originating from the original design. It does make the system to be, again, less deterministic. So that goes back to the compiler system here. And let's let's talk about the Nvidia GPU kernels here, right?

• 10:06Speaker 02

  So they have to deal with the different hierarchies of memories as an example. So for those of the listeners who are familiar with the different memory systems in a computer system, right? You might be familiar with like an L1 cache, which has like an, you know, time of access time of like one nanosecond. But you then have these bigger memories, which are higher bandwidth memories, which are like closer to fifty to hundred nanosecond.

• 10:37Speaker 02

  And for a task to be processed performantly, uh, data needs to be fetched from between these different memories onto the compute, which is there. And that transfer of data adds in more delays. And since this is a conservative system, right? So let's say you have two operations and one depends on the other. It's waiting on that operation to complete. So it adds on further delays, you know? So one operation is stuck on waiting on the data. The other operation is stuck on the, the second operation, right? So that kinds of, this incrementally adds more and more delays into this. So that's the, an example

• 11:18Speaker 02

  of how the traditional, I guess, compiler or the traditional kernel-based system, uh, doesn't scale as well. Uh, what Groq chooses to do, of course, is not have any kernels whatsoever, but have a compiler which controls this at a fine-grained level. So a typical system will have multiple chips, right? So AI, you know, I'm sure people are familiar with models like Llama 70 Billion, right? And these models tend to be split across multiple GPUs and even on multiple Groq chips, right? And this compiler kind of controls

• 11:58Speaker 02

  how this model is precisely split across these different chips and how it's executed. And to get the best performance out of it, down to the level of the chipset and the networking. So as I mentioned before, we've removed a lot of the hardware which adds delay. And this sort of scheduling is done, uh, by, by Groq's compiler alongside with some assistance from, of course, the firmware which is there.

• 12:27Chris Benson?

  And I think I appreciate that. Um, as we talk, I'm trying to get a, a good sense of kind of how the whole stack looks, um, as you're starting to dive into it. And you've talked a bit about the kind of the, the compiler versus having a kernel, uh, kind of at the, at the model layer there. But with, with you guys covering Both the hardware and the software is Groq. Would you say Groq is I try to understand kind of that whole business model that you're approaching it with? Is it is it more of an integrator that's full stack all the way from the hardware up through the OS and into the model layers? Or like from an integration layer, or you do you think you're are you writing most of the the software stack that's touching the hardware?

• 13:11Chris Benson?

  Like, how do how do you choose whether to to go pick and I'm just pulling things out of there, not attributing to you, but going and picking Linux and picking CUDA and picking this and picking that versus what you're writing to create your own full stack? How do you I'm trying to get a sense of kind of how that's distributed those decisions from a design standpoint?

• 13:32Speaker 02

  Yeah, no, that's a great question. So all the way from our starting stack, right? So let's let's start at the top. So most folks end up when think about using AI models in production would end up using some sort of API. So we our cloud organization designed a REST compatible API. It's compatible with OpenAI spec, which is there, which makes it very easy for developers to really integrate with it. And then that ties into all the way into our rest of our stack. And to answer your question directly, yes, most of the stack has been custom written.

• 14:14Speaker 02

  We are, of course, using some Linux-based primitives which are there underneath our system. And there are, of course, some components such as for the compiler, there is this MLIR system which is being used. MLIR is like a compiler term. I don't want to go super deep into it, but it's like a multi-level intermediate representation, which kind of helps to transform things in between. So overall, I would say this entire design pattern has been thought through from scratch. And it's taken the company a couple of iterations to get to that point.

• 15:07Speaker 04

  Well, friends, I am here with a new friend of mine, Scott Deetson, CEO of Augment Code. I'm excited about this. Augment taps into your team's collective knowledge, your codebase, your documentation, your dependencies. It is the most context-aware developer AI. So you won't just code faster, you also build smarter. It's an "ask me anything" for your code. It's your deep-thinking buddy. It's your Stan Flow antidote. Okay, Scott, so for the foreseeable future, AI-assisted is here to stay. It's just a matter of getting the AI to be a better assistant. And in particular, I want help on the thinking part, not necessarily the coding part. Can you speak to the thinking problem versus the coding problem and the potential false dichotomy there?

• 15:48Speaker 05

  Couple of different points to make. You know, AIs have gotten good at making incremental changes, at least when they understand customer software. So first and the biggest limitation that these AIs have today, they really don't understand anything about your codebase. If you take GitHub Copilot, for example, it's like a fresh college graduate, understands some programming languages and algorithms, but doesn't understand what you're trying to do. And as a result of that, something like 2/3 of the community, on average, drops off of the product, especially the expert developers. Augment is different. We use retrieval augmented generation to deeply mine the knowledge that's inherent inside your codebase. So we are a copilot that is an expert and that can help you navigate the codebase,

• 16:32Speaker 05

  help you find issues and fix them and resolve them over time much more quickly than you can trying to tutor up a novice on your software.

• 16:41Speaker 04

  So you're often compared to GitHub Copilot. I gotta imagine that you have a hot take. What's your hot take on GitHub Copilot?

• 16:48Speaker 05

  I think it was a great 1.0 product, and I think they've done a huge service in promoting AI. But I think the game has changed. We have moved from AIs that are new college graduates to, in effect, AIs that are now among the best developers in your codebase. And that difference is a profound one for software engineering in particular. You know, if you're writing a new application from scratch... You want a webpage that'll play tic-tac-toe, piece of cake to crank that out. But if you're, you're looking at, you know, a tens of millions of line code base, like many of our customers, Lemonade is, is one of them. I mean, 10 million line mono repo as they move engineers inside and around that code base

• 17:29Speaker 05

  and hire new engineers, just the, the workload on senior developers to mentor people into areas of the code base they're not familiar with is hugely painful. An AI that knows the answer and is available 7x24, you don't have to interrupt anybody and can help coach you through whatever you're trying to work on is hugely empowering to an engineer working on unfamiliar code.

• 17:51Speaker 04

  Very cool. Well, friends, Augment Code is developer AI that uses deep understanding of your large code base and how you build software to deliver personalized code suggestions and insights. A good next step is to go to augmentcode.com. That's A U G M E N T C O D E dot com. Request a free trial, contact sales, or if you're an open source project, Augment is free to you to use. Learn more at augmentcode.com. That's A U G M E N T C O D E dot com. Augmentcode.com.

• 18:32Daniel Whitenack?

  So DJ, you, you mentioned that the, a lot of the focus around, you know, really that design from the hardware layer up through those software layers and digging into all of those was to achieve fast inference. Could you tell us a little bit about the kinds of models that you've run on Groq? And just some, you know, some, some highlights in terms of when you say fast performance, what, what, what does that mean in, in practice? Now I've seen some pretty impressive numbers on your, on your website. So I won't, I won't steal your thunder, but, uh, but yeah, just talk a little bit about kind of what is achievable with what kinds of models on the Groq platform?

• 19:17Speaker 02

  Yeah, so first of all, you know, I I share some numbers, but we are just getting started. So these numbers are only gonna get better with time. But like, let's say, let's take Lama three seventy billion as an example. Tends to be one of those industry standards for comparing performance. So we've had numbers all the way from like three hundred tokens per second to like multiple thousands tokens per second. Depending on those use cases. Right? And yeah, we've had some smaller models which go up to several thousand tokens per second. We've had our, one of our speech to text models called Whisper.

• 20:00Speaker 02

  Which is again an OpenAI model running on on Groq. And this model, I think we've gotten around two hundred X as the speed of factor as they discuss it in the audio world.

• 20:12Daniel Whitenack?

  Yeah, and and maybe talk a little bit about. And maybe for those out there that aren't, they're trying to process these thousands of tokens per second, what does that imply? I would say, you know, if you're using a chat interface, for example, and and something is responding at thousands of tokens a second, it's. It's a, it's a, you know, potentially a wall of text, it's sort of almost all at once, as far as. As far as our human eyes see it. Could you talk a little bit about the implications of that? So I mentioned the chat interface, which certainly some people are using chat interfaces, right? But at the enterprise level for true enterprise AI use cases, why?

• 20:54Daniel Whitenack?

  Why is fast inference for these kinds of models? What? Why is that important? Because like in a chat interface, I can only read so much text. So fast, right, with my own human mind as it as it comes back to me. Could you give us some, you know, and and I certainly am, you know, have my own thoughts on on this. But I'm wondering if you could could think about. Why does that speed matter in enterprise use cases? And why does it matter to push that maybe, you know, further than, you know, our own speed of reading, for example?

• 21:30Speaker 02

  No, great question. So I think if we were to start with what Google studies from a decade ago, right, people's perception of like search results is like if it takes longer than I think it's about two hundred milliseconds or so, somebody like lose interest, you know, so speed is critical, whether it's for the enterprise or everyday people, right? I mean, we've demonstrated this several times and you can try out for yourself, you can have like, let's say you open chat G P T with something like O one, or you have Groq on the side with one of our reasoning models, and you can try comparing them side by side. So what becomes more critical, as I'm coming to,

• 22:15Speaker 02

  is that we, like, everybody thinks of speed as as being, yes, it's important for real time applications, but then there is the aspect of accuracy, right? So if you could reason for longer, for, let's say in the case of our reasoning models, so we've had like Deep Seek R one, for example, right? And these models, they generate a lot of tokens, and if you can reason for longer, you can get higher quality results as a as a consequence of this. So while not making the system too slow for the user, right, whether again it's enterprise or it's for everyday users,

• 22:59Speaker 02

  speed can translate to quality as well.

• 23:01Chris Benson?

  So to extend that just a little bit, if if you are, and we've kind of been talking, you know, directly about inference speed and stuff like that, more from the practitioner's standpoint, if you're maybe, you know, a business manager or a business owner out there, and you're looking at Groq and you're kind of comparing it against kind of more traditional inference options that are already out there, when you're talking in terms of speed and for instance, being able to to have the time to do the research and stuff, what are some of the the use cases from a business standpoint where they where they need to go, it's time for us to reassess kind of the more traditional routes that we've taken on inference and and look at

• 23:45Chris Benson?

  Groq for these solutions? Could you talk a little bit about what some of those business cases would be?

• 23:50Speaker 02

  Yeah, I mean, if you care about accuracy, speed or cost, you should consider Groq. So not only are we fast, the Groq LPU architecture allows us to give really low cost, or I would say our costs per token are really low and we pass on those savings to all of our customers. So if you are concerned about any of these cases and you want to work with different modalities, if you care about image, text or audio, if you care about rag, if you care about reasoning, we are there for you.

• 24:25Daniel Whitenack?

  Yeah, and just to tie into that as well, some people might be listening to this and thinking in their mind, oh, Groq has this whole platform that they've designed, hardware and software. I don't have a, I don't have a, I don't have a data center. It's going to be expensive for me to spin up racks of these things. Could you talk a little bit about, I mean, I think, I think I could be mistaken, so please correct me. I think that is something that can happen. I mean, there are physical systems that people can access and use and potentially bring into their infrastructure. But I know also, I, you know, I see a login, I see API, as you mentioned REST API on your,

• 25:08Daniel Whitenack?

  in your previous answer about the developer experience. So maybe just talk through some of those access patterns and also how you as a company have thought about which of those you provide, because certainly there are, there are advantages on the hardware side of maybe a fixed cost. But then there's the burden to, you know, the burden to support that. So just talk us through a little bit about the strategy that you all have taken, because you are deploying this whole platform. How have you thought about providing that to users and what sort of, what sort of access patterns, I guess?

• 25:45Speaker 02

  Right, so I'd say to start with, one can go to our website grok.com and just experience the speed themselves. It's a chat interface. And then it's trivial to sign up for our account over there. And on a free tier, we offer like tons of tokens over there for free. You can sign up and get access to our APIs. So, once you get access to our APIs, and let's say you've already been using an existing API. Let's say you're using OpenAI. It's, it's pretty easy for you to switch to Groq. And it's maybe a single or two line change and just try it out for yourself, right?

• 26:28Speaker 02

  We firmly believe in letting people experience the magic themselves, other than us talking about it. I think just actions speak louder. So, yeah. For, of course, our deep enterprise customers, we of course do offer other services on that side, right? So, you're talking about single tenant and then there's of course multi-tenant based architectures over there. So, we do offer dedicated instances where there's a real need for that. And we do manage that. So now Groq kind of deploys its own data centers. And we offer those all, all over an API. So it's very easy for our customers to go and sign up and use them.

• 27:14Chris Benson?

  You could, I'm, I'm gonna ask you if you would, if, if you could kind of talk a little bit about it just because as folks are listening and stuff and they will go try that out after. And I know that we'll have links in the show notes to the site so that they can do that. But could you talk a little bit about, and you could pick your example, but you know, kind of you mentioned like the OpenAI, you know, and something that they've probably had, you know, had experience with. You know, it's, you know, one of those things that kind of everybody has at least touched at some point out there. And you're providing a better experience here. And could you talk a little bit about what that is? When you talk about go experience this yourself and you're gonna see how, how amazing it is. Could you talk through what you've seen your customers experience in that way?

• 27:59Chris Benson?

  Just so that listeners will kind of get a sense or maybe a preview of what they should experience. Having messed around with OpenAI for a while and now they're going over to Groq and they're doing that and they're going, whoa, this is amazing. What is that amazing that you're expecting them to see?

• 28:14Speaker 02

  Well, first of all, people are just amazed by the speed that they get, like the speed of the output that comes up, you know, whether it's text or audio, you just get the output right away, right there. It's really, really fast and it's, I think, really makes people think of new ways of doing things. So, you know, one example from our developer community, and you know, our developer community has grown to over a million developers now. So, one recent example from a hackathon was that somebody developed this snowboarding navigation system based on Groq, taking images and kind of trying to guide people while snowboarding.

• 28:59Speaker 02

  And as, my mind was blown by these creative geniuses that are out there. Just amazing. So, all sorts of new applications out there enabled by the speed.

• 29:11Daniel Whitenack?

  Well, DJ, I do want to follow up on some of what you had talked about there on the developer community. So, could you maybe clarify one thing for me? So, there's the Groq systems that you have deployed and models that you have deployed in those systems, which it sounds like if I'm interpreting things right, people can just use, I'm assuming, your programming language clients or REST API to access that API and build off of those models that are in that environment. So, in that case, it's sort of accessing models, maybe in a, like you say, in a similar way to they would access OpenAI models and that sort of thing.

• 29:53Daniel Whitenack?

  Is there another side of the developer community that is saying, hey, well, we're actually, we have our own custom models, whatever those might be? What is the process? I guess my question is, what is the process of? Getting a model supported on Groq, you've talked about mainly kind of the GenAI level models of, you know, LLM or vision or transcription. How wide is the support for models in terms of, hey, if I, if I have this model, you know, I'm thinking in my mind, in a manufacturing scenario, if I have a model that's a very specific model that needs to run at extremely

• 30:37Daniel Whitenack?

  fast speeds to like classify the quality of products coming off of a manufacturing line. Right. But it's a custom model. And I say, okay, I, you know, Groq has the fastest inference. What is the kind of, what should I expect in terms of kind of model support as of now? In terms of architectures and then your vision for that in the future and also how maybe people could contribute there if there is an opportunity?

• 31:05Speaker 02

  Yeah, I think right now one can just reach out to our sales team and we can figure it out. So based on the workload and the size of the model and things like that, we could, we could figure out what's the best path going forward. Now, going to the future, we have some very exciting developments, but I don't want to spoil that right now since it's still a work in progress. So I guess we'll disclose that whenever we can.

• 31:35Daniel Whitenack?

  And maybe kind of along with that, I know we have, you know, even my team, we've, we've tried out running models on a variety of kind of GPU alternatives. Sometimes what happens there is, you know, the latest model comes out on the market, right? And it's maybe, you know, supported in certain driver ecosystems very quickly. And then maybe on some of these alternates, there's a, there needs to be a kind of longer pathway for support in kind of custom software stacks that aren't, you know, are GPU based. How do you all navigate that right now? I know, you know, of course, our team is, is small and it's hard for us to navigate that.

• 32:19Daniel Whitenack?

  And maybe you have people thinking about those things every day. But yeah, how do you, how do you navigate that challenge as an engineering team to support all of these different models as they're, as they're coming out, given that you have a diff, a completely different software stack than, you know, others are working with in the ecosystem?

• 32:40Speaker 02

  Yeah, if you think about it, uh, we don't have to write kernels per model level, you know? So when a new model comes out, generally on the GPU world and even other custom accelerators, typically people spend a lot of time writing more optimal versions of it. So you might hear about uh new Coda kernels being launched. Let's say, you know, after the original attention, there was the flash attention one. So that's like more optimal way of uh running some of these models on the GPU. Uh, but we don't have to do this um at a per model level. What ends up happening is as we enhance our compiler over time,

• 33:23Speaker 02

  all these enhancements just reflect on to all of the models that we end up supporting. Uh, and the process to support uh different models on Grok is uh kind of similar. We end up spending of some time removing vendor specific hard codings, right? So there tends to be a lot of GPU specific code which we end up removing. And then we kind of run our compiler to translate this into, you know, finally to the Grok hardware. But there are a lot of knobs we tweak and turn to give you the best possible performance of, of that.

• 34:02Speaker 02

  And as the compiler improves with time, we just end up passing on these improvements uh to all the models right away. So our effort per model is, is not as high, you know?

• 34:14Daniel Whitenack?

  So just to clarify on that point, these models would kind of roll out, you would kind of build into the compiler kind of less vendor-specific thing, or you know, more general functionality over time, which would expand your ability to support certain types of operations. But you wouldn't necessarily be able to say, hey, I've got this random model. I created my, you know, some research team created their own architecture, right, of this crazy thing. It may take some effort to kind of map that into the Groq software stack. But maybe, if I'm hearing right, sort of less burden over time as the ecosystem develops.

• 34:58Daniel Whitenack?

  Is that the right way to interpret that?

• 35:01Speaker 02

  Partially, yes. But I would add that if you think about what the Groq system is at the heart of it, right? It's matrix multiplication and vector-matrix multiplications. And that's what most machine learning models are there, right? Yes, when we have a generational shift like transformers, one might want to go and look at what's the new model type and how well does it map to our hardware? We might want to have some strategies to address some of that, right? But fundamentally, models haven't changed all that much after the transformers have been introduced. Now, you know, you kind of hear about diffusion models, even in the text world most recently.

• 35:46Speaker 02

  But as long as these fundamentals don't change frequently, I think our core belief of just supporting this wide ecosystem of models continues to live sturdy. If you look at other AI accelerators, some of them have gone and hardcoded, let's say, to the transformer architecture itself. And their bet is that super-specialization is the way to go. But our belief is that we would like to support a more wider scale of models. And that's pretty much what our compiler system would do, to kind of map between this high-level, let's say, PyTorch model

• 36:31Speaker 02

  into the Groq platform, converting it to, let's say,

• 36:37Speaker 02

  An intermediate layer where the compiler could work independently of what model it is. So there's no hard coupling, let's say, to a particular model or to even an architecture type. It's very low coupling.

• 36:55Chris Benson?

  I'm curious, I've been really kind of spinning on the kind of the speed of you're talking about in terms of inference and some of the capabilities that your stack offers as in general as the model ecosystem has been developing into, you know, second half of last year and into this year, raging into this year, kind of agentic AI. And then that's kind of, you know, evolving into physical AI. And so you're dealing with robotics and autonomy and things like that that you're supporting to where we're expecting an explosion of devices out there in the world that these systems are supporting. What is your strategy forward and approach for thinking about kind of physical AI that

• 37:39Chris Benson?

  we're that we're evolving into where you have agents that are interacting with, you know, physical devices that are interacting with us in the real world. So it's not all in the data center, but the data center is supporting that. How does that fit in? How does that fit into your overall view forward?

• 37:57Speaker 02

  Yeah, I think the AI industry like revolves very rapidly. Like personally, I don't think there can be any long term strategy, but will not need adjustments based on developments. But our belief is still that I think edge based deployments, you know, and calling things over APIs will be the preferred interface going forward for a long time. Right? So sure, your, let's say, mobile chip might be able to perform some basic level tasks over there. But if you need like really high accuracy, high quality model inference, doing this over an API, I think would get you there.

• 38:42Speaker 02

  So compared to the, you know, like the model size, which you can actually deploy on a mobile phone. So just another example for like an edge device.

• 38:52Daniel Whitenack?

  I have one question for you, just as a, as an engineer that has been working on, at the kind of forefront of this inference technology. What has been some of the, some of the challenges that I guess you faced as you've really dug into these problems, maybe that were unexpected, or maybe they were expected for you. What would have been some of the, the biggest challenges? And maybe some learnings that, looking back on your time working on this system, you can share with the audience?

• 39:24Speaker 02

  Yeah, no, great question. As I said, I think the AI industry moves really fast. And sometimes there are these shifts, right? So we saw this shift to large language models. And that's when the company itself kind of pivoted to focus on this. So Meta releasing Llama and the Llama 2 series of models was really what got our company to focus on this side and really push on this, right? So similarly, I think we are, we are startup, we are always pushing on all fronts, always trying to improve on things. So whenever there's some new architectural

• 40:03Speaker 02

  change, we look to see how we could best adapt our system for that to get to maximize throughput, right? So sometimes there are these kind of changes and, you know, this is something which actually excites me about Groq and working at such a talent tense company. My, my colleagues really come up with really great, exciting new ways of doing things to really push the bar on some of these things. So maybe it's like, could be a mixture of experts or reasoning models, whenever something new comes up, right? I think that's getting the maximum

• 40:46Speaker 02

  performance out of that is something we care about, we deeply care about and. Yeah, I think that's been one of the key areas.

• 40:55Daniel Whitenack?

  Awesome. Well, as we, as we kind of get close to an endpoint here, this is, this has been fascinating. I'm wondering, DJ, if you could just close this out by sharing some of the things that you think about personally, kind of going into this next year. As you mentioned, things are moving so fast. There are shifts that are happening. What are some of the things that are most exciting for you as you kind of head into this, this next year of, of development and work?

• 41:25Speaker 02

  So as a developer and like amateur data scientist, I would say that for me, the push on the coding side of the AI world has been very exciting. It helps me kind of think about how can I have more impact, whether it's at Groq or in the world in general. So the push of AI on the coding side, reasoning models, multiple modalities, and the fusion of all of this, right? I think that's what I really want to look forward to for the next couple of years. There's of course the robotics bit which we touched upon, but that I feel is probably a couple of years down the line.

• 42:06Daniel Whitenack?

  Awesome. Well, thank you, DJ, for, for representing Grok and, and congratulations on what you and the team have achieved, which is, which is really amazing and monumental work. So great work. Keep it going. We'll be excited to follow the story and, and hope to get an update again on the podcast sometime soon. Thanks.

• 42:27Speaker 02

  Sounds great, guys. Thanks for having me.

• 42:37Speaker 00

  All right. That is our show for this week. If you haven't checked out our change log newsletter, head to changelog.com/news. There you'll find 29 reasons, yes, 29 reasons why you should subscribe. I'll tell you reason number 17, you might actually start looking forward to Mondays.

• 42:57Speaker 06

  Sounds like somebody's got a case of the Mondays.

• 43:00Speaker 00

  28 more reasons are waiting for you at changelog.com/news. Thanks again to our partners at fly.io, to Breakmaster Cylinder for the Beats, and to you for listening. That is all for now, but we'll talk to you again next time.

• 43:21Speaker 07

  Game on.