Going full bore with Graphcore!

Dave Lacey takes Daniel and Chris on a journey that connects the user interfaces that we already know - TensorFlow and PyTorch - with the layers that connect to the underlying hardware. Along the way, we learn about Poplar Graph Framework Software. If you are the type of practitioner who values ‘under the hood’ knowledge, then this is the episode for you. Sponsors: O'Reilly Media https://www.oreilly.com/changelog – Learn by doing — Python, data, AI, machine learning, Kubernetes, Docker, and more. Just open your browser and dive in. Learn more and keep your teams’ skills sharp at oreilly.com/changelog https://www.oreilly.com/changelogRudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Featuring: Dave Lacey – GitHub https://github.com/davelgraphcore, LinkedIn https://www.linkedin.com/in/david-lacey-743aa53, X https://x.com/davel_gcChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Graphcore https://www.graphcore.aiPoplar Graph Framework Software https://www.graphcore.ai/products/poplarUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This is one of the reasons that we're kind of quite keen on having a very kind of flexible programmable software stack because there are all these choices to make. It's a really great example you've made there of G N Ns because it shows that what we might be doing in two, three years. The time in this field is going to be quite different to what we were doing two, three years ago. And if you think like five years before that, then it wasn't your networks, right? So it wasn't deep learning. So I think key for us as much as having the right process and having these graph techniques, if we can call them as a family. The important thing is to have the capacity to do innovation.

• 0:37Speaker 01

  Big thanks to our partners, Leno, Fastly and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com slash change log. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at O'Reilly. Many even know O'Reilly for their animal textbooks and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos and all their conference talks. Plus you can learn by doing with live online training courses and virtual conferences, certification, practice exams, and interactive sandboxes and scenarios to practice coding alongside what you're learning.

• 1:18Speaker 01

  They cover a ton of technology topics, machine learning, A I, programming languages, DevOps, data science, cloud, containers, security and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, then check out O'Reilly's online learning platform. Learn more and keep your team's skills sharp at O'Reilly.com slash change log. Again, O'Reilly.com slash change log.

• 1:59Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:25Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International, and I'm joined as always by my co-host, Chris Benson, who is a Principal Emerging Technology Strategist at Lockheed Martin. How are you doing, Chris?

• 2:41Chris Benson?

  I'm doing good. It's a beautiful spring day, and we're about to talk AI with interesting people. You can't ask for much more than that, my friend.

• 2:49Daniel Whitenack?

  Spring has sprung.

• 2:50Chris Benson?

  Spring has sprung.

• 2:51Daniel Whitenack?

  Since you're in the South, someone, one of my coworkers this morning was talking to me about every 17 years, the cicadas, like cicada brood.

• 3:02Chris Benson?

  It goes crazy.

• 3:02Daniel Whitenack?

  Entomology is, wait, is that what it is? I forget the name. The one that starts with an E, that scientist. I'm not one of those people. The one that studies bugs. But are you aware of this?

• 3:12Chris Benson?

  I had heard that actually before, and it affects me in a fairly direct way. So longtime listeners, and certainly Daniel knows that I do a lot of wildlife stuff. And so on the side, separate from this whole AI thing, I do a lot of wildlife stuff. And one of the things I do is I am a snake wrangler, among other things, as part of that wildlife thing. So I am commonly picking up copperheads with the appropriate safe equipment, by the way, just so that no one gets the wrong thing. And copperheads love cicadas.

• 3:43Daniel Whitenack?

  That's a poisonous snake for those that aren't in the US. Chris is very adventurous.

• 3:47Chris Benson?

  If you're not in the southeastern United States, a copperhead is a venomous snake. And so I have a lot of experience with those. And so, totally off topic, of course, but the cicadas, yes, favorite meal of a copperhead.

• 4:00Daniel Whitenack?

  Oh, okay. It's their prime time.

• 4:02Chris Benson?

  That's right. Who knew we were going here at the beginning of the show?

• 4:04Daniel Whitenack?

  Yeah, yeah. Well, I mean, I'm sure that there's all sorts of interesting AI to like predict the cicada brood numbers this year. But that's not the topic of this.

• 4:16Chris Benson?

  That's not what we're talking about today. I'm so sorry, folks.

• 4:19Daniel Whitenack?

  Yeah, actually, I think the I'm pretty interested by by the topic and and guest this week because it seems like we talk a lot on the show or, you know, GPUs or accelerators or specialty cards for AI are mentioned. But a lot of times they're mentioned just in the context of the accelerators, not the sort of software component that goes along with them. And of course, I've found, and I don't know about you, but I found that a lot of times when I like, oh, I'm really have access to this really cool card or something and I want to use it, but the problem is not having access to it. It's the like software that I write for it has all of these issues. So

• 5:01Chris Benson?

  Yeah.

• 5:01Daniel Whitenack?

  I'm really excited to talk about that sort of software hardware interface today.

• 5:05Chris Benson?

  Real life AI right there.

• 5:06Daniel Whitenack?

  Yeah, exactly. Real life, practical.

• 5:09Chris Benson?

  Very practical.

• 5:10Daniel Whitenack?

  With Dave Lacey, who is Chief Software Architect at Raph. Welcome, Dave.

• 5:16Speaker 00

  Hi, pleased to be here. It's great. Great to see you, Dave.

• 5:18Daniel Whitenack?

  Yeah, and before we jump into things, do you want to give us just a brief sketch of your background and how you got into doing what you're doing now?

• 5:27Speaker 00

  Sure, yeah. So I'm a computer scientist and I particularly specialized in compilers and that kind of that area of computer science, by life saying. I did research in in that in academia. And then over the last so many years, I don't want to say really, but after that last quite a few years, I've been I've been working in various companies, working on the software stack to target new types of hardware in different areas. Most recently, we'll talk about today in Pathcore and AI, but previously in in HPC and in in meta processing and network processing, all kinds of things. So very much a software guy, trying to make sure that as we push the field forward and and get new hardware to enable new things, we we we have software that can

• 6:12Speaker 00

  support that and and let users actually program these new wonderful things that get created.

• 6:17Daniel Whitenack?

  Cool. Yeah, and you mentioned a few different things. Of course, there's AI specific hardware, there's high performance computing or HPC. Um. I think a lot of times when people think about um like AI hardware, maybe they're thinking about GPUs. But I mean, the story is a lot more diverse than that. Um. So maybe you could give us a little bit of a sketch of the sort of landscape of AI specific or hardware that targets AI workloads, I guess is a is a good way to put it. What are those categories of things out there?

• 6:49Speaker 00

  Sure, sure. So I think the ones everyone knows about are are CPUs, I mean, clearly. And they definitely have their place in this landscape for specific tasks. And then obviously, more recently, GPUs, um because of the nature, the kind of highly parallel compute and what they target, they've had quite a bit of success. I think you see quite a few companies now, and I think ours is GraphCore, where I'm from, and obviously I'm biased. It's is I I think one of the leading uh is probably the leading kind of new type of chip in there, where what we're trying to do is produce a chip that's specifically for machine learning and AI. So one thing's about other chips is you'll find that um they've they've been designed for something. CPUs, we know they do all kinds of things, GPUs originally graphics.

• 7:33Speaker 00

  And our chip is something where we've gone, well, can we design a processor that's just for this space? And this space is really important, right? You've got so much application that it's worth designing a processor specifically for it. I think that's the new kind of breed of processor you'll get, and I I think the IP is the one, obviously, that um that really kind of fits that slot, that allows us to do more in this space than other processors can. Because it's it because we thought about from the beginning, the the kind of attributes you need for an AI chip. So kind of what kind of data are we dealing with? What kind of data patterns does that kind of involve? What ramifications does that have for us? For example, on the memory hierarchy, our memory hierarchy is very different to other processes and so on. Just so we can kind of process data in a different way, all the compute units and so on.

• 8:19Speaker 00

  That's where we're finding we're finding a third place now, where we have specialized hardware, ML, AI, intelligent computing in general, and that's where the IPU sits for us in GraphCore.

• 8:30Chris Benson?

  So as we've talked to different folks through different episodes and everything is trying to understand kind of where you fit in and you're kind of laying out the landscape, kind of still at a high level, could you talk about a little bit about the different categories before we dive fully into GraphCore, just so that we kind of know in our heads where to position it, how that fits in, because people come into this conversation with different levels of understanding and knowledge on what that is. Could you kind of differentiate the different pieces of the landscape, then as we focus in on where you're at, we'll kind of have some context?

• 9:00Speaker 00

  Yeah, yeah. I think if I may, if I can take that landscape and talk about it from the types of processing that different processors do.

• 9:07Chris Benson?

  Sure, sure. That's fine. Yeah, that'd be great. Yeah.

• 9:09Speaker 00

  So like, kind of, a CPU does, is very much deals with scalar processing. So scalar processing is dealing with kind of individual numbers, maybe kind of small groups of numbers together, like small vectors and things like that. And they'll do it, that's very good for the general compute, right? Because actually, nearly anything can be split down into individual numbers, right?

• 9:30Chris Benson?

  Sure.

• 9:31Speaker 00

  So that's kind of the CPU kind of space. GPUs, what they deal with primarily is large vectors of numbers. I think that's very clear, right? So big contiguous blocks of numbers that they come in and to deal with them in parallel. Because they're dealing with them in parallel, they can get better efficiency, so they can go faster or work at the same power. I'd say what you could call the IPUs, say, I've got one or everyone, sorry, you could call them graph processors, in the fact that they're dealing with data that, where you have to deal with a lot in parallel, so they're highly parallel processors. But the connections between them mean they don't come in big contiguous blocks of vectors. There's actually kind of, you're going to have to take one number from over there, one

• 10:16Speaker 00

  number from over there, one number from over there, and then kind of deal with them in parallel. And that's kind of, in some ways, what you, a way you could characterize a graph processor, and it's one of the aspects to think of in categorization. Another categorization, if I may, is also about the type of number you're dealing with. So again, both CPUs and GPUs have a wide range of kind of number formats, if you like. They deal with integers and also fixed point numbers and floating point numbers with certain different precision. But in AI, I think we're finding a lot that what you need to deal with floating point numbers, so kind of fractional numbers to represent, you're kind of representing probability distributions most of the time, right? So you don't need a lot of precision.

• 10:59Speaker 00

  What you need is a lot of range, like if you think of probability distribution, that's kind of what it is. So again, naturally, you're dealing with numbers that are low precision floating point with kind of ranges you have to deal with and kind of randomness involved in there as well, injected in. So you can think about the shape of the data and the type of the data and which processor deals with what. That's kind of probably a reasonable way to describe that landscape across the chips. I expect there are other ways you could split it up as well, but I quite like those as a way of kind of splitting these things out. And it kind of leads you to think, oh, what kind of applications could these be good at, kind of by thinking about them in that.

• 11:38Chris Benson?

  Yeah, I see your point on that, absolutely.

• 11:40Daniel Whitenack?

  And maybe you could connect, so you talked about like graph nature of the IQ, the graph processor, and like taking a number from over there and how that works. Could you sort of map that on to the typical like AI types of tasks that we're wanting to perform and how that connects? People that, for example, are started using TensorFlow before TensorFlow two or maybe familiar with saving photographs or something like that or loading them, does that connect to what you're saying or what's the connection?

• 12:12Speaker 00

  Yeah, yeah, so there are several and there are kind of some different levels to unpick there. So one of the graphs, anything's a graph if you look at it hard enough, it's one of the things. But one of the graphs that people talk about in things like TensorFlow and so on is the compute graph, which is the way data flows around between big tensor operations. And that's certainly one of the kind of larger coarse grain things you can think about. But I guess I'm talking more about the graphs that connect the information together. So if you think about what are you doing in operations and quite often that matrix multiplies. Or their convolutions. Now, let's look at, say, convolutions for image processing. That's a really good example, right? Because there, what you have there is you don't have quite the all-to-all connectivity of a matrix multiply operation.

• 12:57Speaker 00

  Like, you have a kernel which is going to be applied the same thing to lots of different parts of the image that are kind of across the image axis. The one piece of data is being kind of scanned over the day. So you have a kind of a particular fanout graph there. And then you can get into the operations you can get into as you get into other neural networks. Say, if you look at things like ResNeXt or EfficientNet in image-based neural networks, you can see that actually you get kind of grouped convolutions or separable convolutions, again, where you're kind of splitting up the data and then recombining it in kind of finer and finer ways. So it's those kind of things that require hardware that can cope with efficiently moving that data around,

• 13:41Speaker 00

  and also a software stack that can target that efficiently as well. So we should get the software stack, that's kind of fine. I always like to remind people that you need the software to target it as well.

• 13:51Chris Benson?

  It takes them both.

• 13:52Speaker 00

  Yeah, yeah, yeah. And then if you get into even more kind of recent things that people are doing where you have, imagine doing a lot of what we do at the moment when people use TES4 is linear out of a matrix A times matrix B, and that gives you some result. Kind of takes some data coming in and we do a big weighted multiply between them. But really, that can be quite inefficient. We've got this trend to go really big, really big recent, which is great. Actually, it's pushing things forward with these really big, kind of you see in language models and so on. But, you know, it's not very efficient. And you'll see that the shape of that data means there's actually quite a lot of sparsity in there. There's a lot of that data could be zero. That leaves you with even more kind of complex data patterns. And I think kind of that push for efficiency

• 14:34Speaker 00

  is another space where you'll get kind of naturally a graph structure. Naturally, kind of graph structure is anything where you have kind of almost, in my mind, irregular connections between the data. So I think there are lots of different ways of looking at the graph within a kind of neural network or any kind of machine learning program. But I think the key ones for me is like, how are the data, how's the data connected together?

• 14:54Daniel Whitenack?

  Yeah, and I know that at some point, Chris and I were talking about a article about trends. I I forget which of the major AI conferences it was, but they're talking about trends year over year in terms of what people are focusing on. And I know that there was a big boost in people looking more and more at what's called a graph neural networks. Does that connect directly to what you're saying past the other things in in some way? Because I know a lot of people are talking about these things. And my understanding is it's, you know, in some ways, people say a graph neural network, they have just a way of encoding the graph into a tensor. And then you still have the tensor operations.

• 15:33Daniel Whitenack?

  But in other ways, there's actually graph native sort of operations that you can do within within the network itself. I'm I'm not an expert in these things yet. Have those connected with what your company is doing at all?

• 15:46Speaker 00

  Yeah, yeah, it's something that we're quite interested in. And and and actually, some of the research groups we work with kind of are interested in that that kind of network. It's yet another type of graph, but the graph is not the model itself, but the the the data that the model's consuming. What we found is that there's a lot of choice there. There's a lot of choice in how you represent the data structure. Uh, there are many ways to to do that. And then the actual kind of data processing operation you'll do will be different depending on that. So do you represent it as a as a list of edges or a dense matrix or a bit vector or things like this? Actually, I think what people really want is a way to explore through those choices at the moment, actually, um, to to kind of push things forward.

• 16:27Speaker 00

  Uh, this is one of the reasons that we're kind of quite keen on having a very kind of flexible programmable software stack because there are all these choices to make. I I think kind of it's it's a really great example you've made there of uh of G N Ns because it shows that what we might be doing in two, three years, the time in this field is going to be kind of quite different to what we were doing two, three years ago. And if you think like five years before that, then it wasn't your network, right? So it wasn't deep learning it. So I think kind of key for us as much as kind of having the right process and having kind of these graph techniques, if we can call them as a family, the important thing is to have the capacity to do innovation. And part of that for us, I think actually does come down to the software quite a bit, actually. It comes

• 17:12Speaker 00

  down to making sure that people can modify it in a good way, can extend it as well as having the kind of easy out of the box thing where they just, I just want to run a TensorFlow program or something like that. I think that the GNNs is a very good example of kind of where you want that kind of flexibility.

• 17:38Speaker 01

  This episode is brought to you by our friends at RudderStack and we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. RudderStack can be easily integrated into your existing development processes. And because they're open source, you can see all their code. So you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free up to 500,000 events and pricing

• 18:19Speaker 01

  scales transparently from there. Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 18:41Chris Benson?

  So my head is also still spinning a little bit on something that you were talking about a little while ago, and that is, you know, the software and the hardware. You need them both. Over the decades, the world has been very, very CPU focused, you know, up until this most recent time period where we're really digging into AI and, and all of these new hardware architectures have come into being. How do you approach, given the fact that we're having all this innovation and how these hardware architectures are coming out? How do you approach writing the software that goes for that hardware, that makes that hardware run? Especially when you think about the fact that we have a long history of being specific to different types of CPUs, but now the, you know, the sky's the limit on what's happening.

• 19:26Chris Benson?

  How does that change the act and the thought process and the planning for building software?

• 19:32Speaker 00

  I think it does a lot, actually. I think CPUs are amazing. I think that they've been so successful in fixing an architecture, fixing all aspects of the architecture, the way the instruction set runs, the way the memory hierarchy works, and then applying it to lots of different fields and having that success, I think is a remarkable thing. I don't know if it works for all fields, though. And I think what we find in AI is one thing that we've been very keen on right from the start, and I definitely see other people who'll talk about it, is this idea of code design. So code design is this general idea where you want to design several aspects of a system together. And so clearly, one of the big aspects of code design is design your software stack and your hardware architecture at the same time.

• 20:16Speaker 00

  And that's something we're very keen on, graphical, actually. We kind of, right from the beginning, we're very keen on making sure we worked on how the software was going to target the chip, make sure we kind of modeled everything right up to the kind of full neural network applications as we designed the chip and pushing that all through. I think later on in the company, what happens, there's a need to extend that software stack and kind of really kind of invest in kind of building that core to a really robust solution. So now we've got kind of more software people than hardware people. It's a really big part of what we do. And we're as much a software company as a hardware company. But even right at the beginning, it was very much, let's decide these together. I don't think it stops there, actually. I think actually you could also say you're code designing, and we'll see more and more of this. The machine learning algorithm and architecture itself, along with the software and the hardware,

• 21:01Speaker 00

  and the system. So the kind of way you, you know, we're not just, particularly for training, right? For large scale, large data, machine learning training. Which is the world we're often in at Graphcore. You're not talking about one process, so you're talking about hundreds of thousands of processes working together. And they all need to be connected via networking cables and put into racks and the power management and all this kind of stuff has to be put around it. And really, you need to co-design all of that together. And that's good. I heard someone kind of used the term algorithm before to describe this. And I think, yeah, why not? That's a fair enough term.

• 21:36Chris Benson?

  That's fascinating what you're saying. I'm not trying to cut you off. I actually want you to expand on that. The idea of doing software, hardware, the actual ML architecture and the system. That seems daunting because all of those individual topics are, people spend their whole careers doing that. How do you blend that in a cohesive way?

• 21:55Daniel Whitenack?

  Seems like your team makeup is very important for that.

• 21:59Chris Benson?

  I would imagine so.

• 22:01Speaker 00

  Absolutely. I think for us, like, yes, team makeup makeup's important in the company. We've really kind of stressed that. I think partnerships are very important for us. So, so we've always been quite keen on working with commercial kind of internet, consumer internet companies that want to do large scale deployments behind search engine or whatever there is, you know? So that, you know, the kind of thing, which we also want to work with research groups, both kind of university and company research groups to kind of see where that that's going. And we're not alone in that. A lot of companies will, you know, that's why I think there's been part of why that there's this keenness to bring kind of ML researchers into within companies to work on that. So I think our makeup and the communication between those is really important within company to do that.

• 22:45Speaker 00

  And partly it's just an awareness to want to do it. It's just having those, that communication, like freely flowing and actually to have the right kind of culture within a company to, to foster that your software is. Well, actually, no, we, we, we really feel that actually to get this album working, you could work it this way. There is a flip side, I have to say, in the, you've got to be careful not to design out the future, if you like, because you could interpret this idea as being, we'll, we'll harden a piece of software into a chip. So we'll take like, oh, I don't. ResNet 50 or something, to give a good example for you, so I'd say we'll make a chip that will do that really well. I don't think that's really co-designed because it doesn't,

• 23:28Speaker 00

  because you have to also design for generality, as I say, design for not just the algorithms of today, but the algorithms of three years' time or four years' time as well, because a chip has to last that long and people want that flexibility.

• 23:41Chris Benson?

  Is that a little bit of an artifact left over from an earlier time when you were still thinking about CPUs and there would be, things would start in software, you know, you'd solve a problem in software and it would stick and be maintained over the years and finally it would get incorporated into chips. Is that a legacy mindset that maybe has been brought forward but doesn't work when you're advancing on the ML architectures as rapidly as we are? I mean, if you bind it into the chip and you're stuck with that, whereas we're seeing, you know, rapid advancement over the last few years in terms of where things are going.

• 24:15Speaker 00

  Exactly that. I think that's exactly right. I mean, you just need to look at, as everyone who subscribes to like Archive, we'll see the amount, the fire hose of innovation here is just huge and you have to plan for that. And I think good flexible software is the key to that, but and then you have to design your hardware to kind of match that as well. So, so exactly. And I think that's hit the nail on the head. Yeah, we're in a real fast-moving space here. It's not stopped. There's still a lot of way to go before we kind of settle down on exactly what the algorithms are.

• 24:47Daniel Whitenack?

  As Chris knows, I'd love to get into the practicalities. And I guess what I'm thinking of is you have your graph core processor, the IPU over here on this side. And over here, you have already established Unity frameworks like TensorFlow, PyTorch over here on this side. And obviously people are, they want to use those, they want to use Keras or whatever. And I'm sort of looking at a diagram, which we'll link in our show notes, about your popular graph framework software, which is very well-written, but you've sort of got the frameworks on this side and you've got the processor on this side. Could you explain just for people like what it takes to connect?

• 25:28Daniel Whitenack?

  Something written in TensorFlow or PyTorch, like sort of people are used to to this new way of processing on the graph processor. What's in between there?

• 25:39Speaker 00

  Sure. Let's take TensorFlow as an example and I'll see if I can walk through that diagram from left to right. So kind of you have your TensorFlow program written in in Python usually. And what that describes is it will describe, usually it describes some model that you want to optimize and then you apply an optimizer to it. And that gives, that model really is, we're back to this kind of compute graph, right? So it's a, it's a series of linear algebra operations kind of connected together. And out of the end pops an answer which you can evaluate on how close you're getting your optimizer to, to learning a good solution to that. That graph gets explicitly represented in TensorFlow. So a data structure gets created, which is that graph.

• 26:20Speaker 00

  At that stage, it's, it's called the core graph in TensorFlow. So it's, it's pretty much what you've written, more or less. And then what happens is, well, the first thing that happens really is that that gets differentiated. So we kind of create the backwards path to say, how do we calculate gradients on that? And it gets wrapped in a kind of, that, that, that, that whole data structure kind of then represents a big loop outside that says feed data in and kind of update my weights as I, as I, as I calculate these gradients. We take that whole graph and then it gets passed first through the TensorFlow compiler flow. So, and that's not kind of a graphical thing. That's, that's the TensorFlow kind of upstream developers.

• 27:04Speaker 00

  They'll, they'll take that graph, they'll canonicalize it into kind of smaller operations. They'll do some optimizations on it and they'll convert it into what's called HLO graph, the XLA graph. So this goes through their compiler infrastructure, which is called the TensorFlow XLA. So the end of that, you have a kind of slightly lower level split up graph where we're still talking about kind of quite big linear algebra operations like matrix multiplies and things like that, but it's been kind of reduced down and tidied up and made a bit more kind of closer to the hardware. So at that point, our graphical TensorFlow backend takes over. And the first thing it will do is it will do a few more kind of optimizations on that data structure at that level. For example, in the chip, we

• 27:46Speaker 00

  have a hardware unit for doing exponentials and sigmois and the kind of things that come up in certain non-linearity. So it will recognize those patterns in that graph and and say, well, we replace them with one special operator that will kind of go down to the hardware and so on. So it'll do that kind of. And then it will basically convert those operations into an even lower level form of graph, which is much more fine grained than that. So we have something called PopLibs, which are libraries that implement things like matrix multiplies or non-linearity operations or things like that in Poplar. So let me talk about Poplar briefly, because I realize I just introduced that about signal that is.

• 28:25Chris Benson?

  Absolutely.

• 28:25Speaker 00

  So Poplar is, Poplar's our graph programming framework. So that, that is a way of representing graphs that run natively on our device that do these kind of operations. And in Poplar, we have graphs that kind of break it down to the individual processing unit. So on each of our chips, we have kind of about 1400 cores, processes on the car, each of which has kind of hardware threading in there. So you've got about 7,000 parallel computing units. And the Poplar graph kind of represents like the graph at that kind of level. And PopLibs is what kind of then says, well, I've got this matrix multiply to do. How do I split that over those parallel units in an efficient way? So that's where it does kind of partitioning and access splitting and stuff like that. Then we have the Poplar

• 29:08Speaker 00

  graph compiler, which then will take that fine, low level graph and create kind of actual code to the device, which then goes into the graph engine, which then kind of runs it. There are quite a few levels, you know, just quite a few compilers involved. And we have like, kind of, you know, so we counted them. There's like five or six different compilers that have to interact to get that kind of efficient implementation down on on that device. There are some other things that go on, like sometimes you might want to multi-chip models. So at a higher level, you'll do kind of, kind of model pipelining and things like that to to get a kind of efficient model spread over multiple chips and things like that. But fundamentally, that's the flow.

• 29:48Chris Benson?

  That was great. And by the way, I don't think I can recall anyone ever taking us through even a genericized, you know, not specific to your system, but that. So I appreciate that very much.

• 29:59Daniel Whitenack?

  That was super fascinating. And I think, as you mentioned, Dave, it's like, there's all of these layers. And of course, you're connecting, you know, to different frameworks that are out there. It's intriguing to me from like a software development standpoint, like with these frameworks, updating and new architectures and operations being added and all of that. I guess, one, how do you keep up that pace and test like that whole pipeline of things? Is it a matter of having like reference implementations of all of these different models? And essentially running like tests against the compilation of those upon new versions of your framework? Or how does that work?

• 30:40Speaker 00

  Yeah, exactly that. You know, there's a lot of investment. Well, one is the development that kind of know. A lot of software developers and men like a lot of investment in regression system and test infrastructure. That all has to be done because you have to have a kind of really robust, kind of comprehensive software product. That there's no way around that to be usable. And I think to kind of blow away and trumpet a bit here, that I mean, I think Graphcore amongst the kind of raft of AI startups. It's really well advanced in that space. All the elements you have robustness, like kind of documentation, you go to our site, you can see the tutorials and documentation, how you do that and so on. So I think we have to kind of try and keep on top of that. He working internally of course, to keep those questions going. I think the other thing I would say that kind of really helps with this is being very open.

• 31:25Speaker 00

  So those, so making sure that people can, we document and make sure that people can access. Well, this is something that you do get on other platforms, but I think we try and be real leading in this and in kind of being able to know what popular the low level graph compiler kind of explaining that, making sure that people can add custom kind of operators in that. Those libraries I talked about, they're open source on GitHub, that kind of thing. So trying to make our TensorFlow and PyTorch backends are open source and everything as well. So by having a more and more And infrastructure, it makes it easier for the community at large to help you adapt to new things as well. And as we find as we're kind of getting more and more popular, more people

• 32:08Speaker 00

  using us, that we kind of get more kind of community involvement like that as well. And I think that's an important part of it as well.

• 32:24Speaker 02

  We deserve a better internet and the brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default, quick access to the Tor network for true private browsing and an opt-in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

• 33:09Daniel Whitenack?

  So Dave, I'm curious, as an AI practitioner, one of the things I would love to know from you is as you've spent all of this time making AI programs sort of be sympathetic to a certain architecture and for certain tasks, do you have any sort of good advice or help for AI practitioners out there in terms of knowing how to tailor our AI programs or models more generally to be efficient for a certain data set or task? Are there any sort of good advice you have or common challenges that are common pitfalls that you see people falling into that could be mitigated with some best practices?

• 33:50Speaker 00

  Yeah, it's a really good question. It slightly depends on whether you're talking about kind of task performance, like how good your model is at a particular task, or whether you're talking about kind of compute efficiency, kind of, you know, how fast does it run? In terms of task performance, I think people need to be aware that those two are not kind of islands that are separate. The architecture, you're right, even whether you're Aware of it or not, even if you just like take what other people do or whatever, has been kind of affected by the underlying compute platform and what's efficient. Because if you were kind of trying to study whether a particular kind of model architecture, for example, got good task performance, but it was really, really slow,

• 34:34Speaker 00

  then you probably wouldn't like pursue that. Even though it might be quite a good way of going for if you're just looking at pure task performance independent, that's it goes. There've been references to calling this like the hardware loss, right? Kind of like what hardware you've got kind of affects kind of how lucky you are being able to explore certain things. I think it's a good question on practice there though. So I think one thing is that's useful is just a kind of base understanding of what's going on underneath. And probably the important things there are if you're interested in the kind of efficiency, the compute efficiency, knowing kind of how things like, for example, batch size or kind of certain sizes, axes of your matrices actually kind of affect the hardware underneath

• 35:19Speaker 00

  and could be faster slower. I think the other thing is being very aware of the floating point behaviour of various platforms, because that can vary a lot between platform and a good platform provider should document that and you should be aware of kind of the tools they have to show when when things are overflowing or kind of underflowing and so on to to know when you might be losing task performance, not because of the model structure, but because actually because of the data format they're getting there. And actually, we're seeing new techniques coming in to help that, actually. The kind of you'll see things like automatic loss scaling is really good example. But the software stack and event is trying to help there, you know, kind of trying to make things more adaptive, so you don't have to think about those things. But I think it kind of is is worth kind of just having a kind of good kind of surface

• 36:04Speaker 00

  level understanding of kind of what's going on under the board of it.

• 36:07Daniel Whitenack?

  Yeah, Chris, you're probably familiar with Bill Kennedy and the Go world, and maybe you are as well, Dave, but we spent some time working together and he he always in our conversations was talking about this idea of mechanical sympathy. As you're writing code, which I think gets to a lot of what you're saying, Dave, in terms of, yeah, it's maybe not all software engineers don't have to also be hardware engineers. But there is an element of like developing a mechanical sympathy for what you're writing for that helps you write like really robust and good software that I think is really valuable. Like understanding, you know, in Go, it's, you know, understanding like, oh, if I initialize

• 36:49Daniel Whitenack?

  a variable this way, it actually, you know, does something different than if I initialize a variable this way in terms of the memory that's allocated and copies that are made and all of those things. So having that sort of knowledge, I think, is really cool. And it's interesting to hear you talk about it in the context of AI. I think it's something that I definitely want to develop a little bit better intuition for in my own work, I think.

• 37:13Chris Benson?

  That segues into what I was gonna ask too. So we're both thinking along the same lines in terms of questions. And that is, as you've taken us into this and explained kind of that compiler series, and we've talked about kind of the interface being PyTorch or TensorFlow. And I know I'm coming at it as a practitioner and as is Daniel, you have presumably different types of users that need different amounts of that kind of mechanical sympathy, as Daniel described it from Bill Kennedy, or knowledge of what's under the hood at different levels. How far do I need to go? Should I learn Poplar? Am I going that far? Am I going beyond that? What other users, if the answer was no for me, what other users are learning that? How do you break that out? How do you know who should be addressing what?

• 37:57Chris Benson?

  Because that's a question we get all the time from new people in this field is there seems like so much stuff. I don't actually know what to address first and second, third. So could you break that down a little bit from that end user perspective?

• 38:10Speaker 00

  Yeah, it'd be unrealistic to say that like everyone needs to become a full stack developer for the full stack being this kind of, this incredibly complex.

• 38:20Chris Benson?

  All the way through, down through the hardware.

• 38:22Speaker 00

  Yeah, yeah, yeah. I think that's a really rare thing. I mean, you do get people like that. I've met people like that. They're great. And you want to hire them, basically, when you see them. But you're not going to get that. So I think what we find is it's harder for a sole practitioner. I think what we find in the companies we work for is they do that by teams, right? So you have very much an implementation focused team, an algorithmic ML focused team. And the challenge there is to make them work hand in hand, a bit like the code design thing I was talking about earlier, to work together and just being honest about the fact that you're not going to get this all in one person. I wouldn't suggest that people try and really do that unless they're super, super interested in all the parts.

• 39:07Speaker 00

  You always need to pick where you want to go and so on. But I think despite what I said about these lower parts of the stack, the majority of what I call ML scientists or data scientists or stuff like that don't need to know about those lower level details. They need to just have some sympathy with it. And I think that's it. So just maybe dabble a bit to understand a little bit about machines. Don't try and learn it really well. I mean, and definitely our user base is like that, right? You might say that maybe about 90% of people are programming at the PyTorch TensorFlow level, maybe a bit of understanding machine learning. And you've got a small group, but those groups are very important for us because what they can do

• 39:46Speaker 00

  is they can add new functionality, new capability to a stack or an application or a framework or something like that. Actually, maybe this is a case for people to specialize a bit. And if you want to be that person that can understand the details, then do that. But find good people to work with that understand the higher level and vice versa. If you really want to understand the machine learning part a bit, have some sympathy, but then maybe find some people to work either in the community or set up your professional team structure to make sure you've got people with those other skills as well. We have a mix at GraphCore. I'd say we have some people who are that full stack, trying to go up and down it. Obviously, we're in a bit different position because we're an implementation side of things. Right.

• 40:32Speaker 00

  And then we have people who specialize in one end of it.

• 40:36Daniel Whitenack?

  So Dave, I'm interested in, of course, the space is developing rapidly. And I know GraphCore and IPUs have been gaining traction. There's good use cases out there and we'll definitely link to some of the materials on your website and all of that. I'm kind of wondering, as you look to the future, this next year or two years with GraphCore, where do you see things going and what are you excited about seeing in terms of the development of this technology over the next year or two years as we look to the future?

• 41:07Speaker 00

  Gosh, it's very, very hard to pick. There's an awful lot going on, I think, kind of. I think the algorithm space will continue to move very quickly. So I'm very excited to see the fact that we might be doing very different things even two years from now to what we're doing. So people won't be talking about, wow, look at these kind of ginormous transformer NLP ones. Wow, look at this other thing we've done. So I think that's, that's very exciting. I think what's happening in, from a kind of more kind of systems point of view, I think kind of what's happening in the data center will be very interesting, the way, way they're put together, the way to get efficiency, the kind of the software and the, the processors and the, the network will all have to work together. It's a fascinating space that will evolve, I think, quite quickly in the next couple of years.

• 41:51Speaker 00

  From a software point of view, I'm interested in like how the current frameworks and stacks evolve as we try and go towards getting more efficiency. And I guess I'm talking about kind of parameter efficiency here, I suppose. So that's going to kind of give lead to new algorithms. And I think this is just a personal gut feel. I don't know. I don't know, because of all here. But it might be that we kind of are these kind of very linear algebra based frameworks that we have now, the ones that will be very popular, the tensors and Matias, are they the right ones? I don't, I don't think that's a kind of set thing. And that's very exciting, right? That there's still that kind of uncertainty about that kind of thing. There's, there's all kinds of stuff out there about how it's going to grow and scale.

• 42:35Speaker 00

  And that's not even talking about the actual applications that are going to come out of this space as well. So, so all the way through from top to bottom, there, there's loads of exciting stuff coming up and I'm absolutely sure of that.

• 42:46Daniel Whitenack?

  Yeah, awesome. Well, Dave, we really appreciate you joining us. This is super fascinating. I'm really excited by what GraphCore is doing. We'll make sure and link a bunch of links in our show notes for listeners. Definitely check out what GraphCore is doing, read their white paper and all the information about the popular software framework. It's really cool. And of course, the hardware. And yeah, I was just really enthused by the conversation and have a lot going on in my mind that I want to think about more. So I appreciate that. Thank you for joining us, Dave.

• 43:20Speaker 00

  Thank you very much. It's been a pleasure. It's been really good fun.

• 43:26Speaker 02

  Thank you for listening to Practical AI. We appreciate your time and your attention. Follow the show on Apple Podcasts, Spotify, or your favorite podcast app. Your neural networks will thank you. We are also on the web at practicalai.fm. There you'll find recommended episodes, listener favorites, and a free sign up to join the community. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors, Fastly, Linode, and Launch Darkly. That's our show. We hope you enjoyed it and we'll talk to you again next week.