Gaudi processors & Intel's AI portfolio

There is an increasing desire for and effort towards GPU alternatives for AI workloads and an ability to run GenAI models on CPUs. Ben and Greg from Intel join us in this episode to help us understand Intel’s strategy as it related to AI along with related projects, hardware, and developer communities. We dig into Intel’s Gaudi processors, open source collaborations with Hugging Face, and AI on CPU/Xeon processors. Sponsors: Intel Innovation 2024 https://intel.com/innovation?regcode=CMCCHL&utm_campaign=Changelog – Early bird registration is now open for Intel Innovation 2024 in San Jose, CA! Learn more https://intel.com/innovation?regcode=CMCCHL&utm_campaign=Changelog OR register https://reg.oneventseries.intel.com/flow/intel/innv2024/InnovationReg?regcode=CMCCHL&utm_campaign=ChangelogMotific https://www.motific.ai/ – Accelerate your GenAI adoption journey. Rapidly deliver trustworthy GenAI assistants. Learn more at motific.ai https://www.motific.ai/Featuring: Benjamin Consolvo – GitHub https://github.com/bconsolvo, LinkedIn https://www.linkedin.com/in/bconsolvo, X https://x.com/bpconsolvoGreg Serochi – LinkedIn https://www.linkedin.com/in/greg-serochi-a3126933Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Intel’s AI & Machine Learning Ecosystem Developer Resources https://www.intel.com/content/www/us/en/developer/ecosystem/ai-ml-overview.htmlIntel® Tiber™ Developer Cloud https://www.intel.com/content/www/us/en/developer/tools/devcloud/services.htmlIntel Gaudi AI Processors https://habana.ai/products/gaudi/Optimum Habana https://github.com/huggingface/optimum-habanaOverview of AI tools from Intel https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.htmlOpenVINO https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.htmlCase Study: Prediction Guard De-Risks LLM Applications at Scale https://www.intel.com/content/www/us/en/content-details/817317/case-study-prediction-guard-de-risks-llm-applications-at-scale.htmlHow Prediction Guard Delivers Trustworthy AI on Intel® Gaudi® 2 AI Accelerators https://www.intel.com/content/www/us/en/developer/videos/prediction-guard-delivers-trustworthy-ai-on-gaudi2.htmlUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:44Speaker 01

  What's up, friends? Intel Innovation 2024 is right around the corner. Accelerate the Future registration is now open, and it takes place September 24th and 25th in San Jose, California. This event is all about you, the developer, the community, and the critical role you play in tackling the toughest challenges across the industry. Ignite your passion for AI and beyond, grow your skills to maximize your impact, and network with your peers as they unleash the next wave of advancements in technology. Here's what you can expect. Understand the emerging innovation and trends in dev tools, languages, frameworks, and technologies in AI and beyond to empower you and the solutions you're building.

• 1:28Speaker 01

  Get in-depth technical experience doing hands-on workshops, labs, meetups, and hackathons to collaborate and solve problems in real time. You can explore featured partner and Intel solutions. They have partners there, startups there, customers there, and Intel is showcasing the latest in products, services, and solutions across keynotes, tech sessions, and the show floor to help you meet your development needs. Collaborate with experts, learn and have fun, engage in interactive sessions to connect, get certified, gain unique ideas and perspectives, build long-lasting networks, and of course, have fun and get inspired. Hear from leading industry experts, technologists, startups.

• 2:10Speaker 01

  Entrepreneurs and fellow developers along with Intel leadership CEO Pat Gelsinger and CTO Greg Lavender as they take you through the latest advancements in technology. Don't miss this chance to be at the forefront of innovation. Take advantage of early bird pricing right now until August 2. Register using the link in our show notes or to learn more, go to intel.com/innovation. Once more, that's intel.com/innovation or go to the show notes and click that link.

• 2:46Daniel Whitenack?

  Welcome to another episode of Practical AI. My name is Daniel Whitenack. I am founder and CEO at Prediction Guard. I'm joined as always by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris?

• 3:02Chris Benson?

  Doing great today, Daniel. How's it going?

• 3:04Daniel Whitenack?

  It's going well. I am really happy to bring a little bit of content to the show today that is from a little bit of the world that I've been operating in. And some of the groups that I've been collaborating with at Intel, which I think is a really kind of cool set of stuff that maybe people are maybe they're less aware of it, maybe they're aware of it, but really happy to have with us today Benjamin Consolvo, who is an AI engineering manager at Intel. And then also Greg Soraki, who is a developer ecosystem manager at Intel Gaudi. Welcome.

• 3:45Speaker 04

  Yeah, thank you.

• 3:46Daniel Whitenack?

  Thanks for having us. Yeah, well, like I say, I think, you know, people are of course aware of Intel and that, you know, Intel is sort of everywhere in one degree or another and in cloud providers and PCs and all of that. But I think some people might not be aware of kind of the strategic moves that Intel is making in the AI space and what they're focusing on. Ben, I'm wondering if you could give us a little bit of a high level sense of what Intel is doing as related to AI and how that's kind of featuring in there. Strategy right now.

• 4:21Speaker 04

  Yeah, no, I think a lot of people don't, you know, don't know Intel in terms of our, you know, our, our strategy with AI. And so I'm happy to, to talk through that a little bit. So, you know, both in terms of hardware and software, we're really aimed at, at AI, you know, with, in terms of the, the hardware front, which people are more, perhaps more familiar with. We have our, our Xeon product line, which is a, you know, a data center CPU that we, you know, use for inference in a lot of cases for, for AI workloads. And then more recently, we've, you know, announced the, the AI PC and kind of started that, that whole category, which includes,

• 5:03Speaker 04

  you know, the machine has a CPU, a GPU, and then what's called an NPU, a neural processing unit. And so, so that's exciting. That's kind of to optimize workloads on your local, your local machine. And then back to the data center side, we have the, the Gaudi product line, which is really, a really good performance substitute for a lot of the, the modern, you know, GPUs that are out there. So that's really exciting as well, really powerful data center hardware that we have. So that covers some of our, our hardware. I guess the, the other big one that I want to emphasize as well is we have Falcon Shores coming out in the

• 5:45Speaker 04

  future, which is an all purpose, you know, GPU, a data center GPU as well. So kind of leading into that is, is Gaudi and we'll, we'll get more into that in, in the episode. But on the hardware front, we have, you know, multiple products for, for AI. So that's, that's really exciting. And then on the, the software front, there's, you know, Intel again spans kind of the, the whole gamut of, of software for AI, for, for enabling workloads in AI. Rather than kind of going through the whole software stack that we have, I'll just talk about a couple of things that I'm excited about. So the PyTorch 2.4 release includes support for the Intel GPU.

• 6:30Speaker 04

  So right now is the Mac Series GPU and will be Falcon Shores. So that's, that's really exciting that the upstreamed mainstream version of PyTorch has now support for that. And then coming soon, PyTorch 2.5 will have support for the Arc GPU, which forms a part of the discrete GPU product line that we have. And that's also included in the AI PC. So those are a couple of exciting things with, with PyTorch that are happening. And then we also have the OPEA ecosystem. That's the open platform for enterprise AI.

• 7:09Speaker 04

  And it's a kind of this, this open framework that we have that multiple people can contribute to for Gen AI workloads, such as chat Q and A, co-pilot, like a code co-pilot and different other Gen AI examples. So yeah, those are a few of the things that I'm, I'm excited about on the software front.

• 7:29Chris Benson?

  Wondering if you could talk a little bit about how Intel is uniquely positioned in the AI landscape. You have both some very large competitors out there. There's a bunch of smaller ones in the space. How do you, how do you guys see yourselves and what's your, you know, how do you approach that, that point of differentiation?

• 7:47Speaker 04

  Yeah, so we, you know, I think where, where we can really compete well in this space is, you know, cost and performance like and availability. So when, you know, you're running out of availability or don't have the ability to rent that GPU you need from the cloud from Nvidia. And, you know, you, you need something cheaper that also, yeah, runs AI workloads really well. I can say with confidence, there's, there are some good options coming out of Intel that I've used personally. And I, you know, I come from. A background in AI engineering where I was only using Nvidia GPUs and I have gotten to work at Intel and to, you know, to try out our, our Gaudi products for training AI,

• 8:29Speaker 04

  you know, deep learning AI models. And then I've also gotten to run inference on our Xeon products and even run training as well on our Xeon data center products quite a bit to do more fine tuning. So yeah, I would, I would, I would position it, position Intel in terms of, in terms of those things, just being able to have a good, good options for, you know, out of the box performant hardware that people might not be aware of. Yeah.

• 8:59Chris Benson?

  Could you take just a moment, just a quick follow up to what you were saying? You, you brought up the, the big competitor being Nvidia out there and a lot of folks out there, just as you were, had been on their hardware, using their GPUs and such. What was, you talked about that transition. Could you talk a little bit about what is it like coming over to Intel when, when historically maybe you were on Nvidia as a platform and you guys are now coming, you know, as a real powerhouse? What's, what does that migration and transition look like and feel like if you decide to go for that?

• 9:33Speaker 04

  Yeah, I know. And I think I can relate in terms of my background in, you know, in deep learning as I was first working on with TensorFlow and then, and then with PyTorch mainly. And then now since I've been at Intel, there's been a lot on, you know, Hugging Face and Transformers and those libraries as well. And what I'd say in general is that the, the transition is not difficult. You know, a lot of the same tools that I'm used to using for development on the Nvidia platforms, I can use those same tools with some slight code modifications for, for the Intel products. So in terms of like a software leap, it's, it hasn't been too difficult. And Intel actually in

• 10:16Speaker 04

  the, historically has had, we have both the upstream support into those, you know, those frameworks where we have our own developers and the community developing for Intel in the, In the mainstream frameworks, but we've also in the past had, you know, Intel extensions where there are gaps. So for example, you know, Intel has had an Intel extension for PyTorch. Where, you know, there's not yet support in the mainstream framework. You can, you know, install this. This extra package to get, you know, all the support that you need again with just a couple lines of code change. But we were constantly aiming to get our, you know, our changes into the, you know, into the mainstream. Frameworks so that it's just easy for developers to use.

• 11:00Speaker 04

  But yeah, in terms of software development, I haven't had huge obstacles for, you know, transitioning over to different hardware.

• 11:08Daniel Whitenack?

  Yeah, and you mentioned kind of some of those, those important open ecosystem projects. Whether that be things that Intel is maybe driving more directly, like the O P E A stuff. Or it's kind of more community things. Greg, I know being kind of developer ecosystem manager for part, I know you're focused more on the Gaudi side. But we were talking before the show even about like our team is utilizing a lot of these great, great packages. That actually aren't even in, you know, an Intel repository on GitHub. There may be in hugging face repository like Optimum has been a big one that we've used.

• 11:51Daniel Whitenack?

  But there's other frameworks like T G I, I know that, that are important for what Intel's trying to do. So maybe we'll, of course, get into more details later. But just at that kind of open source level, for those out there that are not only wanting to utilize these, these great. Packages, but also contribute to them. How has Intel engaged in that open source community?

• 12:16Speaker 05

  Right. The key thing here is, is wanting to maintain as much of that connection with the open source. And this really started with, with Gaudi when the product was introduced. Four years ago where we started with TensorFlow and PyTorch. And now we've moved, you know, the industry has moved to PyTorch and so have we. And as Ben said, our, our goal here is to have full PyTorch support in native PyTorch. So we're working towards that. The same thing with, with DeepSpeed and with Megatron DeepSpeed for large language models. So we, we engaged with the full ecosystem to support those things. So we can talk specifically about our support for PyTorch. So a customer can run their PyTorch models and migrate them directly onto Gaudi.

• 12:58Speaker 05

  And so for instance, we have a tool that takes a model that maybe was running on a GPU architecture and in real time migrate some of those things and move some of that code that was for GPUs and change them over to things that, that Gaudi can understand. But the key thing is if you're running on PyTorch, you can bring your PyTorch models over to Gaudi. If you've been using Optimum, if you've been using Hugging Face, we've partnered very closely with Hugging Face and have a dedicated library called Optimum Havana or Optimum for Intel Gaudi. And that is a dedicated set of fully performant and fully documented examples of Llama 2, Llama 3, OPT, Mistral, Mixtral, all the important models that people are using today for both

• 13:42Speaker 05

  fine tuning and inference in Hugging Face. So if you're taking advantage of, of using Hugging Face, then it's really easy to bring those models over. And then we look at, again, for, for training, we have our partnership with, we're using DeepSpeed, specifically using Megatron DeepSpeed, which is really optimized for doing those large scale, large language model training where we're taking advantage of the tensor parallelism, pipeline parallelism and data parallelism that Megatron provides to be able to really get customers to scale and, and start using our product very quickly and very easily.

• 14:18Daniel Whitenack?

  I know. One of the things that was really cool for me and just, I know a lot of people have been working on this and contributing a lot, but I, I just love the. Because the reality for us when we were building Prediction Guard is we had a bunch of transformer-based code running. And one of the things I liked about the examples when I was trying out this stuff was you sort of had the example of like loading a model in with Optimum or with transformers. And then kind of the after, and it was just sort of like, like you'd see a git diff in a repo. It's like, hey, change this line to this line, and then you're basically pretty good to go.

• 14:58Daniel Whitenack?

  I remember I was actually on a plane to India during like a hackathon back last June and had got access to some of these Gaudi hardware, these Gaudi processors in Intel Developer Cloud. And I remember doing that and going through those examples. And by the time I had landed in Bangalore, I had the models up and running and on the Gaudi processors for what we needed to have, which was pretty cool. So yeah, great, great work to you all and the whole team in terms of providing some of that, that sort of functionality and tying in very closely with that ecosystem.

• 15:39Speaker 05

  Yeah, it's really important that, you know, because Hugging Face is so huge and so pervasive, we wanted to make sure it was really easy for people to migrate over and just even take advantage of the models that we have already optimized, you know? So a lot of the work we do there is, is really managing at the lowest level of managing some of the static shapes and managing the bucketing and making sure that we have the most optimized models. And as you said, Daniel, they're fully documented, right? So it's really easy for people to go into the repository on GitHub. And you will see examples of running something as simple as doing text generation with GPT on one card or going and running full, a full Llama 3, 70 billion parameter model on eight cards.

• 16:24Speaker 05

  Or if you have access to more nodes, up to 16 or 32 cards, and everything is fully documented. So like you said, you get off the plane. And you're already running and it's a great starting point for people to begin their development, either to take their existing model and fine tuning it with Gaudi and taking advantage of that performance, or being able to run inference and applying that to their applications, just like you've done with Prediction Guard.

• 16:52Chris Benson?

  So we've kind of dived into talking a bit about Gaudi, but I'd like to pull us back and for those of us that are out there listening and are not familiar with it, could you possibly kind of give us a what is Gaudi? And kind of introduce the whole platform in the broad and talk about kind of where it came from, how it came about, what Gaudi is versus maybe some of the other things that Intel has that are not specifically Gaudi and just kind of give us a context setting about what Gaudi looks like.

• 17:25Speaker 05

  Yeah, Chris, that's a great question. So let's talk a little bit about what Ben has mentioned a moment ago about the overall Intel product roadmap. And you look at sort of three key areas. You think about the PC and we have the AI PC that Ben Jin was talking about. Then we have the edge where we, you know, the edge has huge latency requirements and performance requirements. And there's great solutions there with Xeon to handle those low latency, you know, on premise requirements. And then the final is really that large language model training and inference in the data center where we're looking at fine tuning and pre-training models as well as running inference on large batch loads or batch sizes or large batches or dealing with

• 18:11Speaker 05

  users that running application where we have multiple, multiple, multiple users trying to take advantage. And the reason why we have Gaudi and the reason why Gaudi exists really is to give the ecosystem a choice. One thing we've been hearing from customers over and over is they want an alternative to the standard mainstream GPU solutions because of cost and because of availability. So Gaudi really is that low cost alternative to the standard Nvidia GPU solutions that are in the market today. So in a little bit of a history, the company Havana was an independent company. And Intel really saw the value in in the product they were building, the performance.

• 18:55Speaker 05

  So Intel made an initial investment in 2016, 2017, and then fully purchased the company in 2019. So it was a small company and they really needed to scale. So Intel invested in the company and brought them inside of Intel. And brought many Intel employees, I was one of them, into the structure within within the Gaudi team. So we started to build the product and started to ship. And so the first real milestone there was launching the first generation of Gaudi on AWS. So today that's available in the DL one instance on AWS, it's still available today. And the next step was in Gaudi two. And Gaudi two launched a year and a half ago.

• 19:36Speaker 05

  And one of the key, real key milestones with Gaudi two was the submission for ML perf on training and inference. And, you know, you look at, for those that don't understand, may not know, ML perf is is a benchmark that the larger ecosystem uses today to measure these really standardized benchmarks. And so the way that you run those ML perf benchmarks makes it easy to have a direct comparison from product to product. And one of the really key benchmarks was the large language model training benchmark for ML perf. And Gaudi was one of the only products other than Nvidia that submitted an actual benchmark for ML perf, showcasing showcasing the performance. So why Gaudi? What is Gaudi?

• 20:21Speaker 05

  Is Gaudi a GPU? Uh, no, Gaudi is not a GPU. It is a dedicated AI processor that is used to manage and train and run inference on today's largest and most complex workloads for both training and inference.

• 20:36Chris Benson?

  Could you differentiate a little bit for me as I'm learning as we go here? When you say a dedicated AI processor versus a GPU, could you kind of distinguish between those?

• 20:46Speaker 05

  Well, from a, you know, in some cases, a GPU still has the ability to do some other things. You know, it's it's got additional programmability. To do other types of workloads, whereas Gaudi as a dedicated AI accelerator is built specifically for AI training and inference. So we don't have those addition, it's specifically built for AI. So it's a product where if you're wanting to go run these workloads in the cloud, on the edge, or in an on-premise solution in your on-premise data center, Gaudi really is that low-cost solution.

• 21:19Chris Benson?

  As an analogy with a potential competitor that people may know just to transition, Google has their TPUs, which sound sort of similar to that where it doesn't have all the extra stuff that a GPU has. Is it, and I get Gaudi as its own thing, but is it just for people to make that connection a little bit similar to that?

• 21:37Speaker 05

  Yeah, somewhat similar. And you know, I have one more thing that really is a great differentiator in Gaudi is, is part of the hardware architecture, we provide two things. One is 96 gigabytes of onboard on-card HBM memory. And for those in the audience that really know about training and inference, having that local HBM memory is critical for storing your weights or your parameters, the things that actually are stored when you're actually running the inference or running the training. So having that large HBM memory allows you to do one of two things. You can either run a larger model on a single card, or you're more efficient as you scale out to more cards. And the other thing that really is a key differentiator

• 22:22Speaker 05

  for Gaudi is the onboard networking. So on DAI, Gaudi offers 24 100 gigabit Ethernet ports. So instead of relying on the latency of having to use a third-party network controller to scale to multiple nodes in a rack, for example, these dedicated Ethernet ports allow a, in some cases, a direct all-to-all connection for eight Gaudi's in a node or a single server. And then the additional Ethernet ports then can go to the other nodes in a rack. And then scale out to a switch to go to multiple nodes and then a full pod.

• 23:03Speaker 05

  So what you end up with is a virtual alt alt connection, which significantly improves the scalability when looking at large workloads.

• 23:12Daniel Whitenack?

  Ben, you were talking a little bit about kind of your prior work in deep learning, in training models and doing inference and, you know, how you've transitioned a bit and also done some, some projects with, with training on, on Gaudi and various other kind of hardware that, that we've already talked about. Could you speak a little bit more to that kind of practical side? And just to give people a sense of the kinds of projects that are possible on this hardware, you know, just so they can form in their mind, kind of both the, the scale and possibilities with what can be done.

• 23:47Speaker 04

  Yeah, I know. And, and, you know, my, my use cases, you know, might not be the, the full expanse of, of what's possible on our hardware, but I can certainly speak to the things I've been able to, to work on and had some fun with over the last few years. So one of them is working with OpenAI's Whisper model, which is a, a translation and transcription model that's, it works very, very well out of the box. And so I speak both English and French. And so I tested this, you know, the abilities of this model, you know, to transcribe my own voice, English to English, and then also its ability to transcribe from English to French and then going the other way as well. And yeah, first of all, the model does, does a really great job.

• 24:32Speaker 04

  I was really impressed with the fidelity of the, the model that's been trained, that, that OpenAI has released in the open source. So that's great. And then what I was able to do was to, to run this model, you know, on our Zion product line to, to run inference on, on Zion and run that really, really quickly and kind of put together a notebook and a workshop around that. So that's, that's been something that's been fun for me to, to work on, on this, this generative AI model that is, you know, used for translation and transcription. Yeah, that's, that's one. Another one that- That has been exciting for me to work on is with my, I have a background in imaging and computer vision.

• 25:15Speaker 04

  Kind of prior to to working at Intel, where I was applying a number of different techniques like pixel segmentation and image classification, object detection to different problems in computer vision. Especially around geophysical imaging where we're imaging the subsurface. I kind of describe it like an ultrasound for for the earth. So if you know what an ultrasound is, looking at imaging, you know, the the subsurface of the earth to try to find, extract and find certain minerals and that kind of thing. So where I've applied some of the techniques there is to help, you know, get some of those images more more quickly. And both in fine tuning models and and also in inference.

• 26:00Speaker 04

  And so I've been able to use, you know, some of our Zeon again, product line to fine tune some of those models as well as run inference. So that's, that's been exciting. Yeah.

• 26:13Speaker 05

  You know, I'll add this too, just to add on to that. One area that that's really getting a lot of focus now is is the use of RAG, Retrieval Augmented Generation. We've invested a lot of effort there and you'll see that in the OPEA project where we have a lot of RAG based examples. But not only for just general RAG usage, which is important, but also transitioning into multimedia. So we're receiving a huge request for, and we see this in the market, for both video, audio and text all coming together. So whether it's a prompt of of text to get a picture or a prompt for text to get now a video.

• 26:53Speaker 05

  Or to use a RAG type of usage to be able to parse through not only text, but parsing also through video.

• 27:03Speaker 05

  And then get a response. So those are all of the so types of things we have available for use on Gaudi and in the products.

• 27:11Speaker 04

  Yeah, actually, that's a something I'll pick up on too, because yeah, I've been working on some multimodal, you know, models on the AI PC actually running, running some multimodal smaller, you know, two billion parameter multimodal models and actually successfully ran one on the NPU, the neural processing unit to to run inference and to extract essentially some information about an image into text. So using that multimodal capability.

• 27:56Speaker 01

  Hey friends, Outshift, Cisco's incubation engine merges innovation with the art of possible, a launchpad for transformative emerging tech. Outshift blends startup agility with corporate strength to develop next gen technologies from the ground up in AI, quantum technologies, cloud native and more. Their newest AI innovation, Motific, addresses a critical challenge in the rapidly advancing world of gen AI, bridging the gap between concept and deployment. This model and vendor agnostic solution supports the entire gen AI journey. From assessment and experimentation, Motific accelerates deployment from months to days

• 28:37Speaker 01

  while safeguarding against gen AI, security, trust, compliance and cost risks, all while empowering business function and IT teams to rapidly configure end user assistance powered by organizational data. Motific provides advanced customizable policy controls to prevent unauthorized access to sensitive data and helps ensure compliance throughout the entire process. With deep visibility into operational and business metrics, Motific enables you to track ROI, optimize costs and make informed decisions by offering a

• 29:13Speaker 01

  centralized view, Motific deters shadow AI usage and empowers teams to innovate responsibly. So move beyond the traditional constraints of AI implementation, utilizing AI deployment that is both responsible and is revolutionary, ensuring your projects are not just quickly launched, but built on a foundation of trust and efficiency. Visit motific.ai, that is M-O-T-I-F-I-C dot A-I.

• 29:56Daniel Whitenack?

  So Ben and Greg, we've talked about a lot of interesting things, both on the kind of software and migration side, but also on the hardware and what that hardware enables. People might though be out there and be wondering, this is cool, I'd love to experiment with this stuff. How can I get hands on with some of this hardware that I'm hearing about? What are some of those ways that people can? They might have an Intel processor in their PC, or likely many do, but they don't have a Gaudi or a Xeon sitting around at the moment. So if they were to want to kind of explore this ecosystem and get hands on, try some things, how might they do that?

• 30:39Speaker 04

  Yeah, I could start and then Greg, you can you can fill in anything Greg that I'm missing. But I think the best way is to get onto the Intel Tiber Developer Cloud, which is a way, it's kind of our Intel Cloud. There are developers can come and try out our, both our hardware and our software that's set up right there for them. And we'll offer kind of our latest, whatever we have on that platform. So we'll offer, you know, we have our Xeon product, we have our Gaudi product. We have, we even are going to be having like a dev kit kind of for the AI PC, a simulated environment, even though it's not a, you know, a local machine, it's still a cloud. So that's, I think, the best way to get started is the Intel Tiber Developer Cloud. And Greg, did you have anything to add there?

• 31:22Speaker 05

  That's the best, that is the best way. You know, we're, we're going to have some, pretty soon, some, some free access to Gaudi. So right now you need a developer credit. We're working to get some nodes available for free on the developer cloud. So people will have the ability to try out our tutorials and examples and code examples and be able to see and experience the Gaudi usage and see how easy it is to run.

• 31:46Daniel Whitenack?

  Yeah, that's awesome. And I mentioned earlier, some of the experimentation on my end, even on the plane, but a lot of that was enabled by this developer cloud. And I think there are, you all can correct me if I'm wrong, but people can sign up on the site, get access to, and there's also some like training resources. People can spin up notebooks, try a variety of things, maybe if they're not as familiar or they're learning, get access to various things. But there's also a kind of transition to, within that environment, utilize these powerful products in a production sense or in an enterprise sense, rather than just a kind

• 32:27Daniel Whitenack?

  of developer experimentation sense, which is definitely the transition that Prediction Guard has taken. So we've been able to operate very price performant at scale with our LLM and GenAI platform on top of Gaudi and running that in Intel Tiber Cloud. So I don't know if there's anything you'd want to highlight, whether that be kind of success stories or just commenting on that kind of transition to production, that there kind of are people running this stuff in production, not just in a kind of developer experimentation sense. Anything you'd want to highlight there?

• 33:05Speaker 05

  Yeah, Daniel, that's a great point. You know, the Tiber Developer Cloud is really meant to do two things. One is just to give people the access to our products. So they have the ability to experience and run them and test them. But Daniel, to your point, it is also a place for people running a business to be able to have really easy access to our products as well. And specifically with Gaudi, you know, we're enabling customers now with very large scale out to be able to do full production workloads and use the Tiber Developer Cloud as a baseline for their business.

• 33:43Speaker 05

  We invite those, you know, listening to be able to reach out to your sales contacts and Intel and really be able to talk about how we can help those people really using this for business to be able to scale as a real product. And also, as we look to, you know, Gaudi 3, which is our new product that we've announced at previous events. And we're going to make a very large announcement at Intel Innovation in September. Is really, that's also going to be a place where we'll see Gaudi 3 also begin to scale. So it's, it's definitely a place where it, it makes it easy to partner directly with, with Intel or with, with partners in the future.

• 34:22Chris Benson?

  Yeah, I'm curious on whether I, whether it is existing or maybe a roadmap item. Could you talk a little bit about Gaudi at the edge? And when you're not in the cloud and you're out, uh, maybe wanting to use Gaudi in devices that are out there or platforms that are out moving about, what's the roadmap look like on that?

• 34:43Speaker 05

  Right. So going forward, we're going to, so today it is a, you know, OCP compliant part. So it's, it's on a mezzanine card. So it's meant for data center, right? So today, the Gaudi platform is in a, you know, six U or eight U rack mount server. That's the form factor it has today. And that OCP form factor of spec is eight Gaudi's on a single baseboard. And as you noticed, if you may have noticed at Intel Vision, a few months ago, we announced, uh, full packages that you can buy. So you can buy a Gaudi two baseboard or a Gaudi three baseboard that has the full baseboard that's OCP compliant. So you can drop that into a chassis from Super Micro or WeWin or other products.

• 35:25Speaker 05

  But in the future, we're also going to have a standalone PCIe card that'll be available. And that's going to be exactly for those type of more on-prem sensitive solutions. Chris, like you said, at the edge, where people can take advantage of a single Gaudi 3 and its capability on the edge. So you'll see that PCIe card coming soon.

• 35:44Chris Benson?

  Yeah. And maybe that's a good transition to talk a little bit. Like we've talked a lot about the things that are kind of the now of what's available and kind of tooling or hardware wise with.

• 35:57Daniel Whitenack?

  With Intel, but both of you have alluded to kind of the future to one degree or another. So maybe, maybe Ben, I'll start with you, but I know you mentioned certain things, whether it be that Falcon Shores or some cool things that are happening with AI PCs and that sort of thing. But what kind of strategy wise and kind of positioning wise is Intel really thinking about and investing in moving kind of into, into the next phase of what AI is, is becoming? And where, where Intel thinks, you know, the market is, is going, I guess?

• 36:30Speaker 04

  Yeah, no, thanks. Probably the, the best place to start on, on this is, and what I'm excited about is, is Falcon Shores, like you pointed out. So Falcon Shores will be kind of the, the culmination of combining the, the Gaudi product line with, with the GPU, with our, with our current Max series GPU. And it will be a, you know, a GPU graphics processing unit. So it will be a full, full GPU capable of not only the AI workloads, but also, you know, graphics and, and other applications that people want to use GPUs for. And so I think that's, that's probably the most exciting thing that we're kind of aiming at, that we know the market, you know, we, we know the market needs.

• 37:14Speaker 04

  And then, yeah, iterations on the, as you mentioned, that the AI PC. So we'll be, in the future, we'll also have a new, more powerful AI PC with the Lunar Lake chip coming out. And again, it will, you know, include the CPU, GPU, NPU, but it'll be a, you know, much more powerful, more memory form factor that, where developers will get even more out of their local, you know, their local machines. So, yeah, those are, those are a couple of things. And then. The other thing that I'm excited about, just as an AI software developer myself, is the integrations with, like I pointed out at the beginning, the integrations with PyTorch.

• 37:58Speaker 04

  I think it's huge that we're aiming at getting all of our optimizations and everything we can into this framework that is by far one of the most popular deep learning frameworks and one that I use on a regular basis. So that's really exciting to me as well, that just natively I'll be able to work with PyTorch and say, "Hey, I want to use the XPU," which is for the Intel GPU. And same with the AI PC, just have that direct integration. So yeah, those are a couple of things that are exciting for me coming out soon.

• 38:36Speaker 05

  And I'll add to that to say, the key thing here is forward compatibility, right? So people that are using our products today, so I'll specifically speak to Gaudi. If you're running workloads on Gaudi 2, you'll be able to run those workloads directly on Gaudi 3, and that same architecture will move forward into Falcon Shores. So people that make their technological investments now, those will remain viable and relevant far into the future.

• 39:03Daniel Whitenack?

  Yeah, and I guess one other piece of this, which Greg, you mentioned kind of in passing, one of the things that people are really interested in with Gaudi, of course, is the fact that there's some diversity in the market and there's another choice, right, for hardware out there. I think one of the other interesting things that, I don't know if, I know you two are kind of only in pieces of Intel and focused on certain things, but I found it really interesting how Intel is very much investing in chip production kind of diversity as well outside in various geographies around the world as a key part of their business.

• 39:47Daniel Whitenack?

  And I don't know if you have any comment on that or thoughts on how that influences the market as a whole and availability or supply chain sorts of robustness that that could build in. But I know we've seen some interesting things over the years, both in terms of availability and supply chain issues with hardware.

• 40:09Speaker 05

  Yeah, and you could talk about, let's talk about this at a macro level and maybe at an AI level, right? At a macro level, you look at what our CEO, Pat Gelsinger, has talked about when we promoted the CHIPS Act, for example, that we need, Daniel, to your point, we need that, we need to be able to build that infrastructure worldwide. So you can see from an Intel perspective, our investments in our fab in Ohio and our fabs in now in Germany, as well as with our fab worldwide, we have that worldwide capability to support significant growth and expansion. As the world continues to need more and more silicon. From an AI perspective, you know, the growth and the need for AI compute is insatiable and will continue to be that way

• 40:51Speaker 05

  for the foreseeable future. So, again, this goes back to, you know, why we've invested in Gaudi and brought that as a product as part of Intel and as part of just growing the broader AI portfolio. We really want to be able to give the ecosystem an alternative to getting access to AI compute as they need it today.

• 41:11Chris Benson?

  So as we start closing up, I would, I would really love to hear kind of from each of you, and you guys can decide who wants to go first, but kind of where, where do you see it going? Where Gaudi is going and where the overall ecosystem and these technologies are going? And just, you know, kind of, this is a moment where you can kind of take a little bit of poetic license and speculate a bit. Would love to see what you think will unfold and happen in the times ahead and how each of you may see it a little bit differently as individuals.

• 41:47Speaker 04

  Sure, yeah, I can, yeah, I can start. So I think, yeah, just looking at kind of AI broadly and, you know, what's happening, it seems like things progress with these incremental, you know, these incremental changes. And sometimes there's a leap, but there's, there's often just these incremental changes. And one of the questions I often get from friends, maybe you guys do too, as you're working in AI is like, you know, is AI going to take over? And, you know, are we going to have kind of robots controlling everything we do? You know, so that's that seems to come up a lot as I as I say that I work in AI. Sometimes I regret saying I work in AI and just say I should just say software engineering.

• 42:28Speaker 04

  But but but no, it's it's it's always an interesting conversation that I have with with different friends. And, you know, my perspective is is that like the Internet, the the coming of the Internet. AI has come along and has changed the way we work and changed the way we operate. But just like the Internet, we have people behind building these things. And as you know, these technologies evolve, you know, we we will have safeguards and we will have things in place to to help, you know, regulate, you know, the the different technologies of AI that come out. And, you know, lately with the AI agents, I think is one of the, you know, the most recent things where

• 43:09Speaker 04

  you have the agent kind of do more things for you than than maybe previously where you had to ask it to do more things. So that's been a really interesting part of AI that's that's kind of come out and and that I think is going to see a lot more adoption in the future to, you know, to answer your question about the future, I think, just getting the AI tools to to build more things and to be able to kind of do more complex tasks in sequence is something that's that's evolving and happening. Yeah, Greg, did you have some more to add?

• 43:44Speaker 05

  There's I I'd love I I'm so excited about the the personal personalization of AI. You know, I see cases where now things we didn't have when we went to college. But now, you know, you can have your phone or your AI PC open in your college classroom. And, you know, the AI will summarize and and create notes for a lecture or quiz you on a lecture and create all that content for you automatically. I love that. I want to see AI do better with my email. And be able to organize my email better, do searches better, make my life better. Those are things I'm really excited to see from a general perspective. You know, from an Intel perspective, I think you're going to continue to see us lean in on

• 44:26Speaker 05

  giving customers what they need, which is being able to have more and more compute for for fine tuning and for for inference, be either on prem or on the edge, supporting the world's largest models. And as we see, you know, innovation happening on a monthly basis that used to take a year, now we're on a monthly basis. We will keep up with that innovation. Just as an example, Meta just launched their Lama three 400 billion parameter model in the market last week. So, you know, we already been running that model and we've support that model and we wrote a blog on it a couple of days ago. So we're going to continue to to support the most bleeding edge, latest and greatest technology that's coming out again on a monthly basis.

• 45:11Daniel Whitenack?

  Thank you both for taking time out of a lot of things going on in a fast moving ecosystem to come and chat with us. And I would definitely recommend to all the listeners to check out some of the show notes and the links and go try some things hands on and have some fun and start building. Thank you both Greg and Ben. I appreciate you taking time.

• 45:31Speaker 04

  Yeah, thank you. Thank you, Daniel and Chris.

• 45:33Speaker 05

  Thank you.

• 45:41Speaker 00

  All right. That is Practical AI for this week. Subscribe now if you haven't already. Head to practicalai.fm for all the ways and join our free Slack team where you can hang out with Daniel, Chris, and the entire change log community. Sign up today at practicalai.fm/community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.