Advent of GenAI Hackathon recap

Recently, Intel’s Liftoff program for startups and Prediction Guard hosted the first ever “Advent of GenAI” hackathon. 2,000 people from all around the world participated in Generate AI related challenges over 7 days. In this episode, we discuss the hackathon, some of the creative solutions, the idea behind it, and more. Sponsors: Read Write Own https://readwriteown.com – Read, Write, Own: Building the Next Era of the Internet—a new book from entrepreneur and investor Chris Dixon—explores one possible solution to the internet’s authenticity problem: Blockchains. From AI that tracks its source material to generative programs that compensate—rather than cannibalize—creators. It’s a call to action for a more open, transparent, and democratic internet. One that opens the black box of AI, tracks the origins we see online, and much more. Order your copy of Read, Write, Own today at readwriteown.com https://readwriteown.com/?utm_source=changelog&utm_medium=practicalai&utm_campaign=changelogChangelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Fly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Rahul Nair – GitHub https://github.com/rahulunair, LinkedIn https://www.linkedin.com/in/rahulunair, X https://x.com/unrahu1Ryan Metz – LinkedIn https://www.linkedin.com/in/ryanaemetz, X https://x.com/RyanAEMetzEugenie Wirz – LinkedIn https://www.linkedin.com/in/eugeniewirzRalph de Wargny – LinkedIn https://www.linkedin.com/in/ralphdwDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Advent of GenAI Hackathon https://adventofgenai.com/Intel’s Liftoff program for startups https://www.intel.com/content/www/us/en/developer/tools/oneapi/liftoff.htmlPrediction Guard https://www.predictionguard.com/Blog posts: Recap of Day 1 https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Advent-of-GenAI-Hackathon-Recap-of-Challenge-1/post/1552069Recap of Day 2 https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Advent-of-GenAI-Hackathon-Recap-of-Challenge-2/post/1552264Recap of Day 3 https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Advent-of-GenAI-Hackathon-Recap-of-Challenge-3/post/1553059Recap of Day 4 https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Advent-of-GenAI-Hackathon-Recap-of-Challenge-4/post/1555115Recap of Day 5 https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Advent-of-GenAI-Hackathon-Recap-of-Challenge-5/post/1556530Final Challenge https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Advent-of-GenAI-Hackathon-Recap-of-the-Final-Challenge-Custom/post/1556584Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:38Speaker 01

  If you're listening, you know software is built from thousands of small technical choices, and some of these seemingly inconsequential choices can have a profound impact on the economics of internet services, who gets to participate in them, build them, and profit from them. This is especially true for artificial intelligence, where the decisions we make today can determine who can have access to world-changing technologies and who can decide their future. Read, Write, Own: Building the Next Era of the Internet is a new book from startup investor Chris Dixon that explores the decisions that took us from open networks governed by communities to massive social networks run by internet giants. This book, Read, Write, Own, is a call to action for building a new era of the internet

• 1:22Speaker 01

  that puts people in charge. From AI projects that compensate creators for their work to protocols that fund open source contributions, this is our chance to build the internet we want, not the one we inherited. Order your copy of Read, Write, Own today, or go to readwriteown.com to learn more.

• 1:56Daniel Whitenack?

  Welcome to a very special fireside chat, which corresponds with the ongoing advent of Gen AI Hackathon. That will also be reposted on the Practical AI Podcast. I'm very pleased to have been participating in this Hackathon as one of the organizers. But I'm also joined here in the fireside chat by an amazing team from Intel's Liftoff program for startups, who helped organize this Hackathon that we'll be talking about throughout the day. So I'd like to kick it over maybe to Rahul

• 2:39Daniel Whitenack?

  to describe a little bit about what Intel Liftoff is.

• 2:44Speaker 03

  Hey, hey all. Thank you, Dan. This has been an incredible experience. Let me before that even talk about the advent of Gen AI, two sentences. This was probably the biggest generative AI Hackathon that we, our team has organized. And the submissions, all the different chats and things that we have seen. This has been really awesome. They got a lot of positive feedback and a lot of things that we need to improve for the next time. So thank you for participating for the Hackathon. And we would be announcing the winners of the final product development in a couple of hours. Before that, let me talk about Liftoff, right? So Liftoff is an accelerator program, specifically a technical accelerator program for early-stage startups.

• 3:28Speaker 03

  So if you have an idea, you're a seed startup or till Series P, you want to scale, you want to build some cool things in AI or machine learning, please join the program. It's free. I would categorize the benefits basically into three different pillars. World-class technical support and technical expertise. That's me and Ryan here. We lead the engineering side of things. And we have an incredible engineering scale team. Some of the folks are here like Chyotia and Varad. Then you get access to technology, both Intel Software and Intel Developer Cloud. Intel Developer Cloud is a prediction-ready cloud specifically designed for AI workloads. Prediction Guard is one of the startups that came to our program earlier this year,

• 4:13Speaker 03

  and they are running on Intel Developer Cloud now using a Gaudi 2 accelerator. And I'm sure that many of you folks who have participated in the hack app would have used prediction gods L L M A P I's. The third one is co-marketing and bringing your, once you've built that product and deployed it, the next thing is to make some money, right? So we co-market your startup, your idea through all of Intel's channels and also we have a network of accelerators and network of folks that's beyond Intel, that we take the product you build, the company you build, and basically market it all over the world. We also connect you with our sales teams to see if there's a potential for selling the things that we've built through Intel's channels.

• 4:58Speaker 03

  It could be a service on IDC, or it could be a separate that one of our customers of Intel is looking for. I would urge anyone who is looking to really bootstrap and accelerate your startup journey to join Intel lift-off.

• 5:12Daniel Whitenack?

  Yeah, that's great. That's great, Rahul. Could you describe a little bit? So I remember initial discussions between a few of us. You had this idea for the advent of Gen AI Hackathon. Now, a lot of people in the audience out there might be familiar with like advent of code. So where, like, how did you start thinking about this advent of Gen AI Hackathon? And like, what was your initial vision for it?

• 5:42Speaker 03

  A lot of the geeks in the call, which most of us are, would know advent of code. It's a set of programming challenges. You can take any sort of program language you want to learn or to attempt to solve these algorithmic questions. I've been doing that for many years and I thought, you know, Gen AI is something that's new and something a lot of people are talking about, but there are no fun set of exercises or that you can use to learn and also build cool things with the technology that's existing today. So I was thinking, why not just create something, you know, set of challenges? Uh, that's tailored from a person who might not know how to code, but have seen prompt engineering and creating cool prompts to create images,

• 6:27Speaker 03

  all the way to uh people building with uh L L M A P I's. So we designed a set of challenges to bring in as much as audiences possible to introduce them to Gen AI and also learn through that process. Many folks I've seen in in the chats where um they came with just prompt engineering knowledge and have built really cool things uh graduating from one challenge to the other. Uh, and there has been a really good community help also, like people talking to each other and trying to help out uh how to run, how to build these things. This has been a really, really good exercise, even some of the challenges when I was building it, I was like, oh, whoa, I I would like to do that because it's uh we always wanted to add a fun element to it.

• 7:07Speaker 03

  So uh no challenge is is dry, it's just an algorithmic question, but there is something fun element to that. That's where the whole advent of Gen AI came about and we want to do this yearly. Uh, so we next year it might be a new technology, it could be a multimodality uh hackathon. Some other things we already saw, uh some cool engineering stuff, folks are bit this challenge, but uh it would be something, but I'd like to do in December, add window something else that uh lift up us for the community.

• 7:36Daniel Whitenack?

  Yeah, that's awesome. So if you're listening to this uh after the holiday season or as this will be coming out on the podcast later, you didn't miss out on particip- well, you missed out on participating in twenty twenty three, but there's going to be more opportunities to participate in things like this that Intel is going to put on later on. So I definitely recommend people keep an eye out on the lift off program and social media uh to hear about things. Could you speak a little bit to the response to this first advent of Gen AI and like the participation that happened? Um, I think we were all a bit surprised at how many people joined in to this.

• 8:18Daniel Whitenack?

  So could you speak to that a little bit and anyone else from the lift off program, kind of? Any of your observations for um the type of people that joined in and the range of experiences and and all of that?

• 8:32Speaker 03

  This has been fantastic. We had grad students, we had even students who are in school who have taken prompt engineering courses, who just wanted to have fun working on some of the earlier challenges. There has been experts also, some of the experts in LLMs and GenAIs, I worked on it. And some of the products that they've built, right, or some of the challenge solutions is it's like an MVP startup build. But we have many startups who are building a similar solution, who are taking like six months to a year to build a full, solid solution. But some of the challenge answers, especially using the RAG example then, or the Python code explainer, things like that are difficult. And they have even gone further, where when we asked about how to, can you create a story teller chatbot?

• 9:18Speaker 03

  People have created a story plus image chatbot, a multimodality chatbot. So these are many levels of experience, and that's even some of the folks from Intel participated. That's also a very positive thing where we see you can, you know, in a level playing field, work with folks from Intel and solve the challenges together. We had folks from Berkeley, folks from many different enterprises participating. So this has been a mix, and I was amazed at the level of participation. I didn't expect this many people would participate, and we had to stop registrations after 2,000 people registering for the event, and this has been just great. Ryan, do you wanna add something?

• 10:01Speaker 04

  Yeah, I just remember this was Rahul's idea when he first called me. It's like cleaning up after Thanksgiving in my basement or something. He's like, you know, I have in the code? Yeah, of course, of course. I want to do ad in a GenAI. Like, all right, what do you want to do? We worked it out, and it was like, set the goals, like, let's get like, it's like, yeah, we can get big, like, you know, a couple hundred people at least would be a success. But like, stretch it, like, hey, I mean, we could get maybe a thousand even and do the biggest event of the year. Well, yeah, I'm gonna end up cutting it off at twice that, twice the stretch goal. So then, of course, the entire time, we're like, oh, man, did we do a good job? Are people gonna like this? Are we gonna get submissions? Every time, knock it out of the park.

• 10:43Speaker 04

  Way more submissions than we were hoping for and the quality was excellent. And the amount of people we saw in the chat helping each other where it's like somebody, you know, we're all trying. It's 24 hours a day, so we're, you know, our team is trying to stay in and answer questions as much as possible. We set that as a goal, but what we saw like from go, like from when it started was when people would ask questions, other people would jump in and like link them to the explanation or the documentation or whatever. These were all the dreams for the event, so it's been incredible and I want to thank every single person who was involved.

• 11:13Daniel Whitenack?

  Yeah, and maybe it would be good for those, maybe some people jumped into certain challenges and not other challenges, or they might be hopping in at one point or the other, or they're learning about advent of Gen AI as they're listening to this. So what were some of the challenges that were presented to the participants? And how would you consider them in terms of like relative challenge level or skill required to complete them?

• 11:42Speaker 03

  We designed this challenge in a progressing level of, I wouldn't say difficulty, but I mean the ability to code. I would say it's, it's not difficulty exactly and creativity also because a lot of Gen AI is, what we see at least in Lift Off is that AI has become truly commoditized. And you don't really need to know, go through masters or a couple of courses on neural networks to build an application right now. We have the amazing transformer ecosystem. It's really easy to integrate some of these things, the AI applications to, AI superpowers to the applications you build. I would take the first challenge, for example, the first challenge was to create a narrative based set of images.

• 12:24Speaker 03

  And if you look at the challenge the first time, it's, it looks like it's very easy. You just create a couple of images using stable diffusion. It's all about prompt engineering. You don't need to know a single line of code. All the notebooks and the models, everything is available on Intel Developer Cloud. You just create a standard account, log in there, get Jupiter Hub open and just play with it. But the thing is, creating a transition from an image to another and creating a whole story with set up by images. It's not easy, it's really difficult. And we even saw some folks creating a comic book generator using this challenge. That sort of imagination, right? That's what I really wanted people to do, but I didn't want to

• 13:03Speaker 03

  say that, okay, please create a comic book generator as a challenge because it's really difficult. Some of the folks, even without knowing that, built it. You take the final challenge, right? That was Python code explainer, where you give in a Python code, use L L M model to understand the code and give an explanation of it. We had additional challenge, additional sub-challenge there, show the source of documentation or stack overflow questions where I can go into and learn about it more. These kind of additions makes very interesting and a little bit more complicated where you have to use a vector database, you have to use prediction gods L L M A P I's to get the right model. And you had to design a UI for it, all in the constraints of a Jupiter notebook.

• 13:46Speaker 03

  So I would say that there was a progression of difficulty. Difficulty is a very relative word, but yeah, there's a progression of difficulty if you're just coming to Gen AI. And the whole idea is that in a single package, so even if you're not participating in event of Gen AI, right? We have released all the resources that we have built for this. You can take this to basically get your, an idea of what Gen AI can actually help you to build and infuse to your applications. And you can just go through the different challenges. Or if you're an expert in L L M rag-based application, go to that particular challenge and take a look at it. So it's now become a learning resource also. I'll just a hack them.

• 14:24Speaker 04

  If I could just add on to that real quick, it did on purpose go up in difficulty. Like the level of each challenge was supposed to get harder and more advanced, let's say, in terms of coding ability as we went on. But the real focus was about skills and understanding the tools that are being used. Within the industry, there's been a huge focus on creating new levels of abstraction to make neural networks easier to use and build with. It used to be very challenging. Now, not, I mean, not nearly as much. You don't even need to stand up your own neural network anymore, right? You can grab an API. So if you look at the challenges, each one is kind of focused on a different skill. And if you go through all five, you cover prompting specifically for images,

• 15:09Speaker 04

  text image, cover using an L M A P I and the different things you can do with it. Cover image to image, so image editing with A I, then the third one. And then rag based applications with L M A P I's. And finally, we thought, you know, the fifth was the most advanced, like the codex twenty one. That's, you know, there are companies that are basically that are big companies that are working on that exact problem that are betting that, you know, it's going to a lot, that a good solution for code explanation improvement and generation is going to like many billions of dollars of value. So all these things are focused on these different skills. And our hope was that for people who are maybe software engineers looking to move over to A I

• 15:51Speaker 04

  or to students, like whatever, anybody who is interested in learning A I skills, that by go through the ones they chose to or all of them, by the end of them, they'd have kind of a portfolio of knowledge in their head about the different skills, both on understanding how to use them and how to do a good job like with prompting, but also by going through the code and understanding all of the code and all of the notebooks, an idea of what else they could build outside of the narrow set of applications that we asked for over the course of five days.

• 16:19Daniel Whitenack?

  I think one thing that impressed me about the set of challenges that you all came up with was that it focused really around image generation, coherent image generation, and on the L M side, sort of retrieval based methods along with chat. So I think all of these things are the things that people are finding most utility out of when they're first implementing A I solutions within their actual enterprise or industry or startup or whatever environment they're working in. In particular, retrieval based methods and rag systems, at least for our clients, we're seeing that's like the first thing that everybody is building, right? You have your own company set of data.

• 17:04Daniel Whitenack?

  In the case of the challenges that you all put together, maybe that's external Python documentation for the code explainer, or maybe that's Just some external PDFs or YouTube videos or whatever it is for the RAG-based solution. But lots of companies have this data that has this sort of unlocked potential and is unlocked via these retrieval-based methods, which is a lot of times what people are building first when they adopt this technology. So I think it was great that you all tied that together with the participants to give them practical skills in that area and kind of help them learn, you know, what is a vector database? What is a RAG system? How do you implement

• 17:45Daniel Whitenack?

  this with custom data? Rather than kind of immediately hopping to fine-tune a model, which I know, of course, you can do more easily than ever as well. But there's a lot you can do even just by integrating your own data with retrieval or other sorts of methods. I do want to ask here in a second, some of the solutions that you saw and what stood out to you, just to highlight some of those really cool things that we saw. But before we do that, so you can have that in the back of your mind and think through some things you'd want to highlight. I'm wondering if you all could speak to the Intel Developer Cloud specifically, which is something, of course, I've found utility out of. But it was something that was kind of unique about this hackathon, and it

• 18:29Daniel Whitenack?

  might be something that the participants here are a little bit like this was their first time using it. But also, there's a whole lot available there in terms of different ways to run AI models that are maybe, maybe some people are less familiar with. So could you describe a little bit the Intel Developer Cloud and maybe also highlight some of those like different unique ways that people are running AI models outside of just like throwing it on a GPU? There are actually some interesting kind of other either tooling or hardware, software available for people. Could you highlight a little bit of that and the unique ways that people were specifically running AI models throughout the hackathon?

• 19:12Speaker 03

  Sure, um so Intel Developer Cloud is Intel's production-ready cloud specifically for AI and machine learning workloads. And of course, when we say AI and machine learning, it's matrix math, so many other compute-heavy workloads can run really well on IDC. So uh for this particular hack, um we provided um anyone logging into Intel Developer Cloud, registering on IDC as a standard or free tier user. You get a shared uh Jupiter Hub instance uh where you get access to Intel's data center GPUs, uh Intel's Zeon processors. And I would say this uh system, I uh for a free tier user, I don't think any other service provide. I mean, there are many services with a Jupiter Hub front-end.

• 19:56Speaker 03

  Uh but the amount of compute and amount of memory and and RAM and even file storage that you get in the systems. I haven't seen a single cloud service provider uh providing that. And we have seen a lot of people really using it and giving us feedback on how we could even improve it. Today on IDC, we have a lot of models or LLMs already. There are tens of even hundreds of local models that we are planning to add further to boost this. There are stable diffusion models, LLM models and things like that. Beyond that, for productionizing the workload, right? Uh then, uh in your case, uh you are using the Gaudi two accelerators. Those are specifically designed for workloads that that request high bandwidth uh like LLM and GNI workloads.

• 20:40Speaker 03

  And I like this, this, this, uh I lost my train of thought, but yeah. We have Gaudi two accelerators, which we are seeing incredibly competitive and sometimes outclassing the best uh out there uh for your particular workloads. Along with Gaudi accelerators, um uh those are specifically designed for GNI and AR workloads. We have general-purpose GPUs, the data center Max series GPUs, both with forty-eight gigs and uh one-twenty-eight gig uh versions. Um so the folks in the hackathon, they actually used uh both our fourth generation Zeon, that is the latest Zeon uh that we have. Which what it particularly does is that it accelerates your machine learning workloads.

• 21:22Speaker 03

  We have dedicated instructions in the CPU to sometimes even take your workload to two X the performance that you got in an early generation. Uh it's all about making the CPUs, how we see it, is that uh making it as efficient as possible and making it as fast as possible. Just still maintaining the general-purpose utility of a CPU.

• 21:42Speaker 03

  Then, like I said, the data center Maxi is GPU is a little bit more generic solution. Where you can run your AI workload, HPC workloads. Um, each of these machines, when you're uh productionizing, you get a VM, you get an eight node. Uh eight card system, um there are also clustered systems available. Then comes the Gaudi accelerators, both there are single node machines and uh also clustered machines. If you want to do pre-training or or big fine tuning, all those cool things. And soon we'll have Kubernetes, service, uh, object store, file store, all those things uh coming up. Uh, so it's gonna be great. What I see is that if you're building a startup, it would be very difficult to find a performant accelerator cloud.

• 22:24Speaker 03

  Uh, like IDC out there, I'm sure that there are different, different uh hyperscalers. But this uniquely for startups, from my personal experiences, is a really, really awesome solution. Uh, I'd like to more know more from you, Dan, like you, you are one of the first customers of of IDC, right? Like, what are the things that you thought uh that really uh, you know, made you decide to choose to side DC? From the performance and also the team side also, right?

• 22:48Daniel Whitenack?

  Appreciate that and appreciate the support that you all have given. I think it's interesting, maybe for people out there that are less familiar with the various options for model deployment. To understand that there is really good tooling. Um, like you say, whether it's optimizing a model and deploying it on a CPU. Or an edge environment or or um just a cheaper inference solution. Or it's like all the way to these Gaudi two processors that we've been experimenting with. I think there's a lot of interesting and approachable tooling for that. So I first came across some of the tooling around Gaudi two by actually seeing blogs on the Hugging Face blog about Gaudi two.

• 23:33Daniel Whitenack?

  And I think at the time, like the Bloom model, which is a very large model. And running it on either a single accelerator or spread across eight accelerators with really high throughput on the inference side. And doing that with tools like Optimum Habana. So for those of you that are out there and want. To explore things, actually, if you look up the Hugging Face Optimum library, there's a lot of great tooling that you can play around with there, even not for Gaudi, but for other processors too. So whether that be CPUs, GPUs, the Gaudi 2, HPUs, the Data Center Max GPUs. Optimum kind of provides you a way, if some of you can visualize,

• 24:16Daniel Whitenack?

  maybe you're writing in your code and you're importing a model from Hugging Face. It's just like Auto Tokenizer or Auto Causal LM or whatever it is. With Optimum, a lot of the times, either you can just do like a one, a couple line replacement and just replace that with the Optimum version of those classes, or do some wrapping of the various models with optimizers. And this allows you to run, run your, your model very fast on a wide range of architectures. So I think, to your point, Rahul, I think one of the things that we've found really useful

• 24:55Daniel Whitenack?

  is the actual, the ease of use in coming in and saying, okay, well, we have this stuff running on a GPU, let's try it on this various other architectures. I remember even like maybe two or three years ago, trying to do some of this model optimization things for edge deployments is very, very challenging. So like, a lot of times I would try to optimize a model at the time, working on like speech models and other things, and it just wouldn't work because operations wouldn't be supported or something like that. But this tooling, which is cool because you know, Intel's working directly with Hugging Face on this tooling. And of course, the ease of use has just been

• 25:37Daniel Whitenack?

  ramped up drastically and, and we've been applying that with really good results, particularly for inference for LLMs. So that's been a key feature to that, that change happen.

• 25:48Speaker 03

  That's really awesome to hear, right? And particularly the, what also the thing you mentioned, right? So So I think of Intel in two ways, uh probably the biggest semiconductor manufacturer, the coolest chips. Uh the other is, uh Intel is an open source software company also. We are uh contributing to almost all, all big open source projects, uh Linux kernel. Most all things, if you see, we would, we would be anywhere in the top three, PyTorch, TensorFlow, uh Hugging Face. Any, any sort of um open source solutions out there. We work really hard uh across to make sure that your adoption of a technology is as easy as possible. And try to upstream as much as possible to uh the core PyTorch library or TensorFlow library and, and things like that.

• 26:33Speaker 03

  In cases where we uh, we feel that there are further optimizations that could be done. And these things can now be upstreamed in a couple of months to the mainline repositories. We release extensions also. So for example, uh if you take the out of box PyTorch and, and run it on a CPU. You already get a lot of performance because of um Intel's new accelerator library, one D N N, that's powering. A lot of these operations uh when you're running on a uh machine, uh like an Intel Xeon, for example. But if you wanna go a little bit further, right, we, we have things like Intel extensions for PyTorch. That uh with one line of code, um uh essentially it's uh Intel extensions of PyTorch as iPEx and iPEx dot optimize. And passing the model. We add further optimizations uh to run it as fast as possible.

• 27:18Speaker 03

  We are also working on even upstreaming whatever possible uh to PyTorch mainline. So uh that thing you mentioned, right? It's very important to work with the community and enable the software that the community uses. Um rather than having a completely different architecture, um and something that's sometimes is closed source. And working on it, um that's not the way Intel thinks. Even the whole concept of one API, um heterogeneous programming. Everything open about it where other vendors can come in and add their accelerators to the one API. And use a one API standard. Where um if you're writing code for a CPU. Uh there should be minimal to no changes that's required to run that on another accelerator.

• 28:00Speaker 03

  Uh that's the philosophy that we are, we are working with uh overall in the one API architecture that works underneath all these acceleration libraries. Uh Optimo Habana, it's, yeah, we've been working very closely with the Hugging Face team. Almost all L L M models uh work out of the box, um. There are models that we have tested and benchmarked that's available on GitHub and things like BLLM, right, uh, or Info Support. All those things are enabled uh through Intel libraries. For example, BigDL and things like that, giving a higher level abstraction beyond PyTorch. Because when we talk to startups these days, we feel that PyTorch is considered as a low level library right now and that's a little bit funny for folks uh who have worked in

• 28:40Speaker 03

  Tiano or uh, you know, even beyond that uh in in 2016 and 17 uh and coming from the early days of TensorFlow to see PyTorch can being going low level uh and this is higher abstraction libraries uh to work on top of that. It's really an exciting time to be in and work with you all.

• 28:58Daniel Whitenack?

  Yeah, for sure. And I also want to highlight, in addition to like open source code, it's been cool to see Intel recently uh release uh Neural Chat, which was a fine tune on Mistral model, um which is openly accessible on hugging face and permissively licensed. So we've been, we've been experimenting with that and we saw usage of that in the Hackathon. So it's cool to see people like, uh, like a couple of these models, Neural Chat, which is a fine tune of Mistral, came out like, I don't know, maybe a week before the hack. And uh Notice, um, which is another fine tune on Mistral, came out like a few days before the hack and both of those were being used in, in the Hackathon, which I think demonstrates

• 29:42Daniel Whitenack?

  the ability to kind of rapidly adopt this new stuff that's coming out.

• 30:02Speaker 00

  This is a changelog news break. Vana.ai is a Python RAG framework for accurate text-to-SQL generation. It lets you chat with any relational database by accurately generating SQL queries trained via RAG, which stands for Retrieval Augmented Generation, to use with any LLM that you want.

• 30:26Speaker 00

  You load up your data definitions, your documentation, and any raw SQL queries you have laying around into Vana, and then you're off to the races. Vana boasts high accuracy on complex data sets, excellent security and privacy, because your database contents are never sent to the LLM or a vector DB. It boasts the ability to self-learn by choosing to auto-train on successful queries, and a choose-your-own front-end approach with front-ends provided for Jupiter Notebook, Streamlit, Flask, and Slack. You just heard one of our five top stories from Monday's Changelog News. Subscribe to the podcast to get all of the week's top stories,

• 31:10Speaker 00

  and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 31:41Daniel Whitenack?

  Well, I do want to make sure that we have time to highlight a couple of cool things. So what were a couple of the highlights for you all in terms of solutions that you saw, or methodologies that you saw, or just like cool things you didn't expect? What stands out in your mind?

• 31:58Speaker 03

  I'll start quickly and I'll let Ryan talk about this. So we both have been spending, I mean, Dan, you're also there, like every day going through these submissions. And like, it was very difficult to figure out who is the best submission, because each time we think that this is the best and look at the other one, we're like, oh my God, like, this is incredible. Even the first submission, right, the quality of image creation, that was, I was surprised that how can you even create this kind of image, just that the models, I will say. Uh the time that was spent in prompt engineering and even using custom models uh to combine these sort of images and create these solutions.

• 32:37Speaker 03

  Nothing um uh a few, there are a few really interesting rag examples, taking YouTube videos, parsing the audio. Figuring out a YouTube search, that was something that stood out to me and for the Python code explainer, there was a submission that came maybe. Uh I think in three hours, uh the first uh iteration uh of the of the submission for the by this person were. There was uh you could do uh that that solution can do Python explanation, but also give. Uh references to where exactly that the model got this information from a really, really good use of rag and and L L M's. Um Ryan, what are the things that that stood out for you on submissions?

• 33:17Speaker 04

  What always stands out is when somebody, the the Jupiter notebooks which Rahul had put together. I think are really well designed for as learning activities and to like get something done that's cool just by going through them. Um and so people use those to do a lot of amazing work and I was stunned by the quality. But what always stands out to me is when somebody like takes the concept, takes what's in there and then runs with it and like we saw. People setting up A I agents for some of these challenges who like the comic book generator. You know, in the code explainer, in the the fifth challenge of where where code explanation comes in. It's like, oh, explainability, like, listen, we're gonna do things like, we're gonna do the explanation, the the model will then cite the sources it's using.

• 34:02Speaker 04

  Right? Um which made me think, you know, we used to in years gone by. People were very concerned with explainable A I and what they always meant is like, well, if the model is making a recommendation. Or classifying something in such a way, we should be able to figure out exactly why. And so there are all these like discussions of how best to do that, like, oh, you can use shop values, you know, or whatever. And I think what it turns out is like, well, now that we have Gen AI, you know, and we have retrieval based methods, it's like, just ask. What are your, so, okay, this is your explanation, like, where did that come from? And, you know, we, we see like the setting of sources. So that creativity, not just in application, which was astounding, but then also

• 34:41Speaker 04

  in people bringing in methods, cutting edge methods from outside of like what we even included in the notebooks, that always blew me away. And there were some people that just always ended up in the top five. Like Tomas Barshi, I don't know if I'm pronouncing that correctly, but I actually reached out to you because it was like, what do you do for a living? Like, this is, your work is incredible. Um, because, uh, there are so many, Simon's team.

• 35:06Speaker 03

  Pranav.

• 35:06Speaker 04

  Um, Pranav, yeah. I haven't sat down to compile a list. Uh, you know who you are, because you can go back each, to each winner's post that Rahul made and find those names. And that's something that we'll follow up probably with a blog about and, you know, uh, maybe reaching out to somebody to be on a podcast or to talk to us or whatever. I would also like to highlight that, uh, our youngest participant, I think, uh, might be on the, yeah, I see his name. Arien, who is a middle school student who owned us every day. Like at around the time when we were supposed to be posting a video or whatever, with the same like skeleton, like waiting, tapping his fingers. You know, like patiently waiting for this video that was supposed to be here five minutes ago. That was a wonderful part of that for me.

• 35:50Speaker 03

  Yeah, even the, the thing, right, like you were mentioning the, the, uh, Python explaining, uh, I mean, there were, there were suspicions were, okay, now you have explained the code. Now click this button to optimize the code. I'll give you an optimized version of the solution. Taking the challenge in spirit and not just in words and going beyond that, like incredible work. Uh, it's truly, I really feel generative AI and the commoditization way I have really, really helped a lot more folks. Who might not have been here to do this AI kind of work, really democratizing the solution, all the toolings, the API-based approach, for example, from Prediction Guard,

• 36:30Speaker 03

  the hugging physical system, making it as easy to use it. And one thing, and when Dan was mentioning that people were using neural chat for LLM APIs, that was because of Dan's incredible team adding these models and scaling it in like in matter of hours. So it's still a challenge to deploy and scale this. You have an incredible team, Dan, there, who was also participating in the conversation.

• 36:57Daniel Whitenack?

  Thanks. Well, I definitely think so. Appreciate that. And speaking of where people can find out more about some of the specific submissions, even seeing some screenshots, some code that people generated. Eugenie, do you want to comment? You've created some amazing blog posts already, and I think there's more in the works. So do you want to just describe to those listening where they can find out more about some of the solutions? And maybe also where they can keep tabs on future events and things coming through the Liftoff team?

• 37:32Speaker 05

  Thank you, Dave. Daniel, so we posted already three blog articles at our landing page, developer.intel.com/liftoff. And I just want to also give you insights as I reviewed always the top submissions and other like honorable mentions. I had a look at the profiles of developers, and it is really like exciting mix across regions. And as already said, like we have students, we have individual developers, we have founders here, we have here software

• 38:10Speaker 05

  engineers from big companies. But also, I saw very active software developers from Intel. This is very interesting that, I mean, indeed, like Intel Liftoff is more targeting startups, but it was very diverse portfolio of developers and from across regions. So it's really like amazing because in our Slack channel for Hackathon, we see really like always like from like 24 hours.

• 38:47Speaker 05

  Messages there with submissions, with questions because of this diversity. So it's really like a global hackathon at the end of the year. We are very proud about it. And we will post really like articles about each challenges and also about the last challenge with two days development sprint. Now you can read three articles. Not only like it is not only announcement of winners, but also their own comments and results of works what you can find in this blog articles.

• 39:29Daniel Whitenack?

  Awesome. Thank you so much for your work on those. It was cool to see like the traffic coming in basically all day and night, which was awesome and like it's hard to sleep while all of this cool stuff is going on. So as we draw close to an end here, I want to kick it over to Ralph, who leads up the Liftoff program and just get any sort of final thoughts. Like what did you think of this whole process? What were you encouraged to see? And what are you looking forward to in the new year in terms of things related to generative AI and Liftoff?

• 40:07Speaker 06

  Hello, everyone. It's me, Ralph. I'm sorry for the noise here. That's why I was on mute for the all the time. Because there is some, at the office, there is some kind of year end party going on. So yeah, I was completely amazed by what happened during this hackathon and I'm very grateful to the team, starting with Rahul, the rock star developer of this hackathon. And also thank you very much to you, Dan, for supporting this, for really running this with us. And then Ryan, who is the second rock star developer here. And of course Eugeni, who made it all happen.

• 40:49Speaker 06

  And so I really look forward to the impact we can make in the developer ecosystem, in the AI developer ecosystem. I really look forward what's going to happen next year and we want to have a share of what the future might bring to us. And I can tell you, the Intel Liftoff team is ready for whatever comes in the startup world. And yeah, so.

• 41:16Speaker 06

  See you next time and great to have you all here.

• 41:20Daniel Whitenack?

  Thanks, Ralph.

• 41:21Speaker 03

  Awesome. I mean, we co-created this together with Prediction Group on day one, like we had meetings on how to do this and what other things that we need to do on this. Do you want to say, for the folks who don't know about Prediction Guard, you know, to introduce Prediction Guard also to, what was your experience but working on this hackathon with us? And what are the things that we need to do next? I'm sure that we need to do it big, 2,000 is now sort of our baseline. So next time we do, maybe it's 4,000 people.

• 41:50Daniel Whitenack?

  It's pretty big.

• 41:51Speaker 03

  So let's, yeah. What's your take on it there?

• 41:55Daniel Whitenack?

  I think one takeaway is like, when you do a hackathon with Intel Liftoff, you better be ready to scale your servers. So we'll, we'll take that takeaway for next year when it's 10,000 people participating, I'm sure. But yeah, it's been great. I, one of the things like I say is, we really appreciated actually interacting with people, creating practical solutions with LLMs. That's what we're about at Prediction Guard and seeing people actually apply some of the latest models, like the Neural Chat, Notice, Zephyr, Yi, Wizard Coder. Seeing them actually access these things and even like combine them together in unique agents.

• 42:37Daniel Whitenack?

  I think that gave us like such encouragement to see people actually kind of fulfilling this vision that we have, which is providing these open, privacy-conserving, hosted models to people and them combining them in unique ways to create real enterprise value. That's what we're excited to see and do it in a way that is actually trustworthy. Intel, of course, has a great history with security and, and privacy, confidential computing. But to be able to sort of be partnered together and see people creating really both trustworthy, privacy-conserving and scalable solutions with LLMs in this environment is really encouraging, I think, for the future of AI.

• 43:18Daniel Whitenack?

  Because as we've seen even over the past week with Mixtral being released and Striped Hyena and all of these models. The open models are just getting better and better and you know, providing ways for people to access those in a scalable way and build real solutions. Yeah, it's really exciting to see that happen in in the industry. So thank you for hosting this and making it happen. It was a great experience.

• 43:46Speaker 03

  Thank you and thank you to the entire team like Scott, Ralf, the team that I I talk to daily, Ryan, like we practically talk every hour, Eugenie and and the whole of engineering team at InterLift, Chotis, Virat, Basanta, Raj, like you you guys are incredible and all the teams in Dan's side also, like being in the Slack channel and answering questions. There was all of us had reservations, but we kept that to ourselves. We didn't know how it's gonna go, but everyone pitched in with really cool ideas and with mindset to help. And that really shows in the community, all the messages we get, we had messages where

• 44:29Speaker 03

  folks were saying, now I can take this thing to my boss and tell like, you know, I need to implement these sort of things in our day to day work. And this is really, really gratifying to see that. And next time we come in, we'll fix all the shortcomings. We'll do an internal review carefully if there were any shortcomings. I'm sure there are to to fix them. Bigger, better, more scalable, more cooler challenges. We want to continue this and and grow this community. So any sort of feedback Eugenie would be, I'm I'm sure we'll be sending a survey. I know it's very difficult to answer any sort of surveys. It's easier to delete that email, but we would really appreciate, I personally would really appreciate your feedback on what we can improve, what are the things that we could add more and make it more more a community driven effort.

• 45:14Speaker 03

  We we don't really like in lift of the top down top approach. We really want your feedback and and the things that you want to see and build around it. So thank you once again.

• 45:25Daniel Whitenack?

  Yeah, thank you all. Closing out here, I just want to encourage you also to not only keep tab for hackathons, but all of you who are building amazing startups, and I know many of you are who are part of the hackathon, they maybe are too humble to say it, but this Liftoff team is doing amazing things. And as a startup that's participating in it, your startup should join Liftoff and reach out to them because you will find amazing benefit and scale and access to expertise and hardware. So, reach out to the team. They truly are rock stars, like Ralph said. So, reach out and get involved in the gram and the community. With that, we'll, we'll close this advent of Gen AI out. We'll give

• 46:10Daniel Whitenack?

  you the last word, Rahul.

• 46:12Speaker 03

  All right. Yeah, I forgot to mention one person, Kelly. I don't know, I don't know how I forgot. She has been incredible, starting from the website, creating the content, editing the video. I mean, she was sick while she was doing it, but she had some few hours that she had to take off. But she has incredible in the pace at which she was able to help us. And thank you, Kelly, for doing that. I'm sure that we'll be doing many more of these things again, then to a team. If I missed anyone, I'm really sorry, but this was truly a team event. Everyone contributed and without a small contribution, this would have just been an idea. So, thank you all for doing that. Thanks, everybody.

• 47:03Speaker 00

  All right. That is Practical AI for this week. Subscribe now. If you haven't already, head to practicalai.fm for all the ways and join our free Slack team where you can hang out with Daniel, Chris, and the entire change log community. Sign up today at practicalai.fm/community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you next time.