Rise of the AI PC & local LLMs

We’ve seen a rise in interest recently and a number of major announcements related to local LLMs and AI PCs. NVIDIA, Apple, and Intel are getting into this along with models like the Phi family from Microsoft. In this episode, we dig into local AI tooling, frameworks, and optimizations to help you navigate this AI niche, and we talk about how this might impact AI adoption in the longer term. Sponsors: Ladder Life Insurance https://ladderlife.com/changelog – 100% digital — no doctors, no needles, no paperwork. Don’t put it off until the very last minute to get term coverage life insurance through Ladder. Find out if you’re instantly approved. They’re rated A and A plus. Life insurance costs more as you age, now’s the time to cross it off your list. Neo4j https://graphstuff.fm/episodes/2023-finale-llms-and-knowledge-graphs-throughout-the-year?&utm_campaign=UCGenAI&utm_content=AMS-SrDev-ToFuDev-UCGenAI-Audio-None-GenAI1-GenAI-NonABM&utm_medium=Audio&utm_source=PracticalAI&utm_justglobal= – Is your code getting dragged down by JOINs and long query times? The problem might be your database…Try simplifying the complex with graphs. Stop asking relational databases to do more than they were made for. Graphs work well for use cases with lots of data connections like supply chain, fraud detection, real-time analytics, and genAI. With Neo4j, you can code in your favorite programming language and against any driver. Plus, it’s easy to integrate into your tech stack. Fly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Ollama https://www.ollama.com/LM Studio https://lmstudio.aillama.cpp https://github.com/ggerganov/llama.cppOpenVINO https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.htmlMLPerf client working group https://mlcommons.org/2024/01/mlperfclientwg/Article - 5 top small language models https://datasciencedojo.com/blog/small-language-models-phi-3/GPTQ article https://towardsdatascience.com/4-bit-quantization-with-gptq-36b0f4f02c34Article - Which quantization method is right for you https://www.maartengrootendorst.com/blog/quantization/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Hello and welcome to another Fully Connected episode of the Practical AI Podcast. In these Fully Connected episodes, Chris and I keep you connected with everything that's happening in the AI world and hopefully share some resources with you to help you level up your machine learning game. My name is Daniel Whitenack. I am CEO and founder at Prediction Guard, where we're safeguarding private AI models. And I'm joined, as always, by Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris? Doing great today, Daniel. How are you? I'm doing well. Yeah, yeah. Lots, lots going on, lots of fun stuff in the, in the AI

• 1:22Daniel Whitenack?

  world. But Chris, I, I have to say, so one of our, one of our listeners pointed out something which I realized afterwards as well on our last Fully Connected episode, the one about GPT-4O or Omni, we played some clips of the voice assistant and we talked a little bit about the model. And the model, of course, was released and we were using it, but the voice back and forth wasn't yet plugged into GPT-4O. So they're just, I think, releasing that shortly. So I think there was a bit of general confusion in the community about, because when you click that button, it's like you just take over into the, the voice

• 2:06Daniel Whitenack?

  interface and it's like, I was on GPT-4O and so. Probably could have been maybe handled a bit better on our end and maybe their end too. But OpenAI got us, I guess. We were fooled. But yeah, still really cool stuff. Still, the things that we talked about are still what GPT-4O is. But just wanted to clarify that for our listeners if they had listened to that previous episode.

• 2:32Chris Benson?

  While we're doing that, I realized I had made a mistake on that same issue after the show that I wanted to confess on. I had watched all the videos over and over again about the video portion doing that. And then when we were in the show, I had a claim as a senior moment where I was thinking, just I'd watched so much of the video. I was thinking, oh, yeah, I did that. And so it wasn't afterwards, I was like, no, that was the videos I was watching. What was I thinking? So anyway, I wanted to confess. So we had a couple of blunders on that one, but having fun.

• 3:02Daniel Whitenack?

  There you go. In our eagerness and excitement to talk about GPT-4O and record it very quickly afterwards, we got got.

• 3:11Chris Benson?

  So we got, we both got got there. So all good. We'll move on. Well, if for listeners who don't know, the show is fairly spontaneous if you haven't been following us long. And we dive into stuff pretty quick and occasionally, occasionally we fumble a little bit with that.

• 3:26Daniel Whitenack?

  Yeah, yeah. So thanks for journeying with us through this, this wild and crazy world of AI. And now I think something that's been on my mind quite a bit, Chris, with some recent announcements, but also things that have been kind of developing over the past months is this area of local offline AI and AI PCs. There's sort of people are using this terminology, AI PCs.

• 3:56Chris Benson?

  And current marketing hype.

• 3:58Daniel Whitenack?

  Yeah, the current, the current marketing hype. So I thought maybe we could dig into a little bit of, a little bit of that today and talk through kind of what, what exactly does that mean? How are people using AI models locally? What is an AI PC? What are the relevant kind of types of models and optimizations that can run locally? All of that seems to be a big, well, it's a little bit hard to parse out some of that. Maybe if you're new to the, new to the space and you know, what is a? GGUF versus Oh Llama or, you know, all of these, these things, you know, coming out. So yeah, I thought that would be good to dig into. And I know that you have been interested in kind of AI at the edge for some period of time.

• 4:43Daniel Whitenack?

  How do you see, whether it's a staff laptop or maybe a heavy compute node in a manufacturing plant or something where you'd want to run AI quote, "at the edge" or locally, let's say generally locally or offline. What are the reasons that people would, would sort of want to go that direction?

• 5:05Chris Benson?

  Well, I think, you know, this is kind of a thread we've talked about off and on on different episodes over time. And, you know, AI models are hosted in, in software and, and they're going to always be wrapped in that. And as we, you know, the software expands from the cloud all the way out into every device that we're already using. And so it's only natural that as AI becomes more accessible and cheaper to deploy, that it's going to, you're going to start having models, you know, that are kind of ramping up existing software out there. And there's, of course, we're going through all the hype that's associated with that. But the way I see it is, is simply just a, a natural evolution of where software development would go. And to your point a moment ago about the hardware side, we don't talk about that a lot.

• 5:49Chris Benson?

  We're very software focused in general, but the hardware side is really going through revolution. I know that you, I know that in your business, you have partnership with Intel and are seeing that in that capacity. And I certainly see that in my day job. And so there are so many more hardware capabilities coming out to support these functions, many of which will function or are targeting low power disconnected environments. And so this is a timely topic. And, and if you look back for precedent before that, We've always seen over the years in software and cloud development, kind of a shifting back and forth between local capability as suddenly you get a new generation of hardware and things will go a little bit more local and you'll have your own equipment and then things will move back into the cloud.

• 6:34Chris Benson?

  And that natural give and take is part of the flow. And I think right now we've been so cloud focused the last few years because that was really the only available option. That now we're, we're seeing a lot of new capability rolling out on both hardware, software and models that are going to enable edge functionality to really explode over the next few years.

• 6:55Daniel Whitenack?

  Yeah, I was asked, I was at a conference last week and I was asked which direction things would be going, either local AI models or hosted in the cloud. And I think the answer is definitely both in the same way that there is a place for, if you just think about databases, for example, as a technology, there's a place for embedded local databases that operate where an app, application operates. There's a place for databases that run kind of at the edge, but on a heavier compute node that serve maybe some environment. And there's a use case for databases in the cloud. And sometimes those even coexisting for various reasons.

• 7:39Daniel Whitenack?

  And in this case, we're talking about AI models. So I have a bunch of files on my laptop. I may not want those files to leave my laptop. So it might be privacy reasons that I want to, you know, search those files or ask questions of those files within AI models. So privacy, security type of thing, or in a healthcare environment, they may have to be air gapped or offline sort of thing, or public utilities sort of scenario where you can't be connected to the public internet. But then it might just be also because of latency or performance, inconsistent networks or flaky networks where you have to operate sort of online, offline. There's a whole variety of reasons to do this.

• 8:21Daniel Whitenack?

  But yeah, there's also a lot of ways that, as you said, this is rapidly developing and people are finding all of these various ways of running models at the edge.

• 8:34Daniel Whitenack?

  And we can highlight, if you're just into this now and getting into AI models, maybe you've used OpenAI's endpoint, or you've used an LLM API. If you wanted to run a large language model or an AI model on your laptop, there's a variety of easy ways to do that. I know a lot of people that are using something like LM Studio. This is just an application that you can run and test out different models. There's a project called Ollama, which I think is really nice and really easy to use. You kind of just spin it up. You can either spin it up as a Python library or as a kind of server that's running on your

• 9:16Daniel Whitenack?

  local machine and interact with Ollama as you would kind of an LLM API. And then there's things like Llama CPP and a bunch of other things. These I would kind of categorize as local model applications or systems where there's either a UI or a server or a Python client that's kind of geared specifically towards running these models locally. And then there's a sort of whole set of technologies that are kind of Python libraries or optimization or compilation libraries that might take a model that's maybe bigger

• 9:55Daniel Whitenack?

  or not suited to run in a local or lower power environment and run that locally. So if you're using the transformers library from Hugging Face, you might use something like Bits and Bytes as a library to quantize models, shrink them down. There's optimization libraries like Optimum and MLC, OpenVino. These all have, some have, exist for some period of time. Actually, I think in the past, we've had the Apache TVM project on the show and we talked about OctoML. So this is not a new concept because we've been sort of optimizing models for various hardwares for some time.

• 10:35Daniel Whitenack?

  But these optimization or compilation libraries are also usually kind of hardware specific. So you optimize for a specific hardware, whereas other, other of these local model systems are maybe more general purpose, less optimized for.

• 10:52Daniel Whitenack?

  Hardware specifically. I don't know if you've got a chance to try out any of these systems, Chris, running some models on your laptop.

• 11:01Chris Benson?

  I have a little bit. I've used Alama. I think that's my go to.

• 11:04Daniel Whitenack?

  And you have like an M1 or M2, M3, whatever M there is now?

• 11:10Chris Benson?

  Yeah, I have an M2 that I have. I have a couple of different laptops, one that's old and one that's, well, I guess an M2 is old by today's standards. So I may have to upgrade that one pretty soon. But, but yeah, I've used, I've used Alama primarily. I probably haven't used as many of the tools as you have, given, given the business that you're in. It's, I think one of the things that I'm really interested as, as people are, are doing this now is understanding, you know, because we're really focusing on this kind of the infrastructure, the plumbing of making all this work locally and, and doing the integrations with the cloud. But I think there's another, another thing just to throw into the, the mix of the conversation. And that is how, as people start, whether

• 11:54Chris Benson?

  in the cloud or particularly local, having multiple models out. And you have the infrastructure now, you know, as we're talking about to run it, but starting to look at the, the API and the middleware to enable inferencing across APIs without direct human intervention and have it make sense, where you have different responsibilities, much like we have had for a number of years in the software realm. So as we're talking about that, I wanted to throw that in as another topic that I think is going to be really, really important and hasn't had nearly as much attention as the fundamental infrastructure.

• 12:28Daniel Whitenack?

  Yeah, I think that gets probably to a couple of things. One is that one current major difference between sort of hosted cloud models and offerings versus local models is likely you're not going to run a mix of 10 different LLMs on your local laptop all at the same time, all loaded into memory. That would be a pretty, a pretty significant, at least right now, a pretty significant ask to kind of switch between models in that way. But there are certainly cases where, you know, I think the market is showing that people want to not be restricted to one model family and they're spreading out their usage against multiple models. So that definitely needs to happen in the cloud.

• 13:13Daniel Whitenack?

  And from, you know, model providers, but that doesn't mean you couldn't throw in the mix a selection of local models as well for specific purposes. And I think that gets to the other thing that you're talking about, which is kind of data integration, automation, pipelining, all of that sort of thing. I saw a comment on LinkedIn, I think it was even this morning, that those that are really winning in the AI space are those that have taken what they've learned kind of from automation, data, pipelining, data integration in previous cycles of data science and integrated those with generative models at various stages, because a lot of, a lot of the value, and I've seen this too, a lot of the value that you get out of these

• 13:58Daniel Whitenack?

  models is not the models themselves, but the system that you build around them. And, you know, that involves a lot of data integration and automation and maybe even routing between different models. In the case that we're talking about here, maybe even routing between local models and cloud models. And so, yeah, I think that that's really stressed, especially as you talk about running these models kind of everywhere, quote unquote, across cloud and local and on-prem and data center environments. But yeah, it's interesting. I don't, I don't myself have an AI PC yet. Maybe at some point I will, but, uh, but yeah, I'm excited to see where all this goes.

• 14:41Chris Benson?

  Indeed. I think as we push forward, I think one of the things that I'd like to see, especially for local, whereas, you know, we have, you know, we mentioned a few minutes ago, Alama and some of the other tools for infrastructure. I was actually, as we were talking here, I was looking through, um, Jan LeCoon's various posts because he was proposing recently in the last week or two prior to this conversation. Kind of a way of structuring different model interactions and from a responsibility standpoint that they're doing at Meta. And of course, at my employer, we have our own version of that on how we'd structure different things, but really interested in seeing if the community kind of comes around with kind of an open framework, you know, best practice framework around how to do

• 15:26Chris Benson?

  that, you know, be able to do it on a laptop, on an M2, M3, M4, be able to have those interactions locally and a framework that would span between that local and the cloud interactions so that you have something. And I don't think that right now, everyone seems to be doing that on their own. And there's a lot of similarities between them, but there doesn't seem to be kind of a standard approach to how that all comes together.

• 16:02Speaker 03

  If you're anything like me, you have a certain tendency to put things off until the very last minute. Seeing the dentist, going to the doctor, home improvements, that never ending chore list of yours. And while most of the time it works out just fine, the one thing in life that you really cannot afford to wait on is setting up term coverage life insurance. You've probably seen life insurance commercials on TV and thought, yeah, I'll look into that later. No, later doesn't come. This really isn't something you can wait on. Choose life insurance through Ladder today. Here's what we love about Ladder and why we allow them as a sponsor. They are 100% digital. No doctors, no needles,

• 16:44Speaker 03

  no paperwork. When you apply for $3 million in coverage or less, just answer a few questions about your health in an application. Ladder's customers rate them 4.8 out of 5 stars on TrustPilot and they made Forbes Best Life Insurance 2021 list. You just need a few minutes and a phone or laptop to apply. Ladder's smart algorithm works in real time. So you'll find out if you're instantly approved. No hidden fees. You can change any time. Get a full refund if you change your mind in the first 30 days. Ladder policies. Are issued by insurers with long proven histories of paying claims. They're rated A and A plus by A M Best. Finally, since life insurance costs more as you age now.

• 17:27Speaker 03

  Yeah, right now, now's the time to cross it off your list. So go to ladderlife.com slash practical AI today to see if you're instantly approved. Again, that's ladder.com slash practical AI. L A D D E R life.com slash practical AI.

• 18:00Daniel Whitenack?

  Well, Chris, there's an increasing number of options. If you were to explore this space and kind of that interaction of local models with your, with your systems, there's an increasing number of choices of quote AI PCs, which I think is a hyped term now. One of them, you mentioned Intel, Intel coming out with AI PCs. I think Lenovo is shipping some with the Intel's Core Ultra processor. I think the name is Meteor Lake, the code name at least. And similar probably to as a response to maybe what's more familiar. Already mentioned the M1, M2, M3, et cetera, line from from Apple.

• 18:45Daniel Whitenack?

  But Nvidia is also working hard on like the GeForce RTX AI PCs, which there've been kind of gaming PCs with like GPUs in them for for some time. But I think most of these quote AI PCs are more of an integrated. Type of processor or system where where it's not just like an add on to the laptop, but in the case of like the Core Ultra or the M2, there's actual processing in the architecture that is optimized for executing models. And so there's sort of AI ready, shipping AI ready. And that brings up kind of some interesting questions in my mind,

• 19:30Daniel Whitenack?

  which are, well, how do all of these AI PCs compare? If I'm about to get myself an AI PC, where should I go? And in thinking about that and looking at some of these benchmarks, I was really encouraged to see that ML Commons, which is the organization behind ML Perf, which is a set of benchmarks that benchmarks and working groups that have been working for some time to benchmark various systems for performance on running AI workloads, machine learning workloads. They've just announced this spring, an ML Perf Client

• 20:10Daniel Whitenack?

  Working Group, which is really geared towards essentially an application or a workload that you could run across these various AI PCs, or maybe AI-enabled edge machines and that that sort of thing, to really do kind of LLM-based workloads and do some benchmarking for both training and inferencing on these quote clients. So they're they're kind of referring to these generally as as clients. So they say, "The new ML Commons effort will build ML benchmarks for desktop, laptop, and workstations for Microsoft Windows

• 20:51Daniel Whitenack?

  and others other operating systems." So quite interesting. I'm glad that this I, as far as I can tell, someone in our again, you know, sometimes we get things wrong. So someone in our audience can correct us, maybe David from ML Commons, who's been on the show before, he can correct us if something's already been published. But I couldn't find the sort of set of benchmarks. I think it's a work in progress, but really excited to see this when it comes out.

• 21:17Chris Benson?

  Yeah, I think it'll be interesting as as these laptops come out. I'm it's hard to imagine that the entire industry doesn't have to to go full in on this regardless, and thus kind of making the distinction. Of an AI laptop, a little bit of a redundant thing because there's a point, and maybe we're already, maybe we've already arrived now where the idea of purchasing a new laptop that is not an AI laptop is a ridiculous thing. You know, it's a, it becomes a must have feature to have going forward and therefore all laptops kind of have to go that direction at some point in your.

• 21:52Daniel Whitenack?

  Yeah. Well, I, I definitely think that that could be a one downside of this whole thing is that there's, I mean, I know the prices will go down, but these things are really expensive right now. So there is going to be a sort of disparity of those already for some period of time. If, if you're a, a new developer, maybe a, an indie developer, that purchase of that MacBook is a pretty significant expense for you already. And, you know, myself, I just use a refurb ThinkPad from like four years ago, you know, not an

• 22:30Daniel Whitenack?

  AI PC. It's a Core i5. And, you know, not a terrible laptop, but definitely not, not anything that anyone would necessarily be jealous of. Now, I can run some models on this laptop, you know, using Ollama and other, other systems. And I think that that gets down to maybe another element of this. So there's going to be, on one side, these clients that get increasingly sophisticated and build in more AI-enabled or accelerating functionality into their chipset and into their, their hardware. But there's also going to be increased sophistication

• 23:12Daniel Whitenack?

  on optimizing models that can't run locally, such that they can run locally. And this is where people might kind of, I think this is also a point of common confusion that I've heard in workshops that I've given at conferences and other places where people kind of look at, let's say it's Lama 3 or something like that. Lama, I want to run Lama 3. Well, you go to Hugging Face, the top downloaded Lama 3 is the base model. And then you've got these fine tunes for instruction or chat. And then you've got all of these other flavors of Lama 3 like.

• 23:50Daniel Whitenack?

  You know, GGUF, GGML, QAT, AQ, AWQ, you know, all sorts of like acronyms that are really difficult to understand. So maybe it would help to just break this down just slightly. There's usually when a model is released, they release a base model or the pre-trained model or whatever it's called, a base model. So that's that kind of shortest name usually, the Meta, you know, Meta Lama 3. That usually is maybe a good model that you might fine tune off of, but not generally the best model to start with because it's, it's a base model. It's not fine tuned for any sort of set of general instruction following or chat.

• 24:35Daniel Whitenack?

  And they usually release along with that then a set of fine tuned models for instruction or chat. So you've got Meta 3 or Lama 3 instruct, which is usually the better model. And then you've got this whole world of community members out there that build pipelines. So we had NUS Research on. They have pipelines built so that when a model is released, they can create all these different flavors of it, which include flavors for running these optimized in certain ways. So these would be these other acronyms that we can dig into a little bit. But these are most of the time either additional fine tunes

• 25:16Daniel Whitenack?

  or quantized versions or somehow optimized versions of these models that are meant to be run in kind of a diverse set of environments.

• 25:41Speaker 03

  What's up, friends? Do you remember when ChatGPT launched? I do. It felt like the LLM was this magical tool out of the box. However, the more you use it, the more you realize that's just not the case. The technology is brilliant. Don't get me wrong, but it's prone to issues like hallucination on its own, but there's hope. There is still hope. Feed the LLM reliable current data, ground it in the right data and context, then and only then can it make the right connections and give the right answers. The team at Neo4j has been exploring how to get results by pairing LLMs with knowledge graphs and vector search. Check out their podcast episode about LLMs and knowledge graphs throughout 2023

• 26:23Speaker 03

  at graphstuff.fm. They share tips on retrieval methods, prompt engineering, and so much more. Don't miss it. Find a link in our show notes. Yes, check it out. graphstuff.fm, episode 23.

• 26:56Daniel Whitenack?

  Well, Chris, I kind of started getting into the alphabet soup a little bit. I don't know if if you're sometimes as confused as as I am with all of these, all of these model names, but they're getting increasingly long.

• 27:11Chris Benson?

  Yes. You know, one of the, finish your point there and then I have a question for you afterwards.

• 27:17Daniel Whitenack?

  Yeah, yeah, sure. No, I was just gonna highlight a few of these different quantization methods so that people could maybe have them in their mind. So there's the flavors that are GGML or GGUF. You might see those letters. This is GPT-generated unified format. That's, that's what that stands for. And this is a, this is an optimization of a model that will take that model that maybe requires a GPU to run as larger and creates a C++ replica of the LLM.

• 27:51Daniel Whitenack?

  And allows you to run it in quantized versions, meaning that the parameters of the model are taken from, you know, numbers that might have 32 or 16 digits behind the the period and get that down to two or four or eight, that sort of thing, which makes the model smaller and more efficient. So these are mostly geared towards CPU, sort of CPU or laptop kind of environments. There's also GPTQ, which is really a focused kind of quantization method, but it's still meant for GPU only. So these usually ship in similar

• 28:31Daniel Whitenack?

  kind of formats to previous models, but they do kind of some calibration informed quantization to get that model smaller. There's QAT, which is quantization aware training, which as it might sound involves some actual training, retraining of the model to inform the the quantization. There's others like AWQ, and this is another quantization method. So all of these sort of letters that you see, if you're wondering what those are, those are all kind of referring to these different kind of flavors of the model that might be generated for either running the model in an optimized way locally on a CPU, on a laptop,

• 29:16Daniel Whitenack?

  or optimized still on a GPU, but in a smaller format that's more efficient.

• 29:22Chris Benson?

  What are your thoughts on the on the CPU derivative? What are your thoughts on, you know, performance and capability relative to the its own base model and its GPU siblings?

• 29:33Daniel Whitenack?

  Yeah, I mean, I think that the reality right now, and then maybe where it's headed, I think the reality right now is The CPU-based models, even you can run some models that are even, you know, seven billion parameters or something in some quantized version on a CPU. You're not going to get the same throughput as you would with hosting the model on a GPU. In other words, the kind of tokens per second, the speed at which you're generating will be lower. But it's possible to run those. What you're not going to do is, and this is what I also tell people, like the same way that, like you're not going to host your microservice for your company on your laptop.

• 30:15Daniel Whitenack?

  You're not going to host some AI functionality for your company on your laptop, usually. That's going to still live in the cloud. But if you have some sort of private use case where something can't leave the laptop, or maybe it's like you're deploying laptops in a disaster relief scenario, and there's going to be not that much connectivity, right? It's still enough throughput to get responses. So if you had like chat over your docs, disaster relief docs, and that on your laptop, like it's still enough to get an answer in that scenario. And, you know, people can push it pretty far. I think the difference is just, again, it's not one or the other.

• 30:59Daniel Whitenack?

  It's the use case, I guess.

• 31:01Chris Benson?

  Yeah, I think, and just to add to that use case slightly, I think there's also disconnected or partially connected mobile platforms where you can't necessarily rely on the cloud access, and you have devices that are out there as well. I would, just to lump that in, kind of pulling around full circle for a moment back to the laptops, these AI laptops coming out, kind of thinking back to, you know, the natural kind of segregation of responsibilities that we have in software development, aside from just the AI world. Would you imagine that a reasonable level of support in those would be to be able to actually do training on like seven billion size models, the smaller range that are so, you know, that are,

• 31:45Chris Benson?

  there's so much more of those where that maybe in the not so distant future, I'm training a model, it's in that range, my own laptop can handle that, not just for inference, but for training. And then for very large models, that's still cloud based. Do you think that that is a reasonable level that we might see AI laptops able to support?

• 32:04Daniel Whitenack?

  I mean, I think that you can see some people trying some more training or rather fine tuning sorts of things on diverse hardware. I think basically what I'm seeing right now is still primarily inference on local machines and utilization of things like in context learning and RAG type workflows to integrate data rather than fine tuning locally. So I think that that's, that's kind of the reality of where it is. I think there's a possibility in the future, maybe of, you know, some type of training type of scenarios that will happen, maybe not on client

• 32:47Daniel Whitenack?

  devices, but spread across client devices. So, you know, it's been a while since we've talked about federated learning, but it will be interesting to see if that kind of rears its head in this world. I know that there's been efforts to kind of train LLM adapters in a federated way. And there's some papers about this and share kind of parameter efficient updates to weights across different client devices. That seems really intriguing to me. But yeah, I don't know. We'll, we'll see where it goes. Maybe I'll be proved wrong, but I think I'm increasingly more of a proponent of most people don't need to fine tune. You know, a lot of it can be done with RAG

• 33:31Daniel Whitenack?

  and chaining and agents and, you know, selecting the right models. So I think that that, especially as models get better, that will be the case kind of moving forward. But yeah, I am looking forward to getting an AI PC and. You know, putting it through its paces eventually.

• 33:48Chris Benson?

  Yeah, right now, my I have to get my M2 is through my employer. So I'm my next one will be, I'm hoping to be in either an M4. I'm thinking about possibly a non Apple one as well. So the M4s are supposed to are supposed to be able to do quite a bit more than than the earlier generations. Looking forward to to being able to pursue this.

• 34:09Daniel Whitenack?

  Yeah, yeah, definitely. Well, I hope our audience will will include a few links to some blog posts about these quantization methods and some of the systems like Ollama and LM Studio and others that we talked about. I encourage everyone to to get hands on and try your own hand at it and you'll get a sense for the performance of these models locally. So definitely give it a try.

• 34:32Chris Benson?

  All right. Well, thanks a lot, Daniel. That was great information today. It was a good show and thanks for bringing that.

• 34:38Daniel Whitenack?

  Yep, we'll talk to you soon.

• 34:48Speaker 00

  All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to practicalai.fm for all the ways and join our free Slack team where you can hang out with Daniel, Chris, and the entire change log community. Sign up today at practicalai.fm/community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.