Creating a private AI assistant in Thunderbird

In this episode, Daniel and Chris are joined by Chris Aquino, software engineer at Thunderbird to hear the story of how they developed a privacy-preserving AI executive assistant. They discuss various design decisions including remote (but confidential) inference, local encryption, and model selection. Chris A. does an amazing job describing the journey from "let the big LLM do everything" to splitting apart the workflow to be handled by multiple models. Featuring: Chris Aquino – LinkedIn https://www.linkedin.com/in/chris-aquino-9a86044/Chris Benson – Website https://chrisbenson.com/, LinkedIn https://www.linkedin.com/in/chrisbenson, Bluesky https://bsky.app/profile/chrisbenson.bsky.social, GitHub https://github.com/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaLinks: Thunderbird https://www.thunderbird.net/en-US/Thunderbird Pro https://tb.pro/Sponsors: Shopify – The commerce platform trusted by millions. From idea to checkout, Shopify gives you everything you need to launch and scale your business—no matter your level of experience. Build beautiful storefronts, market with built-in AI tools, and tap into the platform powering 10% of all U. S. eCommerce. Start your one-dollar trial at shopify.com/practicalai http://shopify.com/practicalaiUpcoming Events: Join us at the Midwest AI Summit https://midwestaisummit.com/ on November 13 in Indianapolis to hear world-class speakers share how they’ve scaled AI solutions. Don’t miss the AI Engineering Lounge, where you can sit down with experts for hands-on guidance. Reserve your spot today! Register for upcoming webinars here https://practicalai.fm/webinars!

• 0:03Speaker 00

  Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:49Speaker 01

  Welcome to the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I am joined, as always, by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris?

• 1:05Speaker 02

  I am doing just fine. It's been a good day, good fall, and lots of cool things happening to talk about.

• 1:12Speaker 01

  Yeah, yeah, I'm excited to, I'm excited for today's discussion, although I have to say, I feel a bit outnumbered by the Chris's. But there's some cool Chris's on the show today, including yourself, Chris Benson. But we've also got, we've also got with us Chris Kino, who is a software engineer at Thunderbird. Welcome. We won't call you Chris B. Actually, your last name, it starts with an A, so maybe you're Chris A, and Chris Benson is Chris B. And that just works out because his name starts with B, so.

• 1:44Speaker 03

  That's perfect.

• 1:44Speaker 01

  Yeah.

• 1:46Speaker 03

  Hello, hello. Thank you for having me. I know that we had some rescheduling issues early on, but we're here now.

• 1:54Speaker 01

  We're here now, and glad we are, yeah, because we've had a few guests from Mozilla or projects that Mozilla has been involved with for some time, or a couple of times in the past. And it's always great discussions and, of course, love the perspective that Mozilla brings, but also projects like Thunderbird. Could you give us, maybe, just starting out, a little bit about your personal background and kind of how that eventually led you into work on Thunderbird?

• 2:24Speaker 03

  Yeah. So my personal background, I've been a web developer since, oh my goodness, two decades, let's go with two decades ago, is when I started. And I worked for various companies that did different things. I've been, in addition to a web developer, I've done teaching and authoring. Most recently, well, prior to Thunderbird, I was at SurveyMonkey for a little while. And then the great layoffs of 2022 and 2023 hit. I was one of the newer engineers, so I got cut. And as I was applying for jobs, I was like, you know, I've always wanted to work for Mozilla. Let's just, let's just see what their job board looked like.

• 3:09Speaker 03

  And that's when I submitted an application and got word back from somebody who was clearly not a recruiter. The director of product emailed me, you know, just emailed me and said, hey, I'd love to talk with you. And yeah, that was, I initially got hired on to build just weird stuff that was outside of the realm of the Thunderbird desktop. He, his name is Ryan, Ryan Sipes, he had, he's always got something cooking. He had some interesting ideas for a set of products that he wanted to explore. And so he's like, yeah, would you be interested in joining us and working on some of this, some of this weird stuff? And so I said, I said, yes.

• 3:54Speaker 02

  Is a quick follow up on that is you're talking a little, just for the guests that may not be familiar with Mozilla, and the kind of the context of Thunderbird within Mozilla, could you talk a little bit about that for a moment, just to kind of set the stage?

• 4:07Speaker 03

  Yes, I will do my best. It is a little convoluted, as Wikipedia can tell you. So Thunderbird, all right, you can think of Thunderbird, well, first of all, for those who don't know, for the kids out there who are like, what is email? Thunderbird is a desktop email client. And for a certain generation, those three words strung together means nothing. But it is a 20 plus year old open source project, which originated at the company now known as Mozilla. If you're familiar with the Firefox browser, that is made by our sister company, the Mozilla Corporation. And Thunderbird and the Mozilla Corporation all fall within,

• 4:48Speaker 03

  I guess you could say, the guidance of the Mozilla Foundation, which is a nonprofit. And as of, oh, I guess it was five, five-ish years ago, maybe a little bit longer than that, Mozilla was, nobody was maintaining Thunderbird. So this application that was not Firefox had, you know, they didn't want to deal with the, the maintenance that was, you know, engineers, engineers need to maintain this almost 20-year-old C++ codebase. And that was for, you know, management reasons are like, okay, we could just hand this to the community, or we could just shut it down. And the director of product

• 5:31Speaker 03

  here at, who's now the director of product, Ryan said, wait a minute, listen, give, give me a shot, let me, let me try something out, let me see if I can turn this into something more than it is now. And they did whatever legal things and paperwork was necessary to spin it off into its own entity. We are known as Thunderbird or the Thunderbird Project, but if you look us up on the internet, we're officially MZLA, which sometimes I feel like it's one of those acronyms, it doesn't really mean anything. But that's what it says on my resume. So that's, that's Thunderbird, in a nutshell.

• 6:08Speaker 01

  Now, yeah, you, you mentioned this kind of history, this sort of desktop email application history. Just for context, I'm sure there's many people listening that do remember those sorts of days, but, but others that, that might have just always used Gmail in a browser or something like that. But there's a whole, whole generation of us that use something, whether it's Outlook, you know, a lot of people have, have used. I remember installing Ubuntu for the first time earlier in my career and, and using Thunderbird in that context.

• 6:47Speaker 01

  On my computer. Now you mentioned there's kind of maybe a different, a more vision for this now. Any context you could provide there in terms of, you know, the way people use email now versus those days when Thunderbird was being used as a desktop application? What is kind of the, I guess, the focus and transition, if you will? Or at least some of the things that are being thought about?

• 7:14Speaker 03

  Yeah, I'll start with sort of the ethos, like, why is Thunderbird still around? There are still people who use email. It works perfectly well. But unlike using, say, a webmail-based provider, these folks are not interested in having ads sort of injected for them. And maybe they want to be able to opt in to AI. They don't want an AI just like ever-present and reading their emails. So with Thunderbird, it's free and open source, no ads ever, forever. And, you know, your email is your own. It gets downloaded to your computer. So if you lost internet access, you can still read your email.

• 7:58Speaker 03

  If you lost power and your laptop is open to Thunderbird, you've still got it, right? All those emails that you downloaded, they're right there. So I think that the way that email is being used, it's different. It's different now. One of the great things about Thunderbird back in the early days, and it's still true today, is it's very easy to manage multiple email accounts. That was, you know, I remember a time when like, oh, you have more than one email address? Amazing. You must be really important. And now you can just sign up for email addresses, dozens of them if you want, and, you know, have them each dedicated to a different purpose. You can still do that with Thunderbird.

• 8:42Speaker 03

  However, one of the things that people are encountering now as We all are is a certain amount of information overload, right? You're subscribed to so many newsletters and mailing lists and you, you know, you're working on some collaborating on some side project and then you've got your, your main work email. How do you, how do you read? How do you read all that email? How do you deal with all of it? So those are the sorts of things that we're thinking about now is how do we empower the user to better deal with just this huge influx of information that they're getting every day, every hour?

• 9:23Speaker 01

  So first off, thank you for validating my multiple email use. Some, every once in a while, my, my wife gives me a hard time because, uh, you know, I generate a new email like every once in a while when I get frustrated with my email feed or have a specific purpose and it seems like no one knows, no one ever knows which email to email me at, which maybe is a strategy in and of itself. I'm, I'm not sure. So thank, thanks for validating that. But yeah, I, I know that there's, um, there's a good amount of kind of AI intersection with

• 10:04Speaker 01

  email, um, whether that be from like the web email side. So in Gmail or other things with, with Gemini or email, uh, clients that maybe are specifically geared towards AI features like a superhuman or these sorts of things. Not asking you to comment on every single one of those, but maybe, maybe in general, like how from your perspective, have you viewed this gradual integration with AI? And what sort of, um, I guess categories could people have in their mind of the kinds of ways that AI is being applied within email and some of those, uh, trade offs that maybe

• 10:46Speaker 01

  you're making when you're using those features that, that of course I know, um, we'll talk about privacy in some of what we talk about with Thunderbird, but maybe just from a perspective, would, would be curious on how you categorize. AI in email could mean a lot of things. Could you help people understand maybe some of that landscape?

• 11:05Speaker 03

  Sure. Yeah, I think that the two main ways that I see, I mean, in my own Gmail account, because I will use a device that doesn't have Thunderbird on it because I am a, I'm a longtime distro hopper and maybe I'm using a distro that doesn't have Thunderbird prepackaged and I'm just, just trying things out. I have found that the automatic summarization is a thing that Gmail's like, hey, Gemini can do this for you. Or all of the autocomplete that it tries to helpfully offer me, I feel like it's a little creepy sometimes, especially depending on what the email is. Like if I'm talking to my doctor over email and it's like, clearly

• 11:50Speaker 03

  this LLM has read this very private information. I'm like, oh, how do I turn this off? But I do understand that for a lot of people, those two features, they are time savers. You have this way of compressing more human time into your day by offloading it to an LLM. So that's great. It's a, it's a time saver and it's great that people have access to that. Some things that you lose out on, that you're, you're gaining time, but what you're trading are things like tone, right? You, the tone is, for some summarization models, you will, it will kind of strip out all the tone, or if it's bulk summarizing

• 12:33Speaker 03

  multiple emails, that, you know, email from your mom doesn't really sound like your mom. It is literally just like, your mother's coming to town this weekend. So there's, maybe dehumanization of email is kind of the, the wrong term, or maybe a little extreme, but that gets essentially normalized to the tone of the LLM. And yeah, to your point, the, the privacy aspect, that's, that's kind of the, the big one for us here at Thunderbird. We're very privacy respecting, privacy preserving, because that

• 13:08Speaker 03

  A lot of our users choose to use Thunderbird because of that. They want to manage and own their own email. They don't want their personal emails harvested for marketing purposes or for training data.

• 13:40Speaker 04

  Well, friends, when you're building and shipping AI products at scale, there's one constant: complexity. Yes, your rating models, data pipelines, deployment infrastructure. And then someone says, let's turn this into a business. Cue the chaos. That's where Shopify steps in. Whether you're spinning up a storefront for your AI-powered app or launching a brand around the tools you've built, Shopify is the commerce platform trusted by millions of businesses and 10% of all US e-commerce. From names like Mattel, Jimm Shark, to founders just like you. With literally hundreds of ready-to-use templates, powerful built-in marketing tools, and AI that writes product descriptions for you, headlines, even polishes your product photography.

• 14:25Speaker 04

  Shopify doesn't just get you selling, it makes you look good doing it. And we love it. We use it here at ChangeLog. Check us out, merch.changelog.com. That's our storefront. And it handles the heavy lifting too: payments, inventory, returns, shipping, even global logistics. It's like having an ops team built into your stack to help you sell. So if you're ready to sell, you are ready for Shopify. Sign up now for your $1 per month trial and start selling today at Shopify.com/PracticalAI. Again, that is Shopify.com/PracticalAI.

• 15:08Speaker 01

  Well, Chris, we were starting to get into a little bit of, I guess, the intersection of the ethos of Thunderbird with these sorts of AI features. Now, could you, could you help us understand, I guess, like, we talked about autocomplete, we talked about summarization, for example, there are various mechanisms by which these features can be implemented in an email client or an application or web application, whatever that is, in terms of the actual AI model, where it sits, how the data flows, what the model is trained on, maybe. Could you help us understand that piece?

• 15:50Speaker 01

  Like, what are, what are the buffet of options available to us in terms of, in terms of how we might, like, the integration point in the flow of data?

• 16:00Speaker 03

  Sure, yeah, we have, we have discussed at length different ways that we could approach this. So let me, let me begin by saying that this experimental work in bringing an AI assistant to Thunderbird, this is not baked into Thunderbird. We're not going to turn it on for, for users. It's not going to be automatic or anything like that. Instead, what we've done is we have built it as, I'm going to call it sort of like a, like a companion for right now. We'll put a pin in that, we'll return to that because there's, there's a lot of decisions we had to make because of that approach. Now, our options that were available to us, like, we could just do, like,

• 16:45Speaker 03

  add, you know, model inference to Thunderbird itself. This is, in my opinion, like, yeah, but now this is us turning it on for all users. It's like, just add the model in there and allow us to do inference locally. It's private, right? And that's great. However, we're not so invested in this idea that we want to put that on the desktop team's roadmap. When I say the desktop team, just a little plug for the, the mobile team at Thunderbird, there, you can grab it for Android now, works really well. iOS is coming soon. But as far as the, the desktop client goes, like, they already had their roadmap and we just kind of want to run our AI

• 17:30Speaker 03

  experiment parallel to that. Okay, so that is the, that's one option. A second option is like, well, could we, could we run inference in a separate application also locally? Yes, we can, and we've kind of poked around with that, but we don't necessarily, if we want to roll this out, we don't necessarily want to require users to download a second application. Okay, so well, how do we split the difference here? All right, so what if we, what if we called out to an API? Like, one of the cloud providers for, you know, any of the, the models that would be good at helping you out with email? The, the typical tasks of summarization or reply generation.

• 18:13Speaker 03

  So we decided that could be a thing because it doesn't require installing anything, you know, heavyweight and additional. But that brings up a, a separate problem of like, where, where are you sending the email data to? And I'm happy, I will be talking more at length about that.

• 18:31Speaker 01

  Yeah, yeah, I, I think in all my discussions in my day job work, that's often what it comes down to is we would love your AI features. Where is the data going? So I definitely, I definitely understand that. And I assume that, you know, there's definitely this tension of doing things locally and putting that on, like you said, the desktop roadmap. But also there, you know, would potentially be kind of either limitations in terms of the kind of model you could use, or even just if you could use a model that worked well,

• 19:14Speaker 01

  but it might just like destroy all of the battery of, of the device on which it's running or, or that sort of thing. Were those, were those things also part of that conversation?

• 19:25Speaker 03

  They definitely were, especially regarding the, you know, the, the second application that, you know, just ran a model inference process in the background. I don't know about you, but I, my laptop, my work laptop, not super fancy. It generates, you know, a few tokens per second, which is

• 19:46Speaker 03

  It's just not fast enough for this, I mean, literally for me to work on the thing that I'm working on. So that was a big concern. That and, you know, a lot of our users are on Linux and they're running Linux because maybe they're, they're continuing to use their perfectly good hardware from 10 years ago, which most certainly cannot do any sort of local model, anything. But yeah, laptop users, I love my battery life. I don't want to completely destroy it by trying to summarize a batch of today's emails.

• 20:21Speaker 01

  Yeah, yeah, that makes sense. So you mentioned this idea of using a model that is behind a remote API. There's obviously a selection, we're kind of narrowing in, there's this sort of selection of ways that that could happen. And there's like various approaches around maintaining privacy there from just using an API that explicitly doesn't store certain data, at least according to their terms, or doesn't train on your data, at least according to their terms. There's also kind of, you know, I know people are exploring

• 21:03Speaker 01

  kind of homomorphic encryption and all of these things to like, keep, keep data private. And then there's sort of end to end encryption. And there's, there's all sorts of ways that you could think about privacy in that context. What were the main, I guess, the main pillars of what were important for you all to consider? Was that where data is stored at rest? Is it the openness of the, the models, or whether you were hosting those, or a third party was hosting those? Was it, yeah, what were kind of the main, the main topics that came up once you kind of, you know, dipped into that remote inference side of things?

• 21:46Speaker 03

  That is a great question. And I have, I'm going to try to condense the story down.

• 21:51Speaker 01

  Okay, great.

• 21:52Speaker 03

  It'll be kind of like, you know, The Hobbit, and then the three parts of the rest of the story.

• 21:58Speaker 01

  Sure.

• 21:59Speaker 03

  Wow, you should just totally like revoke my nerd card now for not coming up like off the top of my head.

• 22:06Speaker 01

  Fellowship of the Ring.

• 22:07Speaker 03

  That's the one.

• 22:07Speaker 01

  Exactly.

• 22:08Speaker 03

  I just lost half of your listeners.

• 22:10Speaker 01

  The Two Towers, Return of the King. Yeah. You got it. I at least I might not know about the cool encryption stuff that you're about to talk about, or whatever that is, but at least, at least I have that one.

• 22:26Speaker 03

  Nice. Well, our powers combined.

• 22:29Speaker 01

  The powers combined, it takes a community. Yes, yes.

• 22:33Speaker 03

  So to let me, let me start this little story off with the very earliest experiment with this was as a Thunderbird add-on. Okay, what is a, what is a Thunderbird add-on? A Thunderbird add-on, you're familiar with, and your listeners are probably familiar with browser extensions. Okay, so fun fact, Thunderbird under the hood uses the Gecko rendering, the Gecko engine from Firefox. So we, we have access to APIs that can make, make it possible so that something almost exactly like a browser extension can reside within Thunderbird. And

• 23:18Speaker 03

  if you're not familiar with extent, like browser extension development, it's basically like HTML, CSS, and JavaScript. So we started by writing something that was, it was the most 1990s looking webpage that was just sort of like jammed into an add-on and just displayed in a new tab in Thunderbird. And it was, I mean, you know, it looked like an engineer built it, and that's totally fine. But yeah, that's when we started with calling, like, we started off with OpenAI's API and just handing off a number of emails to like, I think it was ChatGPT-4. And that did, that did an okay job. But what, that

• 24:02Speaker 03

  was like, okay, it does work. How do we build this out? And then we started, we started trying to get better results with some prompts tuning and whatnot. And then for, as we started, you know, trying to use it with more people within Thunderbird, we found out like, these people, their emails are sensitive. Like, we don't, what do we, we need to do something about this. So we started shopping around for some sort of, you know, cloud-based provider that could give us a guarantee of, yes, we do not store your data. No, we're not using it for training. And We talked to, we were in contact with a couple of different companies, some of whom just sort of sent us to like a, like one of the pages on their website,

• 24:47Speaker 03

  which told us nothing useful, couldn't give us a good guarantee. So that's when we started talking to the folks at Flower Labs. And I know that you have had several guests from Flower on the show. I just want to say that they are so terrific to work with. They took care of, I mean, really? They took care of like all of our needs. They moved things around on their own development roadmap and gave us early access to things like end-to-end encryption and access to their, their newest product, Flower Intelligence, which it is. So for the listeners who hadn't listened, hadn't heard of Flower or listened to previous

• 25:32Speaker 03

  episodes, they're known for their, their federated learning SDK, right? They build software that sort of does learning on individual nodes and then shares the learnings with a centralized server, all very cool stuff. We didn't need that, though. We needed private API, or rather API access to a private LLM. What we got in addition to that is we got a nice SDK in TypeScript. And we also got, so we got the end-to-end encryption and, oh, right. They found a model for us and then they, they did some post-training on email summarization.

• 26:17Speaker 03

  They built an eval system so they could like fine-tune. They help produce prompts, you know, through the eval system. I mean, they've just, they've been incredible.

• 26:28Speaker 02

  I'm just curious, as you, as you talk about this, and especially having had multiple folks from Flower on the show in the past, you're talking a little bit about kind of how you got into the collaboration, but like, how did you, how did Flower come into the picture to begin with for you guys? You know, how was that connection made? And, and, and how, as you were Looking at that connection potential, how did you know that that was a good fit for this new strategy that you've been laying out?

• 26:57Speaker 03

  I have the most boring yet magical answer to that question. It just kind of fell into our laps because Mozilla, all right, remember I was talking about the nonprofit Mozilla Foundation, under which Thunderbird sits. They're an investor in Flower. And so Mark Sermon connected Ryan, my boss's boss's boss, with Daniel from Flower. And I just ended up on a Zoom with him one day. Ryan introduced us and said, all right, take it away. And that's when the collaboration began. And so that's the thing, I guess, having a human face to go with the company

• 27:38Speaker 03

  made me feel good about those guarantees of like, no, we're, we're literally not in the business of harvesting data. Like, we're going to set up this infrastructure and we literally can't help you debug your prompt because it's encrypted now.

• 27:52Speaker 03

  So that was, the fact that they, they've been so, they were so helpful at every step, except when I sent some bad data, like the fact that they were like, we really want to help you, but we built it in a way that we can't, we can't see it, the data. So that was, it was a fortuitous connection, thanks to the reach of the Mozilla Foundation.

• 28:13Speaker 01

  Well, Chris, you started kind of unveiling some of what made this, this particular route of experimentation useful for you all. Could you help us understand, maybe just at a, at a slightly deeper level, like I could, let's say, spin up a model in VLM, in a VM on GCP or wherever I host things, and then connect to that over an API. What, what kind of makes the, the hosting of the private model within the Flower system? Because it's, it's, like you say, it's not federated, but there is still kind of more there.

• 28:57Speaker 01

  As, as you mentioned, of course, there's the post-training that you talked about, but as far as the inference side, could you help us understand that a little bit more?

• 29:04Speaker 03

  Yes. So as I mentioned, they they take they've taken care of all of that, which is that is the the big benefit. They've really been a great technical partner while we conducted these these experiments. And from the privacy aspect, you know, our my coworkers no longer have to prune their inbox and remove anything sensitive before trying out what we at this point, we have dubbed Thunderbird Assist. It is your personal executive assistant within Thunderbird. That was that was the idea. Anyway, thanks to the the guarantees made by Flower, you know, we were then able to try out. I mean, we didn't land on that, that the current

• 29:47Speaker 03

  model immediately. We tried different births, different parts, Roberta, all the various summarization models. And then one day, the summaries got way better. And I said, what did you do? What is this magic? They switched to one of the Lamas from Meta that was trained for conversation. And it just it worked better for email content. So at that point, we stopped thinking about the prompt, because they had squared that away. And prior to that, they found a model that could do the task very well. So then that got to the part where

• 30:29Speaker 03

  we're like, OK, so it's really great at summarization and reply generation. The third feature that we worked on, you know, this is the the biggest thing we aimed for. Because when you think about like, oh, hey, you're doing email summarization and reply generation. That's great. That's basically the hello world of, you know, LLMs summarize this text. So we started working on something that was, or we had been working on this feature that was, it did not work well. It was we refer to it as the daily brief. And it was intended to be an executive summary of your recent emails. This is when I learned the definition of over prompting.

• 31:13Speaker 03

  What I would do is I would take however many emails arrived in the last 24 to 48 hours, ship it off to the model and then ask it to do, "Oh, can you find the most important messages? Extract all of the highlights and any action items and return those back to me grouped in this particular way with links back to the original emails." I mean, the garbage that I got back sometimes was epic. So I then learned that, okay, what I need to do is I need to split this up into multiple requests, right?

• 31:55Speaker 03

  Let's like, let's send a batch and only ask for importance. Some of your listeners are like, "Okay, that's highly subjective," and I will return to that momentarily. But when we provided the emails, we also provided the unique message IDs from Thunderbird. So that way I could then use that as an index, grab the original emails again and send off a second request, which was like, "Okay, so for these, I want you to find the action items." And then take the same batch, now ask for highlights and crucial information. The formatting task never worked well, so I will, I have lots of feelings about that, so we'll put a pin in that.

• 32:40Speaker 03

  The formatting never turned out because that was what I wanted as an application developer the LLM was not capable of, right? It was, it's a statistical model and I guess it's thinking that maybe they want the header bolded sometimes and maybe they don't. I'm like, "LLM, do what I want." And it's like, who knows? I'll roll the dice. Maybe I'll give it to you the way that you want. And so...

• 33:07Speaker 03

  That was, that was an important lesson was like, okay, so in this currently, you, you really need to be very careful, specific and constrained in what you ask the model for. And then the problem turned into, okay, so how long does it take to make these subsequent requests? What can be parallelized? How do we do that effectively without, you know, burning up all of flowers compute in their, their infrastructure? So we, we started looking for, you know, ways to optimize that. And we switched to a local Bayesian classifier. So we'll take the first of several tasks and we'll just do that locally. So instead

• 33:51Speaker 03

  of asking an LLM to very subjectively decide what sounds important, or what is, um, what the, the cosine similarity algorithm tells them is important, uh, we'll do that locally. We'll let the user use the Thunderbird feature of tagging emails as, you know, like priority one, priority two, whatever. So for our experimentation, we had each user tag a handful of messages as, you know, highest importance and tag a handful as least important. And then the local Bayesian classifier that was, we just included as a JavaScript library in the Thunderbird add-on, uh, works very quickly, um, even for lots and lots and lots

• 34:34Speaker 03

  of messages. And so, okay, task number one, taken off the plate of the LLM. And so now we just have it do the, uh, you know, the rest of the tasks. Um, and likewise, the formatting task, we just handled that ourselves. A quick note about formatting. For a time when we were using one of the, uh, other third-party, party cloud providers, we found that you could provide them a JSON schema that the model would conform to when giving you the response. That was a magical time for me as an application developer because it's like, oh yeah, give me the JSON, I will just put it through my framework and it's just going to render

• 35:15Speaker 03

  the things beautifully. Like, look at my CSS applied so perfectly to this. This is amazing. When we, as we were model hopping and provider hopping, you know, that kind of went away and we haven't returned back to it because at some point, we realized that, okay, so learning our lesson from before about splitting up tasks, I, we realized that we need to take a different approach for the, the daily brief. And this is when I, I got it in my head that like, okay, so the, the future of this feature is not to just keep on sequentially prompting the same language model for like, hey, now do this, now do this. Instead, I think of like, you know,

• 36:00Speaker 03

  you go into a professional chef's kitchen and you don't see like this one giant tool that can slice, dice and microwave and air fry on top of that. You see lots of little dedicated tools that are like in expert hands. It, it does, it does that one thing and it's going to do it the best. So one of the things that we've got that I have written down in my mad science notebook is to explore like, well, what if we could dedicate some small models to specific tasks and then coordinate them in some sort of much more deterministic way. So the, the daily brief currently, all that to say, the daily brief currently has been kind of sidelined and you

• 36:42Speaker 03

  know, we're like, okay, so shipping assist means more or less summarization and reply generation. And daily brief, we still need to work on that because again, the, the approach needs to be more, more granular and more, more deterministic.

• 36:59Speaker 02

  I'm kind of curious, as you've taken us through that process, one of the things on my mind is how different kind of, for lack of a better word, audiences within your customer base, different profiles, how these different approaches that you've taken as you guys have devised the strategy forward on Thunderbird, where are you seeing more uptake? Where are you seeing, you know, people like, you know, I think in early in the conversation, we talked about younger generation who may not have grown up with email like we did. And then on the far side of that, you have kind of the corporate world and stuff like that. And with a, you know, a certain segment of your, of your user base in different aspects.

• 37:43Speaker 02

  I'm just curious how that may have, how people are receiving this in those different capacities, given the fact that you have different interests and stuff.

• 37:52Speaker 03

  Yeah, yeah. Well, the, the short answer is that our users are fairly homogeneous at this point. Our users of Thunderbird Assist are very homogeneous. They are all Thunderbird employees.

• 38:05Speaker 03

  Because it is, yeah, it does, it does. And even though this work is, it is open source, it's on GitHub right now. We haven't released it, you know, for general use because again, Flower, Flower had been tuning the, the models and making changes to their infrastructure. So they weren't ready to receive like a lot of users from all over the world. So within Thunderbird, there, there are even different needs. Some people use the individual email summary feature. Okay, so let me back up. There are, there are three features that are available in Thunderbird Assist.

• 38:49Speaker 03

  The first of which, individual email summarization. And if you aim it at like a quoted thread, you could call it a thread summarizer. There's email reply generation. And then the third one is the daily brief. Now for The different, the, the different kinds of users fall into two camps. There are the, I would like, you know, the, this long thread from the Thunderbird mailing list, I need that summarized because wow, that is too long. The other kind of user is the one who gets way too many emails and it needs an executive assistant. And that was where, as I just mentioned, it's like, okay, so that, that feature is just

• 39:34Speaker 03

  not going to work very well, given the approach that we've taken. But based on what some of the users have requested in response to using Thunderbird Assist, that's given us some ideas of like, well, what we should be focusing on is we need things like semantic search. Or because it is a Thunderbird add-on and has access to more than email, like it has access to your calendar account that's in Thunderbird. Thunderbird also does task management and it even pulls in RSS feeds, which RSS, it's coming back so strong, I love RSS.

• 40:17Speaker 03

  I think that this idea of like, okay, so if we could correlate between these different pools of information, that could be extremely useful to some users, which brings us back to the whole like, okay, well, you need, you need small dedicated models for, for each kind of data because they're going to be formatted very differently.

• 40:37Speaker 01

  Well, Chris, I, I have sort of two, well, an observation and then maybe a clarification. So number one, I, I love how you described this, this progression from kind of the one tool to accomplish every task, which is often how people do think about using these models, down to splitting this out into maybe it could be different models, it could just be different applications of the same model, but that are segmented or these sorts of things. This is so often what I Recommend to people, it's kind of like when you have a junior developer and they come to you and they're like, I wrote all the functionality.

• 41:20Speaker 01

  It's all in this one function, right? Thousands of lines of code and, and you're like, okay, we, we need to split this up, right? It's sort of in this AI world, there's that need for that splitting, splitting up. And of course it makes things more testable and, and all of that as well. But so thanks for highlighting that. I think that's a really, really practical and good point. I think the, the clarification, just want to make sure that people picked up on you. You kind of referenced some of this work with Flower and we talked about that remote inference. If I'm understanding right, because you're running a local application, the data that flows to that remote inference is encrypted on the device. So it's encrypted in transit.

• 42:05Speaker 01

  And then if I'm understanding Flower's implementation, you can correct me if I'm wrong. That would only be decrypted sort of in a confidential enclave in the inference infrastructure, right? So that's when you say like even Flower, even if this is running in their infrastructure, they would not be able to tell you what a prompt is. Did I pick up on that somewhat in the right vein?

• 42:29Speaker 03

  Absolutely. That is, that is totally correct. It is for any web developers or, or, you know, anybody who has had to write software that interacts with an API. You're probably communicating over something called HTTPS, which is sort of like, okay, that is, that is the baseline amount of encryption that we want. It's going to encrypt the traffic between your browser, the client and then the server. They take it a step further. There is, there's a, I would call it a three-part process for making sure that your, your data is protected. So first off, let's say you're logged in, right?

• 43:08Speaker 03

  You, you log into your, your Thunderbird account, which we created specifically for Assist and some other services, which I will talk about a bit later, then. You are issued an API authentication token from Flower itself, right? You're logged in, you're now going to talk to Flower. Flower's like, "Yep, you logged in through Thunderbird. Cool. Here's your authentication token. Use this now to exchange public keys with yet another server." And that server does nothing besides run the, you know, run the language model.

• 43:47Speaker 03

  And at that point, as you, as you observed, that any traffic between your client, between Thunderbird Assist specifically, and then the machine running the model, like, it's all encrypted in between. So you have, yeah, you're double protected, I guess? HTTPS plus the, the public key encryption between you and the, the model server.

• 44:14Speaker 01

  That's great, yeah. And I think this is a great way to maybe expand people's thought process around what's possible with privacy and LLMs, and how that can be split up between like where the LLM is running, whether that's local or not, or, or both. So yeah, appreciate, appreciate you going into a few of those details. I think it's really helpful. As we, as we do get closer to the, to the end here, I would love to maybe just kind of ask you to close this out by, by thinking about, you know, the future, now that you've run these experiments, you've kind of gone through this process.

• 44:54Speaker 01

  I love how we kind of went through the, this kind of story of how this developed. Now, now that you've gone through that process, as you look towards the future, what, what excites you about where things are at now and where they're headed in, in this project? Or maybe in terms of like the wider ecosystem that you're now a part of, using this tooling around kind of remote confidential inference and that sort of thing?

• 45:23Speaker 03

  Yeah, there are a lot of exciting directions that we could take this work. Again, this was sort of an initial experiment, but we are planning on shipping this with what we're calling Thunderbird Pro, which is a suite of services. Like my other web developer teammates, they're working on other things like a scheduling application, a web-based scheduling application. There's an end-to-end encrypted file sending application. And there's Thundermail, I'm just going to say that again, Thundermail, which is our very own email service.

• 46:04Speaker 03

  Okay, so one of the things that could be very interesting, and perhaps even take advantage of federated learning, thanks Flower, is if you could treat the server, the email server, or you know, another machine that is co-located with the server, as another client, right, that has access to your encrypted email that's on Thundermail, that you know, while you're asleep, or while you are disconnected from the internet, it could be creating embeddings or doing some other inference based on your email data, and then transmitting the learnings to your local machine.

• 46:47Speaker 03

  So that, imagine if you could do semantic search without having to generate the embeddings on your laptop. And you can do it in offline way, because as far as you're concerned, the embeddings are effectively pre-generated and downloaded along with the messages themselves. The trick there, of course, is doing it in a way that will satisfy, you know, the most staunch privacy advocates. They're like, wait a minute, if you have a server that's in your infrastructure and has access to my email, then it's really not end-to-end, it's not encrypted. So we need to figure out a good solution to that before we can explore that. But some other things that I alluded to earlier involve expanding

• 47:29Speaker 03

  That's such an overloaded word, expanding the context that the model has access to. And I don't mean like, you know, context window. I mean, like, okay, so giving it access to your calendar, to your to-dos, your RSS feed. What if we added a notes application to Thunderbird? And then effectively turning, making it possible so that Thunderbird could be used as an LLM assisted personal knowledge management and communication tool. That whatever future that looks like, that's, that's more exciting to me personally. I'm one of those people who, like, I have notes from the last handful of, like, a last couple

• 48:13Speaker 03

  of decades that I still keep around, and I would love some, an LLM to help me sift through that. It'd be even more interesting that as I'm making a note, it could suggest related documents and ideas that I've had in the past, or just, I mean, for a lot of users, just helping them stay organized. Because again, there's, there's so much for one tiny human brain to keep track of, and there's just so much information. So I think that for me, as, not as, you know, an ML researcher or an AI expert, I'm just an application developer. I want to work on that. I want to build that and make it possible for people to have more control over their

• 48:58Speaker 03

  information, help them retain their privacy, but, you know, make those creative connections that only they as a human can do. But an LLM, local or, you know, confidential, remote compute assisted, reminding you of like, oh, you wrote this. Here are some things that you've written about that are related to that, or here's some conversations you had in email or in chat. And then for you, the user, as a, just a regular squishy brain human, you're like, oh, I just had this weird random flash of insight based on this constellation of

• 49:35Speaker 03

  information that I generated over years. I think I like that future of AI. And also as an application developer, I think that I really want LLMs to be more deterministic. Like, it's so weird to call an API with the same data and get very different results. And I, we can get into this or not, but I definitely feel like chat is the wrong interface for a lot of tasks.

• 50:06Speaker 01

  Yes, thank you.

• 50:07Speaker 03

  Okay, cool. Just want to make sure I'm in good company. So I've got, I don't know, again, in my little mad science notebook, I've got ideas around like, okay, how do you, how do you swap in deterministic functions? How do you coordinate the efforts? And I think something beyond, I mean, maybe I'm describing a more strict version of MCP, but the fact that your input and your output currently is plain language is, it's a double edged sword. Because the only way to determine if you've got a bad result is for you as a human to evaluate it.

• 50:48Speaker 03

  Unless you spin up another language model to verify the first result. But as a programmer, that feels a lot to me like, oh, I just wrote a function. And the only way to know that if my function call was correct is to write another function to check it. And it just feels like, it just feels wrong.

• 51:08Speaker 03

  So I really, yeah, I want, I want discrete inputs and outputs. I want language models that are small and dedicated to specific tasks. And then I want reusable, shareable ways of wiring them together. I want to create essentially like workflows of information processing within, within Thunderbird. So that's, that's me. I'm, I'm the personal knowledge management cheerleader at, at Thunderbird. That's my new title.

• 51:41Speaker 01

  That's awesome. Well, I, the, that future is a, is one that I could get on board with for sure. After, after struggling with a lot of the things that you mentioned as well. And, and also hoping for, for many of those things. So yeah, thank you so much for sharing this journey and this experimentation that, that you've been on with Thunderbird. And please keep up the good work. Give our thanks to the, to the team for inspiring us with, with a lot of amazing work. And thanks for sharing your, your insights here with us. Appreciate you taking time.

• 52:14Speaker 03

  Yeah, thank you so much for having me. I'm really, really glad that we could make this happen.

• 52:20Speaker 01

  Us too. We'll see you soon.

• 52:21Speaker 03

  All right. Thanks.

• 52:30Speaker 00

  All right, that's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.