Creating instruction tuned models

At the recent ODSC East conference, Daniel got a chance to sit down with Erin Mikail Staples to discuss the process of gathering human feedback and creating an instruction tuned Large Language Models (LLM). They also chatted about the importance of open data and practical tooling for data annotation and fine-tuning. Do you want to create your own custom generative AI models? This is the episode for you! Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Featuring: Erin Mikail Staples – Mastodon https://mastodon.social/@erinmikail, X https://x.com/erinmikailDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Label Studio https://labelstud.io/Slides from Erin’s recent PyData talk on RLHF https://docs.google.com/presentation/d/17GDvbYAf3SfuNTRNzbuX5We3mVKBwnAE-wX5I1XOP6w/edit?usp=sharingUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users, no ops required. Learn more at fly.io.

• 0:42Daniel Whitenack?

  Hello, this is Daniel Whitenack. I am here onsite at ODSC East in Boston, the Open Data Science Conference, and I am super excited because I get to sit down with Aaron McHale Staples, who's a developer community advocate at Label Studio. And yeah, what do you think of the conference so far?

• 1:03Speaker 02

  It's been, first off, a super fan of what you've been doing and a lot of anybody who's creating stuff out there in this space, especially with the current zeitgeist and explosion of interest in AI and machine learning.

• 1:15Daniel Whitenack?

  It's a little crazy.

• 1:16Speaker 02

  It's a little wild. It's a little wild. I would be lying if I wouldn't be saying I'm newer to the field myself, but it's been something I've been very fascinated about. But all that being said, this conference is really cool to see just the breadth of, first off, people here. There are people who are very new to the industry, people who came to just learn more for their first time, but there are people who had been practicing for years and years, and this is their third or fourth time at ODSC. And I'm also really interested about the number of people concerned about data integrity here.

• 1:47Daniel Whitenack?

  Yeah, lots of interpretability, integrity, reliability type talks.

• 1:53Speaker 02

  Yeah, lots of reliability. The other one is also on missing data and how do we approach these problems, especially with the rise of foundational models and generative AI? How does that impact it for the long, which is crucial conversations, I think, to have?

• 2:06Daniel Whitenack?

  Yeah, definitely. And what sorts of different players in the space are you seeing at this conference, both in terms of open source or different targets, like ML ops platforms, that sort of thing? How do you see that developing?

• 2:23Speaker 02

  First off, I'm personally a huge fan of open source. It's not only how I learned to code in the first place, but just a big believer in the ecosystem. I'm a huge believer in open data. I'm a participant in Open Data Week. So I think all of these things are like-

• 2:37Daniel Whitenack?

  And you're wearing a Pi Ladies shirt, which is awesome.

• 2:39Speaker 02

  Yeah, I'm a member of Pi Ladies. So again, super important, I think, to have all these things in the ecosystem. But one of the things I think that stands out is there's so many new innovations that like, if you're starting a tech stack from ground zero, it's really fun to see all the different players in the game. So selfishly, working at Label Studio, one of the best things about being in the space right now is we're a cool platform because we can integrate with so many different data types. That it means that I always get to play with almost every other tool or workshop or players in the ecosystem, which is selfishly fun. It means I get to have more things to integrate with or build. As always, we're huge fans of what the Pachyderm team is always doing. Work very closely with them.

• 3:24Speaker 02

  We work very closely. We've got a lot of friends and fans, the DVC crew. They're not here at this conference round, but did get to work with them at PyCon, which was really amazing to see kind of the work that they're coming out with. The Condo crew is always fun to see around. So it's always exciting.

• 3:40Daniel Whitenack?

  Cool. Yeah, yeah, there's so many awesome things going on. And I've seen like, maybe like three or four open source packages that I like, I don't know if I've been ignoring or haven't heard about. So that's like one of the fun things about coming to these things. I know also you gave a recent talk at Pi Data Berlin about reinforcement learning from human feedback, I believe was the topic. Could you tell us a little bit about like the general like pitch or angle on that talk, which is definitely like a key topic these days with all the instruction tuned models that are coming out and all of that. So what was your kind of angle in terms of what you're thinking about there?

• 4:19Speaker 02

  Yeah, so I think one of the cool things is it was a talk that Nikolai and I gave, and Nikolai is the CTO and one of the co-founders of Label Studio. And what we did at Berlin is we really made sure to expand on this idea that like, yes, these generative models, these larger models are kind of becoming the norm. I think we met yesterday, I was talking to someone who's like, I got interested in AI because I made a thousand things with Mid Journey. I'm like, cool. And I'm very fascinated by, and I'm a believer, like, I don't care how you got into it, but just the curiosity to show up to a conference and learn more is very fascinating. But explaining to someone how it works and then also explaining the best practices behind it is really important. Personally, I have a journalism background. I have a liberal arts background. And I think

• 5:03Speaker 02

  it's really important that we incorporate the humanities in technology for the long run. And so when it comes to reinforcement learning, all of these large generative models, they can all be made just a little bit better with the human signal that we can provide. And we can say a lot of things like, you know, get into prompt engineering, which is another whole other topic, but it will never be as good as like if you can retrain your own data set with subject matter experts or to a specific use case or condition that you're trying to output that data towards.

• 5:34Daniel Whitenack?

  Yeah. And I think one of the things that's been on my mind recently is like this topic, reinforcement learning from human feedback, especially with what's gone on with ChatGPT and all. Sometimes it feels like out of reach for like day to day data scientists. Like, how do I, like, I could leverage this model, but what is the tooling around reinforcement learning from human feedback? Like, how could I use that framework or use tooling around that to like, impact my own models or my own life? Like, how could I connect my domain experts' input and their preferences into a system that I'm designing? Do you have any thoughts there?

• 6:11Speaker 02

  Yeah. So one of the examples I love to point to is actually Bloomberg did this, and they probably did this early April now. And they took the financial data that they had and Bloomberg from all the way back from like, you know, many of us know about it, Bloomberg from Bloomberg News, but that was actually the financial terminal that was used for stock trading. But they have these, I mean, mass amounts of financial data. And how do they stack on top of it? Like, how do they get and access that data even faster and kind of train it to the best use case that we have? Like, currently, our larger models can't do that. They're not experts in financial data. They're not combing just financial data. But what Bloomberg did is they took and they retrained and they built the things. I probably fangirled over, sorry if you're on the Bloomberg team and I fangirled over you at PyCon, because I

• 6:55Daniel Whitenack?

  Great work.

• 6:56Speaker 02

  I was like, I was like, this is the coolest thing ever. I use this as an example. Also, I like learn machine learning off of your repo. Okay, thanks. Bye. But we do have a model. If you want to learn and see reinforcement learning in action, there is an open source repo. It is built by myself, Nikolai, and Jimmy Whitaker, who is, we have as a data scientist in residence at Label Studio and Hartex, but also is at Pachyderm as well. But all of that is built. It's open. You can play around with it. It's based off of GPT-2 right now. So go have some fun and get your hands dirty. And it's all runnable within a Google Colab notebook.

• 7:30Daniel Whitenack?

  That's awesome. Yeah. What, what is, so you mentioned it being run in a Google Colab notebook, which I think is awesome, and using also a bit of a smaller model to start with. And we've seen a lot of kind of directions towards smaller, open models that are accessible to data scientists with like Llama and other things like that. How do you see that trajectory going? And how will that impact sort of like day-to-day practitioners in terms of what they're able to do with, with this sort of technology?

• 8:04Speaker 02

  I think the biggest thing in, I'm actually going to zoom out to answer this is, the biggest thing we need to think about is context. Like, what are you using model to solve, or AI to solve, or ML to solve? And the more that I've been diving into these conferences and the ecosystem, especially at a conference where it's a blended conference, where you have folks that are not necessarily deep in the field or ML practitioner, or they're like new to ML, it is so easy. And there's a meme I always point to that it's like, oh, it's, you know, we're an AI back so-and-so. And it's like, JK, we're just an AI. We're just, you know, calling the API and put a nice, pretty shiny front end on it, which is no shade to anybody who is putting a front end on a GPT API. Like, there is no shade at all to that. But it's like, think about what you need a model for in the first place,

• 8:46Speaker 02

  or what you want to use machine learning. Like, that context is so important. I'm currently playing around with a Naked and Afraid dataset, just to like, play around. There's an open source dataset out there that is

• 8:57Daniel Whitenack?

  That's, that's awesome. Like videos?

• 8:59Speaker 02

  Uh, like, no, it's context from the TV show of how many days they survived.

• 9:03Daniel Whitenack?

  Oh, so literally, yeah, like, statistics about like, and features of the different survival situations?

• 9:09Speaker 02

  Yeah. It's like, it's like, country, their name, gender, and then how many days they made it.

• 9:15Daniel Whitenack?

  Yeah.

• 9:16Speaker 02

  And like, climate.

• 9:17Daniel Whitenack?

  Yeah. Based on that, like, yeah.

• 9:21Daniel Whitenack?

  That's so intriguing. I watch, I watch a lot. So confession, I watch also the Alone show, which is like another survival show.

• 9:29Speaker 02

  Oh, yes.

• 9:29Daniel Whitenack?

  This is like, I'm a huge fan.

• 9:30Speaker 02

  I'm a reality TV terrible junkie. I, that is how I distrust is reality TV, but.

• 9:35Daniel Whitenack?

  So I always wonder, I have this conversation with my wife around like, could I do it? And maybe with a model trained off of, off of your survival data set, I could like, say, I'm from here and this is my background. Could I survive?

• 9:49Speaker 02

  Yeah.

• 9:49Daniel Whitenack?

  I don't know.

• 9:50Speaker 02

  And I can't take credit for the original data set. It is someone who I have made friends with in my reality TV subreddit. So you need to know where I spend my time. But he runs a SQL database. It is actually very good. He's very awesome at updating it. It's available on Reddit. I can share it with you and you can post it in the links. But I'm just kind of playing around with the data set for fun. But in this context, like I'm playing around building demos and just having some fun, teaching myself some new skills. I don't need a large foundational model for that. And I think going back to like your original question of like, well, all these models are getting smaller, more accessible. We can run it in a notebook. We don't need the high powered computer models everything single times. And if we stop and think about the context of the problem that we're trying to solve, it can give us a lot of answers and it can save us time, energy

• 10:34Speaker 02

  and computing power. And I think that's why I get really excited about being on the data labeling side. Again, I have a background in humanities. I'm a self taught programmer, but I think, I don't want to be like, we need more people like me in data science, but we need more of the humanities in data science because we're missing the context.

• 10:48Daniel Whitenack?

  Yeah, we recently had a guest on the show that was talking about like the intersection of art history and computer science and how computer scientists who are analyzing and doing computer vision could actually learn a lot from like what we know about like art and like how scenes are composed or how art has changed over time and how the features that they're actually engineering are connected to some of those things. So yeah, I think that there's a lot of different areas where this could apply and domain experts are so important. And I assume that with all of this like reinforcement from human, reinforcement learning from human feedback, I always mess it up.

• 11:30Speaker 02

  I've been, it's okay. I've been doing the same thing. It's like R L H F. I get it.

• 11:34Daniel Whitenack?

  Yeah, especially since you're from the label studio side, do you have any, could you give like a general picture or workflow for people of like, hey, I maybe want to take one of these models, GPT-2, Llama, MPT now, whatever the one is. But I also want to gather some domain expert feedback and eventually get to some type of like instruction or fine tune model off of that. Like, could you just give a general picture for like what that looks in today's world?

• 12:07Speaker 02

  Yeah, and we'll try. This is like always, I feel like this is better when you have a whiteboard and a diagram and some arrows.

• 12:12Daniel Whitenack?

  Oh, for sure. Yes, it's hard.

• 12:14Speaker 02

  I'll do a quick walkthrough. So first off, you'll create a sort of prompt. So typically, these models work with a prompt. And then you're given a large language model. And then you start to train it. Usually what happens when you're training these models is you get a set of two outputs. And so in this case, we can use, what is opossums? Because we're opossum fans at Label Studio. I feel like that's natural. And you can be like, an opossum is a marsupial creature, or an opossum is a great character for memes. Technically, both of those are correct. But depending on context, and this is where that human signal side comes in, one answer is more correct than the other. So if we were training, let's say, an opossum meme bot thing, or a meme bot generator,

• 12:56Speaker 02

  let's go that direction. We'll have some fun with it. We would take the latter answer of this, a possum is a great animal to make memes. And that would be the better answer. If we were going for what type of animal are we doing, like maybe a biology assignment homework? Probably would pick the marsupial one. But this just gives insight of like, the details that you give your annotation team can really directly influence the model. That's the labeling side. When we move this on, all of this is put through kind of this results from human feedback. Your answers are ranked. I did a binary situation, so just two options, but you can have a multitude of options that you put in. It is all weighted. It is then looped back around, this is when you wish it had the whiteboard, to a reward or a preference model. And this reward or the preference model kind of tells you like, hey,

• 13:40Speaker 02

  I probably want to go for answers that look like this. Now, computers don't speak memes, or marsupial, or biology textbooks, but they do know patterns and trends, which is like what they pick up on. So based on that context clues that we give them, this preference model will start to preference those type of answers. Now, it's really important that these reward preference models also hold in place kind of the original things that we had that it knows, like how is language structured, or other things from our original model that we enjoyed, like, we liked, like, language is always structured like this. Here's a proper noun. We like to capitalize the first letter of the sentences. Things that are important, but like, we kind of overthink sometimes when talking about generative language models, at least.

• 14:24Speaker 02

  After that, we want to make sure that we're not just gaming a system. Models are, again, I don't think models are sentient. They're kind of just like math numbers. They're just trying to game a system. They're playing like, I always compare it to like, you're trying to like, it's Moneyball, essentially. I'm a baseball fan here, so it's Moneyball. You're statsing out the system. And they, in order so that they're just not giving you what you want to hear every time, you'll have to calculate an error rule in there. So we put an error metric or an update rule, and it basically says, alright, we're gonna almost like dunk you down a little bit. So you're not too perfect. And that'll prevent unwanted model drift. Then, once you've done that a few times, you'll combine that with a copy of your original model that you had. Again, you're kind of doing that checks and balances, making sure it doesn't run away.

• 15:08Speaker 02

  After that, you will have a tuned language model. And then rinse, wash, repeat, until you've got that model right where you want it. Set it off to production, and then talk to your friends of the other parts of your ML Ops ecosystem.

• 15:19Daniel Whitenack?

  Yes.

• 15:20Speaker 02

  And it'll come in handy.

• 15:21Daniel Whitenack?

  Awesome, yeah. And I hope that we can link some of your slides from that talk in our show notes. They're awesome, including emojis and the full deal, which helps. So make sure and check out the show notes. The link to the slides will be in there, so you can take a look at these figures while you're listening to the show. One follow-up question. Question on this, we're talking about like gathering this feedback data. People can think about like, okay, in the context of my company or where I'm working, I'm going to gather some of this data, you know, tune a model. But what is your perspective on sort of like the open data ecosystem? And like, what would you encourage people to think about in terms of like,

• 16:03Daniel Whitenack?

  data that they could make openly available to help others who are also trying to do this? Or the other way around, like people that are searching for maybe a place to start? What is the sort of open data ecosystem look like right now? And how important is that as this sort of field advances?

• 16:20Speaker 02

  Yeah, first off, this is you've got me on my other favorite soapbox of the moment. And this goes back to my days when I was, you know, a journalism student, working in journalism. But open data is one of my like favorite topics to geek out on. Basically, it was something that really came actually as part of the Obama administration. He actually established federal funding for a lot of our public and civic data as a part of government accountability and transparency. So there was actually federal grants that went out to make a lot of our civic data public. So there's a really cool example. I believe it's the city of Philadelphia that actually built a SimCity-like game off of their public data. It's so cool. It was like a grant given. Super fascinating. I'll link it to you and give it, it'll be in the plethora of show notes on all of that.

• 17:05Speaker 02

  But open data is just open, freely accessible, freely used data that is made available to the public. Love open data. I'm a participant in Open Data Week. But when it's, it's been federally funded, it's not always the best thing to be federally funded. And like, we all know how government grants go. And if you aren't aware how government grants go, they're very niche, specific, and they run out. And they're not always maintained, and it's not always the cool sexy job that we have. So they're always not the best maintained or context applicant.

• 17:33Speaker 02

  What a lot of these early machine learning models did, and what a lot of machine learning models is, is these open data sets have given opportunities for people like myself to even learn how to do data science. I learned Python in Open Data Week. I remember going back and like, let's get the traffic data in New York City. And it's like basic, you know, using curl and like getting things started for the first like, can you query an API? They're not the most organized data sets out there. They're not the most clean. Sometimes you get some really messy garbage data. You know, the 2020 census is actually a great example. I was speaking to someone yesterday at the conference about this. The 2020 census was the first time that we were able to do it digitally. Well, she gave the example of like, you know, hey, I started the census on my phone. Oh, no, the pot boiled over.

• 18:17Speaker 02

  Oops, I accidentally counted myself twice in the census. Or I didn't fill out my address. Or now I've got two people or a person who lives at this address or a typo. Crap. Now that's a very messy data set. So open data can be a problem. Let's go to like the practical application of this. If you are working in open data or you are interested in getting more involved in open data. One of my favorite sources is like, if you're publishing a story, making tutorials, making content, put your data out there and put how you processed it. And it's not just like one thing to put like your data out there, but also how you processed it. In journalism, you have this phrase, how you frame the story is how the story or is how you tell the story. Leaving out details, context, or even how you came across a source can influence how the story comes across.

• 19:02Daniel Whitenack?

  Yeah, for sure.

• 19:03Speaker 02

  And that's, we see it especially evident in data-driven journalism and solutions journalism, which is interesting and like it's also can be really damaging to trust and reputation. But I think ML runs the same risk right now. If we're not transparent of here's how I prepared the data set. Here's how I trained an annotator or here's the tools that I used. Or here's how I obtained the data in the first place.

• 19:25Daniel Whitenack?

  Yeah, and like you were saying, the certain things like how you give instructions to a data annotator or how you set up your prompt, that has such an influence on the downstream performance of these things. But it's very frequently, I've definitely found like the instructions you give data annotators is something very often left out of the story of how people tell like what they did, right? It's like, oh, we gathered this data with these labels. Okay, well, I can imagine like my own set of instructions for getting those labels, but it could result in a totally different thing that's happening. Like all sorts of biases and other things that go into that, so.

• 20:05Speaker 02

  I mean, well, I have a perfect case example of this. In January, we met many of the team members at HRTechs and Label Studio met up. Basically, we got our entire customer success and sales team and the community side of things and a bunch of our support engineers to all sit together and we had a data labeling competition for fun at the end. And I had just finished, like, how to get started with data labeling and like best practices. And I was like, easy. Like, I'm gonna kick all of your butts. Like, I was totally going in like hot shit and everything and like thinking. Well, I sped through. I was like, whatever. Next. Great. Done. And I like sped through because I was like, speed was a metric, but also accuracy. Well, I sped through this thing because I was like, whatever, I'm gonna ace this. I know the keyboard shortcuts.

• 20:47Speaker 02

  Like, my systems are set up. I had the lowest accuracy score, everybody. My data was all like, you failed, Aaron. It was like, I was like, man, I'm gonna go embarrass myself right now after all that crap I just talked.

• 20:59Daniel Whitenack?

  Yeah, yeah. I think that's the other thing. Like, I don't know if you have any encouragement here, but like, data scientists out there who have not actively participated in the data labeling process, I think, yeah, that's like such a learning experience because it gives you perspective, even if in the future, you're not part of one of those processes, it gives you good questions to ask. If like, oh, someone gives you this data set that was labeled, you should probably ask a few follow-up questions about like, how did that go? What did you do there?

• 21:33Speaker 02

  Well, in academic research, you actually have to disclose things like, did you pay your annotators? Or how did you prepare the annotators when you're doing research? Because that can put so much of a bias on a model that is built off of that data. And that academic, like, you can't get peer-reviewed studies done without disclosing that information. It's part of data ethics now. And one of the biggest things, and we don't talk about it enough, is how do you pay your annotators? Or do you outsource your annotators? Which isn't saying that's a bad thing to do, but again, we have to remember that so many of these models, and I see it, like, I think a lot of it times, it actually is probably, I'm gonna guess here, I don't know, but I would be even wondering if these smaller models that are generated because they're generated at home, or people dorking around on their computer. They might even have more bias because we're not trading an annotator.

• 22:18Speaker 02

  Like, I know when I'm goofing around with my Naked and Afraid dataset, I'm not annotating. Like, I'm just playing some 30-second goofing around stuff and watching YouTube videos, just seeing what's out there. I'm not doing the work, which is a problem.

• 22:32Daniel Whitenack?

  Yeah, I guess kind of bringing things full circle a little bit. Like, we started talking about like some of these like players and ML ops and sort of the ops around this process. We talked about human feedback, reinforcement learning, we talked about open data. What excites you about like the trends that we're seeing and what impact they could have on our industry moving forward? Maybe that's related to like people that weren't able to like participate in this process before, the tooling's better so they can, or maybe it's something totally different like around like tasks or other things that you see like in the future. Like, what are you personally excited about, like, looking forward as you bring this stuff together?

• 23:15Speaker 02

  Well, first off, I've been really impressed with like what the Hugging Face team is doing. I noticed the Hugging Face shirt. The Hugging Face spaces have been amazing. We do have a label studio, a Hugging Face space, but the ability to get up and going in the browser has been super awesome. And there is a talk I went to at PyData Berlin that's running Streamlights. They're running entirely like Python-based models right in the browser and tools. I think there's a, it's Binders, I believe, is another tool that's doing another, again, very similar to notebook processing, all in the browser, makes it more accessible than ever before. And it's just really exciting, especially as like, I love that we have more people interested in this industry, but it's also not only the interest, but the tools to do it correctly and ethically.

• 23:57Speaker 02

  And again, jumping on my soapbox here, this is why the open data is so important. So putting, when we can, putting our sources, our references, building in the public, building in open source and making kind of a, almost, I don't want to say a paper trail, but like a show your work sort of process is really important for the future.

• 24:15Daniel Whitenack?

  Yeah, awesome. That's great. And as we kind of close out here, where can people find you online? And also, tell us a little bit about your own podcast, which sounds awesome and like includes pickles.

• 24:28Speaker 02

  Yeah, so I am available online at Aaron McHale on all the platforms, or Aaron bio has a link to everything that I'm at. You can also chase me down at label studio. So it's label studio, but the last dot io is like a thing. Join the community, come hang out with me there. We have an open coming upcoming town hall and getting into more workshops. So very excited about that. I also run the Dev Relish podcast. So it's everything about Dev Rel and ish. Also, you know, naturally, some people made sourdough bread. I got into fermentation. We got a fun pickle fact and cool pickle logos because you got to relish the developer moments.

• 25:10Daniel Whitenack?

  Well, yeah, I this was definitely not a sour experience. I've relished it very much. Thank you so much for joining, Aaron. It's been a great pleasure to talk to you and looking forward to following up with all the cool community stuff you got going on. Again, people, check out the show notes and thank you so much.

• 25:30Speaker 02

  Thank you so much. This was a quite a big deal that we had going on here.

• 25:35Daniel Whitenack?

  Good one, good one.

• 25:45Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Chang Talk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.