The last mile of AI app development

There are a ton of problems around building LLM apps in production and the last mile of that problem. Travis Fischer, builder of open AI projects like @ChatGPTBot, joins us to talk through these problems (and how to overcome them). He helps us understand the hierarchy of complexity from simple prompting to augmentation, agents, and fine-tuning. Along the way we discuss the frontend developer community that is rapidly adopting AI technology via Typescript (not Python). Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Travis Fischer – GitHub https://github.com/transitive-bullshit, X https://x.com/transitive_bsChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: ChatGPT Hacker Community https://www.chatgpthackers.dev/ChatGPTBot https://twitter.com/ChatGPTBotUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist building a tool called Prediction Guard. I'm joined as always by my co-host, Chris Benson, who's a tech strategist at Lockheed Martin. How you doing, Chris?

• 0:57Chris Benson?

  Doing very well. We're still in 2023, the most exciting year in AI history.

• 1:03Daniel Whitenack?

  It is, you know, it's hard to keep up, but it's also sometimes hard to understand, like, what of these cool demos and models and integrations are like actually production-ready? And how are people actually taking these things into production? And we're really happy to have with us today Travis Fischer, who is a founder and CEO at a Stealth AI startup and is focused 100% on that, delivering products with AI. So we're happy to have you here. Welcome Travis.

• 1:36Speaker 03

  Thank you, guys. It's a pleasure to be here. Looking forward to the conversation.

• 1:40Daniel Whitenack?

  Yeah. Well, on Twitter, you posted this diagram, which I think maybe you have penned right now, which is how to use large language models effectively. And it's sort of like a start simple to complex scale. And I found that really great. And I've actually shared that diagram with a number of people in various Slack channels and all. This is how you should be thinking. How did you, maybe not specifically with that figure, which we can talk about, but like, how did you get into thinking about how to use large language models effectively? Like actually how to build products with these models?

• 2:18Speaker 03

  Yeah, it's a great, great question. So I'll give you kind of my, my quick, what I've been up to in the last six months, which is going to answer some of this stuff. I'm a huge fan of open source. Uh, when ChatGPT launched on November 30th, uh, 48 hours later, I released the ChatGPT NPM package, which was using unofficial API and it just allowed like thousands of developers to go and start building with this cool new thing that, you know, like, uh, the GPT series and L Ls have been along for a while before that, but it was kind of this step function in terms of just their mainstream adoption and really just caught people's attention. You know, after that, I released, uh, the ChatGPT Twitter Bot, which now has about 123,000 followers.

• 2:57Speaker 03

  Uh, I run a group called ChatGPThackers.dev that has, um, about 9,500 AI developers, just a whole spectrum of people, right? We have like researchers in there and then we have like prompt engineering script kitties. And because, I mean, my background is computer science and I do have some formal education in machine learning, but it's not like I'm an AI expert, like my identity means, right? And what has really captured me really over the past year or so has just been the rate of progress and trying to wrap my head around it and understand it. And because it's been moving so quickly, I've been optimizing for my rate of learning. And I personally learn best by building out in public and building open source and just sharing what I learn as I go. So, you know, I think

• 3:38Speaker 03

  there's a lot of like complexity and terms in AI, as you guys know, and to some degree, even just having a mental model of like, what are the different approaches that you could start with? Or how do I approach solving a problem is already like a difficult starting place, right? And I know, um, for the how to use L LMs effectively, my inspiration there was, uh, Andre Caprati, uh, recently, maybe a month or two ago tweeted, uh, something about, you know, all these big companies are interested in using AI. They're aware that they should be using it to some extent. And so they're like, well, we need to hire a team of ML engineers and get on this, right? And the huge unlock now is with these foundational models, like for most problems, you don't need to do that.

• 4:19Speaker 03

  And, you know, yeah, there's the production side, the practical side and I'm sure we'll get into that, but like in terms of where to start and starting simple and actually validating for your business. Use case that you can actually solve it with AI that you understand the problem domain enough that you have, whether it's a training data or that you're actually solving a real customer a problem like, starting as simple as possible with hosted foundational models a lot of times is a great way to get started and just to validate quickly. You know, as you kind of inevitably find points where your workflow breaks down or where you're not getting a quality or the cost or some hard constraints like security, privacy, you know, of your data. There's kind of this ladder of complexity I like to look at and you know, you start with just prompt engineering up at the top and

• 5:04Speaker 03

  then it's about like, well, how much can I reduce hallucinations or or add domain specific context into my prompt by, you know, doing information retrieval. And then, you know, at some point you're like, well, a single prompt isn't doing it. So maybe I add in some iterative process to that where I use another language model to, there's all these techniques for doing kind of multi-step prompting. But you can do all of that with a hosted model and you can get like 95% of the way there for a lot of problems and domains these days in a way that was previously like locked behind, you know, proprietary data providers and you had to have so many resources to be able to do that. So it's really this like democratizing point in in the industry at the applied AI level that we're at right now.

• 5:46Speaker 03

  And I think from the conversations I've been having with folks who are a lot of them, you know, like, like full stack typescript devs who are building applications, right? And they want to use AI. They know it's cool. They don't know how to get started or they're like, oh, I need to learn Python. I need to train these custom models and stuff and and like, all of that is super important and it comes into play at a time or for particular types of problems, but the majority of people for getting started like, uh, start simple is the main takeaway from that.

• 6:13Chris Benson?

  I think that's a great insight and I think that's a one of the places where so many people go wrong is jumping into too much complexity. They don't find a simple need and potentially even don't look for things that work just as fine that are not AI in that way. So I love, I love the go simple and build from their philosophy. I think that's incredibly practical.

• 6:35Daniel Whitenack?

  I get the sense that a lot of this sort of like chaining and like bringing models together, doing the information retrieval, it's sort of like, almost like a hacking culture around this language model prompting, which is really cool and like that can go so, so far. Maybe there's like you say, there's like privacy or domain specific concerns with like enterprise use cases, but in your, like you mentioned the community that you've kind of built up and you're part of on Discord, what are some of the things that have maybe like surprised you that you've seen that, hey, I didn't even think that maybe this was possible with just

• 7:17Daniel Whitenack?

  this layer of like using a hosted model, using pre-training, using retrieval, whatever it is, what are some of those things that that you've seen that kind of surprised you or maybe like helped develop your thinking around this topic?

• 7:33Speaker 03

  I have a few examples in one story. Examples would be folks who are taking these models and like applying it to their personal finances, right? And there's one guy in our Discord, uh, who's like an ex hedge fund guy and he created, um, a very basic agent that uses a large language, I think it's probably using GPT-4, to, uh, take this unstructured information from his bank's website about his expenses and like, you know, extract structured information about that and then, you know, he can graph it and whatever. So I think there's a lot of hacking going on around this stuff. It is very, very early. Another story of something that surprised me, and this is just a fun story, but when I released the kind of unofficial API wrapper for ChatGPT, we kind of had this cat and mouse game going

• 8:17Speaker 03

  back and forth with OpenAI for a while, because apparently there was kind of a group within OpenAI that was like, oh, this is amazing. Look at the open source community is doing. They're building all this cool stuff. And then there was another group that was like, well, we're going to have the official API eventually. We want to control this stuff, right? So there was kind of this, this back and forth, right? And at one point, our community found a, uh, public model, but it wasn't like publicly disclosed. It was security through obscurity, but it was a fine-tuned chat model that ChatGPT was actually Using at one point and all of the open source projects started to use this thing. And there were tens of thousands of actual real consumers at the end, you know, who were building on top of this. And of course, OpenAI knew that we were doing this. I talked with one of our security engineers about this after the fact.

• 9:00Speaker 03

  And they, instead of like what you would expect, just shutting it off. Instead, they switched it out with what they call CAT GPT. And just all of a sudden, one day in our Discord, we started getting hundreds of messages from users saying, I think I got hacked. I'm seeing all these meows like in response to my thing, right? So it goes to show, you know, the moral here, and I ended up hearing from the OpenAI engineer that they were watching our Discord, taking screenshots and laughing their asses off. You know, at this happening, but it goes to show that like, one, the kind of level of, there's no switching cost to these things, right? It's like text in, text out, fairly basic.

• 9:40Speaker 03

  And there's like entire new venues of like vulnerabilities of like swapping it out with a, you know, a cat or something like, what does security look like in this world when it just thought it was an interesting kind of anecdote?

• 9:55Daniel Whitenack?

  Probably that vulnerability of like all of a sudden getting meows like that is a possibility. But I'm wondering, like, as you've spent a lot of time with these models, you've also like you're building products on top of these models. Like from your perspective, taking an LLM integration to that sort of last mile of like, integrated into a product, supporting users. What are the things that should be on either developers or data scientists minds as they think about like, taking the step from like demo to like product integration, I guess would be the question.

• 10:36Speaker 03

  So I like to say that absolutely everything in engineering is about trade-offs. And it's about really thoroughly understanding trade-offs and then being able to effectively communicate those trade-offs and the pros and cons and everything. And it really boils down to those two things like over and over. So let's talk about some of the trade-offs that are most important to using language models in practice. You have the most obvious one, which is quality. Like, can I use these language models to actually perform the task that I want? You know, you have oftentimes secondary but equally important trade-offs. Like, how much does this cost to run in production? What is the latency for my use case for the end users? How consistent and reliable is it? Like, can I have actual, is my use case fault tolerant?

• 11:19Speaker 03

  Which is a great initial question, because we're kind of moving from a world of like very deterministic, human-driving the program to a world where the more control you give to the language models and their reasoning abilities, this is getting more into the agentic side of things, the more that it becomes slightly non-deterministic or very non-deterministic. And so the ability to have guardrails around these things, the ability to have consistency and predictability is extremely important. And one of the first questions that you should ask yourself if you're thinking about like integrating with LLMs is, for your particular use case, for your job to be done, for your customers, to what extent, you know, do you need 100% reliability versus like 99% reliability? And that may sound like a little bit for certain domains of problems, it's everything,

• 12:04Speaker 03

  right? And so that's one fundamental question. There are techniques, and we talk about them, I'm sure you guys are very aware as well of like going from that 99%, getting close, adding extra dimes of reliability. And, you know, that's also a very active area of research where folks are actively figuring out ways to increase the reliability of these models. But the fundamental trade-offs are, you know, quality, cost, latency, and reliability. And using a hosted model is going to be great for quickly, like with minimal resources and validating your use case. For a lot of those types of trade-offs, it may make more sense then to use a local model. And there's kind of been this Cambrian explosion of open source large language models and other, you know, specialized machine learning. Models and we're going to continue to see that proliferate.

• 12:49Speaker 03

  I like to think of the open source kind of state of the art as, you know, six to 12 months behind the proprietary versions. We'll see if that holds, but you know, it's kind of because there's like zero switching costs with these models. Because there's just so much competition, the prices are going to keep going down over time. We're going to see the open source side of these models continue to get more powerful. And so for a lot of these cases where you're dealing with, well, maybe I need ultra low latency on device. Or maybe, you know, cost is a factor and I need to be running in my own data centers. Or maybe, you know, you need to, after a certain point, once you validate a use case, you want to fine tune and distill the model down. And have a really locked in, like a checkpointed, this is like completely unit tested. This is, you know, evaluated version of things.

• 13:34Speaker 03

  And I think we're at the stage right now where there's so much like hype and so many people building AI applications and demos. And that's great, right? Just getting it out there, proliferating through open source, through Twitter, whatever it is, right? This is awesome. But the version of that last mile and the productionization concerns really need to dive deep on all of these kind of fundamental trade offs that I'm talking about in the hosted models versus local models and, and, you know, fine tuning and distillation. They all become really important very quickly.

• 14:13Chris Benson?

  So before the break, as we were talking about these different characteristics that kind of affect applied AI and affect deployment. I was really taken by the fact that so many of them are not really AI specific. You know, if you could almost argue that applied AI in so many ways is about software. It's about the systems. It's now about cloud. It's about all these other things blended together to produce solutions that are productive in the world and, and have value for people and organizations. You know, we talked about unit testing and stuff like that. What is your thinking around kind of the integration of all those things? Cause the model itself, you know, to your point about hype, still kind of gets all the attention and the amazing things.

• 14:58Chris Benson?

  And it is amazing what we're doing, but to make this stuff work in life, there's all these other concerns that there's so many cool things in 2023 happening on the model side that the other 99% to make it real, uh, kind of gets. When you're working with people around understanding how all this fits together so that they can do that, how do you frame that so that their attention gets on the right thing, their budgets are properly allocated to attend to all the things? I've seen organizations really struggle with that because they go into it with hype, focusing on just the model and building skillsets and budgets around the model. And then they try to figure out the whole thing with clouds and deployment and things afterwards, and they have a hard time.

• 15:43Chris Benson?

  How do you navigate that, given the hype cycle that we're operating in?

• 15:47Speaker 03

  My first piece of advice would be that for your particular use case, it's your job to be done, whatever business use case you're solving. And to keep in mind that AI, like all software, is a tool. And it may be the really shiny tool, it may be a tool that is evolving very quickly in front of us right now. It's a very powerful tool, but it is a tool to solve, you know, a business use case and a problem for humans. So rooting, you know, the framing in that, I think is very important. The second thing I'll say is a lot of AI right now, and especially the stuff that gets a light shined on it in the open, because the application layer is so new and there's so much logging and fruit, you know, as you said, like, we need to have more emphasis on the engineering rigor under the hood.

• 16:31Speaker 03

  And so, one practical piece of advice there is to really focus on an evaluation set for your particular use case. And you might have existing data, you might have existing kind of input-output pairs for your particular example, you might have, you might not have that, but like, starting from there and working backwards of like, this is what the end user is going to see. And then working backwards from that to think about, well, how can I use language models or other expert-focused machine learning models to solve that? I think is very important because that also gives you a grounded north star. That like, so much of the prompt engineering and tuning of these models is based around, well, I think this is going to work better, or I eyeball it on this one example and it seems to work for this, right?

• 17:16Speaker 03

  But really applying some fundamental engineering rigor at that level where you have an evaluation set that you can track. That you can improve over time, that you can, and not just tracking the quality of these models, but you know, tracking the other trade offs in terms of pricing, latency, recall, like there's a whole slew of trade offs that can matter depending on your particular, you know, use case. And then the other piece of practical advice I would say is the kind of diagram of this ladder of complexity that I was referring to before. Like every time you take a step down that ladder of complexity from using a hosted model, just using prompting, and then going to, you know, some type of information retrieval embedded in the context to having a multiple

• 17:57Speaker 03

  chains of prompts, to going down to fine tuning, you know, a hosted model or fine tuning a local model. And at the very, very bottom is building your own model, right? Like every time you take a step down that ladder of complexity, it adds engineering complexity. It's going to make your solution more complex to maintain. And so really like having a good handle on how you can start simple and only move down, you know, when you need to, or when you hit a constraint, like, okay, this is great. And I have a working solution with a hosted API, but now I need to worry about the price because I'm going to production and you know, the unit economics, like maybe at that point, then you think about, well, now I have this great solution and I can auto generate an eval set for myself and you know, have a bunch of inputs and outputs and fine tune a model that

• 18:38Speaker 03

  is hyper distilled and efficient and focused. That's great. Don't start there for most use cases, right? The one other thing I would say at the practical level is where language models tend to break down or lack reliability is oftentimes when you're trying to give too much to the model to do at once. And so breaking the problem down into sub problems that are a lot more focused is one of the most practical, like I just find myself telling people over and over again. It's like, okay, that's awesome. Break your problem up into some problems and you know, how to do that is a whole art form in itself. And maybe someday in the near future, language models will do that for us. I don't know, right? That's getting into the more sensational side of things. But as a general principle, breaking your problem up into sub problems, thinking about how you can

• 19:23Speaker 03

  articulate your problem as succinctly as possible in a way that is native to the language models is a really key practice.

• 19:31Daniel Whitenack?

  I love how you talked about like evaluation, forming your evaluation set, getting some ground truth, also breaking up your problem, maybe having an evaluation set for each of those sub-problems would be a good idea. I think there's this general perception that large language models are this kind of unique thing, these chat interfaces are this kind of unique thing, like we can't, how do you like evaluate that in the way like, I think what people have in their mind is, oh, if I'm doing sentiment analysis, it's either positive or negative or neutral, and I can like calculate an accuracy, for example. Whereas they might struggle to think about, okay, well, there's this output from this language model.

• 20:14Daniel Whitenack?

  It's seems coherent and fluent, like, how do I evaluate this? And so I think there's maybe a bit of confusion around the evaluation side. Can you share any tips or thoughts in terms of what you've found to be useful in your own work in terms of evaluation sets? And like, how you think about how good the output of a language model is?

• 20:40Speaker 03

  My first thought would be the less that it's about me thinking about how good it is, and the more that it can be objective, like using some kind of way of evaluating it, the better. There's one project that I really like recently by Lance Martin. It's called Autoevaluator. I don't know if you guys have seen it, but it's specifically for the domain of QA, so question answering. And he recently partnered with LangChain to create a hosted version of it. But the way that I think about this is a little abstract, and it's really like, starting from your job to be done. Oftentimes, sentiment analysis isn't the job to be done. It's like a piece of a job to be done, right? So again, it's like breaking up the problem and understanding how to think about and structure those problems as whether it's an expert model that does a sentiment analysis,

• 21:24Speaker 03

  or it's using a large language model that like it can do sentiment analysis, it's really good at that, but it's also like it can do a whole bunch of other things as well. So the more focused your task is, the more clearly articulated your task is, and the more structured of the output that you have at the individual LLM call level, the better and the easier it is to create reliability around these things and to actually test them with more traditional software engineering practices, like writing unit tests or integration tests. You know, one thing that I'm actively working on right now for the TypeScript community is A way to invoke large language models and have structured guards on them. I know like prediction guard, guardrails, there's a few projects that are doing this.

• 22:06Speaker 03

  But really then having that actually be typed in TypeScript so you can make an LLM call like it's a function, but get some JSON that has these fields that has these types and have it kind of, you know, there are techniques that you can do under the hood to self heal if the JSON isn't properly formed. Or, you know, maybe you want to generate some TypeScript code and you want to validate that's a correct AST or something. Like there's techniques that you can do to constrain the output of the language model for your particular task. But in my view, these techniques are constantly shifting, kind of the best practice in the state of the art there. And so I think libraries like Lainchain and the open source framework that I'm currently working on, you know, I think will do a lot to help developers to abstract out some of the complexity of just viewing this as a general purpose tool. And again, it's like you start simple.

• 22:51Speaker 03

  One of the great things about language models is they can do just about anything. That's also one of the downsides, right? Like when it's so unconstrained, how do you even approach the problem? So having best practices, having like examples constraining the problem. And really it's like the ability to have a unit test or an assertion in a traditional programming language at the large language model call level where it's like, I assert that the output should be valid JSON. Or I assert that the output should conform and be valid TypeScript syntax. And if not, like actually self reflect on that and put it back into the large language model and regenerate it. All of those things I think are foundational primitives at the large language model level

• 23:30Speaker 03

  that will allow developers who want to build real reliable applications to do so more reliably. Cause they can focus on their domain specific, you know, business logic or aspects that are away from a lot of this kind of implementation details that are also constantly shifting under our feet, right?

• 23:46Chris Benson?

  Fantastic explanation. You keep talking over and over again about how things are shifting and the evolution of the engineering around it. That puts a burden on these hackers and developers that are trying to go out and implement these things at this point. Because this year has just been, you know, phenomenal progress. But that makes it really hard for mere humans here to kind of track that and keep up with it. So you kind of talked about some of the concepts right there. But if you're thinking about like, you're about to turn to somebody who's a hacker and they're looking for that guidance. What are some really, you know, not necessarily comprehensive, but hey, go do. One, two, three, or A, B, C, and that will help you kind of keep leveling up.

• 24:30Chris Benson?

  Like, what are some of the things that you're telling people these days to say? If you want to keep up this year with what's all with this insane progress in LLMs and and all the different model types that we're seeing the progress of? What are you going to do in a practical side as a hacker to manage that? Do you have any tips that you can kind of take us through about?

• 24:49Speaker 03

  Absolutely. One, it's super noisy. There's so much happening. We're in the middle of this exponential wave. And, you know, I think a lot of people are like, oh, I have FOMO. I want to be on this wave, right? But where do I start? And there's just, just so much noise, which is great on the one hand, but on the practical side, like, how do you give advice? Where do you start? So, you know, there's a couple of levels to this. I have talked, given some talks to kind of like a chat GPT for beginners type crowd. And really, it's like, my main advice is, one, just use it, just go and try it, right? That's simple. But two, more importantly, like, the next time you have an actual problem that you think, like, maybe I could use chat GPT or a language model for, actually try using it to solve your own problem.

• 25:31Speaker 03

  Because what that does is it starts to build up this muscle in your brain around thinking about using these new type of tools to solve problems. And it's really a different type of tool. It's like exercise. You need to start exercising that muscle early and often. And there's a lot of noise, there's a lot of different AI tools. The side of things which I'm confident will be just as relevant, you know, a year from now, a couple of years from now, is building up that muscle to think about using, how to actually use AI to solve your own particular problems. It's one thing to talk about, like, hypotheticals and general, like, cases of problems where these tools excel. But it's another thing entirely to start building your own personal, you know, muscle.

• 26:10Chris Benson?

  Totally agree with that.

• 26:11Speaker 03

  I have, you know, whether it's a personal problem, and you want to just go, you know, talk to the chat GPT, or you have a problem at work and you're like, well, I think I could use a language model, a hosted API to solve this, or something like, starting simple, starting from your own problems will start to build up that muscle. And, you know, it'll, you'll naturally learn it and take it from there.

• 26:45Daniel Whitenack?

  So Travis, you've mentioned a couple of times like TypeScript, Node, this community that like you're a part of. And I think there's probably a lot of Python people listening to this show, maybe data scientists, practitioners. To me, it almost seems like there's like two communities. There's like all of these data scientists trying to figure out like, oh, large language models, generative AI has sort of broken my intuition around like, how would I need to be doing? Like, do I need to be training models now? Like, how do I solve this problem now? I was training models last year. Do I still need to be doing that? So there's like that side of things. And then there's like this really vibrant community of front-end developers and other developers that are building even

• 27:30Daniel Whitenack?

  like people that are maybe like low code, no code people building really cool products around this technology. You seem to be sort of like exposed to a lot of those things. How do you see these communities developing over time? And like, how have you seen the maybe the TypeScript, the Node JavaScript type crowd kind of rise up to meet these technologies, maybe in a way that the sort of traditional data scientist crowd has not, or has differently, I guess?

• 28:04Speaker 03

  To some degree, a lot of the JavaScript TypeScript world is like jealous of the Python world, right? Because because all the cool new AI stuff is like Python first, or they're using this machine learning framework or whatever. And this is where like hosted APIs, whether it's replicate or hugging face. So you got the hat on, right? Hugging face, or OpenAI, or Cohere, or that like all these hosted models are a massive unlock for application developers. And TypeScript is the largest programming language in the world. Python is it's big. It's the largest by far in the data science in machine learning world for sure. And so, you know, there's this dynamic between the two. Where I think we're seeing at the application layer, folks who are good at building full stack apps that can easily plug in to hosted models and things,

• 28:49Speaker 03

  who really push the envelope in terms of like unlocking people's imaginations, building a good UX around these things, that's so important, like to make it more approachable for people and to really like show people what a lot of the machine learning people have known about for a long time, right? But like you eat both sides of that equation. So one of the projects I did a couple of months ago was I ported scikit-learn to TypeScript. And it's not like a full port, you know, it's like auto generates all of the TypeScript classes, like 260 classes. And then under the hood, it creates a sub-process, a Python sub-process, and then marshals and does the inter-process communication between them. But it works extremely well and you can call and do K-means and PCA and just all these fundamental things that the Python, you know, machine learning world takes for granted.

• 29:32Speaker 03

  There are versions of that that exist in the NPM ecosystem. It's just they're all over the place in terms of quality, in terms of like, there's so many just fundamental aspects of machine learning that the TypeScript world is missing out on. And one of the primary drivers behind kind of what I'm working on, and I'm, you know, happy to share, like I'm building a reliable TypeScript open source framework for building reliable agents.

• 29:56Daniel Whitenack?

  Very cool.

• 29:57Speaker 03

  Thank you. I view agents as this new, it's like if large language models are CPUs, right? And kind of this new compute paradigm, there are these reasoning engines. Like, yeah, they're great at generating text, but the real emergent property, the real game changing property of them is reasoning. If they're kind of the new reasoning engines and you have like that, that's your CPU layer and then you have like a storage layer that's all these vector databases and kind of overhyped on that side of things. On top of that, you have like, how do you actually run programs? And that's, you know, agents. And I view it as like, there's kind of a spectrum of like traditional programming that might happen to use a large language model. And then on the other end, you have like full self-driving agents that are making decisions and creating tasks and just fully autonomous, right? And I'm excited to kind of focus on somewhere in the middle

• 30:42Speaker 03

  and focus on more reliable like use cases that we can actually build reliably today. But to your question about kind of the TypeScript Python. A lot of the frontier, the libraries at the framework level that are pushing the edge here are all Python first, right? And I really want to take a TypeScript first approach, partially because it's the community that I know and love. It's my like best tool on my tool belt and partially because I think people building real applications at the application level. A lot of those folks are more in the JavaScript type world, so.

• 31:11Chris Benson?

  So you have hit an area that I have so much passion for.

• 31:15Speaker 03

  Oh, awesome.

• 31:16Chris Benson?

  I'm sitting here waiting to ask my next question here and Daniel has heard me whine about this for years, what I'm about to say. And so I want to get your take on it. So, um, like there is more to the world than just Python. And I'm a multi-language person and I don't necessarily go all in on any one language or the other. I use, I'm a TypeScript user. In the last year, I've been doing Rust. I had been doing more Go before that outside of the AI and Python stuff. But I, I hit a use case where I was building something and I had to eke every little bit of performance out of the available hardware to do what it was. It was going to be C++ or Rust and it wasn't going to be C++.

• 31:58Chris Benson?

  So went to learn Rust and then I'm in Rust and I'm doing that. And I'm looking at as an analogy here for what we're about to go at. I'm looking at WebAssembly and the Rust community and other language communities are so into, you know, that fact of, you know, write it in with the thing that you need to be in and yet have access to that in terms of deployment and still having great performance and stuff. And every time I'm now messing with WebAssembly in Rust, I'm thinking, when is the AI world going to catch up on having, you know, multifaceted from a language standpoint, access to the models instead of everything being Python first? And so asking the pardon of the Python lovers in the audience,

• 32:41Chris Benson?

  when am I going to be in Rust or Go? And you're obviously doing it in TypeScript. But the language of my choice and taking advantage of, as you called it, the new CPU of reasoning. From that point, instead of having to do a context switch, it is an ongoing year after year after year frustration that I have, as you can probably tell by now. So I'm hoping that you're about to give me the golden path out of here because I need one.

• 33:08Speaker 03

  Okay. Well, first off, I love your framing. I love your passion for this. I also feel very similarly. I think the reason why I'm starting with TypeScript is because of the developer experience at the application level. I think is really important for the type of framework I'm looking to build. But I view WebAssembly, or Wasm, as kind of the ultimate compiled language runtime that I want to target. Because you could imagine a world not too distant from right now, where you have agents that are running in data centers. They're running on edge. So anything that's kind of, whether it's Cloudflare worker or Vercel Edge function or within a service worker in your browser, right? But the common thread there is Wasm.

• 33:53Speaker 03

  To what extent, starting with developer experience at the TypeScript level and then focusing on that at the runtime level, there's still a clear path forwards for a lot of folks for just like using hosted APIs. That's one area that you can have that multi-language very easily. That's a natural point.

• 34:09Daniel Whitenack?

  But you gotta be kind of in the cloud for most of that in a practical sense, which I'm not always.

• 34:14Speaker 03

  A hundred percent. And then there's like the whole open source models or the practical side of things where you're like, well, I need to have full hardware or the latency or something that's like on-device. And I am extremely bullish on WebAssembly as, there's a quote that I like and it was from, I forget who it was from, from the Linux Foundation or something. It was like, if WebAssembly existed 10 years ago, then Docker would have never needed to exist, right? And I think it will have that level of impact eventually. I think potentially.

• 34:43Daniel Whitenack?

  I do too.

• 34:44Speaker 03

  Yeah, I think potentially the kind of unlock here, that the path that could bring it into the more of the mainstream could be AI. I don't know at the model level, there's just so much momentum behind Python. And all the core kind of researchers and stuff are Python first. So when I did the sidekit learn kind of port to TypeScript, there was, I think, a Python port called PyoId. And it's a Python runtime. You guys might know better than I do, but it's targeting WebAssembly and it allows them to run subset of sidekit learn in WebAssembly supported environments, including no JS in the browser. And that's super, super fascinating to me.

• 35:23Daniel Whitenack?

  Yeah, I think that there's a couple of related projects. I think like PiScript from Anaconda is like trying certain things like that. But I'm really interested in that space as well because I've seen. It's sort of like a different kind of diversity than we normally talk about. But the fact that like more developers from more diverse backgrounds are at the table building AI things, I think is an amazing thing. And I think a lot of good is going to come from that. So I'm really happy to see a lot of that happening.

• 35:55Speaker 03

  Well, if we have time, just one more, maybe controversial take on this. Sure, we like controversial takes. Awesome, awesome. You know, as we get closer to building reliable agents, and the way that I kind of was framing it before, it's kind of a fundamental new compute paradigm with large language models of CPUs and you're building these agents on top of them. As they eventually get more and more reliable and more autonomous, right now, a lot of them are just toys, let's be clear. But as that happens, I view it as a new higher level programming language, you know, and we're working with natural language. The AST of that language is, in my view, a directed graph and the nodes are like specific LLM calls or a call to a tool or a call to an API.

• 36:37Speaker 03

  And, you know, there's massive problems around how to add reliability at that level of having structured output or guardrails, like some of these things are clearer than others. And then at the whole graph level, you know, that becomes a program or an agent. To some degree, we're talking about all of these like Python and Rust and implementation issues, and that's all very important. But I wonder to what extent, you know, 10 years from now, we will even be talking about a lot of the current levels of programming abstractions that are hyper relevant to us today as practitioners, or how quickly we'll move towards this world of a higher level abstraction for solving problems that is just significantly more efficient, more approachable, because it's kind of based on natural language. Anyone in this field that talks to you about timelines, you know, is like just throwing

• 37:21Speaker 03

  a dart, ran it with a blindfold on, but that's one thread that I'm really excited about.

• 37:26Daniel Whitenack?

  You kind of went already ahead to where I was hoping you would go, which is what's keeping you up at night? What's in your mind in terms of like looking forward and all of that? And I agree, I think this is a really, really interesting direction, and I certainly hope that we see that timeline progress rapidly. I think we probably will, so yeah. It's been a pleasure to have you on the show, Travis. Really looking forward to keeping in contact and seeing all the amazing things you do and trying out some things in TypeScript. It's an exciting time to be part of this and yeah, looking forward to keeping in contact. Thanks for joining us.

• 38:11Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Chang Talk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.