Automating code optimization with LLMs

You might have heard a lot about code generation tools using AI, but could LLMs and generative AI make our existing code better? In this episode, we sit down with Mike from TurinTech to hear about practical code optimizations using AI “translation” of slow to fast code. We learn about their process for accomplishing this task along with impressive results when automated code optimization is run on existing open source projects. Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Featuring: Mike Basios – LinkedIn https://www.linkedin.com/in/michail-basios-90410436, X https://x.com/maik18Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: TurinTech AI https://www.turintech.ai/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm the founder of Prediction Guard. And I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 0:57Chris Benson?

  Doing well today, Daniel. Just there's so much going on these days in this industry in terms of AI that just constantly learning new stuff and finding out who's doing what.

• 1:09Daniel Whitenack?

  Yeah, it's almost like you need to optimize some things about your life to keep up. Would you say that's accurate?

• 1:16Chris Benson?

  Yeah. Speaking of optimization, I think that's a thread to pull right there.

• 1:20Daniel Whitenack?

  Yeah, yeah. So, speaking of, today we have with us Mike Basios, who is the CTO and co-founder at Turin Tech AI. Welcome, Mike.

• 1:32Speaker 03

  Hello, guys. Nice to meet you.

• 1:34Daniel Whitenack?

  Yeah, well, we alluded to optimization. And I know one of the things that Turin Tech is working on is code optimization with AI. And maybe some people, actually, probably a lot of people listening to this podcast are familiar with certain developer tools that are AI-flavored, maybe GitHub Copilot or something like that for generation. I'm wondering if you could take a moment before we dive into AI-driven code optimization, if you could just help set the stage. For those that aren't aware, what do you mean when you say code optimization? Why is it useful? How has code optimization, quote-unquote,

• 2:17Daniel Whitenack?

  been part of the developer lifecycle for some time?

• 2:22Speaker 03

  We are in an area nowadays that we see more and more applications consuming a lot of cloud resources and a lot of resources and everybody is trying to optimize the performance of their code. Now, when we're talking about code optimization and in particular performance, typically people would like to optimize things like application being faster, maybe memory consumption. We know everybody complaining about Chrome using too much memory, for example, in the past, or CPU usage, which is connected very much with the energy that the different software is using.

• 3:04Speaker 03

  If we talk about mobile phones, application there, we would all like to be more efficient and consume less energy. And practically, that's the area that we are, have been focusing in my research and our group in our research and in the company.

• 3:18Daniel Whitenack?

  Maybe you could talk a little bit about some of the history of that research. I'm sure that that has, just like everything else, been impacted by this kind of latest wave of AI technologies and generative AI. But I know that the company and yourself have been involved in research prior to and all during the development of these things. So could you give us a little bit of background on, you know, how you first started thinking about these problems and how it kind of developed over time?

• 3:51Speaker 03

  Yeah, code optimization is not a new thing. If you read the research papers 20, 30 years ago, everybody would like to optimize and make the code efficient. Like, there are a lot of tools, like profilers, there are, you know, that help developers find hotspots. The biggest problem in this area is, okay, we profile our code, but how we can automatically improve it so we make it faster? Typically, this is a very, very manual process and majority of the people that work on these areas are super specialized. And it's more and more difficult nowadays to find people that know how to Optimize the performance of their code.

• 4:34Speaker 03

  Because the programming languages are becoming higher and higher level. Like now people write more in languages like Python, JavaScript, TypeScript. So nowadays, the real companies that deal with code optimization are companies in the lowers from the hardware space, like Big Intel, Nvidia and those. They write specialized software that take advantage of their hardware, where they show the hardware outperforms, or technology companies that need the scalability. But majority of developers will not necessarily bother about the performance of the code as an immediate first thing that they need to optimize.

• 5:17Speaker 03

  And we have tried, and we have a platform that we helped practical engineers to automate this kind of process, to make it easier for developers to identify places in their code that are slow. And then to optimize it without necessarily having the knowledge that they need, but also do it automatically. Now, the history of code optimization, as I said, it was very, very manual process in the very beginning. Very few people know how to optimize code for specific hardware. You know, they had in the past, I guess, they had to read books and compiler options. Eventually then you had better and better compilers. So you can use compilation option to optimize your code and

• 6:01Speaker 03

  tune the options of the compilers, etc. And then you have a lot of profiling tools that help developers also optimize. But still, all these processes, most of the time, has been manual or semi-manual, right? And that's where we see the advances of AI helping this process. And to give you a bit of. Context, how we started our startup. This started after we published a paper in 2018, I think, that was in the Foundation of Software Engineering Conference, where we showed that we could automatically help developers choose better data structures by taking a code, looking at data structures and optimize it by giving variations.

• 6:46Speaker 03

  For example, sometimes in languages like Java, for example, you don't need to use an array list. If you potentially can use a link list in a scenario that may be better. So we tried to do these small changes and showed that we have good performance impact. But majority of people at that point were manually translating code, like rule-based transformation, like regular expressions. If I see this pattern, convert it with this pattern. So that is how people have been doing those things with code refactoring tools until the LLM came into the discussion.

• 7:23Daniel Whitenack?

  Speaking of that, kind of getting into this latest wave of maybe what I'll just refer to as developer tools that are AI-related. Again, you know, people might be familiar with code generation type of tools or explainers or something like that. There've also been, I've seen, kind of agentic type of tools that will like write a PR for you to do, like you say, "Hey, I want to do this thing," and then there's a PR generated. Here, given the focus on code optimization, could you draw out some of, just for people that are maybe kind of getting into this, could you draw out some of

• 8:05Daniel Whitenack?

  when they would want to use this kind of tool versus some of these other maybe generative application or applications of generative AI to code, I guess?

• 8:17Chris Benson?

  Yeah, how's it fit into that ecosystem?

• 8:19Speaker 03

  So the way we present the code optimization tool currently is part of your CICD process. So you make a pull request, then you will run some unit tests. You will have your integration testing. Potentially you'll have a scanning, security scanning tool like SNEAK or Checkmarks, et cetera. And then the next step is, depending on where your application is deployed, we analyze your code and we tell you, make those changes because you can have this 20% improvement in CPU and execution time, et cetera. That is As the way we present it currently as a C I C D tool in the in the tool chain of the developer tools. However, if you think about the technology underneath and all DevTools, the way I see

• 9:04Speaker 03

  it, are gonna be using AI and they will take advantage of L L M based solutions. From the moment you're using L L M, if you generate code or you translate code, it's a kind of the same kind of approach. It depends on the data that you will apply the L L M. So code generation tools, practically, they see, when you say for the P R example, the way those L L M's have been trained is that they see that there is some comments below the code, so you say, if I give you those comments, can you predict the code? So in the code translation, where you don't know about the speed of the code, you say, I have seen this C plus plus code and the equivalent of Python code.

• 9:49Speaker 03

  And L L M's do it, like for example, Copilot or Chat G P T, they can translate code, like if it's perfect, probably not yet, but that is a fundamental technology. So code optimization is on the same set of tools, but it says, this is a slow code that I have seen, now I have seen a variation of faster code, so I can recommend you this faster code. But eventually, you can expose the L L M, like any other tool that is built on V S Code or L L M based, you can expose it in an editor, and then the developers from our L L M will get suggestion for faster code.

• 10:32Speaker 03

  That doesn't mean that will be beautiful code necessarily, but it will be faster for the hardware that you need to run it.

• 10:40Chris Benson?

  I'm kind of curious, as we're talking about speed, are there any other dimensions that are relevant in there that you guys are interested in, that are either adjacent to speed or contribute to speed, or any other characteristics that may not be directly speed specific, but are things that you're starting to target or expect to target?

• 10:59Speaker 03

  We apply multi-objective optimization, so when you do the translation, you can have different objectives that you try to optimize, like speed is one factor, memory usage is another. CPU usage. Typically, there is a trade off between speed and memory usage. If you have more memory, you would like to use it, that increase speed. So our tool allow you and gives you different suggestions for what you need. But we see users, for example, we see this paradigm to be used in other use cases. Like users have told us, I can improve the readability of my code as far as I know that I don't impact performance, for example.

• 11:41Speaker 03

  Because there are managers in teams that they have five projects and they would like, let's say five projects with five teams with developers. And they would like to guarantee that the quality across is good. So this AI approach, these tools, Artemis, our tool or other tools in the L L M space will be able to help with those. But the biggest problem of any L L M based tool currently, like Chris, you you mentioned you are working for Lockheed Martin, et cetera, is that the code generated by any L L M, it's not guaranteed that will work hundred percent.

• 12:22Speaker 03

  Also, you need existing tools to check if this code is secure. And also, you do not know if you will break your code. So still in the early stages of incorporating those L L Ms. That's why it's also easy for a developer to give it as a suggestion right now. But I'm pretty sure like companies like that working bug fixing, code, security checks, they are using and they will be using more and more L L Ms. Because they can train on the data that they have access, so they can have competitive advantage.

• 12:58Daniel Whitenack?

  Would it be a good parallel just to kind of draw to people's mind, like maybe something they've seen before in other domains? It almost sounds like a parallel to kind of a rephrasing type of prompt in an L L M where you you might say like, I do this a lot with, you know, emails or other things like, here's my really bad email, you know, make it flow better and sound better. Or here's my goofy email, make it business professional or or something like that. You drew the the comparison to like maybe machine translation or something like that. Is rephrasing kind of a good way to think about this?

• 13:40Speaker 03

  Let me give you a very simple example. Let's say you have an essay that you need to write, and you have 10 paragraphs in that essay. And you would like to have a version of that essay that is much better and you can get a better grade. Right? Now, what we do is we will look all the paragraphs and we will provide you better variations. So we will get you version one with three changes that are applied by different LLMs. Then you would need somebody to grade that essay. So in this case, we measure how fast the code is. So somebody says, OK, you have a 70%. Then we take that output, and then we provide and give it back to the LLM and give you another version.

• 14:25Speaker 03

  So practically, you start from version zero of your essay. You apply the different LLMs. You get feedback. So it's like reinforcement learning, live learning, and get different variations. And that score will be increasing eventually. And at the end, you have like a translated version of your original version, that the LLMs and all those refactoring it, which is better on the metric that you have. But we in the platform test that the code passes. We compile it and also we run the performance. In this scenario, we'd have a teacher that would grade. And if you think how OpenAI and everybody has been doing their training, they usually get LLMs and then they use reinforcement learning

• 15:08Speaker 03

  and HLF and all those techniques. So we have done that in the code optimization setting. That's why we have had some impressive results in taking an open source library and we just put in a tool and then suddenly optimize by 30% execution time without us doing anything. The models learn themselves.

• 15:33Speaker 00

  This is a changelog news break. You can add Meta's CodeLlama to the ever-expanding list of code-generating LLMs. Based on Llama 2, CodeLlama comes in three sizes: 7 billion parameters, 13 billion, and 34 billion. And it comes in three different varieties: a general model, one tuned for NLP instructions, and one fine-tuned for Python. How does it stack up? Well, Meta claims it outperforms other publicly-available LLMs, and it shares the same open-ish license as Llama itself, which is free for research and commercial use, unless you compete with Meta. You just heard one of our five top stories from Monday's changelog news.

• 16:18Speaker 00

  Subscribe to the podcast to get all of the week's top stories, and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 16:40Daniel Whitenack?

  So Mike, you got into something that I'm super interested in, in terms of how you're going about this problem, which you alluded to the fact that you're using this sort of reinforcement learning loop, or feedback loop, to improve the performance of your tools. Given that you and your team have worked with sort of code-generation or code-specific models for some time now, before we get into kind of some of the cool stuff that you've done specifically, could you just comment on kind of the state of code-generation models that are out there, on maybe the open-source side specifically, but if you want to highlight

• 17:23Daniel Whitenack?

  any, you know, closed-source ones, that's perfectly fine, too. But from your perspective, how is that, the ecosystem of code-generation models, changing and advancing, and what is the state of it sort of these days, I guess?

• 17:38Speaker 03

  So I was one of the very first believers of L L M assistive code generation tools, like I tried to get better access to GitHub Copilot, even tried to use G P T three to see all this kind of technology. I'm very big believer because I have seen members of our team using L L M to build things much, much faster to code. For example, we had the backend engineer, we needed a prototype for a frontend. He just used one of the closed source L L M's and he could, in one day, he did a new UI that he didn't even know the language, he didn't know TypeScript.

• 18:19Speaker 03

  So I see that they are very, very promising. I see more usage right now for good developers that already know, you know, the basic of computer science. I'm not a believer of, hey, you don't need to code, you will use this L L M for you. Right, that are, you know, some videos, potentially they are over promoting that. But if you are a good enough developer, you know what you need and you use, they can practically help you dramatically to build easy applications, you know, to generate tests, to generate comments about your code. And of course, the performance of, from our experience, I believe GitHub Copilot and Chat G P T are still outperforming the other models.

• 19:03Speaker 03

  But we see more and more open source models starting to become very, very good into the different languages, like we have been trying Lama two, we have been trying Code Gen, we have been trying all of this and we even expose them to our platform so people can compare the results of those. Those tools will need to become a bit easier for developers and V S Code tools to use because by default, people, this A P I exposure that some of the closed source models are giving is solving a lot of headaches for a lot of developers. That's why a lot of developers still have preference for this. But definitely open source models.

• 19:47Speaker 03

  They are very good and I see them becoming even better if they are fine tuned on specific language or specific context. I'll give you an example. If you want, if we want to do translation from SQL to an SQL, let's say we want to optimize SQL queries that people are doing in their databases, you can take one of the open source models and fine tune it on SQL and you probably will outperform GPT three point five or four on the context that you have. But I'm truly very big believer, a lot of, it's a bit of also a bit of psychology between developers that hey, but I am a believer that people that will use, they have advantage over people that don't use those tools currently.

• 20:30Chris Benson?

  So that raises an interesting point. It's a little bit of a tangent, but you've kind of inferred it. This is kind of changing the way we humans are coding. I know you also talked about whether L L M's would just write the code for us and, and the overhype, you know, certainly today about that. But it's kind of changing the way that we code as humans and it's extending our capabilities dramatically in terms of being able to reach beyond what we might have been able to do two years ago, for instance. Do you see that accelerating when you're looking at the fact that if you looked at a traditional coding team a few years ago, as you're producing your product for folks and you're starting

• 21:12Chris Benson?

  to recognize that individual coders are starting to elbow their way out of their traditional swim lanes with these new tooling capabilities that you're providing, how does that change things? Like, how does the market change that you're looking at going forward?

• 21:27Speaker 03

  I believe dramatically and that we have tested this with our internal developers. I have seen dramatic improvement in productivity of developer without access to.

• 21:38Speaker 03

  L L M's and developers that have like there are two things. One is you can make more efficient developers that everybody would like to hear. But from a more senior level manager level, you do not need unfortunately as many developers as you would need before. You cannot avoid this and developer should think for me like, okay, I am competing with some other developer in my team. To produce, let's say, an API or produce a UI. If the guy has access to Copilot or ChatGPT, the other person doesn't have. I guarantee you like with very, very big chances, the guy that has access to ChatGPT and those will outperform and have faster results.

• 22:22Speaker 03

  So it is, it is like you have a very, very good assistant next to you that you should use. Otherwise you are losing. That's how I see it.

• 22:31Chris Benson?

  Yeah. As a two second follow up to that point you just made. When we start pervasively, because I think I've seen some stuff recently over the last month or two that like. The majority of active developers out there are now using L L M's, you know, so there's been really rapid adoption. Here within the developer community. So we've kind of moved very quickly from those who had versus those who didn't have. Into a world where everybody has, they may not all be using exactly the same models as things progress, but everyone has it. Any thoughts about kind of what that means? It's like you're democratizing L L M's across the population of developers. And now they're competing. So it's kind of like me and my L L M's are competing against you and your L L M's as a developer.

• 23:14Chris Benson?

  Just any thoughts, waxing poetic a little bit about what the implications there are?

• 23:18Speaker 03

  Yeah, I mean, it's a weird world that I don't know anymore. Like, like I code less and less, but I now can code again, like. Because for example, we have a data scientist in the team and he says, I don't feel like I'm a coder anymore. I'm just a manager of. You know, a user of this L L M that I validate the output and yeah, it's okay. And it's. It's a bit ridiculous, he says, because even simple things like copy paste, I will not bother. I'll just say, okay, can you? You refactor this, like it definitely has changed. And okay, there may be implication about creativity of people. And if you go into the AI LLM space, like for example, for images, when they those models generate images.

• 24:04Speaker 03

  Somebody, a painter may say to you, hey, you may lose creativity because you generate always the same thing. I don't have answers for those things, and I don't think a lot of people have answers. We'll just see, right? Like how things go sincerely.

• 24:19Speaker 03

  But yeah, it's interesting, right?

• 24:21Daniel Whitenack?

  Yeah, this idea of being a manager of your assistants, I think, is really helpful. I forget who it was we had on the show, Chris, but they were saying, hey, if you think about this thing, just like a high school intern or something, you know? Like, is a high school intern going to solve all of your problems? No, but if they work all day on your problem, or you know, let's say you have infinite number of those, those high school interns that just can do work all the time, is that useful? Certainly, there's a management aspect to that, right? Probably more so with high school interns than with LLMs, I'm not sure. But, uh, but yeah, I love that metaphor. That's really good.

• 25:07Daniel Whitenack?

  I'm also wondering, so this particular application of AI within someone's code base, right? I think similar to what we've seen in other cases with Copilot and some of the things that have happened there. It's a very sensitive area, particularly for like enterprise business users. I think if you're like an indie hacker and like you say, you are wanting to create a TypeScript UI and you don't know TypeScript. Like boom, you can get, you know, some really cool results really quickly. Of course, enterprise code is part of the IP of a company. There's, you know, two aspects of that, one of which is the fact that companies have been hesitant

• 25:51Daniel Whitenack?

  or even sued others over usage of their, their code or data in ways that they didn't expect. I think, though, the other aspect of this that you alluded to is it really is powerful when you start to bring your own data to the table, especially with these open models.

• 26:07Speaker 03

  Yeah.

• 26:08Daniel Whitenack?

  Both because they have kind of privacy conserving deployments.

• 26:13Daniel Whitenack?

  And there's also like code preference things and other things that your company might have. So I'm wondering if you could speak to that a little bit and how you envision you're helping build a product that's doing code optimizations for people. So how do you think about, you know, people creating customized models for their code bases? And the sort of proliferation of these customer specific models and the hosting of those? What goes through your mind when you're thinking about those things?

• 26:43Speaker 03

  That's very, very important topic and we have quite a lot of experience with this. And I will mention an example where we wanted to, we went to a client, very big technology firm, super big, one of the best. And we said, hey, you know, this is our platform. We have these L L M's. You can use any L L M of your choice, like G P T four or open source L L M's like Lama two, et cetera. In the beginning, they said, we don't have any approval for OpenAI because first they don't know the I P issue. And second, they don't want their code to go outside, right? We're talking about proprietary code that a lot of such companies. So the solution there was, okay, you can use the custom open source L L M on your data.

• 27:28Speaker 03

  We do not see anything, so it's a custom solution on premise. And while they're using a product, then practically the, our platform allows them to generate their own training data set. So for example, they use Artemis, they optimize code, they see sometimes the code is optimized, it's not optimized. But they generate the data and those data we cannot see. And the client should have those data for fine tuning their own model. Then through the platform, they can say, fine, further fine tune their own model. That is how the industry will go, especially in financial sectors and technology sector or, you know, defense companies, they will never give their code outside on an open source level.

• 28:12Speaker 03

  But L L M's are super powerful on that. Like with one client that they said, I'm not sure if I can get recommendation. If you give me a recommendation from an L L M, who is? liable if that code doesn't work, right? What is IP issue? I said, how do you solve it now? They have the IP checkers, etc. I say you can just use IP checker for this moment. You know, the same code, the same process that you would do, right? But also LLMs, and we have added this functionality, can do very good similarity search. So if you have other code bases and similar functions in your code base, you can find you nicely or you can look like the same way. People are building chatbots on your documents.

• 28:56Speaker 03

  You can build your own chatbot on your code. So it's practically similarity search. And then we could recommend that we even identified that three teams had implemented the same functionality, a bit different, right? So you even save save time, right? So still, you know, this technology, the underneath technology, if you know how to use it properly, you can take advantage of it. And the biggest example is data bricks about Mosaic ML. Because Mosaic ML for like they acquired Mosaic ML that helps organization fine tune their own LLMs of their data. Because no organization will give their data to OpenAI or any other company to fine tune on top of them,

• 29:39Speaker 03

  especially big organization that there is value on their data. So the same thing applies for coding. That's how we see, and that's why we adopted the product accordingly.

• 29:49Daniel Whitenack?

  My next question is, I guess, sort of selfish, and I like to ask this of people that have, you know, really built impressive things with this kind of new reasoning layer of LLMs. I'm wondering, as you look back on building this product for code optimization with LLMs, are there any challenges that were unexpected that you had to overcome? And are there any sort of takeaways for that you would give to practitioners that are maybe working on their own products or integrations with LLMs? Like, what has been important for you to stress, especially as a CTO and bringing new people into the team?

• 30:32Daniel Whitenack?

  As you're working with these types of models, what's important in your mind? And, you know, any of those challenges that had come up? Anything you'd like to highlight?

• 30:42Speaker 03

  If you're building applications and your application depend on an L L M output, right, then you need to make sure that that as a first stage, I would recommend you something like a closed source API usage because you will, it will solve you the headache of deploying your own L L M, having good G P U's, that is a problem that you cannot scale, right? At this moment, most teams don't know how to do it. And of course, there are a lot of startups, a lot of companies that are working on this, on how to build your own L L M in a scalable way. But that can be a nightmare to build. So if your business is not how to deploy an L L M and value somewhere else,

• 31:26Speaker 03

  if somebody provides it as a service, it makes sense to use it. So in our side, we say you can import with an API key and a secret key, your any L L M that you have access, then your application becomes much easier. But then you have the problem that you need to solve, is the client okay? If those data are used on OpenAI or because there is a data that goes there. This in financial sector or currently cannot be accepted. Then our product will have to deploy Lama 2, have to deploy. So then you have to build it yourself or use a service. So we had to build it because we were one of the early adopters. But there are tools, like Hugging Face provides a very nice API for you to deploy.

• 32:11Speaker 03

  Now they changed the license, I think, but you know, I think majority of people can use it. So the speed of L L M's is a big problem for scaling application, definitely. Then other issues about L L M's, which were a bit in the beginning, now they are trying to be fixed, is token size. Every time you ask and the result may be incomplete. If there is, then how do you deal with the previous context and all this kind? You need to spend time on this to do it properly. I am expecting more and more tools and open source are solving, like Lang Chain and those tools also, open source are doing some of the things. And of course, the biggest problem that a lot of people talk is

• 32:51Speaker 03

  About hallucination of the models, you cannot trust necessarily the models and you cannot just say generate code and execute that code in your backend because somebody may do SQL injection like similar to where people were doing SQL injection in the past, especially for coding, you can have LLM kind of injection. So you need to be very careful on exposing the prompt to the end user because somebody can really damage. So yeah, those are them.

• 33:19Chris Benson?

  When you're thinking about, you know, hallucinations from LLMs and you're but you're working on a problem like optimization and stuff and you acknowledged earlier, you know, some of the problems you face is one of them, you may not get, you know, the right code or compilable code because it's the output of an LLM. How do you approach that specific problem? I was actually wondering that earlier and the conversation continued on without it, but we kind of circle back around like, how do you think about dealing with hallucination when you're dealing with optimization and correctness, you know, and improving in that way, balancing the two?

• 33:51Speaker 03

  Very good question. So it depends on the programming language and the existing tools that you can also use. So if you go for programming language like Haskell and functional programming, in theory, you can have a bit more proofs about code before and code after works the same. NASA, for example, would want this proof. You cannot have code. Second mechanism is ideally would like applications to have unit tests and test all the scenarios. So when you do change ideally, but not all code bases have, right?

• 34:20Chris Benson?

  You mean not all code bases are fully covered with tests?

• 34:24Speaker 03

  I hope.

• 34:27Speaker 03

  Yeah. Unfortunately, like there are open source projects that the unit tests don't even pass. And they are like, we take a code base, we say Pytest or whatever, they don't pass by default. So it's still, this needs to improve. Hopefully LLMs can improve that. The third mechanism is we aim for minimum code changes with the biggest optimization. And we go gradually, like for example, we try to say first target the data structure optimization that is one, two lines that you can check. Then go on one single for loops, double for loops. You go a bit gradually on that. And we

• 35:07Speaker 03

  currently make a pull request with the recommended changes. So we still want the developer to validate those changes. Because you cannot take that risk. And also, from a psychology perspective, if you have a tool there that you consider this is my performance expert and tells you at the end, hey, make those three changes that you can verify, it's not different from if we change the name of the tool and put a developer name and make a pull request, that person will not know where this pull request came from, right? So you follow the same process. That's how we see it.

• 35:41Daniel Whitenack?

  Something that I'm kind of getting in what you're saying as well, which I know is often a misconception that I run into when I'm either doing workshops or working hands-on with people with generative models, is there's typically this misconception that you need to package everything into a single prompt and then output your final result. It's a sort of one-step thing. I'm getting the sense that your workflow, for one, it probably involves multiple calls throughout the codebase because of the context size, I would assume is partly because of that.

• 36:22Daniel Whitenack?

  But then also, you mentioned this kind of iterative element where, hey, there's kind of big rocks that you can move that are the sort of worst-off ending areas. So there's hierarchy in that respect. But also, it seems like, let's just assume, I know it's not a good assumption, but if we assume that a person's codebase is fully tested, integration tests, unit tests, it seems like this is something you could just loop over and over and over and over again to get increasing optimizations, probably with diminishing return. Could you speak a little bit to how you as a team think about that chaining element, I guess would be the way to say it, and then also maybe iterative element?

• 37:05Speaker 03

  First of all, the way we have presented this, like, let's say you take the original version of a code, there are two approaches there. It's one, I apply one LLM and take the first three, four suggestions of this LLM, right?

• 37:20Speaker 03

  And then apply which one works, like even on the papers on how good they are on the codebase, they will say the first top five recommendations from the L L M, three out of five outperformed, something like this. So it's not a one chance, apply the L L M, etc. Now, the best approach from what we have seen is you get the first version, you apply the first version, and if you have the ability to get feedback from what you applied, that is where those L L M's are very, very good. So for example, you say optimize this code, you try to pass the unit tests, or you try to compile your code and you get the compilation error message.

• 38:03Speaker 03

  Then you go back automatically to the L L M and say, the recommendation you gave me didn't pass because of this error. And then they can give you better. It's like the Wolverine technique, or I think somebody did a demo where they were showing how you can compile and learn. And because this is also, if you think how you are using those L L M's, let's say, write my email, they recommend, they say, sorry, this is too official. I want it a bit more friendly. This is, I believe, currently the best approach. So it's like iterative approach. If you have a way to measure and give the feedback back, you can have the best result. Like if we take the logs and give it to the L L M, we'll have even better. But even if you say, hey, sorry, this is not good, give me something better,

• 38:47Speaker 03

  it will again try to improve in the context. You can even do things like, again, I think, you know, I was in a presentation that Langchain and similar one, where you can combine two, three versions of different L L M's. And then the three base, you can have three and then combine them, say, take context from here, this work didn't work. So there are different approaches.

• 39:09Chris Benson?

  You just stole my question right out of my mouth. I was gonna ask when you extended that to multiple L L M's and integrated. I was kind of wondering how you were thinking about that, because you were addressing like when one L L M gave you multiple points back in terms of optimization and you're trying those out, and kind of how you might extend that to multiple L L M's, because we're getting in this world with an increasing number. We're gonna just be awash in L L M's before long. And as you have so many APIs or so many deployments available to you, how does that change you? You know, it sounds like your workflow would work for that regardless. But, you know, does that add value? Or do you think there's diminishing returns as you keep adding L L M's into it?

• 39:48Speaker 03

  No, I think there is a value and also it gives a flexibility to the user to always not be locked to a single L L M. Like if you, for example, it may be different user, it may be pricing issue. It may be a new model this week can have bigger token size. It may be the performance of this, so you cannot rely on one single L L M. Nowadays, because it's so easy to build L L M's if you have the data. Our business doesn't depend on, hey, we have the best L L M and somebody else suddenly in a week can give you a better L L M and then you are out of business because you need to spend hundred million to train on C P U's.

• 40:26Speaker 03

  So in our case, combining L L M's, using L L M's and having that workflow, L L M agnostic. It's a way that you can utilize. Now, if people pay for better L L M's and access, the end result is better here. Though we have to mention one issue, which a lot of people may not know. It is a bit tricky when you use output from one L L M or the other L L M. So there are I P issues, et cetera. So you cannot, we are also investigating exactly. Yeah, but you cannot use in theory chat G P T output to fine tune Llama in a commercial setting. Maybe, you know, the alpaca paper, I think the first one showed that you can say to chat G P T, give me examples, then fine tune another model and then have it.

• 41:11Speaker 03

  But you are not allowed on a commercial side. But I'm sure there may be two open source L L M's eventually that you will use. As far as you have the framework, then we allow those things to happen.

• 41:24Daniel Whitenack?

  As we kind of near the end of our conversation here, I'm wondering if you can paint a bit of a picture for us from your perspective as someone working day to day in developer tools that are A I driven. What are some of the most exciting things that keep you up at night as you look forward to sort of the next year? It could be things you're working on, but it might just be generally how this, how this field is developing. What's really exciting for you as a person building these sorts of tools as you look to the next year or so?

• 42:01Speaker 03

  I personally believe that this is the start of the power of this technology. We already see how much it has changed the way people are coding. What I want to see, and I see more and more things make it like people nowadays want to use like talk, speech to text and then code, this kind of things like you. They're trying to make developers even lazier, you know, this kind of things. So from my perspective, from what we are trying to do, I want us to, what we are in the process of looking open source project, going through our platform and really get great optimization that we can give to the community.

• 42:42Speaker 03

  So for example, if I can get a very slow machine learning library, 30, 40% all automatically, make a pull request, see, show to the people how easily we can optimize the speed and everybody can, can benefit from that. And it excites me to know the, we still haven't found out what are the limitations of the current technology and how much inefficient code is outside there. And I'm excited to find out how much we can improve in an automatic way. Like, I don't know, I read this, this kind of things that everybody's using in their

• 43:20Speaker 03

  own day because they can make our, you know, laptop faster, like already my funny is going crazy. And yeah, that's what this combination of L L M's and coding is something very, very exciting because we don't know the limitations of this. That's, that's what I want to find out.

• 43:40Daniel Whitenack?

  That's awesome. Well, we will certainly be on the edge of our seat as you're exploring those, those limitations and those possibilities. Really appreciate you joining us, Mike. It's been a great conversation and I'm very much looking forward to my code running faster despite my ignorance of how to make it do that. So thank you so much and we'll talk to you soon. Thanks a lot.

• 44:04Speaker 03

  Thank you, thank you, guys.

• 44:13Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changetalk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.