AI vs software devs

Daniel and Chris are out this week, so we’re bringing you conversations all about AI’s complicated relationship to software developers from other Changelog pods: JS Party, Go Time & The Changelog. Sponsors: Neo4j https://www.neo4j.com/developer?utm_source=changelog&utm_medium=podcast&utm_campaign=practicalai – Is your code getting dragged down by JOINs and long query times? The problem might be your database…Try simplifying the complex with graphs. Stop asking relational databases to do more than they were made for. Graphs work well for use cases with lots of data connections like supply chain, fraud detection, real-time analytics, and genAI. With Neo4j, you can code in your favorite programming language and against any driver. Plus, it’s easy to integrate into your tech stack. Visit Neo4j.com/developer https://www.neo4j.com/developer?utm_source=changelog&utm_medium=podcast&utm_campaign=practicalai to get started. The Hacker Mindset https://thehackermindset.com – “The Hacker Mindset” written by Garrett Gee, a seasoned white hat hacker with over 20 years of experience, is available for pre-order now. This book reveals the secrets of white hat hacking and how you can apply them to overcome obstacles and achieve your goals. In a world where hacking often gets a bad rap, this book shows you the white hat side – the side focused on innovation, problem-solving, and ethical principles. Fly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Kent Quirk – GitHub https://github.com/kentquirk, Mastodon https://hachyderm.io/@kentquirk, X https://x.com/kentquirkSharon DiOrio – X https://x.com/sharondioSteven Pyle – LinkedIn https://www.linkedin.com/in/stevenpyleJosé Valim – Website https://dashbit.co, GitHub https://github.com/josevalim, X https://x.com/josevalimJerod Santo – GitHub https://github.com/jerodsanto, LinkedIn https://www.linkedin.com/in/jerodsanto, Mastodon https://changelog.social/@jerod, X https://x.com/jerodsantoKevin Ball – Website https://www.kball.llc, GitHub https://github.com/kball, LinkedIn https://www.linkedin.com/in/kbal11, X https://x.com/kbal11Nick Nisi – Website https://nicknisi.com, GitHub https://github.com/nicknisi, Mastodon https://nicknisi.com/@nicknisi, X https://x.com/nicknisiJohnny Boursiquot – Website https://www.jboursiquot.com/, GitHub https://github.com/jboursiquot, X https://x.com/jboursiquotAdam Stacoviak – Website https://adamstacoviak.com/, GitHub https://github.com/adamstac, LinkedIn https://www.linkedin.com/in/adamstacoviak, Mastodon https://changelog.social/@adam, X https://x.com/adamstacShow Notes: JS Party #317 https://jsparty.fm/317 (This will 404 until Thursday!)Go Time #306 https://jsparty.fm/306Changelog & Friends #28 https://changelog.com/friends/28Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents. So you can launch your app near your users. Learn more at fly.io.

• 0:43Speaker 00

  Hello, Jared Santo here, Practical AI's producer and managing editor of all the shows here at Changelog. Daniel and Chris took this week off, but we didn't want to leave you hanging without anything to listen to. So today's episode is going to be a little different than our usual fare. AI is permeating the entire software industry. So we found ourselves talking about its impact, sometimes in practical ways, other times in less practical ways on many of our pods. So today we are serving you a sampler platter. You'll hear a segment from this week's JS Party podcast, where me and my two co-hosts, K-Ball and Nick Nisi, discuss the recently announced Devon Project, which is making waves in developer land.

• 1:27Speaker 00

  You'll hear a segment from a recent GoTime episode called, "How Long Until I Lose My Job to AI?", where Johnny Borsico and his experienced panel of friends discuss using CodeGen AI to augment your dev skills instead of replacing you. And finally, you'll hear a segment from the Changelog where my co-host, Adam Stacoviak, and I talk to Jose Valim. Jose is the creator of the Elixir programming language, and he's been a guest here on Practical AI in the past, talking about Elixir AI tooling. Today, you'll hear us question him regarding Elixir's place in a world increasingly influenced by large language models and how he thinks about it as a language author and promoter. Hopefully, there's a little something for everyone on this episode.

• 2:10Speaker 00

  And if it's not approaching AI from a perspective that's compelling to you, don't worry, your regularly scheduled programming with Chris and Daniel will be back next week. Okay, first up, it's JS Party and Devon.

• 2:40Speaker 00

  Uh, there's some news that's good. There's also some news that's maybe bad, maybe good. I don't know. The thing that everybody's talking about this week, at least as we record, and last week as well, is Devon. Devon, D E V I N. The first AI software engineer, according to the makers of Devon, which is Cognition Labs, a new company which raised a Series A led by Founders Fund, headed up by Scott Wu, who seems to be a very intelligent person even from a young age. If you watch that video of him doing math very quickly at ages when it seems like

• 3:20Speaker 00

  you shouldn't know math very quickly. And um, they got a demo out there of this new AI software engineer. So I could say more. I'll stop right there. You all have probably seen the demo, Cable and Nick, or

• 3:34Speaker 01

  Oh, yeah.

• 3:34Speaker 00

  at least heard about what's going on. This is a, this is a new tool which can start from scratch and do some cool stuff. I'll just leave it there for now. We can

• 3:44Speaker 01

  Yeah, I mean,

• 3:45Speaker 00

  talk about the details.

• 3:46Speaker 01

  If you're excited, you too can pay for the right to have a software engineer that can only fix one in seven of your tickets and spin up lots of new ways for AWS to charge you money without your oversight.

• 3:59Speaker 00

  Sounds like, like an intern. No, just kidding. Uh, sounds nice. And what are you, what are you referring to? Or is this some, some specific things that Devon's been up to?

• 4:07Speaker 01

  I, so high level, there's a couple of things that I'm referring to here. So one is like, they're pumping up the marketing on that. This is a standalone software. You know, why get a coding assistant? Get something that can go and do your software. And they published some data on it and like, It does do better than the state of the art in terms of tackling a going from a GitHub issue to, okay, I'm going to actually solve this implemented change and get it to happen. But like, the number they published, I think, was 13.86% of issues unresolved. So that's about one in seven. So you pointed at a list of issues and it can independently go and solve one in seven. And first off, to me, I'm like, that is not an independent software developer.

• 4:52Speaker 01

  Like, that's... And furthermore, I find myself asking, if its success rate is one in seven, how do you know which one? Right? Like, are the other six, those, it just got stuck? Or is it submitted something broken? Right? Because if it sets up something broken that doesn't actually solve the issue, not only do you have it only actually solving one in seven, but you've added load because you have to go and debug and figure out which things are broken and like, like you have a whole bunch of additional load. So like, I think the marketing stance there is a little over the top relative to what's being delivered. The other thing, and this is around, I think, a part of what they do is, oh, it can spin up resources for you.

• 5:34Speaker 01

  Right? And they showed this cool demo of like, you pointed at this thing and it allocates a bunch of different production resources for you. And the person who's handled DevOps in me before, and now the, you know, engineering leader who has to sign off on our Digital Ocean or AWS or Google Cloud or whatever expenditures you might have, looks at that and is terrified by, I'm going to give an LLM, which is known for hallucination, which is, you know, these things are not, you have to design application. And I'm building applications with LLM, but you have to design around their unpredictability and their willingness to lie. And I'm going to give that raw access to spinning up resources in my cloud? Like, that sounds, well, it sounds like something I would not sign up for. I'll say that.

• 6:19Speaker 02

  Okay. Okay, well, let he whose success rate at issues that is greater than one in seven cast the first stone.

• 6:27Speaker 00

  Yeah, so I was gonna, I was wondering how what Nick's ratio is over there, you know? Like, one in seven sounds about the way I would do. I'd pull off the easiest one first. Does Devin know what the easy tickets look like, you know? Because that's the skill right there.

• 6:40Speaker 02

  I'm over here counting on my fingers, trying to see if I'm within that ratio.

• 6:44Speaker 01

  But do you know when you fail? Or do you just throw out broken code and you're like, ah, here you go?

• 6:50Speaker 02

  It's, it's more of a question of do I know when I succeed? I guess.

• 6:54Speaker 00

  Right.

• 6:54Speaker 02

  Which is, yeah, same thing.

• 6:56Speaker 00

  You think you succeeded until you find out later that actually you failed. You know? That's, that's been my experience. Or you succeeded under the constraints that you put yourself under, right? Or that was actually specified in the ticket itself. But you actually failed at some other unnamed, unlisted constraints that were unknown at the time, but are obviously clearly there in production. And so in that context, you failed.

• 7:19Speaker 00

  It's not easy. It's not easy to succeed in this world. Well, what about K-Ball? What if you, can't you point Devon at like a $5 a month digital ocean? And say, you know, deploy to this and like, can't you cap your risk? I guess on the, on the DevOps side.

• 7:34Speaker 01

  Probably. You probably can. And like, I do want, so I'm taking a hard skeptic stance on particularly the claim that this is an AI software engineer. Like, don't hire a person, use this thing.

• 7:46Speaker 00

  And this is their claim. So I think it's fair for you to be that harsh on them because they say, meet Devon, the world's first fully autonomous AI software engineer. That's a very bold claim. So I think it's fair that you're being that harsh. Go ahead.

• 8:02Speaker 01

  They're showing some cool stuff. It looks like a pretty interesting tool to put in the hands of someone who knows what they're, they're doing and is able to validate it and is able to say, okay, go and solve this, you know, relatively well constrained problem where I can easily validate the correctness of your output. Go at the sandbox where I know that you're not spinning up massive amounts of resources in a way that I'm going to regret. Or even go at this non sandbox situation, but I can, I have the knowledge to check what you did, look at the logs and be like, yeah, that's okay. Those are really cool things. That could be really valuable. That could dramatically increase somebody's productivity. And those are so far from being something that I would trust independently to replace a software developer, that they're not even in the same country.

• 8:47Speaker 01

  Like, maybe not even in the same world. Like, these are just completely different claims.

• 8:52Speaker 02

  Yeah. I think that the sensationalism of this comes from not what it can do now, but what it represents and the progress that it's made when comparing to other things like. Whatever it was comparing that, you know, 13% to, to other AI chat things that can do things, it's way better than all of those. It still sucks compared to a human, but it's made like monumental progress in terms of AI. And I guess the question is, does that continue? Can it get further than that? Or will it reach some kind of limit? And then the other piece of it, I think, just from a marketing thing, and I'll be honest, the only thing I've seen on it really is a, a fire ship video, is that it's, it's already

• 9:34Speaker 02

  doing some work on Upwork. So in a way, like, that's a marketing claim that they, it competes against real humans for jobs.

• 9:42Speaker 00

  Truth. According to them. I haven't confirmed, but what you said is true that they say that. Yes.

• 9:47Speaker 01

  So this is the struggle with it, with all of the LLM world right now and all of the AI world. Cause on the one hand, you get, we, we have been in a place where we're in the rapid part of an S curve. There have been some very rapid advancements in the core capabilities of these things. And they are super freaking cool. Like really cool. And also, they have a lot of limitations. A lot of those limitations are baked into the architecture that's being used. And so you get kind of a situation where, like, there's a bunch of people doing really cool stuff with this and like figuring, trying to figure out what it's good for, but it demos way better than it does anything reliably

• 10:29Speaker 01

  in production, because you can get a really cool outcome, you know, 40% of the time, some situations, 70% of the time. And like, you show that and people are like, oh my gosh, this is going to take over the world. And I would not trust a, for example, AI software engineer that even that they could handle 70% of my tickets, but 30% of the time spins up millions of dollars of costs for me. Right? Or like other things. And once again, like, I'm not trying to take away from the technology, but I don't think these hyperbolic claims actually serve anyone, except for getting attention. They get attention. Okay, great. And you're going to get a whole bunch of people who buy this thing are disappointed.

• 11:12Speaker 01

  If it costs them a bunch of money, they'll sue your ass off. And like, why would you do that to yourself?

• 11:18Speaker 00

  It's somewhat similar to generative AI in the image. Let's just stick with like static image world where everywhere you see is impressive results. And they'll be like this new like mid journey seven is off the charts amazing. Here's nine examples that'll blow your mind, right? And if you click through on that, they're all going to be very impressive. Like those are amazing things, but then you have to stop and think, well, mid journey didn't create nine examples. That blew my mind. Mid journey probably created 40, 50, maybe 500 examples. And then you human decided which ones were amazing and you cherry pick those out as the examples.

• 12:00Speaker 00

  And like, that's a great teamwork, guys, right? Computers plus humans equals better results. And so there's the cherry pick and that's the, I mean, that's what code review on these things will be. That's what happens when you tell copilot, no, I did not want that function. Right? It's all, as hipster brown calls it in the chat room, human in the loop. And that's exactly what is necessary. And I think the reason why you call them hyperbolic claims, K ball, is because they're saying it's a fully autonomous AI software engineer. Human out of the loop, let it rip. And maybe fans of the bear will like to say, let it rip. But those of us who aren't fans of Devon are thinking, let's not let it rip too much because it might just tear the whole thing down.

• 12:45Speaker 00

  Now I'm being hyperbolic. Nick, you're not in a long, do you agree with me?

• 12:48Speaker 02

  Somewhat, yeah. I think like, yeah, it's humans who are deciding what is good out of that and kind of helping to train that going forward. But in a way, like I was trying to think and trying to relate this to another article I saw. It wasn't about Devon specifically, but was about like prompt engineering as a quote unquote profession being taken over already by AI. Because an AI can iterate and more quickly come up with a way to answer the questions that you want by upending exactly what it wants to hear at the end of a string. And I think. The example that I I heard from that was like, we want you to answer this question and it, the AI is quote unquote incentivized to answer it a little bit better if you put it into a scenario

• 13:31Speaker 02

  that it likes. So the AI is Captain Kirk on the Enterprise and it has to answer this question to save a planet from whatever. And the question could be, what's two plus two or something like something really simple. And by putting in all of these extra prompt words that the AI is coming up with on its own, it's making better results overall. And I'm just wondering how that marries to the idea of humans being the ones who curate the, the good ideas that come out of it.

• 13:59Speaker 00

  Well, prompt engineering, I have been convinced by Swix that it's a code smell.

• 14:03Speaker 02

  Yeah.

• 14:04Speaker 00

  Like we, I was, at first I was convinced like this is the new thing that everybody needs to learn. And I think it's just the, a leaky abstraction that's we're currently dealing with as humans because the tooling is not good enough so that we have to engineer the prompts. I mean, Google's search box is prompt engineering, right? Like knowing how to Google is, it's the exact same thing. It's just way harder and it's like way more magical now to like tell it the magical incantations to get the best results back out. And so the fact that it knows what results are better to me is not intelligence or anything. It's just like, we just need that to go away. And I think that's, I think Devin's actually an example of where they've productized and hidden a lot of the innards that we've currently been exposed to in order to make

• 14:49Speaker 00

  the tool work better than it would for an inexperienced user to use it. Like they've actually turned it into a product. And I think that's great. I think it's one step on a long line of iterative improvements that will make it so that prompt engineering, I mean, you're just going to basically talk to it in layman's terms and it will know how to feed itself the correct prompts, so to speak, in order to get the, to get the goodness out. But I don't know. Okay, well, back to you.

• 15:16Speaker 02

  Yeah, I mean, I think so, high level on all of this AI stuff is, there's really cool stuff there. We're figuring out how to use it and the current state is, is clearly intermediate. However, the thing I want to keep coming back to with this is like, there are things that is like, okay, this technology is immature and we're going to evolve around it and, you know, figuring out how we handle prompts and managing prompts and what's generating them and whatever like that, that fits well in that bucket.

• 15:44Speaker 01

  And there are things that are fundamental pieces of the way the technology is designed, right? LLMs, machine learning models in general are statistical, probabilistic. They're very different than most things you think about in software where you're trying to make something that is logical, consistent, like you could put A in, you get B out. And that is not there with these things. And so you can design applications around that. And there are things that you can do to, to sort of pin that down, to add validation that is outside of the LLM and do other things. And maybe Devon is doing that. But I think the more we start looking at these sort of, you know, places that require judgment,

• 16:28Speaker 01

  places that require precision, places that like, if you just make some random [bleep] up, it can cause a lot of problems. Like, those are not actually, like, there's a fundamental thing about what the technology does that means it's not necessarily going to be a good building block for that. And so making hyperbolic promises about where it's going to develop that depend on it being a fundamentally different technology than what it is, feel like they are setting yourself up for a lot of heartbreak.

• 16:55Speaker 00

  What about the job market? Do you think it's fundamentally affected by tools like Devon as they progress over the next three to five years? Because we're not talking about humans out of the loop. I think we're all in agreement here that that's not feasible or smart, at least in today's technology plateau of LLMs, but less humans in the loop. You know, that seems like it's very feasible if these tools continue to iterate and even just not have revolution, but evolutionary advancements from here.

• 17:26Speaker 02

  Yeah. If it makes me three to five times faster, do we need three to five times fewer engineers?

• 17:31Speaker 01

  Yeah. I mean, I think there is. So this is a technology that has the potential to dramatically impact the productivity of software engineers.

• 17:40Speaker 01

  And I think there's a couple of different things around that as we think. So, short term, that can create some disruption, right? Short term, that means that a company that had been running on, say, five engineers and might have needed to hire and expand to 15, now they don't have to expand nearly as soon and things like that. So I think there is the potential for relatively short-term disruption. I will say both the history of economics broadly and software in particular is that every time we make it easier to code, we discover there are whole worlds now that we can address and build software around that we couldn't before. So if, for example, and this actually, there's a particular example of this that I think is interesting to dive into.

• 18:21Speaker 01

  So, one of the big economic challenges in the tech industry in the last four or five years is that we had these massive tech companies with incredibly high revenue per employee. Google, Meta, Netflix, like the fangs mostly, right? And so they were able to set the salary bar that was super high. They were paying ridiculous amounts of money. That's a technical term, ridiculous, for software engineers. And then when we had very low interest rates and a ton of VC money flowing in to the industry, there were lots of companies whose fundamental business economics do not support that level of salary per software engineer,

• 19:05Speaker 01

  who were nevertheless paying that amount of salary per software engineer based on VC capital. And sort of this thesis that, okay, we'll be able to scale out of this and we'll get whatever. And I think that caused a lot of distortions and problems in the field. Now, if suddenly software engineers are three to five times more productive, the range of businesses that could use software but previously could not afford to compete with the fangs, et cetera, of the world, there's a whole set of business models in there that become viable because it's that much cheaper to develop software. And so I could imagine this actually dramatically expanding the number of viable either software businesses

• 19:48Speaker 01

  or businesses that are non-tech but would like to include software or could have custom software, and dramatically expanding the number of those that happen. So I think, long-term, I don't think it's a negative impact on the software engineering career path. I think that there's what it means to be a software engineer looks a little bit different when you have different types of tooling. That has been true as long as I've been around. Like I just JavaScript land, right? I remember when jQuery was a revelation. Oh my gosh.

• 20:18Speaker 00

  You're ancient, dude.

• 20:19Speaker 01

  This is gonna make me so much more productive. It did make me so much more productive, all these other different things. And now the level of tooling that we have there that supports our productivity, building things in the front end is astronomical. And has that taken away from the number of people writing JavaScript?

• 20:33Speaker 00

  Speaking of astronomical, Astro has a new database.

• 20:51Speaker 03

  What's up, friends? Is your code getting dragged down by joins and long query times? The problem might be your database. Try simplifying the complex with graphs. A graph database lets you model data the way it looks in the real world instead of forcing it into rows and columns. Stop asking relational databases to do more than what they were made for. Graphs work well for use cases with lots of data connections like supply chain, fraud detection, real time analytics and generative AI. With Neo4j, you can code in your favorite programming language and against any driver. Plus, it's easy to integrate into your tech stack. People are solving some of the world's biggest problems with graphs. And now it's your turn. Visit neo4j.com/developer to get started.

• 21:35Speaker 03

  Again, neo4j.com/developer. That's N E O four J dot com slash developer.

• 21:48Speaker 00

  Up next, we have a GoTime Podcast fireside chat between longtime programmers Kent Quirk, Sharon Diorio, Stephen Pyle, and host Johnny Borsico.

• 22:02Speaker 04

  It's one thing to have Jan AI, you know, pump out snippets of code that is part of a larger whole, right? Whereby I'm the engineer, I'm engineering a solution, not just, I'm not just a code monkey, just clacking out, you know, syntax. I'm trying to fix a problem. I'm engineering a solution to a business problem. Now I could go, you know, as high level as I can, I can just open up, you know, Copilot in chat mode and say, hey, this is what I'm trying to accomplish, start spitting out, you know, files, right? Now, maybe today, right, it can build somewhat trivial apps. I've seen YouTube videos and clips and things of it spitting out entire working React apps and all these things. And that's, that's great. And I think over time, it's going to

• 22:45Speaker 04

  come even better at doing those things. But I have a hard time trying to sort of correlate that or trying to replace solution building. Cause to me, solutions aren't static, right? When a business comes to me and says, hey, I need you to build a solution to this problem. I build it, they take it into production, they do stuff with it and they come back and says, hey, you know what? This is great. Now I need to change it in this way, or I need to account for this, this exception, or I need to account for this particular use case, or this specific customer, where 90% of the time it works like this way for every customer of this type, but for this customer of that type-

• 23:25Speaker 05

  But on alternate Thursdays during the full moon, it does this completely different thing.

• 23:29Speaker 04

  Exactly. So now what, what, what am I like, how, how are we supposed to treat those like entirely made up solutions that, you know, am I just feeding that back into the system and saying, hey, so I can't now account for these, you know, alternative approaches?

• 23:46Speaker 05

  Is it gonna be like it was when the first generated code frameworks started hitting the scene? And you'd go in and there'd be all this code. It was like, yes, super fast. If you had to do an ORM, it wrote all the code for you, et cetera. And then you needed to change something. And all of a sudden it was like, eh. You know, change management in some of those days was

• 24:07Speaker 06

  Yeah, or regenerate the whole thing from scratch and oh, sorry about all your customization.

• 24:13Speaker 06

  Yeah.

• 24:14Speaker 05

  So that'll be another big test that AI has not yet proven it can do.

• 24:19Speaker 06

  Well, so, so let's talk about art for a second, though, because this is again, a similar thing like everybody's really excited. Look at the images I can generate with, you know, Mid Journey or whatever.

• 24:28Speaker 05

  With stolen art.

• 24:29Speaker 06

  Well, right. But, but the point is, again, it's going out and giving you the average solution. It's going out and going, here, here are the things that look most like what you described that are somebody else's created already and kind of gonna cobble pieces of that together.

• 24:42Speaker 05

  Or here's an opinion formed by the loudest voices out there that I sucked up as source data.

• 24:48Speaker 06

  Yeah, hell, yeah. But like, I sat in a, a meeting today where an artist went over her design, basically her design process for a big design project. Like, here's the resources I looked at. Here's the, you know, the feeling I was going for. Here are the things I considered. I looked at these typefaces. This typeface reminded me of this, you know, building architecture, which is relevant to the site where, you know. And then that artist proceeded to churn out, over the course of a couple of months, 200 pieces of, of support art for an event.

• 25:25Speaker 06

  That was a brilliant design exercise by somebody deeply steeped in art and creation, who then studied the event and what the event needed and integrated all that. And yes, some random person could have sat down with Mid Journey and said, make me this stuff. And it would have been much less good.

• 25:50Speaker 06

  But people who don't know the difference would have been, sure, it looks fine. You know, I mean, we've all seen that, right? You know, my, my document has 37 fonts and 12 colors, but it looks fine to me. But like, there's a big difference between, you know, something crafted and something just slapped together. And, and yeah, I guess I think that AI is gonna make it easier to slap together, but.

• 26:14Speaker 04

  But for, for most people, though, would you argue, so here's what I'm not saying. I'm not saying that these things generated by AI, like if you're a cornoisseur of a certain particular art, you know, you're an architect of a particular kind of application or solution or, you know, or thing, whatever it is, you could, you can critique. Right? The output of GNI, as it stands today, again, arguably, it's gonna get better at what it does, right? But you can critique the output today and be like, ah, this is subpar, right? This is not as good as what I could have come up with. But for most people, it's good enough.

• 26:50Speaker 06

  It depends on what they're using it for. And so again, if you're just doing something for yourself, who the hell cares? I mean, yeah, I'll slap something together out of two by fours if I'm building it for my garage. I don't care. But i- if I'm gonna sell it, if I'm gonna make a business around it, that's the part where I'm saying, I don't think the AI stuff is there. If you're just doing a hacky project for personal use, yeah, I mean, maybe you would have had to pay somebody to come in and slap that shelf together in your garage if you didn't have the skills to do it yourself. And so now there's this kind of, you know, yes, there's a few things that I couldn't do before that now I can do today for myself. Design that invitation for my kid's birthday party. Hell, yeah.

• 27:31Speaker 06

  I can't draw, but I can use an AI. That's, there's nothing wrong with that. And yet, you know, so yeah, there's probably some, you know, the kid next door you're not paying 20 bucks to to do that for you.

• 27:41Speaker 04

  But that's now. What happens in, you know, in the future as AI evolves and it improves, or, you know, do we get to this uncanny uncanny valley level of like, oh, no, it's not just good enough. It's like.

• 27:51Speaker 07

  It's like the standard. Now it's building your whole kitchen.

• 27:54Speaker 04

  Right. See? Do we need to worry about that?

• 27:57Speaker 05

  And how much coding out there is most of what's out? You know, how many crumbs have we ever created in our lives? How many crumbs are still being created every day? Yeah, okay, so that's a problem that's largely been solved, greater or lesser degree, but.

• 28:12Speaker 06

  Yeah, I mean, it's white box, right? You make white box easy to do.

• 28:15Speaker 05

  Yeah. And, and tech has always been making things that, that had gates around them or real or created limited availability or, and making them more available, like artisan things that used to be only certain artisans could do that. Only musical artists had a studio and an audio engineer. And now they can go and, you know, create their own tracks with one app at home. For me, that's the part that's like, oh, I can't complain. I can't be the curmudgeon complaining about this latest thing that might make something that I do more accessible to other people. Like I've benefited from these other things that came along. It's time to share the wealth. I don't want to.

• 29:00Speaker 05

  I'm really rooting against it.

• 29:04Speaker 04

  Yeah.

• 29:05Speaker 06

  I don't think it's about gatekeeping. I mean, I'm not like, I feel like a big chunk of my career has been spent trying to help people learn to program. And so I'm not thinking that the reason I'm skeptical is because I don't want other people to do what I do. I think it's more because I feel like the hype and the reality are distinct. That what the reality is producing is mostly devoid of creativity. People are confusing knowing what to look up with being creative. And I think knowing what to look up is a skill and a lot of us have it. And the better engineers, I think, are better at it. And so, yes, AI helps to ease that problem.

• 29:49Speaker 06

  But knowing what to even ask about,

• 29:54Speaker 06

  you know, like, or looking at a new solution to a problem. That's something that I think is well beyond what's what AIs are capable of now or in the reasonably, like, the LLM model, I think, is fundamentally non-creative. That's my take on it.

• 30:11Speaker 04

  Hmm. Spicy. We're not at the unpopular opinions yet. So one thing you mentioned, like, the whole teaching, like, y'all remember when, um, maybe it was during, uh, maybe first or second Obama term or something, but there was this giant push to teach everybody how to code. Right? Like, it was, it was everywhere. It was in the media, it was in newspapers, it was everywhere, like, we need to teach our young how to program. Now, I'm looking at a clip from, you know, Nvidia CEO, like, I don't know, three or four days ago or something, saying, hey, people shouldn't learn how to program. You should now let, you know, the new programming language is a human language.

• 30:51Speaker 04

  I'm thinking, man, you are sitting here, you stand to gain billions of bajillion dollars, right? If your wish comes true, you know, because you're producing, you know, chips and stuff for these things. Of course, you're gonna say that, right? So this, this, so, but I mean, what? I'm definitely not on the don't teach people how to program camp.

• 31:08Speaker 05

  No, no. I'm gonna, I'm gonna take a slightly spicy take here. I don't think he's completely off. Now, in the time we've all been engineers, we've seen waves of different things that are gonna come and take our jobs. You know, offshoring and, you know, code generation, now AI. And they haven't. And my theory is that the key thing that an engineer has is the ability to communicate. And even when you're supposed to be communicating to people on the other side, product, the business, whatever affectionate term you use for them, aren't always as good at that, although it should be part of their job. But having somebody who can think back

• 31:50Speaker 05

  and forth, there will, I think, always be a need for those people. Because every, every CEO thinks they have the answer to every question.

• 32:00Speaker 08

  That's what they hate to do.

• 32:02Speaker 05

  Right? But they, they really shouldn't. If they have a business that's big enough to grow, their biggest skill is finding the people they put in the right place. So if your job right now is like doing creds for a company that can't even explain what they want, I wouldn't worry, because they're not going to be able to explain what they want to AI.

• 32:21Speaker 06

  Right. Yeah, no, it's the thinking logic. It's, yeah, there's the think about, break something down into steps and think logically. Like, I once did have a client, very early in my career, who was a pretty good business person, who really wanted to automate his business, and he was able to sit down and explain it to me. Like, if he had had the tools to program, he could have written his own code, because he thought about it really logically. And, and it was just my job to basically take dictation and turn it into Pascal for him, then, back in the day. But, but that was, that's few and far between. You know, quite honestly, most people who specialize in business aren't specializing in thinking logically. They're specialized in thinking about people, and, like

• 33:06Speaker 06

  you said, about communications.

• 33:07Speaker 04

  So does AI then make you more what you already are? If you're a logical thinker, you'll benefit? And if you're not, you still struggle?

• 33:15Speaker 05

  And who gets to train the agent?

• 33:19Speaker 04

  I wanna be in the training side. I wanna be the one doing the building of the things that you use.

• 33:27Speaker 06

  Fascinating. It's a good question.

• 33:30Speaker 07

  I think at the point where it becomes like a personal AI, where it's just like, it's tuned to you, your data doesn't get shared. It's just, you know, then it becomes like a superpower, right? You can just, it's like you're co-pilot.

• 33:44Speaker 05

  But how would that work if it's only got your data? It's basically replicating to appoint you.

• 33:50Speaker 06

  There's a generic, it's trained on the universe.

• 33:54Speaker 07

  Right.

• 33:55Speaker 06

  And then specialized for you.

• 33:56Speaker 07

  Right.

• 33:57Speaker 06

  Is, is the way that we're seeing all of this.

• 33:59Speaker 07

  Yeah. It's like creating your own, your own GPT, but it's based on a larger model. Right? Yeah, exactly.

• 34:05Speaker 06

  So, I mean, like my company, we built, we have a query engine that looks like SQL. And then we built an AI where we train that AI, like as part of a, part of a prompt. We basically can go out and get your data and all the names of your fields and the data types of your fields. And we can plug them into the AI query so that when you say, show me my slowest service. It can go, all right, what are the fields that are named according to, you know, time duration? And what are the things that look like a service name? And now I can write a query for you. So it knows how to query Honeycomb.

• 34:46Speaker 06

  And then it can write that query for you from your inept prompt. Because it's been specialized for that particular type of application. And I think that's a really cool use of AI.

• 34:58Speaker 04

  That, that is, that is a productivity boost for, for people who are already technical. Like, you know, like full disclosure, I, I am, my startup is a Honeycomb customer. So, you know, I, I, I will go in the dashboard and I can formulate those queries. But I'm already a highly technical user who knows how to use these kinds of tools to get to, and know exactly what kind of data I'm looking for and when I've found it. Now for the layperson, right? Who doesn't know, like the layperson, like.

• 35:28Speaker 04

  Well, the more I think about it, the more I'm thinking, okay, if I'm a, on one end of the spectrum, you have the complete layperson who is using perhaps, you know, chat G P T or something like it to maybe generate copy and not hiring a copyright person like you might traditionally do back in the day. Right? I'm sure the copywriters of the world are suffering right now because Content creators. Like content is like, yeah, content creators are suffering because this stuff is now being generated. So are all our Google searches. Right. So if, if that was your job, absolutely you're impacted, right? And the layperson can now bypass you and get to something that, again, the good enough, right?

• 36:12Speaker 04

  They can get something good enough to achieve some means, right? On the, the, the complete opposite end of the spectrum, you have people who engineer software, right? That, again, given the context of the conversation, we're talking about like how safe is our jobs, right? And so when I'm asking this question, I'm not asking, is the layperson going to find ways of reducing, right, their reliance on sort of, I don't want to say lower skill, just the different kind of skill, right? I'm thinking like for people like us, as software engineers, who presumably will be impacted by this to some degree, right? And we already are, right? For us, there's also the micro spectrum whereby if you're on the sort of the lower end of that spectrum,

• 36:55Speaker 04

  and if the only thing we are doing is generating crud, well, I'm sorry, your job is indeed in jeopardy. If that's the only thing you've been doing, right, with your career. On the opposite side of it is the highly specialized person who understands a business problem has to debug and troubleshoot and then talk to people and integrate different things and Institutional knowledge. Right. All that stuff. I mean, I don't see that skill, right? I don't see that being replaced by A I anytime soon. Am I wrong here?

• 37:27Speaker 06

  I don't think so personally.

• 37:29Speaker 05

  How many of those people do we need?

• 37:31Speaker 07

  And that's the thing, right?

• 37:32Speaker 04

  I mean, is it the game of musical chairs? We should be looking for a chair now?

• 37:36Speaker 07

  No, any business is thinking, do I need a thousand engineers? Right? When 500 will do.

• 37:42Speaker 06

  I mean, I think as we've all said, right? It makes us more productive today. So I'm writing more lines of code per day than I was five years ago.

• 37:51Speaker 07

  Right, right, right.

• 37:53Speaker 06

  So that's good, you know, but we all kind of expect productivity to continue to rise. So this is a productivity tool.

• 38:02Speaker 07

  Not a replacement tool.

• 38:03Speaker 06

  You know, we're also using like using languages that are more expressive than they were. You know, like the code I write in Go is probably one third the length of the same code I write in in C++ or used to write in C++ back in the day. So that's also a productivity boost at some level. At least if you believe the old metrics that it's basically you can write the same number of lines per code of code per day, no matter what language you write it in.

• 38:29Speaker 07

  But I think actually knowing how to use it is a skill. Like to put on your resume, but not before too long or if not now. I mean, I even if I didn't want to use it, I would, because it's becoming of like, it's going to be a point where like, oh, you know, I use copilot all the time. Oh, good. You've the point for you to get the job.

• 38:51Speaker 04

  Is prompt engineering on already in your language?

• 38:55Speaker 07

  It's gonna be soon.

• 38:57Speaker 05

  No, but if you put it in your interests as AI, it shows up in the keyword search.

• 39:02Speaker 07

  There you go, right?

• 39:06Speaker 06

  I mean, that's a good point, but it but to me, it's like back to my woodworking thing. It's like, I know how to use a power saw. I know how to use a drill press. I know how to use lathe, you know? Those are those are kind of expected. Today, if I'm gonna do woodworking and say I only use hand tools, people are gonna look at me like, I don't have time for you.

• 39:24Speaker 04

  Same thing.

• 39:25Speaker 06

  And same is true of like, if you're not using Copilot, what am I paying you per hour?

• 39:29Speaker 04

  What are you doing?

• 39:30Speaker 06

  You know?

• 39:31Speaker 04

  Why are you not, why are you not as productive as you could be? Yeah.

• 39:34Speaker 05

  I think at this point, the only people who really aren't using it are people who are doing like very arcane languages or people who their businesses, they don't allow it, their company doesn't allow it. I think everybody else has at least tried it.

• 39:46Speaker 04

  I mean, if you're a company that doesn't allow such things, like I understand not bringing sort of open source code into your organization that might be the wrong licensing model for you or something like that, right? You don't want to be in some hot water. All you have to do is look at, you know, Oracle and Google over the whole Java thing. I think those were the companies involved, but if you allow your engineers to use a model where you can control the kinds of things that were used in the model, right, for the training? And you can have maybe you can run your own internal, right, GNI for code generation, whatever it is. Right? I think if you're an organization that is afraid of these things, you should at least follow that route as opposed to saying, hey, nobody can use any GNI

• 40:30Speaker 04

  coding tools whatsoever, because I think you're gonna lose people if you do that. Because, because I'm gonna look at my peers that are, that get to use these things and are, they're learning those skills. Right? And then now I'm falling behind because everybody's using, you know, some sort of code generation tool and I'm not. Right? I mean.

• 40:49Speaker 05

  This is where I hope somebody reaches out, somebody who hears this and reaches out and can answer that question of like, can you have a copy of Copilot that you train on a specified set of repos? And only those repos?

• 41:03Speaker 04

  Private repos.

• 41:04Speaker 05

  Well, I mean, I would, you know, as we joked, I mean, if I'm a Go developer, I'm training it on Johnny's code. Because if I don't like it, I can mail it to Johnny. But.

• 41:15Speaker 04

  Nice.

• 41:16Speaker 05

  You know what I mean? There's people out there.

• 41:18Speaker 04

  One neck, a choke. I get it. I get it.

• 41:20Speaker 05

  It doesn't matter what language it is. There's people out there that that you respect their code. And you'd be like, yes, I would like my code to be more like this. I wish it would learn that that's what I was thinking, or that's the way this problem should be thought about. I mean, yes, there's precious few of those people, and those people will probably never lose their jobs. But for the rest of them, you're mortals that are going to have to work with the the tools that are out there. And I would love if this is a possibility now that you could train Copilot on what you'd say to train it on and not all of GitHub.

• 41:48Speaker 06

  I think there are companies that are working on that product. I feel like I've even seen a product announcement like it. But yeah, I mean, you know, the thing about it is you can take one of these LLMs, and you can essentially subset it, and you can make a tiny, compact LLM that will run in a box that you can actually stand on your desktop. And then you can further train that with new information. So so that's exactly what you want to do here. You want to take a coding centric LLM like a Copilot, and create the mini version of it, and then train it on your repositories. And now it knows how to write your code, and it's also not talking out to the cloud while you're doing it. So there's got to be businesses like that if there are.

• 42:25Speaker 05

  And this is where we redact all of this and put it in at our business plan.

• 42:39Speaker 03

  What's up, friends? There's a new book out there called The Hacker Mindset. This is a productivity cheat code to unlock new levels of success in your career, in your creative pursuits, and in your personal growth. This book is about leveraging the principles of white hat hacking and applying those skills to the broader world. It's available for preorder right now, and it's not your typical productivity guide. This is written by Garrett G, a seasoned white hat hacker with over 20 years of experience. This book reveals the secrets of hacking, and how you can apply those skills to overcome obstacles and achieve your goals. So don't miss your chance to get ahead and get this book, The Hacker Mindset. You can preorder

• 43:22Speaker 03

  your copy today at thehackermindset.com. Be among the first of many to tap into this power of hacking for your success. Join the movement and embrace a new way of thinking. Again, that's thehackermindset.com.

• 43:45Speaker 00

  Last up, we have Jose Valim, creator of the Elixir programming language on Changelog and Friends. That is our talk show flavor of the Changelog Podcast.

• 44:00Speaker 03

  To Jared's point earlier, AI, as it's known today, GPT, ChatGPT and others are not that good with assisting with Elixir programming. And so I guess the question is, what does it take to make it good? You mentioned embeddings earlier. You mentioned documentation being more readily available. What does it take from a, I guess, a leader in the Elixir world to enable LLMs to be better? Like, what role do you play in that journey for them to better consume the documentation and better know how to programming in Elixir to help folks like Jared and myself, or our team, or others to really become better and more proficient in Elixir versus just like anytime Jared asks ChatGPT for assistance, it's just like, no, it's not good. So just quit.

• 44:46Speaker 09

  You know, so I think if I got the question right, I think we did our work correctly in the sense that, at least from the language point of view, in the sense that like documentation was always first class. So documentation is very easy to access. So if what you want to do is to like configure an LLM, it's actually very easy to access that programmatically send that, extract information. And we talked about like, one of the things that you also have to do is like try to get understanding from the source code. So you can find, oh, this code is using those modules, is importing those things. And those are things that you can do relatively

• 45:28Speaker 09

  easily in Elixir. We can most likely improve that. So I feel like we have the knife and the cheese. It's just a matter of somebody going and like cutting the cheese, you know? It's, yeah, I feel like the foundation is there in terms of like having this information is structured, but somebody needs to feed it somewhere. But again, we can go back, like maybe it's a corpus size, like maybe ChatGPT like indexed HacksPM already? Not sure, right? Maybe it has done that. I don't know. I don't know if I can send a letter to somebody, hey, please index my website.

• 46:07Speaker 09

  Or maybe it's a matter of, so one of the things is that Redmalk, they have, they release twice a year, like kind of a graph plotting GitHub against Stack Overflow. And I think they're having, like, the most popular language is according to GitHub and Stack Overflow. And then there's like a, you know, a linear thing in the middle. And it's very funny because Elixir is high on the GitHub side, but quite low on the Stack Overflow side. And one of the reasons for that is because we have always had the Elixir forum. So that may be one of the things.

• 46:43Speaker 03

  Right. Where's the knowledge? Where's the back and forth from the community?

• 46:46Speaker 09

  Yeah, the knowledge is in the forum. Is that thing being indexed? Because we know Stack Overflow is, right?

• 46:52Speaker 03

  Yeah.

• 46:52Speaker 09

  And.

• 46:53Speaker 03

  Right.

• 46:53Speaker 09

  So, and ironically, that's one of the reasons, I think, I may be misquoting that Redmalk, they are considering removing Stack Overflow from their plots because it's prone, like, I think it has been losing relevance in the last years, right? But, you know, maybe in the effort of trying to have a closer community where everybody can engage with each other, where I'm active in the forum and I'll probably not have this patience if I was doing with Stack Overflow, right? We created our community a special place, but it's not known. So, yeah, so I think, I, I think it's still like too many, too many unknowns. But I think at the core, at the core, we unwillingly did a good job.

• 47:38Speaker 09

  Because we were worried about documentation being accessible, documentation being first class. So we did that and that can be, and we promote people to write documentation, lots of documentation, right? So there is a lot there. And yeah, and maybe the RAG is going to be the thing that is going to be enough. That's one of the hopes, right? Going back to, we want everybody to be able to use this. If RAG is good enough, then a lot of people would be able to augment their ecosystems without depending on, on OpenAI or whatever. But we are still evaluating.

• 48:10Speaker 03

  When you, when we talked about sort of the, the long-term future of Elixir, artificial intelligence and that sort of larger topic of, you know, how long will be relevant and, you know, can AI generate it well, that whole conversation? This makes me think of this necessity to not have a black box that is whatever AI is. Because just like you said, who do I send a letter to to index my stuff so that my very relevant language today remains relevant tomorrow because tomorrow says AI will continue to be more and more relevant to developers in their journey to develop, right? So who do we send the letter to? How do we know?

• 48:51Speaker 03

  Well, currently, the status quo of AI is, for the most part, a black box. Obviously, open source LLMs and indexes have become more and more pushed because of this challenge, but I think this illustrates and highlights really the long-term challenge because even you can't say for sure why what wasn't indexed was indexed for the Elixir corpus, whether that's the forums, whether that's the documentation through Hex documentation or whatever. It's unclear to someone like you how to enable ChatGPT or the likes to better support Elixir assistance for developers using those things to use as tooling. And that's just not cool because long-term, we need to have inroads into those places

• 49:36Speaker 03

  so that we can be part of the future if AI is predicting how we'll get to the future.

• 49:42Speaker 09

  Yeah, and I think, yeah, it's too early. I think we're going to improve a lot. I was listening to a podcast today where Sam Altman, he was saying like, they improved ChatGPT 3 about 40 in the orders of magnitude in terms of size, performance, and things like that since they started. I think 10 times for ChatGPT 3 and 1/2. And I think open source is going to catch up, I think, and I think that's the hope. But yeah, it's also like we go back to this when we are thinking about Live Book because what I want is for open source to win, right? But when I'm building a feature for LifeBook, right? I need to build the best feature for the users, right?

• 50:26Speaker 09

  And when I can use ChatGPT 4, right? And I can immediately see the results and they are really, really good, right? I can use other tools off the shelf, they are not as good, right? So we are a small company, we are doing open source. So my options, if I have to choose for my users, is going to be ChatGPT 4 because it gives me the best result for the least amount of effort, right? I just, it's there. And this is like, so we were back in like about my indecision about investing this stuff. Exactly because I want open source, right? I want things to be open source. But right now, the quickest return of investment is GPT. And then I am in this contradiction space, right?

• 51:11Speaker 09

  But yeah, and it's just, I think it's just patience. We have to be patient and, you know, I think probably in one year. And the whole thing is like, it's crazy to think about is that this thing has been happening for a year only, right? It appears that this thing has been out for so long, but it's a year. And I think like, if I'm back on the show in a year, we may potentially be having a very different conversation. So yeah, we'll see.

• 51:35Speaker 03

  Do you have any fear about this? Like even as you respond to that, you sort of had some, I wouldn't say like trepidation in your voice, but you sort of had some uncertainty. Do you have any like fear and uncertainty and doubt, the FUD that people sort of pass around? Do you have any fear about this?

• 51:48Speaker 09

  No, not really, in the sense that I consider myself like very lucky, very fortunate or whatever, or blessed, whatever you want to say it. I think maybe it's a, I'm not being overconfident here, but more like thankful that I think whatever happens to me, it's going to be fine. I truly believe that what's going to make Alexa survive is the community more, you know, than whatever technological changes, unless there is something very drastic. I talked to my father about this, about investments, right? So like when Bitcoin wasn't a crazy, right? And then my father is like, oh, have you heard like about this thing? That if you put your money there, like people got this huge return.

• 52:32Speaker 09

  And then I always told him, father, if we got to know about it, it's because it's too late. You know, it's like, or if something happens, right? It's like, oh, if something happens is because, like, if something goes this bad, it's because it's going to be bad for everybody. So like, don't try to fight it, right? So again, like, unless there is a very major change, I think I will be fine, right? So I'm not worried about me in the sense, I always think more about, it's more about ideals, you know? Again, like, I like to say, well, me 10 years ago, that's where my trepidation is, if things go like closed source, you know? And those things, they happen by, we don't see the results.

• 53:14Speaker 09

  Like, I think another polemic topic about this, it's like, hey, I use Chrome. As soon as Chrome came out, I, today I don't use Chrome anymore. But as soon as Chrome came out, I immediately swapped to Chrome, right? And if I had known that this would lead to a point where, you know, Google is in this position where it has a lot of control over the browser, over the web, right? And over how we use the internet, like 10 years ago, I would probably not have used Chrome. If I could have seen it, right? Or, so I think, I think that's where my trepidation comes from, of like, things being closed source, like developer experience.

• 53:54Speaker 09

  So today, another example today, like, Elixir was the first programming language that GitHub had like the new navigation, code navigation things that were provided by the community, right? So there were some programming languages, and there still are, where they have very good navigation and exploration on GitHub UI. And the path for that, to get that feature, to get that behavior, was, and I'm very welcome that, I'm very thankful that the GitHub team, they, you know, they discussed with us and allow us to do that, but that's closed source, right? And GitHub plays a major role over, you know, how developers use, right?

• 54:34Speaker 09

  So it all comes back to this idea of like, if you want to provide a good experience for your users, how much of that is behind something closed source that you have no control? And you are depending on somebody, you know, paying attention. To you, like, or you having a contact, or, you know, me having a name because I was very active in the Rios community that GitHub uses, like, 10 years ago, right? Those are the things that, but I, like, I feel lucky, you know? But I, it worries me, right? Like, how much is being closed, right? How much is, is going to be out of our control? And then the trepidation, I guess, is like, what does that matter? For the small José out there, right?

• 55:18Speaker 09

  Like, let's start building his thing today, and they won't be able to.

• 55:24Speaker 00

  Well, you killed the vibe there, should I say?

• 55:26Speaker 09

  Oh, thank you.

• 55:30Speaker 09

  That's me at parties, you know? So you're just...

• 55:33Speaker 00

  Not invited. Just kidding. Oh, funny. All right, well, let's, uh, should we try to close on an up note? On a high note? On a, on an upper?

• 55:45Speaker 09

  What's that?

• 55:47Speaker 00

  Wow. I had no idea. I think we should end it right there, Adam, don't you think? We ended on a high note. Cheese and knife tactic there, I love it. It was.

• 55:55Speaker 09

  Do you want higher? I don't think it will be good for the listeners.

• 55:58Speaker 00

  I think that was plenty high enough for me. Adam, were you satisfied with that? That was a high note, literally. Yes. And I'm, I dig it.

• 56:12Speaker 00

  Thanks for listening to this very special episode of Practical AI. These were just extracted segments of much longer conversations. If you want to hear more, there's a link in your show notes to each of the episodes featured here. Thanks again to our partners, fly.io, to our beat freak in residence, Breakmaster Cylinder, and to our friends at Sentry. Save yourself 100 bucks on the team plan when you use code CHANGELOG while signing up. That's all for now. Chris and Daniel return on the next one.