Pod Insights Software Engineering Daily Hype and Reality of the AI Coding Shift About this episode AI coding tools have gone from novelty to core infrastructure in under three years. Today, many devs use AI daily, a substantial share of new code is AI-generated, and expectations for automation are rapidly increasing.
Sonar https://www.sonarsource.com/ is a company specializing in analysis of code quality and security, and they recently released a new survey – the State of Code Developer Survey. The survey provides a deep examination of how developers are using AI in real production environments, and where the real-world gaps and risks still exist.
Chris Grams https://x.com/cdgrams is the CVP of Corporate Marketing at Sonar, and Manish Kapur https://x.com/kapmani is the VP of Product Marketing and Developer Relations at Sonar. In this episode, they join Matt Merrill to discuss what the survey reveals about AI-assisted development, why 96% of developers still don’t fully trust AI-generated code, how deterministic verification layers fit into agent-driven workflows, and what engineering leaders should prioritize as AI shifts from experimentation to production infrastructure.
Sponsorship inquiries: [email protected] https://cms.megaphone.fm/organizations/619b680e-d813-11ea-9750-e7ecac406436/podcasts/e60108fe-e328-11ea-b2b6-1348249f900a/episodes/3913370a-bc44-11ed-ace6-8b1f96d82bba/[email protected]
The post Hype and Reality of the AI Coding Shift https://softwareengineeringdaily.com/podcasts/hype-and-reality-of-the-ai-coding-shift/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.
Transcript 10,946 words · 4 speakers
0:00 Speaker 00 AI coding tools have gone from novelty to core infrastructure in under three years. Today, many devs use AI daily. A substantial share of new code is AI-generated, and expectations for automation are rapidly increasing. Sonar is a company specializing in analysis of code quality and security, and they recently released a new survey, the State of Code Developer Survey. The survey provides a deep examination of how developers are using AI in real production environments, and where the real-world gaps and risks still exist. Chris Grams is the CVP of Corporate Marketing at Sonar, and Manish Kapoor is the VP of Product Marketing and Developer Relations at Sonar. In this episode, they join Matt Merrill
0:44 Speaker 00 to discuss what the survey reveals about AI-assisted development, why 96% of developers still don't fully trust AI-generated code, how deterministic verification layers fit into agent-driven workflows, and what engineering leaders should prioritize as AI shifts from experimentation to production infrastructure. Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product-focused organizations. His background is in backend development, cloud architecture, and distributed systems design. He currently architects and delivers software products and leads a team of engineers at Depp Agency.
1:26 Speaker 00 You can learn more about his work at code.theothermattm.com.
1:43 Speaker 01 I am here with a couple of folks from Sonar to talk about the State of Code Developer Survey report. So before we get started, Manish and Chris, would you guys mind introducing yourselves and talking about your background and what you do at Sonar? Manish, you want to go first?
1:57 Speaker 02 Sure. Nice to meet you, Matt, and thanks for having us on the program. Chris Grams, I'm the VP of Corporate Marketing here at Sonar, which might immediately make people want to stop listening, except for the fact that I would say- I'm also sort of our resident data and survey nerd. So probably know the survey results coming out of this better than most. I've been in enterprise tech for a long time. I started out early in my career at Red Hat. I spent about a decade and then went on to work with a bunch of other software companies in the consulting role. And then was one of the early employees at Tidewift, which is a company that sponsors open source maintainers and was acquired by Sonar about a year ago. So I've been with Sonar in this role just over a year.
2:41 Speaker 01 Nice. Well, nice to meet you. Manish?
2:44 Speaker 02 Yeah, nice to meet you.
2:45 Speaker 03 Hi, Matt. Nice to meet you. Thanks for having us here. I'm Manish. I'm based in Austin. I've been with Sonar for about two and a half years, almost three years now. Like Chris, I have got a long history with the enterprise software background. I started with Sun Microsystems and then I joined Oracle and here I am at Sonar. I've worn different hats in my life. I am a VP of Product Marketing and Developer Relations at Sonar right now. But I've worn the hat of a product manager, developer relations lead, and different other technical roles and pre-sales and other technical roles as well. So I'm fairly hands-on, technical, looking forward to speaking with you today.
3:21 Speaker 01 Cool. And so how about Sonar itself? For anybody that's not familiar with Sonar, do you mind talking a little bit about what the company does and what products you offer?
3:30 Speaker 02 Yeah, so our main product is a product called Sonar Cube, which has been around for quite a long time and is used by over seven million developers around the world. Basically, what Sonar does is you can think of us as the essential verification layer for code, whether it's AI-generated or whether it's written by developers, that we help ensure the quality and security of all of the code that's created by developers, by AI, and now increasingly by AI agents. So to give you a sense for the scale of Sonar, we analyze 750 billion lines of code every day. So we view our mission as helping organizations ensure the code that they're shipping into production is high-quality, secure, well-maintained.
4:17 Speaker 01 Awesome. That's a great overview. Thank you. I read over the survey report and it was really interesting. And I think as somebody, my background is in engineering leadership and prior to that, mostly backend engineering and things like that. My immediate question was, okay, another survey report, how does this one differ from kind of the Stack Overflow one, which is pretty well known? And as I dove into it, I was like, wow, this is really unique and I really enjoyed it. So if you could, what do you think that we can learn from this survey? That you may not be able to learn from the Stack Overflow?
4:50 Speaker 02 Well, first off, I would say it's nice getting mentioned in the same sentence as the Stack Overflow survey because we aspire to be part of the pantheon of big developer surveys. The Stack Overflow and the GitHub State of the Octaverse report, I think they call it. To us, those are the ones that are help leading, ensure that people have the information they need to make smart decisions as developers and development leaders. So if we are successful with this, we're becoming part of that pantheon of great developer surveys. And I would say where we wanted to be additive is we were like, what unique perspective do we at Sonar have to bring?
5:30 Speaker 02 And I think it really comes down to the fact that we know code pretty much, like I said earlier, we analyze 750 billion lines of code a day. So we started earlier this year, we started this State of Code series of which we did a lot of analysis. And hopefully we'll get a chance to talk more about some of the work we're doing on understanding the coding personalities of the leading LLMs. But we also analyzed code from a perspective of its maintainability, its security, and released a set of reports on that. What we wanted to do with this particular report is we said, okay, we've looked at code through the lens of how maintainable, how secure, how reliable our code. We looked at the code that LLMs are creating.
6:12 Speaker 02 Now we want to look at what is the perspective of the developers who are using these new coding tools every day? What do they think of the current state of the art? And so this is almost like the human side of Sonar's version of the State of Code. Going in and really understanding what people think of the world as it's changing extremely quickly right now. And trying to get a sense for where we are. And one last point I would make on that is that we fielded this survey in October. I think of last year and the world's changed quite a bit even since then too. So as we go through this, I think. We'll want to spend a little bit of time talking about the stats that we feel like are holding true. And I think.
6:55 Speaker 02 Manish and I could give you a perspective on the things that we think have changed as well since we fielded the survey last fall.
7:00 Speaker 01 That will be good to talk about. I love what you said about the human side of it. That was what came through to me is that you really tell. A really nice story with this data. And so I'm curious about, can you talk a little bit about how you collected and analyzed it? Cause the report itself is lovely. So.
7:19 Speaker 02 Again, I've been doing these for a long time. I actually started doing these sorts of developer surveys, probably 20 plus years ago at Red Hat. And did some of Red Hat's first ever research data. And the thing I've always felt about good research is that. By the way, appreciate what you said about it tells a story. Well, because that's exactly what we were going for is that we wanted to not come in. I always come into these projects from the point of view of, don't give me data, give me knowledge, like what is the takeaway? Or set of takeaways. And we also go into the design of the survey with a strong point of view on. Some hypotheses of things that we think may be the case. And we want to go test and see whether people agree with us or not, and whether that's their perspective as well, or.
8:04 Speaker 02 Some places where maybe we're wrong. And there were a couple of examples of places where our hypotheses were wrong in this too, but we. Come into it with a perspective on the stories that we want to tell. And then. Sometimes we come out of it with a whole new set of stories. And in this case, we had a combo of both. We had some. Data points that I think were validated by what we found and some that surprised us.
8:25 Speaker 01 I would love to hear about the ones that surprised you as we go along. So if you think of it.
8:30 Speaker 02 Yeah.
8:30 Speaker 01 Yeah, just definitely drop that. So, all right. We've talked enough. We've done enough meta talk. Let's actually get into the results and let's start with a bang. I think this question is to each of you and Chris, you can go first. What's your biggest takeaway? Right out of the gate, what was the thing that you think resonated with you all the most?
8:45 Speaker 02 Well, I think just at the top line, there is a couple of data points that at the time when we first saw the data were staggering. So for example, seventy-two percent of developers who have tried AI are already using it every day. I wouldn't be surprised if now that number is even higher than seventy-two, but last fall, when we first looked at the data, seventy-two percent seemed pretty amazing and probably a validation of what we expected to be true. But in addition to that, we asked developers to tell us what percent of the code that they're generating or that they're writing today is AI-generated and how we asked them both about their thoughts on that today
9:27 Speaker 02 and then their thoughts about that in the future, like what they would expect it to be in a couple of years. And what we heard is that forty-two percent of developer code is AI-generated or assisted today already, forty-two percent, which is kind of crazy. But by 2027, developers expect that number to jump to about sixty-five percent. So as you look at that again, this is one of those things where we're seeing this, even last January, when I started at Sonar, I think most developers were still looking at the code that was coming out of these AI tools with a little bit of a skeptical eye, like, was it doing a really good job? But the fact that forty-two percent of code is AI-generated
10:10 Speaker 02 as of last fall already is pretty interesting. But here's the other data point that I would add to that, is when we asked developers, how much do you trust that that code that AI generates is factually correct? Ninety-six percent of developers said they do not fully trust the code that's coming from AI. So you have this juxtaposition of forty-two percent of code being written by AI, going up within the next couple of years to seventy-two percent, but developers don't really trust it. That creates like a verification gap or a trust gap that needs to be solved. So that's probably the top takeaway, I would say, from the survey.
10:52 Speaker 01 Right. And that rings true with me, based on my experience in my job. And I'm hearing more and more about agentic code reviews and kind of almost, this isn't the right way to put it, but pitting agents against each other to verify and things like that. So that is a very interesting finding. Manish, what was your biggest takeaway?
11:09 Speaker 03 Yeah, I think I agree with what Chris said. Mostly, I was surprised. The one thing I'm surprised about is how quickly AI coding has been adopted as a use case. If you look back, it was just about two and a half years back when the first GPT versions came out. And here we are talking about more than seventy-two percent developers already using it every day. It's a daily affair for them, right? And this number is from last quarter. I'm pretty sure it's gone up higher now. Fifty-nine percent of developers are using it every day. So it's just how quickly this use case has taken off and how much it has been used across the industry. What I've also seen is while AI is speeding up code generation, it's slowing down everything that happens right after the generation part of it,
11:53 Speaker 03 which is a lot of software engineering work which still needs to happen. Code generation is the first part of the equation, right? Next comes reviewing the code, verifying the code, basically debugging the code, integration testing, long-term maintenance of the code. So I think those things, we are seeing they are not caught up yet in terms of the speed at which code is being produced. So agents will even be a force multiplier. If you look at coding tools and then agents come around, just like what you mentioned, Matt, then there will be a series of agents, a legion of agents talking to each other. It's basically how quickly things will evolve, we have to still see how it pans out. So I'm just waiting for what happens next. So I think for me, writing code is no longer a challenge.
12:35 Speaker 03 It's about what happens after the code is written and what happens in a world where agents are talking to each other and developers are offline or not so much involved in the day-to-day coding process and the SGLC process.
12:48 Speaker 01 Yeah, I am wondering myself with bated breath, though I have to say I'm pretty excited. Every day I get more and more excited about what I see, so. Speaking of that fast adoption, one thing that I'm encountering in my day-to-day life at work is that people, developers want to use these tools. And first of all, they're getting pressure. Second of all, they're starting to become more interested in it. And people are using personal accounts for work things to be able to do this just because they want to move faster or they want to try. And so you did have some findings there in the report that were pretty interesting. So do you want to share some information about what you found there?
13:24 Speaker 03 Yeah, we were surprised by this. There's a significant amount of shadow AI which is happening. By shadow AI here, I mean, we found that almost thirty five percent of the developers are using their personal accounts to access AI tools rather than using the corporate sanctioned ones, right? I think why this is happening is because you are a developer, Matt. I've come from a technical background. So I've been sort of developer, but not truly a developer, but written enough code to be deemed a developer. So developers by nature, they are builders and tinkerers, right? We all want to tinker with the latest and the greatest technology. We want to explore the most modern tools that are out there. So we want to stay at the cutting edge of things.
14:05 Speaker 03 So I think that is part of the reason developers want to go outside of the sanctioned tools as this world is changing so fast. And with agents coming around, it goes even further where developers are beginning to use agents for scanning the repos, for generating migration scripts, for refactoring modules of the old code. And all of those things are happening every single day. But this also means that the code, the prompts, the data, the context information they are sending to these third-party tools, the shadow IT tools or non-sanctioned tools. You're basically taking a risk with the IP and data privacy of your organization when you start using those tools. So you do need a governance and the governance landscape is not very well defined, unfortunately, as of now.
14:47 Speaker 03 It is going to be get very complicated with the agents coming along and the legion of agents working together. It's going to become challenging for some time, but I'm sure we'll solve it like we have solved other problems.
14:58 Speaker 01 Yeah.
14:58 Speaker 03 One thing which remains common in terms of governance is anything that is being generated, whether you use corporate sanctioned tools or whether you use your own personal tools, the code that is being produced has to be verified. After the next steps have to still take place. You have to pay even more, do even more due diligence in terms of verification and assurance that the code is reliable and production ready.
15:21 Speaker 01 Just today, I had one of our privacy council folks giving us information on, you know, make sure if you're going to use these tools, make sure that these are in contracts and things like that. I work for a software agency and it definitely feels like the tail is wagging the dog, right? Like, everybody expects you to use it, but there definitely is not that forethought yet in terms of how that gets governed. So that's awesome.
15:44 Speaker 03 I agree.
15:45 Speaker 01 Chris, is there anything you want to add about those findings or?
15:48 Speaker 02 The only other thing I might add is just in terms of the number of tools that people are using internally, it seems like people are testing lots of different things, right? So I think we saw an average of, the average individual used four different tools.
16:03 Speaker 01 Oh, wow.
16:04 Speaker 02 And people on either side of the curve went higher or lower than that. But if you think about that, too, what it says is that we haven't, as of again, when we fielded the survey last fall, that we haven't really settled out into like who the winner is on the AI tool side, although you could make an argument that maybe Claude is making a pretty strong play for it here in the last month or two.
16:27 Speaker 01 Yeah.
16:28 Speaker 02 But people are still testing a lot of different things.
16:30 Speaker 01 That rings true for me, too. Yeah, I mean, even up until November was kind of like, I'm not sure which, but now once I've seen what Claude does, it's pretty incredible. And also, companies can't react that fast to these changes and people are bound to just sign up for something and try it. So it's no wonder it's happening.
16:49 Speaker 02 Yeah, like if your company had contracted to use a tool that now is the third or fourth best tool in the market and you have to go through the whole enterprise procurement cycle to get another tool versus being able to do your job 10 times faster and do something on the side, you can see why, even though it's maybe from an enterprise standpoint, not risk, ideal from a risk avoidance standpoint, you can see why it's happening.
17:13 Speaker 01 Yes, definitely. So let's pivot a little bit. One of my favorite things that I saw in the report was two favorite things, but one of them is this concept of the great toil shift. Can you talk a little bit about what that is and what you found there?
17:27 Speaker 02 Yeah, Matt, this is one of the ones we were saying earlier, some things that surprised us is, so we expected to go in and have people tell us that AI had reduced a lot of the toil work that they dealt with. And examples of toil work would be things like writing documentation, writing tests, stuff like that, that just take up an enormous amount of time and it is sort of like the bureaucracy of writing code. So when we directly asked people the question, do you feel like AI has reduced the toil work that you do on a regular basis? The vast majority of people said yes. Seventy-five percent of people said, yes, AI has reduced toil work. But the answer was actually a lot more complicated because when we later asked them
18:12 Speaker 02 more questions and based on whether you were using AI a lot in your job, like whether you use it every day, or whether you use it less, the people who are using it less still had a lot of those traditional toil work tasks, like the ones I just mentioned. But the developers who are using AI every day had a whole new set of toil tasks that were very different. And the kicker is they both represented that they were spending about the same amount of time on toil work. It was different types of toil work. So the new toil, as opposed to the old, like writing documentation, AI writes documentation great. So that just like took that off the table. And then all of a sudden, people have this new task, which is what Manish was talking about earlier of now they have to verify
18:57 Speaker 02 the quality and security of all this code that's being generated at hyperspeed. And that becomes the new toil work is the verification process, getting there and actually because AI is never going to be held accountable for the quality of code that it produces. The human is going to be held accountable for the quality of code it produces. And so to me, this is like one of the biggest challenges when it comes to toil work. If you know that the person is being held accountable, all of a sudden going and checking all this code that's being written by the robots is you've got to do it. And it's not going to be the most pleasurable work necessarily, but you have to do it because it's just like you wrote that code yourself as far as accountability goes.
19:39 Speaker 01 It's so interesting to, and I've used this analogy with people that I've talked to is, and it might not be like quite an apt one, but I, it works for me, which is when spreadsheets came along, accountants didn't go away. Right? It just changed the type of work they needed to do. And I feel like it's exactly the same thing, but I love the naming of it, the toil shift. Probably going to use it.
20:00 Speaker 03 Yeah. Thirty-eight percent of the developers say that reviewing code written by AI is harder than reviewing human written code. So that's another thing, right?
20:08 Speaker 01 Yes.
20:09 Speaker 03 And going back to the reviewing part of it. So that was something that stood out to me as well.
20:13 Speaker 01 Yeah, it feels harder to me to put yourself in that context and follow the thread through. So, yeah, that makes sense.
20:21 Speaker 02 We've also found that sometimes there's a little bit of a needle in a haystack challenge when it comes to AI-generated code too. But as these models are getting better and better at writing more performant code, and even more secure and higher quality code, sometimes the issues in that code just get harder to find. So you as an actual human going into this and not having written it yourself, or having somebody who you know write it and you're reviewing your peers' code, that it becomes just, there may be less issues, but the issues that are there can actually be more pernicious because they're harder to find.
20:57 Speaker 01 Yes, definitely makes sense. So as a company who, one of your main products is a static analysis tool, can you all talk to some like creative ways that you see your customers using static analysis to kind of combat some of these things?
21:11 Speaker 03 Yeah, so I think for the last seventeen years, folks have used us and considered us to be the de facto code quality standard. We actually do much more than code quality. We actually, our analysis engine, it does not only review your code for code quality, code security, code reliability, maintainability, complexity, and now even architecture. We also look at the architecture of your code base to see how quickly it is drifting from a good architecture to a bad architecture. So yeah, our customers have been using us in quite a few different ways. In the new agentic world, I think we have added support for all the modern AI native IDs, whether it's Windsor, Cursor, GitHub Copilot, any of those IDs that you're using, or any of the CLI.
21:55 Speaker 03 CLI's, as you know, Matt, are becoming very popular now, whether it's Gemini CLI, Codex, or Claude Code, or any of these things. Basically, we have support for all of these. And when I say support for all of these tools, we are available for them as a deterministic verifications layer for them, independent deterministic way, right? AI writing code, AI reviewing code, it's like, basically you will end up in situations that the AI will say, whatever I have written is the right code. They may not catch everything. They are trained on the same dataset as what they are using for writing the code and reviewing the code. So may not be able to. So we have a deterministic way of reviewing the code. And it is embedded right into the S D L C where the modern S D L C where the A I is writing code.
22:40 Speaker 03 Whether it's the I Ds, the C L I s, or even in the pull request where agents are sending us pull request for review, we can do it. So we are embedded tightly into the S D L C. We also have a M C P server, which I'm seeing quite a few of our large customers are using M C P servers, Sonic M C P server. It's basically a gateway into the code analysis of Sonic Cube for the agents talking in the protocol that agents like to use, right? So and that interface is available for agent tech, I Ds, C L I s and whatnot. We also have in the product, we also, apart from the analyzing the code, we are also looking at how can we improve our detection engine to detect A I specific issues.
23:21 Speaker 03 So we have a facility to add custom rules. So I've seen some customers looking at adding custom rules within the product to detect A I specific patterns. So they can do that. We have added several rules to detect A I specific issues like prompt injection attack, rules file backdoor attack. So these are A I specific issues that A I introduces, which a typical human developer will not introduce.
23:46 Speaker 01 What was the last one that you said there?
23:48 Speaker 03 Rules file backdoor attack. It's a vector which is referred to as rules file backdoor attack, but what basically what it is, is the configuration files, the dot M D C files and the dot M D files, the rules files that are used by these coding agents and I Ds. You can have hidden Unicode characters in those. Those are hard to detect by A I. So we have rules to detect those kind of issues which are introduced through your configuration files such as a rules file. And these are called the rules file backdoor attacks. And then the L L M prompt injection attacks also, we have a rule for that, which can detect issues pertaining to L L M prompt injection. So these are things which we have added on addition to what we already do. So these are specific issues tied to L L Ms.
24:32 Speaker 01 So that backdoor attack, that would be you copy a skills file from Claude somewhere else and you unknowingly copied a bunch of Unicode. Code characters that poisons that prompt or something like that. Is that?
24:43 Speaker 03 Exactly.
24:44 Speaker 01 That's really interesting.
24:45 Speaker 03 Exactly.
24:46 Speaker 01 Okay. Note to self.
24:47 Speaker 03 Exactly. A bad actor can introduce some hidden Unicode characters in the file, right? So.
24:53 Speaker 01 That is really interesting. So as I put this together, right? You know, if I'm writing a skill or I'm I'm describing what my C I C D process would be like, I can use your M C P server or other integrations to basically say, as part of my checks, I want to run static analysis and report out on the results. And if the results look like X, Y, Z, fail the build or whatever it might be. Is that correct?
25:16 Speaker 03 That's absolutely correct. So we have quality gates and you can define the policy conditions when your bill should fail or pass.
25:23 Speaker 01 Nice.
25:23 Speaker 03 So it's, there will be applications that you build which are internal facing, which are not external or which are not sensitive or not production meant for production. It's for a small team. You may not want to have a very strict enforcement policy for that one. Whereas if you're writing a banking application or a medical application where cost of that application going down or breached is very, very high. You would have a higher set of policy conditions applied to that. The gate will be set at a higher level.
25:47 Speaker 01 Yep.
25:47 Speaker 03 So you don't, you fail the gate, even if there's a single security bug in it. That's the condition. Whereas if you have a low priority bug and you want to pass it for a lower internal application, which is not critical, mission critical, you can do that. So you can define your gates and policy conditions based on your needs. And that's a use case that our customers very often use as well.
26:07 Speaker 01 Very cool. Thank you.
26:08 Speaker 02 Yeah, one thing about that I find particularly interesting is a couple of months ago, we were talking to a very prominent analyst, the technology analyst. And we were talking to them about the process for code review in the A I world. And one of the things they observed is they said, look, our recommendation as a leading analyst firm is people use the same review process for reviewing code that they did for human written code as they would do for A I generated code. Like if you had a good process in place with all the things like quality gates and things that Manish was talking about. If you had a good process in place for your human generated code, you can actually use the same process effectively for A I generated code because it's still ultimately just code.
26:54 Speaker 02 That really resonated with me and you had mentioned at the beginning, A I code review tools. And that's one of the things that I'm scratching my head a little bit about is it seems sometimes like. People all of a sudden feel like just because code is written by AI, that there needs to be a new AI review process for reviewing that code. And maybe in some cases, that's more helpful, but also some of the sort of standard tried and true static analysis processes for reviewing code actually work with AI-generated code as well and can give you repeatable results over and over as well.
27:30 Speaker 03 Yeah, I think one of the key things here is the false positives from what I have used some AI code review tools and use obviously use our own product all the time. The false positives in certain situations can be a lot higher with AI code reviews. So I think you need to have a best of both world is my thinking. Whatever the ideal use case is for each type of technology, we should use it for that. So yes, so I think absolutely Chris, what Chris said is our customers are actually wanting to use a deterministic, which is always consistent in its result, which has low false positives as a verification layer. And for some use cases, they might be exactly to the point that you and Chris mentioned earlier, writing documentation, writing PR review descriptions.
28:14 Speaker 03 Those are better suited for LLMs for sure, right? LLMs can do a much better job in that and we should exploit the strength of LLMs for those kind of use cases.
28:23 Speaker 01 Yeah. So one of the things that you guys mentioned and link off to in this State of Code Developer Survey is a report that you have on LLM coding personalities. And I thought that that report was fascinating and I'm hoping that maybe you can talk a little bit about that and what you guys are finding with that.
28:41 Speaker 03 About a year back, we started working on evaluating LLMs for code quality and code security and code complexity. The history here is we have benchmarks, we have industry standard benchmarks that almost every LLM vendor comes out with, whether it's human eval or any of the standard benchmarks for coding, testing the coding performance of LLMs. So while those benchmarks are very good and they're relevant and they are the starting point, they're half the equation, we think. Why do I say that? I think benchmarks basically check LLM output for correctness. They check whether the code is able to write a particular algorithm or solve a particular problem and whether it's correctly solved or not. What they don't check for is how each L L N produces the code for solving that algorithm
29:25 Speaker 03 or solving the challenge, right? Benchmark problems. So we looked at, we have got a evaluation criteria where we evaluate, it's a proprietary criteria. So we evaluate four thousand four hundred coding problems, we give it to all L L Ms, and we evaluate how they perform in those coding problems. And these are problems which the L L Ms are not aware of because they're not part of the standard benchmark. So these are unknown problems, new issues for them, right? And when they run to produce the code, we score them just like any other benchmark in terms of how many times, whether they have a good enough pass rate or not, whether it produce functionally correct code or not. But we take it a step further, where we look at how many bugs did it produce per thousand lines of ten thousand lines of code, a million lines of code? How many security issues did it produce? How much complexity it produced?
30:09 Speaker 03 Whether it's cognitive complexity, what does the cognitive complexity look like? What does a cyclomatic complexity look like of the code they generate? So benchmarks are the good starting point, but we took it to the next because we all all about code health and reviewing the code. That's all our whole business is built on, right? So we took it to the next level to take the analysis and figure out what they are. And based on our findings, we assigned personalities to the L L Ms, the first six or seven L L Ms that we did. We found that they had unique personality traits. We call them archetypes of personalities. Some of them were writing really good code, but the functional performance was superb, but the cognitive complexity and cyclomatic complexity was way too high. The others were cyclomatic complexity and cognitive complexity was very less.
30:52 Speaker 03 They were writing less number of lines of code, but they were also correct. So we assigned the personalities to these L L Ms. So it's a matter of how each L L N produces code, how abstract they are, or how overgeneralized they write the code, or how do they do the error handling, or how much of duplicate logic they have, or what kind of code smells they have, what kind of security issues they have. So we did a deep thorough analysis of each and every type of problem that they create, right? So we started with personalities report, but we have now evolved into an L L M leaderboard that you can find on sonar.com/leaderboard, where we have about thirty five models as of today. L L M's that we have evaluated for security, reliability, maintainability, cyclomatic
31:35 Speaker 03 complexity, cognitive complexity, issue density, all of those things are noted there. Not only that, we also tell you if it's producing ten security bugs, what are the types of security bugs it produces, whether it's path traversal issues, or it is secrets, leaked secrets, issues, what kind of issues are there? So we have a detailed report. Perhaps we don't have enough time to go through it, but I encourage you to take a look at it at sonar.com/leaderboard. You can dig into all of the thirty five L L M's in terms of evaluating them, in terms of the traits that they have.
32:08 Speaker 01 You mentioned the personalities. Can you talk a little bit about the personalities? What's a quirky example or an interesting example of one of the tools and how it was categorized?
32:18 Speaker 02 So when we first did this last summer, we were really focused on that idea of the personalities themselves for each of the models. But what increasingly became clear is that the models were evolving so quickly and new models were coming out that we've actually, that's one of the reasons we switched over to the leaderboard thing and away from the personalities.
32:39 Speaker 01 Yep.
32:39 Speaker 02 Is because the personalities were changing so fast.
32:42 Speaker 01 Yep.
32:43 Speaker 02 That it was hard to just assign something to Claude Code, like I forget even what we said last summer was Claude Code's personality, but its personality now is best coding expert in the world. So I think the biggest thing about each of the personalities, like Manish was saying, is that we found as the models got more performant, they also got more verbose in terms of writing more and more lines of code, which added to the cognitive complexity as well. And so we found that to be the case all the way up until about probably November of last year, when all of a sudden, some of the models, you'll see this as you're looking through the results on the leaderboard, some of the models are actually both performant but not
33:28 Speaker 02 getting complex, they're in decreasing the amount of issues. So where it was linear for a while there last year, where the smarter, you know, in terms of performance. The models got the more lines of code they would write, the more complex. It's getting a little more nuanced than that now, and some of these models are getting really good, really good. So I would say the personalities, at least of the top, you see where the cream is rising to the top on some of these. The top personalities are all just getting to be to the point where they're expert coders. I don't know, Manish, what would you add to that?
34:03 Speaker 03 Yeah, I think you're right. So we had the personalities defined for the first six models that we evaluated back in the day when we evaluated GPT five, Claude Sonnet four, Claude Sonnet three point seven, we assigned personalities to those. So for example, we picked models which was relatively small language models like Open Code or eight billion parameters. And we picked up a large model like GPT five. So we looked at both the ends. And based on that, we created some personalities like Open Code or relatively small language model. We called it a rapid prototyper. Why did we call it a rapid prototyper? Because it was basically solving issues relatively fine, using less number of lines of code. But it was not thorough in terms of the number of issues that it was introducing.
34:46 Speaker 03 It used to prototype it. A prototype does have bugs, but it can be quick and efficient way to test, approve or concept. So we called it a rapid prototyper, whereas Claude Sonnet models were like a senior architect. They took a lot of things into play, like scalability of the application, how many users will be there and kind of, you know, performance and all of that criteria into writing the code. So we called it a senior architect. So we had given some names to the first six models. But now that we have over thirty five models, it's hard to give a name for all of those models.
35:16 Speaker 02 Yeah.
35:16 Speaker 03 We are moving away from giving them model names or personalities as they're getting better and better.
35:21 Speaker 02 It was a lot of fun, though, when it was endearing to anthropomorphize these L O Ms.
35:26 Speaker 01 That is a lot of fun and it's like a nice way to remember that. But now we're getting into cattle, not pets, right? We're getting into that territory with, yeah.
35:37 Speaker 01 For anybody listening to this, I'm looking at this leaderboard and OPUS 4.5 thinking is slightly ahead of OPUS 4.6 thinking, which is really interesting. But then when you look at it from a security point of view, which is security vulnerabilities, issues per millions lines of code, GPT 5.2 high is at the top of this. Reliability, which is bug severity and issues per million lines of code, Gemini 3 Pro highly. That's fascinating to me that all those different aspects. I think also one thing that stands out to me too, is that this was done specifically with Java. That's what it looks like. So there is that aspect to it as well, is that these
36:19 Speaker 01 models could be trained differently on different languages as well. But still, very fascinating. Anything else you guys want to add on the leaderboard or the personalities before we move off that topic?
36:30 Speaker 03 I think if any of your listeners want to get the models tested, there's a form in there. We can have you to test it for them.
36:35 Speaker 01 Oh, cool.
36:36 Speaker 03 We keep getting requests, so we try to keep up as much as we can, but with almost a couple of models coming out every couple of weeks, it's hard. So if there's something missing, we can look at it if there's a huge amount of interest in that.
36:48 Speaker 01 How long does one of those benchmarks take to run? Just out of morbid curiosity, it seems like it would take a while, but maybe not?
36:57 Speaker 03 No. Initially, when we started doing it, it did take a while, but now we have made it a framework, which can evaluate models rapidly. The bottleneck for us in the last few tests were, you know, when OPUS 4.6 came out, there was a whole lot of API requests being made to the model. And it was over flooded with requests. So it was not fast enough to respond, or sometimes our test would timeout. So we had to restart the tests.
37:20 Speaker 01 So you DDoSed Anthropic, basically. So no, I'm just, I'm just joking.
37:27 Speaker 03 The university DDoSed.
37:31 Speaker 01 That's awesome.
37:31 Speaker 02 The big takeaway, I would say, on our work on this over the last six months from our perspective is as you're evaluating which LLMs to use, don't just look at performance alone. Look at it through a more holistic layer of how verbose is the code that's being written? How many security issues are being created? Not just how well it performs at completing a coding test is that I think there's more nuance to it than that. And some of the models that perform very well, when you take into account all of the other things like the cognitive complexity, the verbores of the code, even the cost of running the models is obviously a very big thing too, right? Not every model is created equally from just the cost of the tokens that you have to do.
38:15 Speaker 02 So you have to take all those things into account, not just the sheer performance aspect, which I think had been where a lot of the focus was previously on evaluating models.
38:25 Speaker 03 And the reasoning in addition to cost, you can go to different reasoning modes. For each model, these days have two to four reasoning modes. The higher the reasoning, the higher the cost, the longer it takes to solve the problem, but it's more thorough then.
38:38 Speaker 01 And the skeptic in me too is also thinking about how much are these prices going to steeply increase, right? Like, are they trying to just get people on the hook to be the leader and things like that? So that is definitely on my mind as I think about what my organization uses. All right, let's pivot over to years of experience and how that affected survey results. So I've been doing this a long time, 20 plus years, and I thought that that was really, really interesting. So can you talk about how the perception of these tools comes into play with years of experience?
39:11 Speaker 02 Yeah, I can start and then maybe you can go into a little bit more detail. But this was another one of the results that was fairly surprising for us is that what we learned was that there's a big gap in terms of how people at different levels of experience use AI. So junior developers told us that AI makes them 40% more productive, but then 66% of them also admit that the code that it writes looks correct, but is actually broken. So there are Just like more ready to roll in and start writing the code, but then they're also sort of scratching their head and not sure whether they can actually trust the results of what came out. Now, senior developers were a little bit more measured.
39:53 Speaker 02 I would say they're using it in different ways, like again, this is data from last fall, but sixty five percent of them say they were helping it mostly to understand old, complex code or write documentation, do things where they're cleaning up the past. And using it in ways to maybe check the accuracy of things versus the junior developers who are ready to like roll it and let the AI do all the coding. Now, again, I would asterisk this by saying that I think as a lot has changed in terms of the quality of the code that's created right around. If you're following along the conversation about this on Twitter, there's sort of a general consensus, it seems that around mid-December, maybe when everybody was taking a holiday break and got a chance to play around with some of these new models
40:38 Speaker 02 that they realized that these things had gotten really good. And so even some of the more senior developers were starting to do a little bit more sophisticated things than they were doing before. But I would say that's the big gap is just junior developers are ready to jump in and try the tools. And senior developers maybe are trying it more measured ways because they're also maybe a little bit more jaded about. They know the risks of low-quality code getting into production and how it's going to hurt them later. And so I would say that's probably the key difference and maybe a little different than what we expected. But Manish, anything else you'd like to add there?
41:17 Speaker 03 No, I think you covered it well, so I don't have anything to add there. Yeah, senior developers used AI tools as reasoning assistance and they understand the code what is being produced and they try to make sense of it. But there's more of a trust factor when it comes to junior developers. They jump right into it and develop it. And that's just the nature of experience, right? So inexperienced developer wants to question everything or look at all angles. Because the newer, younger generation of developers, they are more trustworthy in terms of the new technology, so.
41:47 Speaker 02 It's got to be really scary right now to be a junior developer too and see like the thing that you spent maybe years like learning how to do that a machine can go and do that part and it's the higher order engineering tasks around designing the architecture of an application or things like that that maybe they don't have as much experience on that now are the things that are separating us from the machines, right? So for senior developers, I think it's a really interesting time, especially if they can figure out how to turn into orchestrators of AI. And Manish was talking about the idea of legions of agents of how can you orchestrate? That's another conversation on X slash Twitter
42:28 Speaker 02 these days seems to be people playing around with how to build their own army of agents out there doing individual tasks and who's doing the best job at coordinating all of the agents working together. But I think increasingly junior developers are going to have to move beyond their coding skills and figure out some of those upstream skills in order to stay relevant.
42:48 Speaker 01 One of the things that the report pointed out, I have a note here, it said junior developers report higher job satisfaction related to the use of AI coding tools. And that makes intuitive sense based on what you guys are saying too. However, what you said, I'm not on Twitter or X, but what you said about using the holiday break and also, you know, Anthropic gave you those free credits to do exactly that. It's exactly what I did. And I walked away from it with a completely changed opinion about how effective it was. And I've been doing this a long time. I just, I found that absolutely fascinating that it sounds like I was not the only one who had that experience.
43:26 Speaker 02 You're not alone. You're not alone. We're seeing this. I mean, it's like every day, it feels like I see some either blog post or something where a senior, well-established engineer with a great reputation says, hey, if you're not doing this and you're not starting to build your own team of agents to do the work for you. Like I've seen several people say, I don't write a single line of code by hand anymore. Most of my work is figuring out is the sort of architectural level work or is training individual agents to do particular tasks, you know, like. Giving agents each their own tasks so that they can help work with each other and review each other's work. It's really fascinating how quickly that's moved, literally just in, I mean, it's what is it, February 17th here as we record this today?
44:11 Speaker 02 And a lot of this has just happened in the last month and a half.
44:13 Speaker 01 Yeah, it's truly astounding. You said that passion or enthusiasm, excuse me, that junior developers have for these tools. I honestly walked away from it being much more enthusiastic and I'm really curious. We talked about differences from October to now, and I'm really curious if that's going to change with senior developers. Have you guys seen anything like that since October or?
44:37 Speaker 02 Yeah, we're actually thinking about doing another like pulse version of this survey just because things have changed so quick. We feel like we need to get out there in the field and do a little bit more research just so we can compare now that we have a baseline to see what's changed. So, you know, stay tuned. If we're able to make that happen, we'll be happy to share the additional data on it too. But we have, again, like we walked into it with hypotheses last fall. I have some new hypotheses as well that I want to test and a lot of it is around how much of this, because sometimes, you know, there's that conversation on Twitter slash X, which good for you for staying off of that and staying out of that world. But some part of me wonders whether the people that are having that conversation are like way, way out on the cutting edge
45:21 Speaker 02 or how much of the sort of mainstream of enterprise engineering groups are already harnessing agents. And we did have some data on that as of last fall that we saw a pretty significant number of organizations had already been playing around with agents. But again, those numbers, I think, just in the last month and a half have probably changed, but I think we need to test it and see for sure.
45:43 Speaker 01 From my experience working with larger clients, it's ticked up in the past six months. It's ticked up quite a bit. Yeah, it's really interesting. Another thing that I noticed that was really interesting is the use of AI in greenfield versus brownfield projects. So greenfield as in new code, brownfield as in legacy code. What did the report say in terms of clues as to where it's used most effectively and where people thought it was used most effectively?
46:08 Speaker 03 Yeah, so AI is best for projects which are starting from scratch is what we found. Ninety percent of the developers use it for new projects and it's much less effective when you have to work with an existing codebase, particularly if programming languages used in that project are not commonly used these days. I think if you look at L L M's, they are generally very good with other flagship languages like Python, JavaScript, TypeScript, Java and all. But if you go back to some of the legacy applications and old codebases which have some legacy code in there, and I won't name languages or frameworks there. But some of the legacy projects, they're not so good at. So only forty-three percent of the developers find AI effective in updating and optimizing code that is already in use, particularly for
46:52 Speaker 03 older frameworks and older languages. So that's one of the observations. So I think it also comes down to correctness. So AI excels in greenfield because surface area is small when you start with and correctness is generally high with the newer state-of-the-art L L M's. With the brownfield applications, they sometimes struggle with the legacy API assumptions and there are some non-obvious couplings in those applications or implicit rules that are not very well documented or hard to figure out from the just by looking at the codebase. That's less of a use case. Currently, L L M's are being less utilized for brownfield applications.
47:29 Speaker 01 That makes intuitive sense to me as well. Have you heard anything since? One thing that I'm, this is something I'm curious about but haven't tested, is the use of like an agent's M D file or something analogous to it that can help give hints for legacy applications. Have you heard about anything like that since October?
47:48 Speaker 03 Yeah, I have not heard for specifically for legacy applications, but that does make sense. It's logical. I think with Claude Code having the concept of rules, skills, hooks. The hooks can enforce certain guarantees, rules can give you some constraints. With those things in place, I think it definitely, there's a chance that we might see an uptick in uses of AI in the older applications, brownfield applications as well.
48:15 Speaker 01 I'll be really curious to see. So one of the clients we're working with has a 20 year old system that's mostly written in C++ and the tech lead that's on it is struggling with it. I was like, you should run an experiment, see if you can embed some of the tribal knowledge that is in that app in some of these agents files and see. So for the listeners, we'll report back on that as well as what you find in your pulse check. So.
48:38 Speaker 03 Yeah, I think context is very important, as you said, right? So context where the files that you give to agent and information that you give to agent should help with the brownfield applications.
48:49 Speaker 01 Just real quick, I am curious. You had respondents for this all over the world. Did you see any interesting geographic patterns at all?
48:56 Speaker 02 Nothing that we wanted to report out on. I was, I mean, I think everybody is living in the same world, basically. And so we looked to see whether there was anything that we could tweak out that was statistically significant and not enough that made it into our top findings of the report. I think we're all kind of going through this together regardless of where you live in the world.
49:18 Speaker 01 That makes sense. I was just morbidly curious. All right, let's start wrapping it up. And I think I learned a ton from the survey. But for me, it's been a while since I've been like a boots on the ground developer. I've been in leadership for a while. I still code, but I'm still I'm in leadership. So as I put myself in the shoes of a developer who's having executives or managers push these tools, sometimes those asks are grounded and sometimes they're not. And so for anybody listening that might be giving what they think is an unrealistic goal for using AI, what's your advice based on what you saw?
49:54 Speaker 02 I think the big thing would be to tease out whether your organization is focused on the speed of writing code using AI or the speed of shipping code using AI. And the latter, shipping code using AI is harder. And that's why you see many organizations not really getting the sort of promised benefits of AI is because the promised benefit maybe they're focused on is you can write code a lot faster. But as Manish was saying earlier, the name of the game is being able to verify the quality and security of that code before you ship it, or you're not really gaining as much speed as you think you would. So. I think it's it's an education process for developers who are working at organizations
50:37 Speaker 02 that already get that. Like I was saying earlier, it may be that if you had a really robust code review process for the age of developer written code, that you're actually probably set up pretty well to succeed in the AI-generated code world where you have automated code review in place. You have tools like Sonar Cube or whatever you were using before to check the quality and security of code. And that would be the big advice is to make sure that organizations have a really good process for verifying the quality and security of the AI-generated code. Or if you don't, then sort of ensuring that
51:16 Speaker 02 you do your part to like educate the leaders on your team that just because AI can write code really fast, doesn't mean it's good code that you want in production that's going to potentially increase your organization's risk or create spaghetti code that's hard to maintain, or some of the other challenges we talked about there. I don't know, Manish, anything you would add?
51:38 Speaker 03 No, I think said it right. So I think don't lose sight of quality. Ship it faster, but don't lose sight of code quality and software quality and the application quality. Don't compromise on that.
51:47 Speaker 01 Be careful what you ask for is coming to mind. And also, you know, what's coming to mind is Lucille Ball on the chocolate line, eating the chocolates. It's like if they start coming out faster, you gotta, yeah, so.
51:57 Speaker 02 Exactly, exactly.
51:59 Speaker 01 If I'm a developer, what do you think the most important takeaway from this survey is?
52:03 Speaker 03 Yeah, so for developers, the most valuable skill is no longer knowing how to write code. That is pretty much a solved problem, writing code. It's more about understanding the code, making sure the code is correctly written by the agents or your tools that you're using, and making sure it's being reviewed and you're following all the, putting some guardrails in place to make sure. Ultimately, developers are still accountable irrespective of who writes the code. Like Chris said, you want to do the right things. You want to put some guardrails in place. You want to validate the code. You want to make sure the code is written well. It's not so much about learning a new programming language anymore.
52:42 Speaker 01 Anything you want to add, Chris?
52:43 Speaker 02 Maybe just repeating what I said earlier about if you're a developer who is trying to figure out how to stay ahead and keep your skills relevant, probably the best thing you can do right now is to gain the ability to manage and orchestrate and train agents. And if you can do that really well, and you can harness the skills of these leading edge LLM tools, and you can sort of stay on top of knowledge, be curious about learning, because these things are evolving so fast, what you learned during your Christmas break may not be even the current state of the art anymore. And so staying on top of that and spending at least a few hours a week just learning and testing and trying new things.
53:27 Speaker 02 I read something today where somebody said, this may be the most important year of your career, because things are moving so quickly right now, that if you're not paying attention and you're not staying curious, you could quickly fall behind in your career and have a hard time catching up. So one side of me, I was like, oh, am I doing that? And I was like, I think I kind of am, but it was a little bit of a call to action for me, too, of like, making sure that in that hour at the end of the day, when you're winding down and everything, you try a couple of new things every day and just sort of see, did you get better results on it today than you did yesterday? And increasingly, I'm finding you are.
54:04 Speaker 01 Yeah, I'm hearing two things, which resonate as true with me. It was Manisha saying, you know, you can't lose sight of the basics. The code needs to be bugged, you know, you got to ensure quality, but at the same time, you got to kind of dance on the bleeding edge in order to stay relevant, quite frankly. Yeah. And then last question, if I'm a development leader or an engineering leader, what do you think the most important takeaway is?
54:26 Speaker 02 You can't ignore the trust problem in code is as AI is getting better, but still, like that stat I said at the beginning is the sort of big takeaway for me, 96% of developers still don't fully trust the quality of AI generated code. Not necessarily because it isn't good, because in many cases, the code is really good and continuing to get better. It's because they know AI isn't going To take the fall when the code fails or there's a security issue. So if you're a leader, you have to ensure you still have human accountability for the code that gets shipped into production. And so you have to build the right systems and processes to ensure that you're able to verify the quality and security of the code before it ships.
55:10 Speaker 02 Don't just close your eyes and ship AI-generated code unless you're in some sort of leading edge vibe coded startup where you can risk doing that if you have actual customer data and customer information and things like that. Probably makes sense that you go and really figure out how to trust the code that you're shipping. And that to me is the 2026 problem of AI-generated code. In 2025, the problem was how do we generate even more code? And that's like Manisha said, a solved problem. Now we can generate pretty good, high-quality code. It's continuing to get better, but ensuring that there's a human who's willing to put their stamp on it can say, I'm willing to ship this into production and take all the risks that entails.
55:55 Speaker 02 That's the biggest challenge for 2026.
55:58 Speaker 01 Yeah, I think also the thing that if I'm a leader, I'm thinking about, okay, you're telling me that 66% of people don't trust the code and I need to put that human stamp on it. That's a good thought-provoking thing to end on, I think. But is there anything else that you guys want to cover that we didn't tonight?
56:14 Speaker 02 Matt, I appreciate the time. Thanks for inviting us on. We both, I think Manisha and I and the whole team are, we're passionate about doing this sort of research. And the LLM leaderboard project that we talked about earlier is a sort of like a continuing passion where I think we go look at it every day and see the new models that go up and, and the results and just watch these things getting better. It's a wild time. I've been through some different turns in my career, as I'm sure you both have. But I'm not sure I've seen anything quite like this. So it's a fun time to be of all's a little scary, but it's a lot of fun. So I appreciate you having us on.
56:49 Speaker 01 Yeah, thank you so much for being here.
Previous Unlocking the Data Layer for Agentic AI with Simba Khadder Next Open-Weight AI Models Page updated November 6, 2026 · transcript and summary are machine-generated