#332 Dan Faulkner: The Code Is Clean. The App Is Broken. Why AI Development Has an Integrity Problem

What happens when AI writes code faster than anyone can test it? In this episode of Eye on AI, Craig Smith sits down with Dan Faulkner, CEO of SmartBear, to explore one of the most underappreciated risks of the AI coding boom. As tools like Claude Code and Codex push software development to unprecedented speed, the systems built to validate that software are being left behind. Dan makes a distinction that every engineering leader needs to hear: clean code passing unit tests is not the same as an application that actually works. Dan introduces the concept of application integrity, continuous and measurable assurance that your software does everything it was intended to do and nothing it was not. He explains why the gap between what AI builds and what teams actually validate is already creating hidden risk in production, and why that risk compounds the faster you ship. We also get into the new failure modes that agentic AI is introducing. Slop squatting, instruction inversion, cascading errors. These are not theoretical. They are happening now, at scale, in codebases that no human has fully read. Dan also walks through SmartBear's autonomy ladder framework and their newest product BearQ, a team of AI agents that explores your application, builds a knowledge graph, authors tests, runs them, and updates everything as your app evolves. The key distinction: it is built to augment human teams, not replace them. Finally, Dan shares his honest take on the future of software engineering. The fallacy was always that coding was the hard part. The hard part is knowing what to build. That skill is not going anywhere. Subscribe for more conversations with the people shaping the future of AI and emerging technology. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Introduction and Dan Faulkner's Background (01:05) What SmartBear Does: Testing and API Lifecycle Management (03:27) AI Is Outpacing Application Testing (07:51) Slop Squatting, Instruction Inversion and New AI Failure Modes (17:31) Black Boxes, Technical Debt and the Expertise Crisis (22:00) How to Avoid Self-Validating AI Systems (24:11) The Autonomy Ladder and BearQ (31:30) Why Testing Must Be Continuous and Everywhere (36:31) Infrastructure Risk and Automation Bias (44:11) The Future of QA and New Specialist Roles (50:44) How Teams Use SmartBear Tools Today (58:57) The Future of Software Engineering and Human Roles

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  My name is Dan Faulkner. I'm the CEO of SmartBear. I joined SmartBear about five years ago, initially as the Chief Product and Technology Officer. I've been the CEO since the beginning of 2025. Prior to joining SmartBear, I was at a small martech startup called Planner, which we sold, where I was the CTO. Prior to that, I spent most of my career working for a company called Nuance Communications, which was one of the pioneers in really field-hardening speech technology and making that work well for real people in real-world environments. Now, what we do at SmartBear is we provide tools for development teams to improve the quality of their applications.

• 0:44Speaker 00

  We do that in two primary domains. The first is in application testing. So that means testing applications above the waterline of the code, the actual real application in its real deployment environment, and in API lifecycle management. So APIs obviously are now the vocabulary of agents, so they've become even more important than they were, and they were critical anyway. So they're the two main areas that we focus our product investment into.

• 1:13Craig S. Smith?

  Yeah, and did you say testing APIs or configuring APIs?

• 1:19Speaker 00

  It's the full lifecycle management. So our product is built on top of a very well-known open source product called Swagger, which is essentially the codification of the open API standard. And the commercial product that sits on top of that is an API catalog that manages all the artifacts across the lifecycle, so the specification, its documentation, and tests all of them. So the contract testing, the functional testing, performance testing, and so on of APIs. So you can kind of keep that API catalog well-governed and up to speed.

• 2:01Craig S. Smith?

  Yeah, and as you said, APIs getting more important because of the explosion or...

• 2:10Craig S. Smith?

  What's anticipated as an explosion of agentic AI that's connecting and calling APIs for tools or databases or things like that. It as AI increases software production, the velocity, the speed of software production. If application level validation doesn't in scale alongside it, are we increasing systemic digital risk? This is something I've spoken to a few people about.

• 2:46Speaker 00

  Yeah, I mean, certainly what we believe that there are pillars throughout the SDLC are resilient and meaning they will still be there in the AI disrupted SDLC and those are failing to keep pace with the acceleration in coding. One of those pillars is application testing. There are others like security and monitoring, you know, even strategy and requirements, creation and definition. Those things all still need to be done and coding specifically has accelerated way past the capabilities of what those pillars have been able to do.

• 3:25Speaker 00

  And the risk that that creates, kind of the systemic risk, is that the vocabulary we use to describe it is you lose the integrity. You risk losing the integrity of your applications. And what that means is, do you really know what your application does? Is it doing everything you wanted it to do and nothing you don't want it to do? And one way that we're addressing that is through the application testing space. So, yeah, it's definitely an issue.

• 3:53Craig S. Smith?

  Yeah, yeah. And, uh, yeah, so application integrity is, we'll describe integrity again.

• 4:02Speaker 00

  So, application integrity is, you can kind of view it as the assurance that your application does everything that it was intended to do, and that it will just work for your end users. And you need that to be continuous assurance, because the applications themselves are evolving so quickly. So, what's the flip side? How would that break down? You would start to see that there are missing capabilities in your application that were specified upfront. It was not behaving as you anticipated. It had some unspecified behavior in it. Or, even though the code is clean, it's not actually solving the needs

• 4:43Speaker 00

  of your end user. Those are all ways in which the integrity of an application can break down, as we rely more and more on these kind of coding agents to do the code creation for us.

• 4:55Craig S. Smith?

  This episode is brought to you by Tasty Trade. On Eye on AI, we talk a lot about how artificial intelligence is changing how people analyze information, spot patterns, and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data, and the ability to understand risk clearly. That's one of the reasons I like what Tasty Trade is building. With Tasty Trade, you can trade stocks, options, futures, and crypto, all in one platform with low commissions,

• 5:35Craig S. Smith?

  including zero commissions on stocks and crypto, so you keep more of what you earn. The platform is packed with advanced charting tools, backtesting, strategy selection, and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI-powered search feature that can help you discover symbols aligned with your interests, which is a smart way to explore markets more intentionally. For active traders, there are tools like Active Trader Mode, One-Click Trading, and Smart

• 6:15Craig S. Smith?

  Order Tracking. And if you're still learning, Tasty Trade offers dozens of free educational courses, plus live support from their Trade Desk reps during trading hours.

• 6:29Craig S. Smith?

  If you're serious about trading in a world increasingly shaped by technology, check out Tasty Trade. Visit tastytrade.com to start your trading journey today. I'm going to myself. Tasty Trade Inc. is a registered broker dealer and member of FINLAW, NFA, and SIPC. Yeah, you know, the AI is probabilistic and traditionally software has been deterministic. So when you're using a probabilistic system to write a deterministic codebase,

• 7:08Craig S. Smith?

  I would imagine that on the margins, maybe, you know, maybe not, uh, maybe there's 97% uh accuracy in the, in the code writing AI, but that 3% can slip into the code. And these codebases get so large because they're being written so quickly, it becomes a real challenge for humans to go through line by line and, and figure out whether there's uh anything wrong. Is, I mean, that's basically what you're talking about, right?

• 7:41Speaker 00

  Yeah, I mean, I think there's kind of two angles to it. One is in the code. Um, so let's start with that one. There's just a giant amount of code that's being, um, uh, the, the phrase I've been using is like firehosed into the repos of the world, right? At, at record speed. And we do need to like to stay current. In the last few months, it's very clear that Claude Code and um OpenAI's latest version of Codex, they've kind of crossed some very important threshold where the code is really good. It's clean and it's pretty good at testing itself. And, and it's really now become something that I think people feel confident in generating

• 8:21Speaker 00

  code that's um, that's quite reliable. Our point is that you can have reliable code and you can have very hygienic, clean code that passes all of its unit tests. That doesn't mean the application works as intended. That means they're very different things. And you need to Actually take the compiled application in its end to end environment. So it's, you know, distributed AWS architecture, for example, then you need to figure out, does that work end to end for your end users? Does it solve the business problems that really need to be solved? And does it do it across all the operating systems and browsers and disparate devices that your users are

• 9:04Speaker 00

  employing to use your application? And that has been something that wasn't keeping pace with development before AI. And it's kind of, it kind of gets left in the dust without a different approach in the AI disrupted S D L C. And it's a real problem because any business that wants to release an application and they have, you know, material consequences for that application failing has to reckon with how much do I test this? How much am I willing to give up the velocity gains of the coding process in order to rush this out into production because my, my testing practices can't keep up. So, so that's where they, if they make that trade off in the

• 9:46Speaker 00

  wrong direction, then the integrity of their application is at risk because they will have issues that they're unaware of in production.

• 9:53Craig S. Smith?

  And, and if, if you're relying on AI tools, they can introduce patterns or vulnerabilities at scale, right? Across thousands of applications simultaneously. Is that, is that an issue?

• 10:07Speaker 00

  It's a real thing. And certainly for the industry, there's new categories of issues that are emerging and they're being, I would say exacerbated by the emergence of agentic workflows. So now we're thinking about there's the coding, but then there's the agents who are actually, you know, using capabilities. And there's pretty interesting sets of issues emerging. Some of those lie in security. There's a pretty common issue that's emerged, has a delightful name. It's called slop squatting. And it's where agents will import or refer to libraries, third party libraries that don't exist. And a bad actor can create that library.

• 10:49Speaker 00

  Put it into the chain and suddenly that's that's deployed at scale. There's the well-known issues around prompt injection to get agents to create malicious code. But there are also more subtle issues where if you've interacted with anything like Claude Co-Work or ChatGPT or Gemini, you'll have had the situation where it does something that you don't want it to do. This could be as simple as if you're writing a document and you can say, hey, can you can you stop doing that? And for me, one of my pet peeves is they all seem to have this real tick where they like to say, hey, this phenomenon isn't just A, it's B. And it creates this ludicrous kind of straw man that makes cringe and it just screams of AI. And I'll say, hey, can you stop that?

• 11:33Speaker 00

  And he goes, oh, absolutely, I'm going to stop that, creates a new draft and does it again. Um, and that's called instruction inversion and it happens in coding too. So you can say, can you please not do this behavior when you create code? And it says, I will absolutely not do that, I promise. And then does it straight away. Um, and so there there's this whole raft of of systemic issues that are appearing in code bases, some of which are clear and some of which aren't. And it even permeates how the frontier models are tested. There's a very, very famous and well-known thing called SWE Bench, which is used to assess the quality of the latest frontier models.

• 12:15Speaker 00

  And that has also been shown to mark things as being correct when they're not and to mark tests as passed when they're not. Um, so we need to be incredibly excited about the advances in the state of the art. We also need to be realistic that there's a whole set of new challenges that need to be thought through and wrestled with and addressed. So it's a it's an interesting time.

• 12:39Craig S. Smith?

  Yeah, I mean, it's fascinating that now there are terms uh that are being developed for all of these various behaviors. Uh, whereas at the beginning, it's just, you don't know what to call it, but it's doing these crazy things.

• 12:55Speaker 00

  Right.

• 12:55Craig S. Smith?

  I mean, one, one of the things about the uh the probabilistic. Nature of AI is that when it makes a mistake, which it's it's going to do because it's never a hundred percent accurate. And you try and correct the mistake if there's if it makes a mistake in the correction. It just you end up magnifying the problem and you it you just can never get ahead of it. Because every time you I mean, maybe I haven't used the latest iteration of Claude code. But there was a time when that that was very frustrating. You know, you'd ask it to fix that error and it would just say it had fixed it, but it had actually created more problems.

• 13:38Speaker 00

  Yeah. I do think Claude Code and Codex and maybe other models too. But those two specifically, literally in the last two or three months, have made a leap in improvement. And I think addressed many of the frustrations, but they're fitting into a system of other agents and other tools. And it's unquestionable that they're as they're going at such velocity and such a volume. There are issues that are being hidden, not not intentionally, but they're being hidden and and we will have to reckon with those. And you also get what's called cascading errors. So agents working with other agents, one makes a small error, that gets amplified. We get errors in the context files that people are providing to guide their agents.

• 14:20Speaker 00

  Striking how many people don't really understand how what their software processes are, what their company's rules are. So it's great that we can give, you know, an agent's dot M D file a ton of context and say, this is how we want you to build. But of course, if that's wrong, you're building a whole bunch of software with baked in errors. That's not the agent's fault. That's that's the human's fault, but the human's in the loop.

• 14:41Craig S. Smith?

  Yeah, I know a company called Diff Blue out of the UK, out of the UK, actually. I don't know if you know them out of Oxford. And they have a solution very specifically for writing unit tests for JavaScript. I believe it's JavaScript. And they had a real hard time getting traction with it. And I don't know where they are now, but since then, there's been this tsunami of. Of of other code test writing solutions. But what was interesting is we would talk about how as AI coding takes hold,

• 15:21Craig S. Smith?

  uh, the world is going to end up with these massive code bases that no human has ever read. And and maybe, I don't know, is it possible for a human to read these code bases? Once they're they're installed.

• 15:37Speaker 00

  Um, the code itself is readable. It's much harder to read someone else's code than your own. Um, but that's okay. It's doable and the the the human code review has been a real thing for a long time. But code review itself is being automated heavily and we start to get to a little bit of a black box problem. We also end up, and I think the market is cottoning on to this right now. We have a not even long term, but a short to medium term expertise problem. Um, because to the extent that people are starting to say, hey, I'm not going to hire junior developers. I'm going to rely on my agents. You're going to lose the human resource pool of people who can actually open and understand and think deeply about code.

• 16:19Speaker 00

  So there's real uh debt problems, technical debt, knowledge debt, um, design debt, all sorts of debt issues that will need to be addressed. And we're in this kind of very giddy environment at the moment where we're behaving as if all of these issues get solved um by by one tool. And and they don't that they there are some really important things being very well solved by AI. And I'm actually very bullish on it, but it's a complicated environment that we live in. Um, and people are going to need to be in the loop for the foreseeable future and in existing ways and in new ways.

• 16:57Craig S. Smith?

  Yeah, and when I said uh that the code bases that no one has read or can read, when I engineer developers would be incapable of reading it. It's just if you have a code base of a couple million lines of code, I mean, that's uh.

• 17:15Speaker 00

  Yeah.

• 17:17Craig S. Smith?

  That that was written by AI and has to be gone over by a by a human. That's that's a pretty daunting task. So, uh, and and you mentioned, you know, then using AI to to test or validate the code. But that in itself, as you said, the black box problem, you don't know whether it's validating systems that shouldn't be validated. I mean, it's uh, it it just compounds the the problem. Uh, so how do you avoid self validating systems that simply confirm their own assumptions?

• 17:54Speaker 00

  Yeah, I mean, you you need some, you need in the systems that are independent of each other um to do to validate and you're going to need humans supervising uh that those processes. So, you know, the next iteration of our application testing products are agentic. They are there. They have to be in order to keep pace um with agentic coding. Um, so they can explore an application and create a test strategy and create tests and run tests. And it's good that they're doing that outside of the environment that the code is being generated in. Um, that alone should give you some kind of third party confidence. But you absolutely must have a

• 18:37Speaker 00

  human who's looking over that system's shoulder and saying, well, show me what you're doing and show me your thinking and show me the results. And are you testing the things I want you to focus on? Um, so we have to design agentic systems, I think, to work in partnership with people. And it's very, it's both provocative and seductive for folks at the moment to be thinking that uh the agents are working independent of people. I think it has to be a partnership and and that's what we're gonna, that's what we're gonna see. It's certainly what we're designing for.

• 19:07Craig S. Smith?

  Right, yeah. So can you talk to me about how SmartBear is approaching this specifically?

• 19:15Speaker 00

  We have this sort of simplifying analogy that we stole from automotive driving that we call the ladders of autonomy. And we've all, many of us will have heard of like levels of autonomy in cars. You know, with Waymo is the fully autonomous and, you know, your old fashioned manual car with no bells and whistles is a level one. And if you think about that from a coding perspective, there's kind of a level one idea where there's a human who's kind of typing all of their code. And then they get an IDE that does code completion and they're bringing in libraries and then it gets more and more intelligent. Up to kind of a vibe coding idea where you're saying, hey, build me an app that does X and it's, it's off to the races and does most of the work for you. As you go up that autonomy ladder, think about the tools that exist to help you.

• 20:00Speaker 00

  So you might have tools that a manual tester uses to help them explore an application and do their testing. Then you might have a test automation tool or an open source framework that will allow you to record an interaction. And it turns that into an automated test. But that still requires a person, kind of a GUI, pointing and clicking and doing stuff. And what we have tried to do at SmartBear is build tools that will help people wherever they are on that autonomy ladder. Because there's tons of industries that are very cautious about adopting agentic systems and autonomous systems. And we've got to make sure that they've got what they need, but they all want to eventually, when the regulations allow them and their lawyers allow them.

• 20:45Speaker 00

  They'd like to get to this kind of highly automated, reliable world. The new capability that we've created is a platform called BearQ. And what that does is-

• 20:58Craig S. Smith?

  I'm sorry, Bear?

• 21:00Speaker 00

  Q.

• 21:01Craig S. Smith?

  Q.

• 21:02Speaker 00

  Yeah.

• 21:02Craig S. Smith?

  Spare.

• 21:02Speaker 00

  Q. No, the letter Q. What that does and how that's different is whereas pretty much every other tool we've created, including ones that contain agentic workflows, they're geared towards a human as the primary doer, the tester. And Bear Q is designed with the agent as the primary doer. It's a team of agents that work together and they have different skills. And it's able to explore an application, figure out what it's doing. It builds a knowledge graph and says, okay, I understand what this is all about. And then it can build a testing strategy, author the tests, run the tests.

• 21:45Speaker 00

  It can introspect if a test fails and say, well, did I write this test badly? Or is this really an issue? And as the application evolves and there are, you know, each time it's iterated by the development team, it will update its application exploration, its knowledge graph, its testing strategy and its tests accordingly. Very importantly, though, we have not designed this to be independent people. We've designed it to click into the testing, to fit into the testing teams that exist today. It's, we view it as an augmentation to the organizations that exist today. We don't think people want to abandon human oversight of application quality at all. But we do know that there's a, that they can't keep up.

• 22:29Speaker 00

  So what we're trying to do is think of it as, you know, if there's a pie chart, there's a slice that's being addressed by the tools and the people to date. We're basically saying, hey, let us fill up the rest of that pie chart and help you keep up. And that's kind of our approach on the application testing realm. So they get the benefit of the pace. It's independent of the coding agents. It's above the code level. So it's testing, will this really work for real end users? And we think that's solving an important problem.

• 23:01Craig S. Smith?

  Yeah, and to avoid the self. You said you need very independent systems. Does that mean you each the one system that's that's testing another or one tool that's testing another has has to be based on a different foundation model?

• 23:21Speaker 00

  Not necessarily. I mean, we exploit the frontier models. But what you're really doing is testing an output of the model, not the model itself. And so when you present a compiled application to our product, it doesn't know what frontier model was used to generate the code. It doesn't care. It doesn't care if it was a person or a model or a combination of people and models. What all it's doing is is essentially using the application to learn what the application is intended to do.

• 23:55Craig S. Smith?

  Yeah. And things move so quickly. Does testing have to move from pre deployment to sort of continuous testing?

• 24:04Speaker 00

  Yes, I do believe that you will need a continuous testing. I think it's not so much a pre deployment and post deployment. It's both. You should be continuously testing pre deployment and you should be continuously testing in production because there are new factors come into play when you go into production and real world end users that you don't control in your testing environment come into play. And so you you absolutely need to be testing in production as well, but it's got to be continuous because the updates to applications will become continuous. It's in the name of C I C D. It's arguable whether it's really been continuous up until now, but it's going to get more continuous when you have coding

• 24:47Speaker 00

  agents creating updates and new applications much, much, much more quickly.

• 24:51Craig S. Smith?

  Yeah. And in the old days, not very long ago, programmers wrote unit tests or and tested as they went along. It was kind of a tandem exercise. You would write code, you would test, you would write, you would test. At this point in smart bears tools, do you envision, or maybe it's already happened? It's a role within a separate role within the the the software development team team. Somebody who understands the tools for testing and just focuses on that. While the the programmer continues writing, you know, with a copilot of some sort.

• 25:36Speaker 00

  Yeah, I mean, I think that unit testing and other types of code level testing have pretty much been subsumed into the frontier models that or the the coding specific frontier models. Claude code creates the code, creates the unit test, creates better unit tests than most developers would, more fleshed out, more complete, more counter example types of tests. So all of that, that that creates good, clean code. What will be needed is the continuous ability whenever the application is actually compiled to say, okay, now let's actually use that and see if that works in the end environment, in the deployed environment.

• 26:21Speaker 00

  And I do think that that will need to become much more integrated into the SDLC. So I can easily imagine a scenario, if you can imagine continuous code updates, continuous builds. Every time that happens, having an immediate testing of the built application feeding back to the the developer. Um, I I I think that that kind of it's not so much shift left, it's kind of smeared everywhere. It's it's got to be left and it's got to be right as well. It's got to be in production as well.

• 26:54Craig S. Smith?

  Yeah, and and that, as you say, above the code line of application testing. Even there, I mean, it used to be there, there were third party firms, U A T firms that that did that for you. I mean, is this, who are Smart Bears customers, I should say, are they separate testing firms that that are contracted to to test applications? Or is this intended to be in-house application testing?

• 27:26Speaker 00

  Yeah, for the vast, vast, vast majority, we sell into in-house development teams within companies. Um, and how they're constructed, whether they have kind of integrated quality and development or separate quality and development tends to depend on. Um. The tech stack, the industry, the age of the application, their tool chain. And really, our goal has always been to say, yep, we'll, we'll kind of, we'll meet you where you are, depending on, you could be super progressive, you could be doing something once a quarter. Um. And, and any of those are fine with us. Our fundamental kind of simplifying point is whichever one of those you're doing,

• 28:09Speaker 00

  you should be testing that your applications actually work. And we know that you don't have enough people, you don't have enough seats. Um. Of any software to be able to do that sufficiently quickly and sufficiently well. And you need to do it more acutely than even than you previously needed to do it. Because you've allowed consciously, you've made the choice to allow a degree of abstraction to come into your development processes. I, I listened to a, an interview just the other day where the head of product at Codex said that none of their developers open their IDEs anymore. They're just orchestrating teams of agents.

• 28:54Speaker 00

  So by definition, you are further away from the code. And I think people are saying, that's okay, as long as the coding agents are reliable and create clean code. But it, it creates a distance and an uncertainty and the um, likelihood that you're going to get more drift between what you really intended the application to do and what it really does goes up. And there's going to be a need to validate that what you really intended is really happening. And it doesn't matter how well your context file is defined, it, you're going, it doesn't matter how well your user requirements are defined.

• 29:39Speaker 00

  You've got to validate that and understand what the drift is between those upfront plans, that upfront context and the thing that that has been built at the end of your of your tool chain.

• 29:54Craig S. Smith?

  And if that's continuous, then it does that, is there one person in the organization that's doing this application testing with SmartBear? Or or is it distributed across everyone that touches the application?

• 30:11Speaker 00

  Um, I don't think we really have the right to be prescriptive about that. In the same way, every company, you can go to ten different companies, you're gonna find ten different tech tech stacks, ten different tool chains, ten different organizational models, responsibility models, and I don't think that's gonna change. There's there's not gonna be one uniform approach to software development to rule them all. So that's one factor. There's just it's an incredibly diverse um ecosystem that we live in. The second is that while innovation happens quickly, adoption happens slowly, and we see that time and time and time again in technology.

• 30:51Speaker 00

  It's the adoption of tools and agents and AI is going to move at different paces, company by company, industry segment by industry segment, for diverse reasons. Sometimes it will be mandated by regulatory requirements. Sometimes it will be mandated by a huge cost of failure, either financial, reputational, legal. Um, sometimes it will just be mandated by conservatism. We need to assume that that diversity of approach will persist, even as AI is probably adopted more quickly than any technology that's that's happened before.

• 31:30Craig S. Smith?

  Yeah, as AI-generated software proliferates, it's it's already in some areas.

• 31:40Craig S. Smith?

  Becoming infrastructure really rather than just a productivity tool. How do you see that affecting systemic resilience?

• 31:50Speaker 00

  There's a lot of fear in the mo- in the market at the moment. We can see it in the public market and in private markets about what this all means for the future of software that we've previously thought of as critical infrastructure. I would actually say that it's it's uncertainty about how this will all shake out rather than a conclusion that the existing infrastructure itself is going to be replaced. We will continue to need audit trails and proof and therefore those systems can't be replaced by by black boxes. We cannot have ecosystems of technology that say, here's what I'm going to do.

• 32:31Speaker 00

  I I'm doing it now. I've done it and it's perfect. And we just say, great. I think infrastructure will evolve. I think that we will see portions of it

• 32:43Speaker 00

  be hugely impacted positively by by AI. But if we fundamentally look at AI as a new type of automation and we think of agents as workers. I think what we're going to see is that the overall population of workers grows and the overall proportion of workers will increasingly be agents. There's going to be a whole set of new tasks that we need to do to oversee the agents, to validate their work, to validate intent, to make sure that things are are as intended. And I think that's actually going to create new opportunities for humans and they will need their own infrastructure and their own tools.

• 33:29Speaker 00

  So we're not going to see this kind of unidirectional change where everything becomes agentic and everything stays the same for humans and it's a zero sum game. I just don't see how that's possible without people abdicating responsibility in a way that they don't and won't and can't from a any business or government perspective. If we choose, the more we choose to allow AI and agents to operate in a black box. The more systemic risk we will introduce into our infrastructure. There was just two days ago, a story about the, I think it was the head of AI security at Facebook or Meta,

• 34:11Speaker 00

  who was working with Clawbot, this kind of open claw, malt book. It's changed its name several times, the thing that OpenAI just acquired. And it was an experiment, I think, so you know, but the fact is it, she had given it very clear instructions and said, do not take any actions until you've checked them with me and I give you permission to take the actions. And it deleted her email inbox and it it nuked all of her emails.

• 34:38Craig S. Smith?

  Oh, my God.

• 34:39Speaker 00

  There's actually this published conversation where she's going, no, stop, don't do that. Don't, why did you do that? I, you know, and the agent, the agentic system says, yeah, I, there was a rule, I understand that you gave me a rule and I broke it. I'm sorry about that. It won't happen again. And it's like, how do you, how do you know? Like, it seemed to arbitrarily decide not to follow the rules. What's technically feasible and what we can imagine one day needs a tremendous amount of hardening and scaffolding and process and insight that doesn't exist today. There's huge opportunity for us in there.

• 35:18Craig S. Smith?

  Yeah. And so you're not concerned that, you know, there's something called automation bias where I can imagine this job of of testing applications must be exhausting. But if you you're you're testing something with a tool that's automating or you're watching the automation and it does it perfectly 10 times or 100 times. Pretty soon you just thought you start trusting the automation. Is that a concern of even if smart tool? I mean, you were saying a smart bear, I'm sorry, the smart bear requires a human to be in the loop.

• 36:00Craig S. Smith?

  It's designed for that or where is there a risk that people will just start hitting that button over and over and letting it run?

• 36:09Speaker 00

  Well, there's the, we can see what's happening in the market today. That people are trusting AI code gen.

• 36:18Speaker 00

  Without checking it properly and it's going to create, it has already created a giant technical debt that will need to be unwound. Um. So it's a hundred percent going to happen. People will over depend on automation and we will, it will require significant errors and, and their consequences to cause people to, to change. And I think that that is completely consistent with the adoption of any automation I've experienced. For me, that was kind of the migration from on prem to cloud, the emergence of mobile and those caused huge change. Some people rushed out early, almost kind of proved what's possible and what will

• 37:01Speaker 00

  be possible, but didn't survive because they, um, they got over their skis and catastrophic failures and we will definitely see that, definitely see that play out here. Um. And I think that, you know, we're not a business to consumer product. We're a, we're a B two B company. And what we're trying to do is look out for businesses and say, hey, you know, let's anticipate environments where we can get the benefits of the velocity and the quality that these AI solutions can bring us. Now are tremendous benefits that we should be excited about. Let's also make sure we can do that safely.

• 37:40Craig S. Smith?

  Yeah. So QA, uh, as it's been traditionally is sort of entering this period of structural transition. Where do you see it in, in say five years or is at the pace, things are changing? Is that too distant or rising?

• 37:59Speaker 00

  Five years is too distant. I see tight partnerships between people and agents and a much more fluid SDLC where, and where the people have much more generalist skillsets. There's going to, so people who actually know what they want to be built, what the business constraints and priorities are, what they want the end user experience to be. If you can describe those effectively, then I think we'll see more and more reliance on agents to do that work and to do it quickly. I really struggle to see a world where people are not in that loop in multiple ways.

• 38:41Speaker 00

  I also think you're going to see a need for new roles. You're going to need to have people who deeply understand software architecture, who deeply understand security, who deeply understand quality, deeply understand code, because things will continue to go wrong. They will continue to break and you have to have people who understand that. And they may too be using specialized agents and specialized tools. I'm not saying that that doesn't include agents, but they're different roles. And, and so it's going to be a different organizational model, a different balance of human and agentic work.

• 39:25Speaker 00

  Um, and I think that we will see a huge amount of experimentation in companies as they try and find the thing that works best for them. But as I said earlier, today and for as long as I've been in software at all, I've never seen two companies that approach software development the same way. I don't see that changing.

• 39:45Craig S. Smith?

  Talk about some of how, how someone uses a smart bear's tools, because you, you, you have a fairly broad platform now from APIs, testing, observability. What, what's sort of the vision, what the company is, how it fits into the, the application development, necessarily just the code? And how is that about, how are you keeping up with the market?

• 40:12Speaker 00

  Yeah, I mean, we do have quite a few products, but you can really boil down our focus to being around API governance. So API lifecycle management, we think is increasingly going to become a governance question. Uh, your, your catalog of APIs, which will sometimes be used by humans and sometimes be used by agents and probably mainly be used by agents. You need those agents to really understand what the API does. We've mentioned that word drift before. You need to make sure there's no drift between the spec and the tests and the code and the doc. They all need to be absolutely lock tight, which means they need to be managed and governed together. All the artifacts. Need to be governed together.

• 40:53Speaker 00

  And then that catalog also needs to be able to keep pace with the agents. So it needs to be agentic. That's our strategy there with with APIs. To the extent that APIs, you know, kind of the lingua franca of of agents and software in general, that just becomes more important. So avoiding drift, providing great governance and audit trails and keeping pace with agentic coding. That's what we'll do in that family of products. On the application testing side, we already address the critical application level testing tasks. So whether it's web or desktop or mobile,

• 41:36Speaker 00

  cross browser, any endpoint, load testing, functional testing, all of those capabilities are capabilities that Smart Bear supports. Our goal and our strategy and, you know, when we launched BearQ, I would say our reality is that we're bringing those capabilities to people wherever they are on that autonomy ladder. Those are all things you must do. They're all, if you're not doing them, you should be doing them. But if you're in a kind of heavily people driven process, great, we can help you do that. If you're in a heavily agent driven process, we can help you there as well. It's really our mission and we're not looking to get out of those zones.

• 42:19Speaker 00

  We we feel that they are going to be needed and needed more as agentic coding really takes off and becomes truly reliable. Those questions or the questions that we answer are going to become kind of more urgent and more acute, even more than they are today.

• 42:37Craig S. Smith?

  Yeah. And and how does somebody use your primary tools? I mean, is there a chat interface? Do you?

• 42:47Speaker 00

  Oh, I see.

• 42:48Craig S. Smith?

  Give it access to your code base and?

• 42:51Speaker 00

  Yeah, so so it depends on the product. We have some products that are like systems of record. So the API lifecycle platform Swagger basically will will ingest the APIs from your gateway or from your systems. And then we will organize the assets around that. And so today, that most of our users for that product today are people. So it's a GUI interface, or you can get kind of, you know, command line interfaces, but it's all also accessible via MCP server. If you do have a gentle workflows, we've integrated

• 43:29Speaker 00

  agentic workflows into the product as well. So literally, kind of like with a push of a button, we can read your code and generate the spec. We can read the spec and generate the documentation. So we can do all of that clever stuff. And what we'll be doing over the course of the year is introducing more autonomous agents to do a bunch of those tasks for you. But for the API catalog, I anticipate that those are being overseen always by by people. I think that's just a key requirement.

• 44:02Craig S. Smith?

  What is the role of the human that's managing this? I mean, you said they would be in charge of governance of the API library. But for example, in the testing of an application, so much of this is agentic. What is the human doing, just watching it or reading summaries or, you know?

• 44:26Speaker 00

  Yeah, so so what we do with BearQ, the new agentic and highly autonomous application testing product, is we show our work. So we create daily reports, we have great analytics and insights. We literally say, here's what we did for you today. Here are the tests we created, here are the tests we run, here's what failed, here's what didn't. People can click into that as much as they want to and inspect it down to a fairly atomic level. We mentioned the word trust earlier, as their trust increases, they may prefer to just stay more at the dashboard level. But they can also directly intervene. So they can say to BearQ, hey, I would like you to go and focus a bit more on this portion of the application, go deeper down there.

• 45:11Speaker 00

  And that can be done through a chat interface. Or they can even just say, you know what? I'm going to come in, I'm just going to write a test because there's a very specific test that I personally want, and they can inject that into the process as well. We're trying to accommodate the ways in which people will want to work with agents and with an autonomous system. For some people, that's going to be more hands off, more summary driven. And for others, they're going to say, no, I've I've got team members who I actually want in there, working alongside the agents. So so it's incumbent on us to try and accommodate those use cases, use cases as well. But in general, when you ask, how do people use our products?

• 45:51Speaker 00

  We're trying to tailor the user experience to the appropriate, you know, step on the autonomy ladder and make sure that it's optimized for for for the right end user, whether that's a person primarily, agents primarily, or a combination of both.

• 46:07Craig S. Smith?

  And Smart Bear with BearQ, is it, or yeah, BearQ?

• 46:12Speaker 00

  Yep.

• 46:12Craig S. Smith?

  Is, uh, how do you see the the market? Is it kind of an endless market right now? Because Agentic enters the production processes.

• 46:25Speaker 00

  Oh, yeah. I mean, I think I think the the demand, the need is uncapped. I feel like there's a another shoe that needs, another shoe or two that need to drop in the world's evolving understanding of how software is not just developed, but developed, deployed and maintained. And right now, we're incredibly focused on how software is generated. But there's an awful lot, like eighty or ninety percent of your work is ahead of you once, once you've built an application. That's where we we need to shift our focus next. Or else we'll find that the benefits of just creating applications or updating applications very quickly will end up being somewhat diluted.

• 47:07Craig S. Smith?

  Yeah. And you do you feel like the market that you're focusing on or the, you know, testing and validation, it's traditionally been a crowded market? Is it thinning out as it gets into agentic because people aren't there yet? Or do you feel that it's that there's a lot of competition? And if there is a lot of competition, what's the strategic bet that you're making to differentiate SmartBear?

• 47:35Speaker 00

  What we're trying to do is create clarity for the market and clarity for our customers and prospects about what precisely we and others mean when they talk about having AI-powered solutions. There is a world of difference between pointy clicky SaaS app that has a couple of AI features or has a chatbot on the front and a product that's genuinely got is a team of agents. They're wildly different and that's why we've been leaning so hard on these concepts of application integrity and also on this autonomy ladder

• 48:18Speaker 00

  is so that we can create a framework to explain where we are. We think that what we have with BearQ is unique. We don't know what other folks are working on and it's unlikely that we'll be the only company who has a product that does that, you know, forever. I think there's a disservice being done to portions of the market at the moment by a number of vendors saying, hey, we have this agentic product. We have this autonomous product. We look at it, it's not really, it's not really. You're just kind of jumping onto a word and sadly diluting its meaning, which which will actually end up holding the market back a little bit. We will also see some new companies emerge. There's a lot of YE Combinator startups who are addressing the kinds of problems that BearQ does.

• 49:03Speaker 00

  And there will be smart startups who address this this issue and the types of problems that we care about in in novel ways. And, you know, so it's always incumbent on us to be innovating as well. It's something that we prioritize. Prioritize is is a kind of continuous pressure testing of our own ideas and thinking about what's next.

• 49:23Craig S. Smith?

  Yeah, and we're coming up to an hour, but how do you see this developing? I mean, as you said, five years is too distant a horizon, but generally, I mean, the volume of code that's being written, the volume of software, the volume of of security vulnerabilities that are being created and the demand for human QA that's being eroded to some extent by people over relying on on AI. You said that there'll be that you need people who have a deep understanding of security or, you know, different aspects of an application.

• 50:06Craig S. Smith?

  I mean, I imagine that the workforce is is changing around these things. As tools like smart bear become available, it allows teams to to sort of balkanize in their expertise so that they can go deeper. Or or not, I don't know. How how do you see it developing?

• 50:27Speaker 00

  Um, I mean, it's a huge question and it's it's like, I think inherently subjective. And the only thing I would I would say in your summary there, which I generally agree with is, I do not think that there will be an erosion of human talent. I I think that we may see that the agent workers fill in the gaps that are already there and the gaps that will be growing very quickly as everything accelerates. And I think that the humans are there. I think their jobs will change. There will be things that they do today that can be automated away. But to the extent that you have expertise, whether that's in quality, security, architecture, operations,

• 51:13Speaker 00

  there's going to continue to be a demand for those skills. And kind of in a, you know, we went from assembly to C and everyone thought C was ridiculously high level and you weren't a real programmer. And then we went to Python. It's like, well, you're basically writing a story. You have no idea what's going on under there. And now we are literally writing natural language and still most people can't write functioning software. Well, we've answered the question, like you literally write down what you want the software to do. Most people don't know what to do. And by the way, most people don't have a good idea that they can express well. So what we're kind of exposing is that there was a fallacy that software engineering was typing.

• 51:58Speaker 00

  And it's not, we are going to need people who who understand business problems, business processes, consumer problems, consumer needs, then need to be able to express those in a way that creates a compelling application and the application itself is going to have to be performant, well-architected, secure, private, and all of the things that we've come to recognize. And we're still going to need people giving those instructions. Otherwise, you're going to end up with unbearable convergence to the mean. People or businesses will will accept.

• 52:31Craig S. Smith?

  This episode is brought to you by Tasty Trade. On Eye on AI, we talk a lot about how artificial intelligence is changing how people analyze information, spot patterns, and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data, and the ability to understand risk clearly. That's one of the reasons I like what Tasty Trade is building. With Tasty Trade, you can trade stocks, options, futures, and crypto, all in one platform with low commissions, including zero commissions on stocks and crypto.

• 53:16Craig S. Smith?

  So you keep more of what you earn. The platform is packed with advanced charting tools, backtesting, strategy selection, and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI-powered search feature that can help you discover symbols aligned with your interests, which is a smart way to explore markets more intentionally. For active traders, there are tools like Active Trader Mode, One-Click Trading, and Smart Order Tracking. And if you're still learning, Tasty Trade offers

• 53:56Craig S. Smith?

  dozens of free educational courses plus live support from their trade desk reps during trading hours. If you're serious about trading in a world increasingly shaped by technology, check out Tasty Trade. Visit tastytrade.com to start your trading journey today. I'm going to myself. Tasty Trade Inc. is a registered broker dealer and member of FINLA, NFA, and SIPC.