AI-Powered Threats to the Software Supply Chain

Open source software underpins virtually every modern application. That ubiquity is a superpower for developers, but it is also an expanding attack surface. Software supply chain attacks were once rare but are now happening daily, with malicious actors exploiting the trust developers place in public registries, package managers, and CI/CD pipelines. Chainguard https://www.chainguard.dev/ is a secure software supply chain platform. The company started with hardened container images and has expanded to cover domains including VMs, language libraries, GitHub Actions, and agent skills. Matt Moore https://x.com/mattomata is a co-founder and CTO of Chainguard, and a veteran of Google’s open source, container, and security infrastructure work. In this episode, Matt joins Gregor Vand to discuss lessons from recent supply chain attacks, why CI/CD pipelines are now a primary attack surface, the challenge of meaningful software inventories, the EU Cyber Resilience Act, and what the arrival of Anthropic’s Mythos model means for the pace of vulnerability discovery and the urgency of patching at machine speed. Sponsorship inquiries: [email protected] mailto:[email protected] The post AI-Powered Threats to the Software Supply Chain https://softwareengineeringdaily.com/podcasts/ai-powered-threats-to-the-software-supply-chain/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  Open source software underpins virtually every modern application. That ubiquity is a superpower for developers, but it is also an expanding attack surface. Software supply chain attacks were once rare, but are now happening daily with malicious actors exploiting the trust developers place in public registries, package managers and CICD pipelines. ChainGuard is a secure software supply chain platform. The company started with hardened container images and has expanded to cover domains, including VMs, language libraries, GitHub actions and agent skills. Matt Moore is a co-founder and CTO of ChainGuard and a veteran of Google's open source,

• 0:44Speaker 00

  container and security infrastructure work. In this episode, Matt joins Gregor Van to discuss lessons from recent supply chain attacks, why CICD pipelines are now a primary attack surface, the challenge of meaningful software inventories, the EU Cyber Resilience Act, and what the arrival of Anthropic's Mythos model means for the pace of vulnerability discovery and the urgency of patching at machine speed. Gregor Van is a security-focused technologist, having previously been a CTO across cybersecurity, cyber insurance and general software engineering companies. He is based in Singapore and can be found via his profile at van.hk or on LinkedIn.

• 1:41Speaker 01

  Hello and welcome to Software Engineering Daily. Today is a fun one where we get to talk to a person and a company again. So at the end of many episodes, we say we'll be following along and that's exactly what we've been doing here. So very happy to have Matt Moore of ChainGuard back with us. So yeah, welcome, Matt.

• 1:59Speaker 02

  Thanks for having me.

• 2:00Speaker 01

  Yeah, so some of you may remember the episode we did, probably around two years ago, I think, at this point, something like that. With ChainGuard, everything, talking about hardened containers, about security. So spoiler alert, a lot has happened over the last two years from that perspective. So I think it's going to be super interesting to dive into how ChainGuard has evolved with that and like how it's protecting things and evolved. But as usual, just in case anyone didn't catch that episode, just a very like TLDR of who is Matt Murrah? How did you get to Chain Guard?

• 2:34Speaker 02

  Yeah, I'm Matt. I'm one of the founders of Chain Guard and the CTO. Chain Guard, my co-founders and I, we all met over many years of collaboration at Google. And we collaborated on things across open source, the container space, the security space, the developer experience space. And we have been working on parts of these problems for over 10 years together. But one of my co-founders and I had left Google briefly and I was on a break, barbecuing and playing video games, trying to figure out what I wanted to do next. And our CEO, Dan, was trying to recruit me back to Google. And I was like, I don't really want to go back to big tech just yet. And so I Uno reverse carded him and I was like, have you thought about starting up?

• 3:19Speaker 02

  And we'd been working on aspects of this problem space for a while and really nobody cared. But at that time, SolarWinds had just happened and overnight everyone cared. And so the time was right and we'd spent a lot of time studying this space. And so it felt like right team, right time, and we decided to go for it. And it's been a pretty wild ride ever since.

• 3:42Speaker 01

  Yeah, amazing. And yeah, it's interesting how you mentioned SolarWinds there, like right time. And then we're going to get into almost equivalents that might have happened over the last two years. And then again, for anyone like not familiar, just what is Chain Guard? What are hardened containers?

• 3:59Speaker 02

  Yeah, so we started with containers. Chain Guard on the whole, like we sort of want to give folks a safe way of consuming whatever open source they're writing. These days you can't really write modern software without open source influencing it in some way. Basically, every language runtime or compiler, whatever, is open source, Java, Python, Node, right? These are open source language runtimes. And then on top of a lot of them, those sit on a lot of open source projects, like the system packages that go into every Linux distribution, the Linux kernel. If you're running containers, the container runtimes are open source. Kubernetes is open source. So open source just permeates everything that we do. And it's basically unavoidable these days if you want to write a modern application.

• 4:44Speaker 02

  But it's also sort of a superpower, right? Like, if you think about before the proliferation of open source, how much you had to write in order to like ship an HTTP server. Like back 25 years ago, Amazon was racking servers, right? Google was racking servers to build a startup. You were racking your own servers and going in and dealing with the hardware aspects of this. But like, cloud has made that so that you can just take that and use it. Open source has made it so that the amount of software you need to write to like write an API is so much smaller, right? You write a little bit of glue on top of frameworks that sit on top of HTTP servers, right? And so it's this amazing accelerant that allows folks to write much, much, much less software to build a functional app.

• 5:30Speaker 02

  But really, you need a way of safely getting all of that software. And so we started with containers because what we found was the way folks were consuming containers today, you sort of start with a baseline of hundreds of unpatched vulnerabilities. And it's not that patches for these don't exist in the upstream. It's that the way modern distributions work, they're just very slow to get those into folks' hands. And so we wanted to give folks a better way of doing it, where we would fully patch everything that goes into the containers that they're getting. And really, for many years, early on, one of our big struggles was convincing folks that we weren't somehow lying. Like, this was actually a tangible result. They had this learned helplessness that like what the result that we were showing them wasn't actually possible.

• 6:15Speaker 02

  But it is. It's just a lot of work that nobody prior to us had really done to the level that we did it. And so we've sort of evolved as a company and gone beyond just containers. Really, the way I look at what we do now is being a secure software supply chain as a service. All right, we give folks a safe way of consuming that open source. One of the analogies I really like, and I realize that it dates me a little bit, but like in the early days of digital music, right? Like, folks got their music off Nap. Right? And ignoring the licensing aspects of it, it was a way you could get a lot of music. And sometimes you'd get music, but sometimes you'd get malware, right? That's the state of open source today. If you're just consuming stuff from the public registries,

• 7:00Speaker 02

  it's a lot like the result you'd get from getting stuff off Napster. And so the role we fill is a lot like when iTunes came onto the scene and started allowing folks to buy MP3s. Right? You could buy a song and then you got access to that song. It was coming from Apple, a reputable company, and you had a safe way of getting that music and you weren't going to get malware from Apple, right? And our initial product, we sort of oriented around unit cost. So you buy an image from us, you get that image, just like you'd get that MP3. And then over the years, our catalog grew a lot and we had customers who wanted to standardize on us and wanted access to that whole catalog. And so we shifted towards, we still have that model if folks want that, but we sort of evolved into more of a Spotify model, right?

• 7:44Speaker 02

  Where folks could get access to the whole catalog, listen to what they want and consume whatever container images they want from us. And it's a developer seat pricing model. So depending on which model folks are interested in, they have a safe way of getting their open source from us. We started with containers and then we've expanded to a variety of other different ways folks consume open source.

• 8:06Speaker 01

  Yeah, I love that analogy. I also use Napster back in the day. Moved on to, I think, Kazaa and all sorts of other things. And then, yes, now I'm Spotify because it's very quick and easy and it's definitely the actual song and all that kind of stuff. So I think it's a great analogy. When we talked a couple of years ago, as you've called out, it was originally hardened containers and it is now more of a platform play here. And then almost back then, it was like a CVE reduction story, if you want to call it that. We've got smaller images, we've got fewer packages, actually, like fewer but hardened, fewer known vulnerabilities. But like, how has that conversation, I guess, changed with your customers and with just people interested in the product over like, over the last two years?

• 8:50Speaker 02

  Yeah, it shifted a lot. I mean, hardened containers was always just the beginning. And when you're trying To solve for a space like supply chain security, it's an enormous place. We had to pick somewhere to start and containers was a space we knew exceptionally well. So we decided to tackle that first. But the real question is, how do you secure your entire software stack? All of that open source that your developers aren't writing, but it's running in your environment. Right. So it's subject to all the compliance of your environment. Right. And every unpatched piece in there or every piece that may carry in malware. Right. Potentially presents risk to the business. So we have rapidly expanded beyond containers. We started to target VMs and language libraries a bit over a year ago.

• 9:34Speaker 02

  And then in our most recent user conference, we announced that we were expanding into hardened actions and hardened agent skills. And so one of the ways I think about the framing beyond CVE reduction is I think about CISA's Secure by Design initiative that launched a couple of years ago. And one of the tenets in there is seeking to eliminate entire classes of vulnerabilities, not just patching vulnerabilities, but this is things like writing things in memory safe languages or leveraging like an ORM to avoid SQL injection. Right. So part of the way I look at it is by building things from source, we're actually able to eliminate entire classes of vulnerabilities that come from trusting those public artifact registries. I mentioned the Napster analogy, right?

• 10:17Speaker 02

  Like just by building stuff from source, we're able to eliminate 98 to 99% of the source of recent malware attacks. Right. It's just a huge class of vulnerabilities. Basically, the vast majority of how most of these things happen, right? Like just building from source gets us that. But it also gives us a level of control that allows us to eliminate other classes of vulnerabilities, like folks running unpatched software downstream of us. Right. We can make sure that all the patches are applied from upstream. And so I think two things have also materially changed in the past couple of years since we last spoke. The number of CVEs being reported has just exploded.

• 11:00Speaker 02

  And this is only going to get an order of magnitude worse post Mythos. Right. We saw data coming from Firefox and some. Other high profile projects where the number of CVEs patched in a recent release, I think it was 150, was 10 to 20x what most prior releases had found, right? And so you've got the CVE graph just growing exponentially. But then you've also got malware attacks. These used to be relatively rare and they've gone to monthly occurrences, to weekly occurrences, to daily occurrences. And I posted something on LinkedIn talking about how these things were now happening daily. And someone left a comment being like, soon it's going to happen a couple of times a day. And like literally the next day, there were three attacks that happened on the same day. And so while no layer of defense is going to be able to promise 100% protection.

• 11:45Speaker 02

  One of the things that also hasn't changed in the last two years is that none of our customers have been impacted by any of these attacks where they were using our products, right? So our library's product has been able to defend against all of the malware attacks that have happened over the last several months. And the rate at which they're happening is just frankly, mind blowing.

• 12:06Speaker 01

  Yeah, absolutely. Certainly, open source supply chain attacks have had a moment through the last two years especially. And I mean, it was kind of for many of us, we could almost see it coming. I think it was just you have actually taken action on it and started a company to defend against it. Most of us were like, ah, it will happen probably, but hey, it's like, nothing's going too wrong yet. But here we are, and it really has become very clear to a lot of unfortunately malicious people that that is a great way to actually, that's how you should distribute your malware. So let's go on to like a case here. So there was the XC or XZ util's case. We did actually cover this and one other that we're going to jump onto in another episode, but from a different angle.

• 12:50Speaker 01

  So some listeners may be familiar, but this was XC util's was basically where a maintainer got more and more trust with the community that maintained that set of packages. And eventually they were able to then push in a malicious, I think it was a malicious macro basically into the build system. This went into a package lib LZMA. I'll stop there, that's the super factual. So if someone gains trust. Manages to push a malicious macro into lib LZMA. How did Chain Guard, I guess, respond to that?

• 13:24Speaker 02

  I think one thing worth calling out is like, still in my mind, that attack is still in a class of its own with respect to both the level of sophistication as well as the patience, right? Like, a lot of what we've seen recently is much more opportunistic, large blast radius attacks that try and just smash as much stuff and then get as many credentials as possible and then figure out what other fun they can have with those. This was an incredibly long period of time.

• 13:55Speaker 01

  It's like two and a half years, I think, they spent.

• 13:57Speaker 02

  Person, I say that with air quotes, right? Like, Giatan, right? Like, they, they spent time, you know, gaining the trust of the community, doing legitimate contributions and like, basically, it's resembles spycraft, right? Like, it is a level of sophistication that I don't think I have seen since, right? That doesn't mean it's not happening. I'm almost certain it is, and I think people are worried about it happening all sorts of different places. But that said, right, like, the attacker had maintainer level access, but the attack still required the use of an intermediate artifact. So Giatan basically published an altered release distribution, which a lot of distributions build from, right? And in this case, they were like, haha, the source is this, but then they altered the

• 14:41Speaker 02

  release distribution that they attached to the release that they knew downstream distributions were building from, and it carried modifications that like, activated the malware. Building from source, defended against it in this case, but it's also possible that if Giatan had taken another path, it might not have, right? Because they had done all of that legwork to get that sophisticated and that deep level of control over the project before they were doing their attack and they were doing it effectively in plain sight. But I think this touches on why defense in depth is so critical. And my point about no one layer of defense is ever going to be 100%. And if someone's telling you that, then.

• 15:21Speaker 01

  The Swiss cheese model, right?

• 15:23Speaker 02

  Yeah.

• 15:23Speaker 01

  Yeah, the Swiss cheese model.

• 15:24Speaker 02

  I love the Swiss cheese model.

• 15:26Speaker 01

  Yeah.

• 15:26Speaker 02

  So like building from source, minimizing what you're running, scanning for anomalous behaviors. I think one of the things that's interesting about like scanning for anomalous behaviors, not just necessarily scanning for like known malware signatures, but scanning for things that like programs you want running in your environment aren't doing. So it's not always malware. One of the things that we've been starting to talk about scanning for and blocking is what we're calling grayware. Right? Classes of software that maybe aren't compromised, but do similar things to malware that like you don't want running in your environment. Like they may establish remote access terminals. They may read or transmit sensitive credentials to places they probably shouldn't. There are projects out there that do these things, that the projects that haven't been compromised and people use them like remote access terminal.

• 16:11Speaker 02

  That sounds really great for debugging. Maybe I'll stick it in there, but maybe you shouldn't, right? Maybe your product security team doesn't want that running in your production environment. So yeah, I mean, this notion of grayware where like you flag behaviors that you don't want running in your production environment, whether it's matching a malware signature or not, is I think one of the things that I think we need to start getting real about what you'll let in and what you won't let into your environments.

• 16:37Speaker 01

  Yeah. What's interesting here as well is so the packaging of LZMA that ChainGuard actually uses, it runs on Wolfie as opposed to Debian or you know, x86. So this wasn't actually specifically targeted. It was sitting like, I guess, within a build that you had put together, but because it wasn't actually targeting Wolfie, technically it wasn't a threat, at least in that moment compared to anyone running on Debian. But like, can you walk me through the mechanics of that as well? Like, do you think that was accidental that they didn't also target Wolfie? And I guess like also maybe walk us through if they had been targeting Wolfie, what would that have changed for you guys, for example?

• 17:17Speaker 02

  Yeah, so I mean, I think I mentioned there were a few things like one, when through the release distribution, I think another one was I believe there was something in the modifications that targeted specifically systems that were producing, I think devs and RPMs, which we don't build devs and RPMs. And so, yeah, I mean, like we built the release that was tainted, but like it didn't impact us because the way we built it did not activate the attack code. So I think that this is my point about source level access gives folks a way of potentially doing malicious things. And Giatan had the level of control to potentially do some of

• 18:00Speaker 02

  those nasty things. But it's also why we complement the fact that we build from source with so many other techniques, including running different behavioral scans around the type of Ciscall's that are being made in one release versus the next. And it's not to say that any of these things are foolproof. And I think one of the interesting things about dealing with open source is so much stuff tends to be out in the open. And how you scan for some of this stuff, actually, you almost want it to be somewhat secret, which is counterintuitive to an environment that biases towards transparency and whatnot. But like, say I have an open source

• 18:40Speaker 02

  malware scanner that we run to detect these kinds of things. If I were an attacker, what's the first thing I'd do? I'd scan it to make sure you aren't going to catch me, right? And so there's, I think, limits to just how much you can be completely transparent about everything you're doing and still be protected against folks who know how you're going to scan what comes out the other side and can check in advance, basically, whether you're going to detect them or not. And so it's a balancing act that we have to deal with. But yeah, we are doing various behavioral scans of the package builds where we try and classify behaviors of what they're doing, but also check them for known signatures of malware and things like that.

• 19:23Speaker 01

  And I think ChainGuard Factory flags like an update, like if there was code update that makes a new network connection, that's actually not mentioned in chain logs, for example, that's interesting.

• 19:33Speaker 02

  Yeah, we try and look at a wide variety of signals. I think one of the behavioral classification things that one of the tools we do, and we do sort of differential scanning. So like, if the previous release, the dot one release didn't have it, and the dot two release did, like, patch releases are typically places where you don't expect significant behavioral changes, either at the build time or at in the resulting binary. Maybe you enabled some new piece of functionality. In which case, flagging is probably the right thing to do. You can review that. And then from there on, that becomes a baseline. But there was actually a breach on the library side that was able to compromise the source code. And our scanning was able to block that.

• 20:14Speaker 02

  It was, I forget the name of the recent attack, but it basically embedded a PTH file into a Python library that forced the interpreter to basically load that as, even without importing the library, it loads that at runtime and executes it. And that got flagged by our malware scans and blocked. So we didn't rebuild the malicious release, even though the source for it was available because we detected that PTH. We actually built the release right before, which was totally fine. And after they detected this, they actually cut a new release and we immediately built that, but we didn't build the bad release because the malware scans that we run detected that malicious behavior and blocked it in our system.

• 20:55Speaker 02

  So there's a wide variety of stuff that we are doing to both scan the broader ecosystem, scan the stuff that we are building and running through our systems and then taking an approach to how we build these things from source that tries to eliminate classes of vulnerabilities. We can't eliminate all classes of vulnerabilities, but we try and do what we can to eliminate classes of vulnerabilities where we can. There's building from source. There's things like compiler hardening flags that enable different types of memory safety things. There's potentially exploring replacing components with memory safe alternatives. So things like sudo rs, and I think we were one of the first folks to ever package sudo rs, which is a Rust reimplementation of sudo. But there's a whole wide variety of these.

• 21:40Speaker 02

  We've had the U U util's project, which is a Rust port of the core util's project in our distribution forever. And so there's a lot of these different projects that are coming onto the scene that where possible, we try and make those available. Rust CLS is another really good one where you can swap out the TLS implementation used by curl, lib curl, et cetera, with a version that's been written in Rust for handling cryptography.

• 22:06Speaker 01

  Yeah.

• 22:09Speaker 03

  You're shipping faster than ever with AI coding agents, but those agents don't vet the packages they pull in, and they don't have security context built in. Ori by Endor Labs fixes that. It plugs directly into your editor via MCP, catching vulnerabilities, blocking malicious packages, and flagging exposed secrets in real time. No separate tool to switch to, no dashboard to babysit, security that fits how you actually build. Teams using Ori see 10 times fewer security tickets and six times faster fixes. Free for developers. Get started at www-endorlabs.com/auri. If you're running Postgres in production, you've probably felt the moment analytical queries start fighting your transactional workload. Most teams end up adding a second database and all the pipeline complexity that comes with it.

• 22:53Speaker 03

  Tiger Data, creators of Timescale DB, takes a different approach. We extend Postgres with hybrid row and columnar storage, so one table handles both writes and analytical scans. Native compression cuts storage costs up to 95%. Continuous aggregates keep dashboards live without bash jobs, and it scales to petabytes without you re-architecting. Companies like Cloudflare, Octave Energy, Schneider, Axpo, and Flowco run production workloads on Tiger Data today. No stale data, no second system to operate, just Postgres. Managed for you, ready for the workload you're building toward. Try it free at tigerdata.com.

• 23:27Speaker 03

  Agents are getting smarter every day, but even the smartest agents get stuck without the right context and the right tools. That's where Notion comes in. With the recent launch of custom agents, Notion became the collaborative AI workspace where teams and agents work side by side. And now, their new developer platform is turning that workspace into infrastructure developers can build on. What stands out to me about Notion's developer platform is how much they've built underneath the surface. The platform ships with a set of new primitives that change what you can actually do. Workers are Notion-hosted sandboxes, where you run custom code, database syncs, agent tools, webhook triggers, without spinning up your own infrastructure. The CLI lets developers and coding agents read from Notion, take actions, and deploy workers through one interface. And with the external agent API, you can bring agents like Claude or Codex into Notion as workspace participants

• 24:12Speaker 03

  with real permissions and triggers, not just a sidecar tab you're copying from. Pulling in research sources tracked Externally, syncing them into Notion and having an agent help triage what is worth covering, all in the same place the team already works. Learn more about Notion's developer platform today at notion.com/SED. That's all lowercase letters, notion.com/SED, to try Notion's developer platform today. And when you use our link, you're supporting our show, notion.com/SED.

• 24:41Speaker 01

  Just moving on, I just want to briefly touch on one other case, which was TJ Actions. So again, we did cover a bit of this from a different angle in a different episode. So some may be familiar. This was related to GitHub Actions. So basically this GitHub Action, it was compromised. And what that did then was extract like CI/CD secrets. And we have seen this through other attacks as well, like CI/CD pipelines are now a primary attack surface. And at the end of the day, Chain Guard images, they are built in CI. So like, how do you guys think about security of the build pipeline itself as well?

• 25:20Speaker 02

  All day, every day. So I think one of the things we've been saying from the very beginning is treat your build systems like production systems. CI/CD workflows operate with some of the highest levels of privilege in modern software delivery, right? But they often remain one of the least protected components in the development stack. You worry about least privilege in production. I don't want this one microservice doing more than it should be doing. But then you've got your CI system that has permissions to like, push everything to production, right? And so it's got these incredible credentials, this incredible power to publish things into your production environments. And so we need to protect those. So I think the principles laid out in the Salsa framework are a good start.

• 26:05Speaker 02

  But I think one of the things that, in my opinion, I think is the biggest miss in Salsa, was not mandating the use of short-lived credentials. Because if you look at virtually every supply chain attack that has happened, someone got, and you mentioned some of the fun ones, TJ Action, Shai Halud, they exfiltrate long-lived credentials. Some of them were made possible because someone got a hold of a GitHub Pat or some other long-lived credentials. Right. I think that Solsa fundamentally missed by not mandating the use of short-lived credentials everywhere. And I think at Chain Guard, we feel a little bit like Cassandra, the whatever, the story of Troy, right? Like they bring Cassandra back and she's a prophet, right? She can see the future, right?

• 26:46Speaker 02

  So several years ago, we built Octo SCS to solve this problem for us, right? It is a credential federation service for GitHub because GitHub does not operate one. I've heard one's coming, but I think it's on their public roadmap. But this would have prevented a great many of these attacks. And we operate it as a public good. Anyone can use Octo SCS. And the one long-lived credential that sits underneath it lives in a KMS system that nobody can access without it basically paging our security team. But like, it gives you short-lived credentials. And I even flagged that to the TJ Actions Maintainer when this happened, because like I said, we operate this as a public good. I think another good one that's happened recently is the Trivi breach.

• 27:31Speaker 02

  This built on a technique that Billy Lynch, who is one of our engineers, disclosed to GitHub three and a half years ago, we call it impostor commits, which is this technique where folks are pinning a GitHub action to a commit. That commit actually doesn't have to live in the repo that it looks like it comes from, from the rest of the action line. It can actually come from a fork. If you Google Chain Guard impostor commits, you'll find the blog post. But this is part of the technique that was used to compromise Trivi. And so treat your build systems like production systems, harden them, use short-lived credentials, practice least privilege. I make damn sure that every link in your supply chain is auditable, right?

• 28:15Speaker 02

  Just like you would in your production environments. So you know what came from where, what happened where, et cetera.

• 28:22Speaker 01

  Yeah, that's great, great advice. So yeah, if anyone takes away anything today, take that away. So looking at like Chain Guard as a product that, as we've been talking about, has evolved over the last couple of years, I'm kind of interested in I guess a few things, actually. So the first thing, I guess, is actually just the number of images that you now actually maintain, which I think early 23, it was about 400, then it was, I think, over 1,000 by the end of 24, and now we're pushing over 2,000. And I guess it's like, how do you guys think about just maintaining that consistency and quality? I mean, that's just a crazy amount of stuff to maintain.

• 29:01Speaker 02

  Yeah, there's a lot. I think we're adding 200 to 300 images every quarter. And we've got an amazing team that has been building a lot of those, in some cases by hand, increasingly leveraging automation. But on the upkeep side, right, like 200 to 300 isn't necessarily growing super rapidly, but maintaining those images keeps growing, right? It grows by 200 to 300 every quarter, right? So we're looking at automation really across the board. And a lot of people talk about human in the loop, right? If you think about the purpose of automation, it's generally getting humans out of the loop, right? We're slow, we sleep, and we too can make mistakes. And we've always emphasized automation long

• 29:43Speaker 02

  before things like agents arrived on the scene. And we would try and automate as much as we could, and then humans would get involved when things broke down. And when we did get involved, we would look at ways of avoiding repeating that breakage, as you might look at a postmortem, right? You always reflect and you're like, oh, how can we do better next time? And over time, those systems have improved. But agents have come onto the scene, and we aren't ignoring them, right? We are looking at ways we can safely use them. But the way we look at them is they're sort of yet another tool for pushing our automation further, handling more of that without needing human intervention. But ultimately, just because our automation is becoming agentic, just because you stick that word in front, it doesn't change any of our commitments or our SLAs. So we're trying to find ways to

• 30:28Speaker 02

  adopt it responsibly and figure out those best practices, because the way I see a lot of what we're doing is operationalizing best practices at extreme scale. One of the very first places we actually employed agents was to bolster our testing across our inventory, right? Like, can it generate more tests that exercise different behaviors of libraries or packages and do things that could help us catch regressions coming in through new updates? And so we still very much have humans in the loop, but the mindset shift is increasingly going from how do we solve a problem to how do we solve entire classes of problem?

• 31:08Speaker 01

  Yeah. And the average patch time, I believe, is still incredibly impressive. You know, like a critical is like less than 20 hours and then high is two days, something like that.

• 31:17Speaker 02

  And we recently announced an SLA for the KEV list. So if a vulnerability is known to be exploited, we have a 24 hour SLA for getting that fixed and in folks' hands. So.

• 31:29Speaker 01

  That's amazing. I guess looking from a product perspective, but I guess more on the competitor landscape as well. I think, you know, maybe a few people might be thinking, hey, doesn't Docker now do this? So yes, they do. And I guess that must have been an interesting moment for you guys. It does happen unfortunately to very great ideas of companies, you know, the bigger guy comes along and says, that looks interesting. We can do that too. GitHub, I think, have done that. And GitHub Actions was basically a clone of other startups into GitHub, made sense. So yeah, how has that affected things? How you approach this problem? Docker has a free tier. That's I think interesting to talk about as well.

• 32:06Speaker 02

  Yeah. And speaking of things that have changed in the last two years since we last spoke, competition has really come out. And, you know, I mentioned we had to convince people that what we do is possible. Thankfully, that's actually a conversation I have to have a lot less now that other competitors have come out with products that at least claim to do the same thing that we do. I think one of the confusing things with VHI is that they claim two things, only one of which can be true. Like one is that they are building on top of existing distributions like Alpine and Debian. They also claim that they're building everything from source, right? But if they're building everything from source, then they're not Alpine or Debian, right?

• 32:47Speaker 02

  They can't carry patches. They can't build versions that Alpine or Debian aren't carrying and, you know, and build those things from source and then claim it's Alpine or Debian, right? Those two things are sort of in conflict. I think the other thing is we are very familiar with the Alpine ecosystem and the Debian ecosystem, and we use them for many years. Our CEO, Dan and I really built the first real hardened images back in 2017, 2018, which were called the Google Distraless Images. And we actually did build that on top of an existing distribution. We built it on top of Debian packages. And it was because of this that we knew that to get to the results that Chain Guard delivers, that we couldn't build it on top of another distribution. If you take any of those images, which are running on the latest Debian,

• 33:31Speaker 02

  they're minimized and have the latest versions of all those packages. You take any one of those images and scan them, with the exception of one, they all have unpatched CVEs because Debian is, you know, deciding what to backport, what not to backport, et cetera. If you scan our equivalents, none of those have any CVEs. And so, you know, I think I'm glad that more folks are starting to take this problem seriously and not just because I don't have to explain that what we do is actually possible. But I don't think all of the approaches are necessarily created equal. And I think our approach is informed by an extremely long history in this space, doing things like this to harden these images and figure out the path that we actually have to take in order to, you know, achieve the result that we are getting.

• 34:16Speaker 01

  Yeah, I think usually in this case, it is so that, you know, rising tide raises all boats, you know, I think, or just like a bigger pie or whatever analogy you want to give it, but it should overall be positive. But yeah, it's obviously also interesting, yeah, when such a large company does come along claiming to do sort of a very similar thing. And I think that distinction that you've already made is very helpful. One other thing that was interesting, I think, end of last year, you guys announced Emirates OSS. You know, this is like maintaining deprecated open source projects.

• 34:48Speaker 02

  Emeritus, it's a play on.

• 34:49Speaker 01

  Oh, Emeritus, yeah, I was like, Emirates, is that like to do with?

• 34:52Speaker 02

  The OSS at the end, yeah.

• 34:54Speaker 01

  Emeritus, there we go, nice. I was going to ask what the name meant, but you just, yeah, there we go.

• 34:59Speaker 02

  Yeah, so this is, I think that the origin of this, you know, is that certain projects that are, in some cases, you know, critically load bearing for folks' workloads stopped getting maintained. And so there was one that was near and dear to us, so. Many, many, many, many projects that Dan, I and others collaborated on Google is actually the starter project for one of our engineering managers when she joined, reporting to Dan at Google, Priya, was a project called Conico. She has since left Google and just joined us and leaves our container theme, actually. But so, you know, this project, like, Google never really had, they never productized it, they never supported it, but this project literally closed

• 35:43Speaker 02

  a three, four-year-old issue for how do I build a container with a Docker file on Kubernetes without mounting the host daemon socket, without running with privilege so that I can, you know, start my own Docker daemon, you know, Docker and Docker, stuff like that, the kinds of things you don't want folks doing in your, you know, cluster if you're in SRE or your product security, like, Conico closed that issue. It showed folks like this is how you can do it. And I think GitLab documented is one of the ways to build container images. I was, I joke that I can't meet with a bank without them talking about how they use Conico for image builds. And then, you know, one day, Google stopped really merging any pull requests. And I think it took about six

• 36:26Speaker 02

  months before they openly acknowledged, like, they were basically archiving the repo. And so, we had a lot of history with this project, obviously. And so, we were like, and we knew how important it was to a lot of folks. And so, we decided we would take it on and we would make sure that it was receiving patches for various security problems in the libraries that are in there. But we weren't going to do a lot of feature work. It's the, the nature of sort of, at some point, a lot of open source projects reached this quote unquote done status. And it was, you know, it was pretty mature. It was, you know, had a lot of the features that it needed from supporting the Docker file spec. And we wanted to give folks a stable and safe way of consuming that piece of software where, you know, we would keep patching the security stuff,

• 37:07Speaker 02

  but, you know, not do a lot of feature work that would potentially destabilize their workflows. And so, that was really the seed from which this idea for Emeritus grew. And there have been a few other projects that we've taken on with a similar sort of philosophy, right? We will build images from them, you know, we will patch any Abilities that come in through dependencies. If things are disclosed in the package itself, we will look into that and try and remediate those things. But we're not going to do feature work, right? I think one of the great things about open source is if someone really wants to take on like making substantial changes to a, you know, a project, they can fork it and they can add their features to that.

• 37:48Speaker 02

  And if that fork becomes, you know, a successful alternative, then people can just start consuming that. And like, that's great. That's we're open source working as intended. And that can become the source of truth that folks use if they find the features that are being added valuable. And so I think there are some great examples of like really successful forks that become bigger than the original project. I mean, I think a lot of folks don't appreciate these days that like Jenkins started as a fork of a project called Hudson. Because Hudson was owned by Sun and then Sun got bought by Oracle. And the, the person who started Hudson, Koshukei, left Oracle and wanted to continue as a source project.

• 38:31Speaker 02

  So he forked it as Jenkins and now who talks about Hudson? I mean, I think it's probably a vanishingly small set of folks who even know what it is.

• 38:40Speaker 01

  I didn't know that story. I had not heard of Hudson, so that's yeah, super interesting. I guess as we kind of like look forwards, I guess in the present and forwards, you have mentioned that Chain Guard is not just hardened containers anymore. So kind of using that framing, you know, I think looking at things like S bombs, you know, so this is software bill of materials. We've covered these in past episodes as well. This is more like runtime behavior, like what is being fetched, like what is the provenance? This feeds into the fact that we do have regulatory frameworks coming in towards the end of 2027. One is the EU Cyber Resilience Act, which, okay, it's EU, but it will no doubt frame still a lot of what happens on the US side. And no doubt you have EU customers, so they have to abide by that.

• 39:25Speaker 01

  And that is saying, you know, mandatory S bombs, things like 24 hour vulnerability reporting. Like, are you kind of taking in that kind of thing into where the product's going as well? Or like, you're already thinking about that? I just, yeah, what are you guys thinking about all of this?

• 39:41Speaker 02

  I think it's a nice tailwind for us because I think we do a lot of the things that folks need in order to satisfy these things. All of our images carry S-bombs that we generate at build time, but we also go out of our way to make sure that SCA tooling that generates S-bombs works exceptionally well with our software and the binaries that we include in that software. And so we definitely do a ton of stuff to make it so that folks can either get the artifacts that they need from us or generate them from the images that we are producing. And I think one of the interesting things, just to sort of circle back to the Trivii breach, I think the European Commission itself was actually compromised

• 40:22Speaker 02

  by that Trivii breach, right? And so they disclosed that they were compromised by it because of that 24-hour reporting deadline. I think it was beginning of May, and something like 340 gigs of EU citizens' data was leaked as a result of that, right? And that's something that wouldn't have impacted them if they were using Changard's images, right? And so I think that even if folks are seeking this really just to be compliant, right, there's really nice security properties that you get out of adopting hardened images, adopting software that knows exactly what's going into it. I worry that when folks talk about things like S-bombs, that they aren't talking about the complete picture a lot of the time,

• 41:06Speaker 02

  though. And so everything that goes into our images is captured in our S-bombs. And I can assert that because everything that goes into our images comes from one of our packages. And all of our packages capture everything, like every file that ends up in the final images file system comes from one of our packages. And so we have this sort of chain of provenance. For every single file, we know exactly where it came from, and you can trace the builds all the way back. But that's because that we own that full end-to-end. And so I think that folks who aren't doing this, right, you look at the way folks traditionally have built container images with Docker files. Like Docker files, most of the official images are bootstrapped from like a random tarball somewhere that has a root

• 41:51Speaker 02

  Hopefully, that's a set of packages that that RutaFest was built from and then they start to add stuff up on top of that. Some of those come from the package manager, some of those curl pipe to bash in the Docker file build. One of my favorite things to call out around S-bombs is what our CEO Dan calls the WordPress test, right? If I have an SCA tool, can the SCA tool tell me that WordPress is in the official WordPress image on Docker Hub? And I think we found one that can actually tell you that WordPress is in there. I think it's the Black Duck scanner, which uses a sort of signature, like file signature based scanning. But because the official image like pulls down the source code and builds it in the container and like that's what you get.

• 42:36Speaker 02

  There's no metadata telling you that WordPress is actually in that container, right? If you scan our WordPress images, because it comes in through the system package manager, it's represented in our package database. Every SCA tool on the planet can tell you that WordPress is in our images, but it's also in the S-bomb that you're getting from us. So I think one of the things that I worry about, it's not just mandating S-bombs that I think is important, is this notion of dark matter, right? How much coverage does the S-bomb actually have of the files that are in your system? And like, I mean, can I give you an empty JSON document and be like, here's your S-bomb? Satisfies the schema, but there's nothing in it. It has 0% coverage. Does it pass the mandatory S-bomb check, right?

• 43:19Speaker 02

  And so, okay, that's an extreme example, but say I give you an S-bomb that covers like 10% of what's in the image in terms of, you know, packages that are installed. Does that count for mandatory S-bombs? Like, this is one of the tensions that I think that I've always struggled with, with people like being like, yeah, mandatory S-bombs. It's like, this isn't a yes or no thing. This isn't a binary thing. We should look at it sort of like code coverage, right? What is the coverage your S-bomb has of your image, right? Because an empty JSON document would obviously be zero, but I think the intent of it is to have 100% coverage. But they're not saying that, right? And I think that's one of the things that I just take issue with in general when folks talk about S-bombs.

• 44:00Speaker 02

  They're not talking about the bar you should be clearing when you are generating that kind of metadata about images.

• 44:06Speaker 01

  That's super interesting. I guess these frameworks, and especially if they're trying to like implement them as mandatory at all, then, well, I guess it's this question yet. Is it mandatory? Should it be mandatory? Or should it be, there's a bar, but then you say mandatory, and I guess they had to pick one. But yeah, I totally agree. I think this will initially lead to probably a lot of subpar SBOMs. I think at least getting the terminology and getting a lot more, I mean, I can think of so many developers, if I was to say, just say SBOM, have no idea what I'm talking about. That's going to be helpful. Yeah.

• 44:35Speaker 02

  When the original executive order happened in the US, there was also this tension where like, they didn't want to be prescriptive about what format you use. And there's two major formats, right? There's Cyclone DX and there's SPDX, right? Which both have, you know, community backing, but they didn't want to, you know, pick a winner, right? And so there's no prescription around like what format you use. And I think because of that, they also didn't really do too much around prescribing like the structure, because you'd have to do it twice and do all this other stuff. I would go so far as to argue that like, the package database that distributions include in their images is effectively an SBOM, right? And so I think that you could arguably check the box. Every SCA tool on

• 45:15Speaker 02

  the planet can generate you a fantastic SBOM from a distributions package database. It's the first thing any of them look at. It's all the other stuff that doesn't come in through the package manager, that where things start to struggle, like the WordPress test I was talking about. And so that I think is one of the things that there's some tension around. And, you know, yet another gap in the like, okay, well, like what mandatory SBOM? You're not talking about coverage. What format? Like, what are the minimum elements? What is the depth of resolution? I could tell you, you know, at a package install level, I could go down into the file level within like a go binary file. I could tell you the transitive dependencies with respect to every version of every go package that

• 45:59Speaker 02

  got pulled into that binary. Then there's other fun things too. Go is really good about including that dependency information, but Rust doesn't do it by default, right? And so you have to enable auditable mode in Rust compiler builds in order to get that same metadata. And we do that by default in our tool chains, because like I said, we go out of our way. To make it so that SCA tools work really well with Chain Guard produced images and binaries. But like you can get images today from AWS. They built a lot of stuff in Rust, but like they're not turning on Auditable Mode. And so there may be vulnerabilities in some of the libraries they're using, but you can't tell because that metadata is not there. Right. And so, and if that's not something an SCA tool can recover because the metadata is not there and it's not captured in the SBOM.

• 46:43Speaker 02

  Who knows, right? You don't know whether or not you were vulnerable to that. And so that's part of why we go out of our way to work really well with SCA tools. In addition to capturing as much metadata as we can, but there's file coverage, package coverage, there's depth of resolution. There's how you actually like structure all of that information as well. There's so many things that are not well scoped. And so like saying mandatory SBOMs. Without any of that other stuff means it's really hard to satisfy that in a way that is particularly meaningful. And so as a result of that, right, like we do produce SBOMs, but like they don't necessarily have the same shape everyone expects.

• 47:25Speaker 02

  Because every one thing, you know, you ask 10 different tools, what the right shape to produce an SBOM is, you'll get 10 different answers. But if we work really well with SCA tools that can generate their own SBOMs and you can verify the image came from us via our image signatures. You can generate your own SBOMs from our images that we know will work with your tool. And so that's why we do it sort of both ways. Right. And we don't think that the SBOM space has really matured enough that like just generating an SBOM is enough. And it's because of all of these unanswered questions about shape, coverage, depth, et cetera. That I think make it a hard thing to satisfy, at least in a useful way.

• 48:05Speaker 01

  Yeah, absolutely. Kind of related, like as we're sort of cruising to the end of the episode, but there's a couple of things. First is related to what you've just been saying, enterprise adoption. What friction points could you, I guess, speak to that's not a friction point, it's just a perceived friction point. And actually, I guess, using Chain Guard is like, you should just talk to us.

• 48:26Speaker 02

  Definitely some fud out there about stuff like migration and things like that. Fear of the unknown. It's like, I don't know how hard it is to migrate, so it must be impossible. Right? And so, you know, this is some of what we deal with coming from other distributions, you know, like Debian, like a RHEL-based distribution. And so I think that, you know, we've gone out of our way to accommodate in various ways as broad a base of compatibility as we can, right? So we use the same package manager that Alpine does. But one of the big sources of incompatibility in Alpine-based images is the muscle LIPC. They use a different LIPC than virtually every commercial distribution on the planet, as well as many of the non-commercial but widely used distributions.

• 49:07Speaker 02

  So like Debian, Ubuntu, all the RPM-based, you know, Amazon Linux, RHEL, et cetera. They all use GLLIPC. And so when we bootstrapped our distribution, even though we decided to use the APK package manager, we bootstrapped our distribution around GLLIPC to get that sort of maximally broad compatibility. And so I think there's definitely a lot of fud around migration, but it's definitely a spectrum. And I think the main thing is, okay, depending on the package manager you're coming from, you know, if you're coming from Alpine, now it'll look very familiar, APK ad, package name. If you're coming from Debian, yeah, you'll have to change apt-get into APK ad. Or, you know, if you're coming from the Red Hat ecosystem, you'll have to change, you know, yum install into APK ad.

• 49:51Speaker 02

  But we have tools that help streamline this, make this easy. We have a new tool that we call the gardener that helps drive a local migration for you, leveraging, you know, our agents to help you do some of that package mapping and stuff like that. So if you're coming from Debian, we can be like, oh, this package is called this in the Debian ecosystem, but it's this within the Chain Guard ecosystem. Or if you're coming from the RHEL ecosystem, it may be called something else, right? And so mapping, package names, doing those kinds of things, and really helping you drive that further and further to completion. And we have a bunch of other stuff that we're working on to sort of smooth migration. But, you know, in some cases, it's really just a matter of dropping one of our images in. So the vast majority of our images are application images, which most folks just take and run.

• 50:36Speaker 02

  Right, like something like an Envoy, you typically plug it in and run it as a proxy with some configuration. And maybe you're running something like, you know, Istio. How many folks are modifying their Istio images? They're mostly just taking it and running it through Helm Chart or something like that. I think that one caveat to that is folks who want to like inject their own root certificates into it. Maybe they are using something like Zscale or something like that. But other than that, right, like it's, it's actually relatively uncommon for folks to be modifying the application images. And so those will just drop it and run as the application. You don't have to worry about it having a different package manager. You're not installing packages. And then when it comes to like migrating stuff like Docker file-based builds, right?

• 51:20Speaker 02

  We have tools that will help drive that migration and help you get as close to that finish line as possible, if not all the way there. And then with things like our libraries products, a lot of times our customers' goal is to effectively just turn a switch. Like a lot of our customers will proxy our language packages through something like Artifactory or Cloudsmith or Nexus, right? And they've already done that to sort of insulate their internal networks from developers just being able to pull whatever. And so when you have that kind of mature setup where you can regulate what packages your developers consume, and it's coming in through Artifactory, maybe it's just passed through from the outside world, but you have that cut point.

• 52:04Speaker 02

  You can cut that over from pulling from, say, Maven Central to pulling those from Chain Guard. And your developers may not even know that you've switched it to Chain Guard. So on the library side, it's much more transparent in terms of, you know, dropping in and replacing things. That's our goal, to be as drop-in as possible. But I think some of where that can, you know, deviate a little bit, it's the same migration problem you'd have. If you wanted to move to one of our images that doesn't have a package manager, it's the same as moving to a Google Distrilist image, which half of this cloud-native ecosystem is using these days. So it's clearly possible. But if I were to move from like a Debian-based image to a rel-based image, I would have to rewrite my package install lines anyways and, and figure out the package map.

• 52:46Speaker 02

  And get it working again anyways. And so any sort of migration like that will have a little bit of work, but folks are already going through those kinds of migrations to like one LTS release to the next. And, you know, where a lot of system packages are changing. And so we try and give folks something where we aren't an LTS based distro, we are a rolling distro. So if they migrate to us once, which we can help them do as smoothly as possible, they shouldn't have to go through that every four years when there's a new LTS as well. Right? We can hopefully, you know, be the last migration that you ever do. And we, like I said, are going out of our way to make that as easy and painless as possible.

• 53:28Speaker 01

  Yeah, amazing. For anyone, I guess at that sort of enterprise level, but medium company, whatever you want to call it, like just that on ramp isn't maybe as scary as it sounds. Very briefly, just, we were talking about it before we started recording anthropic mythos fable model or fable, which is derived from mythos. Super curious, just like this is very bleeding edge because it's only been announced in the last 24, 48 hours, but how does that affect your day to day or like you've been, I think, using it a little bit. Like, what are your first thoughts?

• 53:56Speaker 02

  I think it's, when it first came out, everyone freaked out. And then there was a whole group of folks who were like, this is just marketing. I've seen some of the findings. It is not just marketing. These are real vulnerabilities. It has very good capabilities. There's several different aspects of it. I think I mentioned the Firefox numbers, right? They found a whole slew of vulnerabilities that other agentic tools that they had been running weren't finding, right? And I think one of the things that makes it exceptionally interesting is its ability to chain together multiple vulnerabilities, which, you know, it's in real time, we're watching it close the skills gap that's needed, like this notion of sort of scarcity, right?

• 54:36Speaker 02

  Like right now, there's the skills needed to find and weaponize vulnerabilities are scarce. It's hard. It takes time, right? And that time is dropping precipitously. And so I think the fable model that they released today has all kinds of security controls. The first thing I threw at it mostly for Grin's was to audit security isolation tool that I've In, you know, having OPUS just audit in a loop and find and fix vulnerabilities in this abstraction. I've been very impressed with how good OPUS has been, you know, I've been trying four, seven, four, eight. So I threw this at Fable and I got this prompt back being like, it looks like you're trying to do security stuff. I'm downgrading to OPUS four eight. So it wouldn't let me do it.

• 55:17Speaker 02

  So they said in the announcement that they had put in security controls around keeping folks from doing this for vulnerability research. And it caught me. So I think there are controls that will keep folks from using it in that way. Folks always find a way around those controls though. And so I think the thing to bear in mind is this is not marketing. This is very real. This will find tons of vulnerabilities that other models probably haven't found yet. And folks need to be prepared to patch things at machine speed. Because you could already weaponize vulnerabilities at machine speed. And now with Mythos coming into the picture, being able to find even more and more effectively weaponize vulnerabilities at machine speed.

• 56:00Speaker 02

  If you are prepared to patch things at machine speed, which is really what we've been building towards at Chain Guard since before we even realized this was a problem folks were going to have. You're going to be pretty unprepared for the world of hurt that I think is going to play out over the next six to 12 months. As at least the initial tranche of Mythos vulnerabilities become, you know, known to the world. And then it's a muscle you need to build even if you get that initial tranche patched. Because I would be very surprised if we didn't continue to see step function improvements in model capability. The quality of the harnesses for leveraging the models to find these vulnerabilities. And it's going to be an interesting next year.

• 56:44Speaker 01

  For sure. It really is. So on that note, thank you so much again for coming on again. Learned a lot this time as well, as I and I think the audience did last time. So yeah, who knows, we'll maybe get you back on in another two years. We're here because you know, this landscape is just changing so much and so fast. So thanks again, and I'm sure we'll catch up again.

• 57:02Speaker 02

  Thanks for having me.