Mozilla Firefox CTO on Browser War Stories and the Path to Distinguished Engineer

Bobby Holley went from an intern to the CTO of Mozilla Firefox. I asked him about everything he learned in that process. We cover his full career including some interesting stories on living through the browser wars and advice on career growth. 𝗣𝗼𝗱𝗰𝗮𝘀𝘁 𝗹𝗶𝗻𝗸𝘀: • Transcript: https://www.developing.dev/p/mozilla-firefox-cto-on-browser-war • Spotify: https://open.spotify.com/show/0MX9PyeCzDhdlyRv6slwIX • Apple: https://podcasts.apple.com/au/podcast/the-peterman-pod/id1777363835 𝗘𝗽𝗶𝘀𝗼𝗱𝗲 𝗟𝗶𝗻𝗸𝘀: • NYT article link: https://www.livemint.com/Industry/q2EjgGX6d5Ouwec479WSqM/For-Mozilla-Google-group-hugs-get-tricky.html • Mozilla VP twitter thread: https://www.computerworld.com/article/1722183/former-mozilla-exec-alleges-google-torpedoed-firefox-with-oops-excuses.html • Internal memo on writing: https://docs.google.com/document/d/1518xKjijjEWHQb6wZjAWJrUN8liZGGI9v5pRFr9eFHo/edit?tab=t.0#heading=h.1gfr5hva69qx 𝗧𝗶𝗺𝗲𝘀𝘁𝗮𝗺𝗽𝘀: 00:00:00 - Intro 00:00:57 - Starting at Mozilla 00:04:57 - Browser wars history 00:10:55 - Google relationship changing 00:16:11 - Why work for free 00:19:02 - Projects that drove his career 00:33:12 - No performance reviews 00:34:42 - Rust adoption 00:43:33 - Career progression 00:47:54 - Should you focus on promos 00:57:14 - Distinguished promo rejection 01:00:56 - Examples of distinguished engs 01:10:54 - Advice for aspiring distinguished engs 01:14:40 - AI browser wars 01:26:32 - Biggest technical regret 01:29:11 - Book that impacted his career most 01:32:09 - Advice for his younger self 𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗕𝗼𝗯𝗯𝘆: • LinkedIn: https://www.linkedin.com/in/bobbyholley • X/Twitter: https://x.com/bhology 𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗥𝘆𝗮𝗻: • Newsletter: https://www.developing.dev/ • X/Twitter: https://x.com/ryanlpeterman • LinkedIn: https://www.linkedin.com/in/ryanlpeterman/ • Threads: https://www.threads.com/@ryanlpeterman • Instagram: https://www.instagram.com/ryanlpeterman • TikTok: https://www.tiktok.com/@ryanlpeterman

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  AI is coming to consumer technology like water running downhill.

• 0:05Ryan Peterman?

  This is Bobby Hawley. He grew from an intern to the CTO of Mozilla Firefox and lived through the browser wars with Google.

• 0:12Speaker 00

  Every time Google would do something that would hurt Firefox, it would always be framed as unintentional.

• 0:18Ryan Peterman?

  He also had interesting stories from his projects, including some difficult security bugs he tackled.

• 0:23Speaker 00

  I was disassembling it. And then I had this realization that like, I knew who had written this exploit.

• 0:28Ryan Peterman?

  How'd you know that that security researcher wrote that vulnerability? He shared tips helpful for any career in tech.

• 0:36Speaker 00

  One thing that I think that everybody can work on at whatever level they are is writing. And I think it is one of the most powerful things.

• 0:44Ryan Peterman?

  There's this new era of browser wars, bringing AI into existing browsers.

• 0:49Speaker 00

  I have pretty strong opinions about this.

• 0:51Ryan Peterman?

  Here's the full episode.

• 0:57Ryan Peterman?

  Before actually working at Firefox, I saw that you, you intern at Nvidia. I'm kind of curious, why'd you pick Mozilla over Nvidia?

• 1:05Speaker 00

  Yeah, so I mean, my first internship after freshman year was at Nvidia. And it was, it was a great internship. I think it was much more standard of internship. You know, you show up, there was orientation, they showed you what to do, you sat at your desk, gave you a very basic starter project, said, well, you know, we'll check on you in a couple hours. And it was really designed to be sort of thoughtful, you know, growing experience for somebody who didn't really know what they were doing. And Mozilla was, I would say, quite different from that. I remember showing up, really, I think on my first day, my manager was the director of platform who ran all of Gecko. It was this

• 1:45Speaker 00

  guy, Damon Seacor. And he really, he had a very particular vibe. He sort of embodied, I think, the dude from The Big Lebowski in like, all of the best ways. Like, it's just extremely laid back. And I remember asking him, you know, what should I work on? And he was like, I don't know, like, go get on IRC and get on Bugzilla and like, start figuring out some things to do. And that was just a very big contrast to the sort of more standard internship. And what that spoke to was the nature of how Mozilla operated. Because Mozilla had been an, you know, open source project that largely happened in some ways by accident,

• 2:27Speaker 00

  where lots of people just showed up and started working on things. And there was not a lot of support. And there wasn't, you know, this was like way before video conferencing. There was just the bare minimum of what you needed if you were really dedicated and really creative and really persistent to figure out like how you would download the code. You had to figure out how to download it. You have to figure out how to build it. You had to figure out where to go to, you know, file bugs and like get help. And there was just like not a lot of guardrails. And what that selected for was a particular type of person who really just needed the bare minimum and was very motivated to just go in and figure things out.

• 3:10Speaker 00

  And so the management structure at the time was very much oriented towards nurturing and empowering this culture, right? Very much like, let's get the right people together and let's just get out of the way. And so this, for me, I would say initially was a little bit disorienting. But I ended up getting really inspired by all of the people who were working on it, some of whom were paid, some of whom were not paid. And a lot of these people, you know, maybe they'd started being paid recently as Mozilla started to have money, but had been, you know, getting involved in the project from their college dorm room, just trying to figure out how to make their browser crash less. There was this guy, sort of one of my longtime heroes, Boris Zabarsky.

• 3:53Speaker 00

  We didn't really, I don't think there were really titles at the time, but he eventually was one of Mozilla's first distinguished engineers. And he was just...

• 4:02Speaker 00

  Very awe inspiring to me because I only, you know, for I think several years I only met him on IRC. But Mozilla was a Wild West, you know, the Firefox codebase was a Wild West of code, just like tens, you know, millions and millions, probably about 10 million lines of code that were inherited from Netscape and largely unowned. And the major challenge was just dealing with all of this code and figuring it out. And Boris probably owned like half of it. He was the person of last resort that you would always ask if you didn't know what to do. And you would realize that you'd be having this conversation with him on IRC. And for me, it would be like taking all of my focus of trying to understand and like, you know, keep up with him. And then I would realize that he was in like five

• 4:45Speaker 00

  other channels at the same time, like having these conversations with like other senior Mozilla engineers, other interns, contributors, users who were just having bugs. And he was just a machine.

• 4:57Ryan Peterman?

  So I'm kind of curious, at this time, how does this tie into the timeline of Google Chrome and the browser wars? Was this before Google Chrome?

• 5:07Speaker 00

  Yeah, so this was before Chrome. I guess as a brief history lesson, so the original round of the browser wars was in the 90s. And this was largely defined by Internet Explorer versus Netscape. And they competed with each other by trying to, you know, ship different proprietary APIs and make the site work better in IE or work better in Netscape. But eventually, Internet Explorer won largely because Microsoft bundled it for free as part of Windows and was very aggressive with respect to distribution and really pushing Netscape out of distribution. And Microsoft eventually was, you know, taken to court and lost and, you know, had an antitrust judgment about precisely this. But the way the market works, right? Like, it didn't matter. Like IE had already won. And Netscape

• 5:52Speaker 00

  was going under and as sort of a, as a parting shot, they open sourced the code and created this Mozilla Foundation to steward this gargantuan mess of browser code that was Netscape Navigator. And

• 6:07Speaker 00

  Then what happened was that Microsoft decided that this web thing was not particularly aligned with the strategic interests of Microsoft, right? Because this web thing was a new platform where you could use the same, have the same experience across different operating systems. And so they largely just declared that they had won and they put the browser in maintenance mode. But there were a lot of people who really saw the potential of what the web could be and what it already was with like Internet Explorer 5 and Internet Explorer 6, but that it could be so much more. And a lot of these people, like Boris, were just drawn to, I want to make this experience better and I want my browser to stop crashing. And so people started downloading this source code for Netscape Navigator and working on

• 6:48Speaker 00

  it and gradually improving it and adding new features like the ability to block pop-ups and the ability to have browser tabs rather than a new window for every website. And this sort of gradually turned into a really more useful browser than Internet Explorer was. But it was still very much a suite, right? It had an email client in it, it had an address book, it even had an IRC client for all of the developers, you know, it was kind of built into the browsers, they could talk to each other. And eventually, a few, I think relatively junior people, you know, just like some, you know, contributors, interns, you know, came along and decided to build a stripped-down version of it. And that was originally, there was just a browser, the bare minimum of what you need

• 7:33Speaker 00

  for a browser. And that, originally, they called it Phoenix, but then that was a BIOS, then they renamed it Firebird, that turned out to be a database, and then finally they called it Firefox. And Firefox just took off like wildfire because the world really wanted a better browser and it was a better browser. And so Firefox became, you know, just this thing that everybody, you know, the first thing you do when you got a computer, what you do for all your friends, what you do for your family members is install Firefox because it's just such a better experience. And Google was extremely well aligned with this, right? Because Google was this burgeoning product and business that was largely built around a search engine that was about discoverability of the web.

• 8:15Speaker 00

  And so that the more that the web was discoverable and usable and useful, the better it was for Google. And Google quickly realized that Internet Explorer was a big problem and that Firefox was really this coming solution to this. And so Google was very, very invested in Firefox's success. When I started as an intern, we all received badges to go over to the Google campus and eat lunch whenever we wanted. There were snacks in the kitchen that I heard, I'd never quite confirmed, were directly delivered by the Google facilities people. And of course, the most important was that Google signed a deal with Mozilla to be the default search engine for Firefox. And this was perfectly fine with Mozilla because everybody liked to use Google anyway,

• 9:00Speaker 00

  and this was a great user experience. And it also suddenly allowed Mozilla to be financially sustainable and to hire a lot of these contributors who'd basically been working nights and weekends. And Google had engineers that were directly working on Firefox and contributing to it. And there was really this sense that Google and Mozilla together were working to unlock, you know, the future of the open web. And when I started in June of 2008, that was sort of the peak of that sentiment. But shortly thereafter, Google announced that they were building their own browser, which was Google Chrome. And I think that you can obviously see why in retrospect, right?

• 9:42Speaker 00

  The real, the business strategic posture of Google as a company was really hinging on this ragtag group of very weird people who were working on this open source browser. And there was a desire to have a little bit more control over their own destiny. And so when they initially announced Chrome, you know, we had a meeting, John Lilly told everybody who worked at Mozilla about this, but indicated that it was still very friendly. The desire was to take market share away from Internet Explorer and that we would still continue to be close partners and on the same side. And initially, it really did feel that way. But as time went on, things started to change.

• 10:24Speaker 00

  And I think, I think it was like 2009, there was a New York Times article that I think really captured the zeitgeist. I think the title of it was something like, "Between Mozilla and Google, group hugs are getting tricky." And that referred to this very positive hippie atmosphere of working together on the future of the open web. And the degree to which Google's more direct business interest in owning and controlling the user's journey on the web was starting to create tension.

• 10:55Ryan Peterman?

  You mentioned that at some point, you had Google badges and you had free food through Google and you had a good working relationship with them. How did it transition from that to a full competition in your personal experience?

• 11:10Speaker 00

  So my sense is that as time went on, things changed slowly. And I really don't think that it's a situation where any individual person at Google was like, "Ha, ha, now we are going to get them and they don't know." But it's just the way that organizations work. And so we continued to have strong collaboration with many engineers over at Google. But there were two things that really happened at the same time. The first is that Google really ramped up its attempt to acquire market share with Chrome. And this was, you know, probably very truthfully aimed at Internet Explorer.

• 11:55Speaker 00

  But Mozilla was very much collateral damage. And they were spending hundreds of millions of dollars a year on advertising and distribution. And if you were to include the effective cost of the advertising that they put on google.com, which was their own homepage, the advertising spend, I think, would reach into the billions. And there was very much a sense that whenever we talked to them, that it wasn't supposed to be directed at us. Jonathan Nightingale, who was a VP of Firefox at the time, you know, a couple of years ago, he did a tweet thread explaining his point of view on this. And I think he referred to it as the year of 100 oopses, where every time Google would do something

• 12:39Speaker 00

  that would hurt Firefox, it would always be framed as unintentional, right? And the obvious way this would be is that like Firefox was sending source traffic over to Google. And so when users were in Firefox, they typed something in the address bar, they would end up on Google, and then Google would show these Firefox users an ad that would say, or a big banner directly on Google.com, like not even on the search engine result page, saying, "Download Chrome, it's better." And they always claimed that it was unintentional, but these things just kept happening over and over. And I do think that on a sort of organizational strategic level, it was unintentional. But the individual incentives of the people involved, and what they got by doing this, and then, you know, maybe fixing it in a release a couple of weeks later, you know, they still won as far as their metrics were

• 13:21Speaker 00

  concerned. So that was one piece. But there was definitely another piece where Firefox had legitimately fallen behind. And part of this was that when Chrome came out, it had a lot of really interesting and new architectural innovations that created gaps with Firefox. Some of these included having a multi-process architecture, and they had a plugin architecture like for things like Adobe Flash that was much more robust against stability issues. And they had a very fast JavaScript engine. Firefox also had a fast JavaScript engine,

• 14:02Speaker 00

  and we, you know, traded back and forth, you know, taking the performance crown. But they had some real structural advantages. And at the same time, Mozilla made a big and ambitious bet in the early 2010s to try to build a mobile phone operating system. Because there was this sense that mobile is coming, and both Android and iOS are potentially hostile platforms that are not going to allow Firefox to do what it did on desktop on their platforms. And the belief was that the only possibility was to create a new mobile operating system. That was based on web technology. And, you know, there was something to that idea. And there was quite a lot of interest from the industry, a very surprising amount of interest. And we had a lot of partners, but ultimately it didn't work.

• 14:46Speaker 00

  And the big bet that Mozilla made on Firefox OS created a lot of resource competition with the work that needed to happen to keep Firefox competitive exactly at the time when it was needed the most. And so Firefox really had a lot of gaps with respect to Chrome. I think the multi-process architecture was a big one. The stability issues, particularly coming from Adobe Flash, which was widely used particularly for streaming video, which was becoming a thing. That was another big gap. And Firefox also had some legacy baggage around its browser extension API, where the way that you would make a browser extension, I mean, the whole concept of browser extension really arose from the Firefox architecture, where you could just inject anything, anywhere.

• 15:32Speaker 00

  And the Firefox application was written in JavaScript and markup. And you could just modify it with an extension by just adding more JavaScript and more markup and more CSS. And this was really cool. And that's where people first got the notion that you could configure and change your browser and extend it with new functionality. But the problem was that there were no well-defined interfaces. And so all of these extensions were depending on all of these internal details of how the browser worked. And anytime we changed anything, we would break an extension. And so that was a very real and significant drag with respect to trying to architecturally modernize the browser.

• 16:11Ryan Peterman?

  You mentioned prior to Google's funding, this was a group of a bunch of people that were working for free in many cases. I'm kind of curious, because I haven't done a lot of open source stuff. What is the incentive structure? Why would people spend so much time working for free on a project like this?

• 16:32Speaker 00

  So I think it probably varies. But the world was quite different back then. And open source, there were not, first of all, a lot of opportunities for people to contribute to something major and meaningful, right? Like, these days, much of the software stack that everybody uses, many projects are open source. But there were not so many of those back at the time. And so I think that it was an interesting opportunity to have impact. And I think for a lot of people, like I mentioned Boris, he literally got involved in Mozilla because he had no way to browse the internet in his dorm room. One option was, he was like at MIT at the time, I think, and, you know, he could either VNC into some like Solaris thing with Internet Explorer 5, or he could use this build of Netscape that he downloaded that was just constantly crashing.

• 17:17Speaker 00

  And so he got involved just because he wanted to fix something for himself, and he had this vision of how he wanted it to be better for him. And then he realized that there were so many other people out there that were also looking for the same things and trying to fix this stuff. And once you dive into a codebase like this, it is just an amazing intellectual challenge because the codebase was just so vast and so complicated. And so he was like a math major, right? But this was, in some ways, like even harder. And I think that that challenge was another major aspect. But I think the biggest aspect was the motivation of the Mozilla mission and what we were trying to achieve. Because the, I think, a lot of principles that people take for granted about, you know,

• 18:00Speaker 00

  how the web should work, that it should be open and it should be interoperable, that it shouldn't be controlled by one company, and even that that about the internet itself, right? You know, the Mozilla mission is about making sure that the internet is a global public resource that is open and accessible to all. And I think if you grew up in the 90s, when Microsoft was eating the world, that was not A given. That was not the dominant way of thinking about how technology platforms should work. And Mozilla had a vision of how it could be different, and it had a means of executing that vision, right? It wasn't just an advocacy organization arguing that tech companies should do something different. It was actually going out and doing something

• 18:39Speaker 00

  and succeeding, and creating a browser that gave users choice to block pop-ups and control over their experience, and the ability to block ads, and the ability to have tab browsing, and all these things that they weren't getting before. And creating that reality and making the world better in that way, I think, was very inspiring to a lot of people.

• 19:01Ryan Peterman?

  Yeah, I'm kind of curious, like, you know, once you got there, I want to hear a little bit about the projects that you're working on and the various stories of things that happened in the early days. Are there any projects that you felt were particularly interesting or contributed to your career?

• 19:16Speaker 00

  Yeah, so I got started working on all sorts of things. Like, I think as an intern, the first thing that I worked on was the front end for the new mobile browser, but this was before there was Android and iOS, so it was actually for, like, Nokia Memo. And that's what I started on. Then I started working on the graphics engine. I worked on the image rendering library. And there was very much a sense back then, just because there was so much code that, you know, was largely unowned, that as soon as you show up and you start working on something and you, like, fix one thing, you start looking around, and you're like, hey, who owns this? And they're like, you do now. And so I really, I ended my first internship owning, like, quite a lot of code in the Firefox browser. And that felt to me like a really big, important responsibility to make sure that, you know,

• 20:01Speaker 00

  this stuff went well. I think I was particularly motivated to work on the, really, like, the hardest and gnarliest parts of the code, the parts that felt most critical. When, you know, after Chrome came out and then Firefox was coming back with its first counterattack of trying to build a browser that was more performance competitive, particularly around JavaScript. So Chrome launched with this JavaScript engine called V8 that was, you know, had a just-in-time compiler and it was very fast. And we were intending to ship our own just-in-time compiler for our JavaScript engine. But there was a lot of challenge at the end of actually getting this thing shipped,

• 20:44Speaker 00

  particularly around this very complicated area of how the JavaScript engine talks to the rest of the system, you know, how the bindings between the JavaScript and the DOM, which were referred to as XP Connect. And so, literally, there were months of delay of getting Firefox 4 out the door because of all of the tangled web of that architecture. And I, you know, also with the advice of Damon at the time, decided to dive in and try to help make that stuff better. And so, I ended up working on that code. And the thing about that code was that it was very tied up in a lot of security considerations in the browser. When the browsers and the web were first designed, you know, JavaScript was famously designed in 10 days.

• 21:29Speaker 00

  People did not fully think through all of the security implications of this architecture, both in terms of the web platform and the browser itself. So Firefox as a browser, one of the cool things about it was that all of the application on the front end was also written in markup and JavaScript. But all of that relied, from a security perspective, on making sure that a website would be properly isolated from touching other websites and from touching the browser replication itself. And this was all a very complicated mess of security enforcement in this DOM-binding code. And it was not perfect. And there was this cat and mouse game between people working on Firefox and this

• 22:12Speaker 00

  external community of security researchers who were constantly finding new ways to poke holes in this, right? And you do some very, very clever thing with function dot bind and you end up slithering up the prototype chain and suddenly you're like in the address bar and tab code. And once you do that, you know, the browser and perhaps the computer are compromised. And so there was, there were a number of researchers. There was one, I think their name was Mozbug RA4. We never learned what their actual name was. Rumor had it that they worked in, they lived in a country where security research was illegal. But there was very much like a weekly back and forth between people at Mozilla working on Firefox and Mozbug RA4 of new potential paths of vulnerability.

• 22:56Speaker 00

  So Firefox 4 definitely, I think, introduced a much more robust architecture. And that was when I got involved with some of the stuff. But there started to be other and additional security researchers who showed up who were finding new ways to poke holes in it. And what really started to worry me was that there were certain categories of things that we didn't really have a way to fix in a robust manner. You know, we could put a Band-Aid over that particular part, we could wallpaper it over, but there were usually ways that you could apply the same tack in another part of the codebase. And I started to get pretty worried about this.

• 23:39Speaker 00

  And I launched a very large cross-functional effort that I spent, you know, at least over a year on, which I referred to as Slaughterhouse. And that was because, you know, it was a joke because there were these things called Chrome object wrappers that we needed to get rid of. But there was generally a sense that this could be a really big problem if this, these types of exploits were to be weaponized in the wild before we did something about it. And there was one particular researcher who I worked with quite a bit, who was really good at identifying and finding these issues. And so what would happen was be these external researchers, they would send us something, they would file it in a confidential bug in Bugzilla in our bug tracker. And then we would look at it and be like, yeah, this is a real issue. And then we had a bounty program, right?

• 24:24Speaker 00

  Where we would pay them a small amount of money as a thank you and an incentive to bring it to us and, you know, not sell it on the black market. But the black market certainly paid more. And sometimes there would be zero day exploits where you'd have to sort of jump on some new thing and fix it and ship a fix within, you know, 24 hours to make sure that users were protected. And sometimes these attacks got ahead of us. And so there was one situation where somebody flagged me that there was a zero day exploit in the wild that had been discovered. I think it was on like moscowtimes.com and I was disassembling it.

• 25:08Speaker 00

  And then I had this realization that like I knew who had written this exploit because it was one of the security researchers that we worked with. And without thinking too hard, I fired off email to this guy being like, hey, like you're double crossing us, right? Like, you're not supposed to sell these things if you report them to us. And he got back to me, and the thing about security researchers is that they're extremely paranoid. He's like, no, no, no, I didn't. This must mean that my devices must be compromised at a very deep level. And the only thing I can think to do here is to go off the grid. And we then discovered a, basically like an hour later, that actually it was Mozilla that had been compromised, that Bugzilla, our bug tracker, had been compromised by a threat actor.

• 25:51Speaker 00

  And I tried to write back to the researcher, but at that point, he'd already gone offline and we couldn't reach him. And so it was several weeks later that we got a phone call from a bowling alley in upstate New York where this researcher had gone into hiding and was trying to, you know, throw off the people that were after him. And we managed to explain that actually, sorry, this was our bad and had to help him get back on his feet. But it was a really big problem that Bugzilla had been compromised because that is where you have the records of all of the interesting security attacks that people have discovered against the browser. But the amazing thing was that Slaughterhouse had just wrapped up and that

• 26:31Speaker 00

  out of, I think, 45 of the, I don't know, something on the order of 45 of the bugs that the threat actor had exfiltrated, we had fixed all but two of them. And with this sort of new architecture that basically fixed it. And so that really impressed upon me, the importance of getting ahead of things on security and making sure that you have the right architecture, because sometimes you find yourself in a situation where you don't have the luxury of the time to fix it.

• 27:00Ryan Peterman?

  How'd you know that that security researcher wrote that vulnerability?

• 27:05Speaker 00

  People have very particular styles. And in this particular area of security or of exploits, which is different than other kinds, there was just a lot of people's fingerprints on it, right? You could tell. Because these were things that were handwritten logic attacks, where you were using very particular parts of like, I'm going to take this API and then I'm going to bind this function to it and then I'm going to send this call like around async through the event loop twice and then I'm going to do this, right? Like, there was a very particular way and these were all handwritten. There is a different type of security attack that was also a major issue at the time, which were memory safety vulnerabilities.

• 27:47Speaker 00

  And these were much less of a artisanal thing and more things that you would produce by tools, particularly by fuzzing tools. And so this was also a major issue that Mozilla was grappling with, because you would have people who would run these fuzzers against the codebase and the codebase was written in C++. And there are lots of ways to have memory hazards in C++. And really, any one of them is the sort of like webpage takes over your computer situation. And so we were really struggling to find ways to deal with that in a robust manner. And that was really one of the things that motivated Mozilla's investment in Rust.

• 28:28Ryan Peterman?

  So I could see, and I'm just trying to understand here, the security issue here is there's some front-end code that abuses or does some special path in Firefox and then can take over Firefox in that case in the past. But how does it take over the computer? Wouldn't the operating system kind of shield ideally from Firefox doing crazy stuff?

• 28:52Speaker 00

  So certainly not at the time. Originally, particularly on Windows, applications largely ran with system privileges. And if you took over the, you know, the classic thing that security researchers would do to prove that they'd pwned your machine was to launch the calculator, like the Windows calculator, right? And so you'd run this exploit, you'd really have to trust the person who sent it to you that it wasn't going to do anything bad, but you know, calc.exe, that was the proof. As time went on, and Chrome had a major hand in improving the table stakes of how browsers worked, there were additional layers of protection that were put on, right? And so some of these include in-process sandboxing as well as operating system-level sandboxing.

• 29:35Speaker 00

  And so the architecture that browsers like Firefox and Chrome have today involves a very, you know, defense-in-depth, multiple levels of protection, including operating system-level sandboxing that is applied to things like, you know, the rendering and content processes. That said, this was all bolted on over time on Windows and Windows, unlike a platform like iOS. iOS is designed from the beginning with an application sandboxing model where applications are just applications and they have their own little world. That was not how Windows worked. And so evolving Windows and evolving the security-critical applications that sit on

• 30:19Speaker 00

  top of Windows, like browsers, to deal with these sorts of attacks is honestly an ongoing challenge.

• 30:24Ryan Peterman?

  Did you ever figure out how Bugzilla was compromised?

• 30:27Speaker 00

  I think it was a credential leak from an employee. Like, I'm 85% sure that that's what it was. And that was, I think, before we had things like two-factor authentication for logging into Bugzilla. You know, Bugzilla was a very old tool that dated back into the 90s. And so similarly, it was something that had to evolve with the threats over time.

• 30:45Ryan Peterman?

  One thing that I was curious about is, as an organization, how do you prioritize security? Because, for instance, at that time, there was also these big investments in performance, and I'm sure there was a bunch of other things.

• 31:00Speaker 00

  For sure, yeah. Security is one of those long-term things that is very easy to take a short-term perspective on and ignore. And that is perhaps a decision that some products will make, even where they are, but it's not something that you can do as a long-term browser. And so, honestly, from my perspective, it's a cultural thing at Mozilla that, from the very beginning, back before there was any sort of management structure and performance measurement and, you know, company objectives, it was just understood that we had an obligation to protect our users and to keep them safe. And I think that persists. That means that there's a lot of, I would say, ongoing

• 31:43Speaker 00

  bottom-up energy towards making sure that we improve security. And there's a lot of pride that people at Mozilla take in the security of Firefox. Because Firefox does not have, you know, I think that the Chrome Security Org, you know, on its own, is like on the order of, you know, 70-plus people. And we don't have anything close to that. But it is considered to be a shared responsibility, and we have a lot of people who are very dedicated to that, and trying to think ahead, right? Like, Slaughterhouse, the project that I led, that was not any kind of top-down edict of, we need to go fix this. It was that I was responsible for the code, and I felt that this was important, that we needed to get it right. And one of the things that I'm very proud of at Mozilla today

• 32:25Speaker 00

  is that there are, there's an annual security competition called Pwn2Own, where researchers show up and test their exploits and receive prizes for compromising the browsers. And generally speaking, they always find a way to compromise every browser, right? Like, no browser is perfectly secure. And Firefox for the last two years has won the award of being fastest to patch within, you know, on the order of 24 hours. And this is only possible because we have people literally in time zones around the world who are just working nonstop to get it out. And it's not something that we ask people to do. It's there's no management edict that somebody needs to stay up late or work on a weekend to do it.

• 33:09Speaker 00

  People are just motivated to do it because they care.

• 33:11Ryan Peterman?

  You mentioned no performance reviews and this scrappy bottoms up culture. How does that kind of culture work?

• 33:18Speaker 00

  I think in the world of prior to being much management, it was just uneven, I think, as you would, as you would guess, right? And I think that it was generally, things worked well because the caliber and dedication was very high, but it was very much a bottoms up culture, right? And if there was somebody who was owning that particular part of the code that wasn't doing a good job, everybody just kind of lived with that. And there also, I would say, there was very much a notion of people having ownership of areas and that being somewhat final, right? Mozilla had a governance structure that was very much insulated against any sort of executive power.

• 34:01Speaker 00

  It was this module system and people were owners of the modules and even the initial access control system, I think, was called DESPIT. And it was very much a notion of distributed and decentralized decision making. I think that that culture was very powerful and there are still elements of that culture that I think are important to preserve today, but there are also ways in which it fell short and needed to change. And one of the things that I have spent a lot of thought on over, you know, recent years is finding myself in a leadership position is how do we preserve those aspects of the technical culture that make it great while also addressing some of the shortcomings?

• 34:42Ryan Peterman?

  You mentioned a little bit about C++ and memory safety issues. That's a very classic thing. And I know that Rust was developed at Mozilla to tackle those problems in large part. So I'm curious, can you tell me a little bit about the adoption of Rust and, you know, how it solves the problem?

• 35:00Speaker 00

  Right. So in the early 2010s, it was pretty clear that Mozilla and Firefox, the code, had some real structural issues. And a lot of these were issues that we could address through sheer hard work. Right? This included building a multi-process architecture and evolving our extension architecture such that stuff, you know, didn't have its tendrils into all the undocumented interfaces within the browser. And building native video playback so that we didn't have to rely on Adobe Flash anymore. Like, these were all large but incremental projects to address specific architectural issues.

• 35:45Speaker 00

  But there were two architectural issues that were very difficult to address in that way. And the first one I already mentioned is memory safety. Right? Because when you have a codebase that's written in C++, a large portion of the defects that you can make are the sort of defects that compromise the browser and without suck-- in the absence of sandboxing, compromise the computer. And that was just a very difficult thing to deal with in a non-whack-a-mole-y fashion. And there's another piece, which was that particularly at the time, there was a increase in the number of cores that were shipping on devices. But browsers were still largely single-threaded.

• 36:28Speaker 00

  And this meant that you were leaving a lot of the available performance on the table if you didn't have multi-threaded code. But if you did have multi-threaded code, especially in C++, you just had all sorts of very difficult, non-deterministic bugs that were very difficult to track down. And some of these also became memory safety issues that became exploitable defects. And there was this notion that both of these were rooted in deficiencies of the C++ programming language. And all browsers were written in C++. But, you know, in 2010, Firefox had just taken on the world and opened up the web. And there was a sense that we could do the impossible. And so we began an effort, you know, sponsoring some research that some people internally

• 37:11Speaker 00

  had around building a new programming language. And this programming language was Rust. And the idea of Rust eventually became that we would have a programming language that had static guarantees at compile time, that number one, you didn't have memory safety issues, and number two, that you didn't have race conditions. And-- This idea was that first, we would build a new programming language with these properties, which was difficult to do. And then, we would use this programming language to build a new browser engine for Firefox. And so, the Rust project co-evolved with this project called Servo, which was a ongoing, evolving testbed for the Rust programming language with the intention of

• 37:55Speaker 00

  building a browser that was memory safe, but more importantly, was parallel. Because there was a sense that we could solve these architectural issues and close the gap with Chrome, but that also, we needed something to leapfrog Chrome. And there was a belief that concurrency and safe concurrency powered by a new programming language was the way to do that.

• 38:14Ryan Peterman?

  Yeah, I want to talk about the competition with Chrome at the time, because you said they launched with a more performant browser, and also, they were hammering you guys with distribution advantages. I'm curious, what was the strategy at that time from the Mozilla team to compete with Chrome?

• 38:34Speaker 00

  So, the strategy to compete with Chrome was multifaceted. One of the parts was a recognition. I think there was a little bit of denial in the very early days about the deep value that some of Chrome's architectural advantages would provide. And this was, you know, they came out with a JavaScript engine with what's referred to as a Method Git. We had a Tracing Git. We thought that the Tracing Git was good and was going to be better. But in the end, it wasn't. Chrome came out with multi-process architecture. I remember the first thing that I heard was, "Yeah, good luck doing that on Mac." And then, they figured out how to do it on Mac.

• 39:15Speaker 00

  And there was a belief that some of these things were not possible, until they were. And then, once they were, that they weren't the most important thing. But eventually, it became clear that things like a multiprocess architecture and things like native video playback were really essential. And we had this issue particularly on video playback where, because Chrome first had a better plugin architecture for Adobe Flash and then was also early to introduce native video playback, they were able to have a much more stable experience on YouTube, which was blowing up at the time. Whereas Firefox on YouTube, which was what everybody wanted to do on the Internet, was just crashing left and right. And it wasn't stuff that we could fix.

• 39:56Speaker 00

  It was Adobe's fault, and we really couldn't do anything about it. And so, we had to work on a lot of these really hard but discreet challenges of closing the gap with Chrome. But then, at the same time, there was a desire of you can't just close the gap. You have to do something new, and you have to do something that is better. And Rust and Servo were our strategy for doing better.

• 40:24Ryan Peterman?

  So, you mentioned the strategy with Servo and the rewrite in Rust. How did it go? Was it successful? Well, we know that Google's distribution eventually dominated, but how did the efforts with Servo go?

• 40:39Speaker 00

  So, on the one hand, Servo and Rust had, against all odds, captured something really real. And Servo, as a browser engine prototype, had a lot of really amazing stuff in it, right? It was not complete, but it did have the skeleton of certain features that had never been done before, including a multi-threaded CSS engine. But the problem was that we had a team of, you know, a dozen people working on Rust and Servo. Google had hundreds and hundreds and hundreds of engineers who were working on Chromium. We had less than that, but still, most of Mozilla's efforts were directed towards Gecko

• 41:22Speaker 00

  and Firefox, because that was the product that we had in market. So, we had this situation where, in order to build a competitive browser engine... It's not like you need to just do this one time. It's the accumulation of all of the work over all of these years of hundreds of engineers working on improving it and improving the architecture, improving the performance, improving the features. And so every year, there was more and more stuff that servo would have to catch up with. And I think if you just look at it, there's, it was very difficult to believe that we would ever end up in a situation where servo would be competitive with a regular production browser engine.

• 42:02Speaker 00

  And so circa like 2014, 2015, that was a situation where there was really a sense that we needed to deliver a fully refreshed version of Firefox that really got us back in the game. It included this parallel CSS engine that we uplifted from servo. And this was a project that I personally led and was really the highlight of my career working on it because it was so exciting to be doing something really new in a new programming language that, and every step of the way, there was this notion that like, that's definitely never going to work. Right? Like, how are you even going to like hook this thing up? And then we hooked it up and we had it traversing the DOM over C++.

• 42:45Speaker 00

  You know, we have a C++ DOM and we're having this Rust CSS engine and how you get these two things to work together. There was no tooling for it. We had to build it all ourselves. At the end of the day, it was really a dramatic performance improvement. I think it improved like rendering time on amazon.com by on the order of 25%, which is just enormous in terms of web performance. And because it was parallel and it can run on all of your cores, it was and still is the fastest CSS engine in the market. Firefox really got back in the game and was really performance competitive with everything else out there. And that was a really exciting moment for everybody, realizing that if we just coordinate

• 43:28Speaker 00

  and we do all the technical effort and we make the right bets, we can win.

• 43:33Ryan Peterman?

  Okay, so you mentioned that this was one of the most proud moments of your career and like a major launch, very exciting project. I'm kind of curious to talk about the career parts of it. So where were you in your career at this point? And maybe you can talk about how it grew as you worked on these projects.

• 43:49Speaker 00

  Sure. So I think when I started, I don't know if Mozilla had levels. I certainly wasn't aware of them, that they existed. It was a culture of choose your own title. The title that I chose for myself was Negative Entropy. That was sort of what I wanted to do on this incredibly complex and gnarly codebase. As a fun fact, my favorite intern made his title Entropy, but we were really on the same side.

• 44:17Speaker 00

  I think by the time we were shipping Firefox Quantum, Mozilla had developed, I would say, a more industry-standard career progression. And so you had, you come in as just title of engineer, and then you've got senior staff, senior staff. At the time when I was working on Quantum CSS, I believe I was my, I had reached the level of senior staff. And upon the successful launch of Firefox Quantum, I and the engineer who led the other half of Quantum, Quantum Flow, were promoted to the level of principal engineer. And at the time, that was, there were not very many principal engineers, you know,

• 45:02Speaker 00

  like probably fewer than five. And so that was a pretty significant jump. But my long-term dream was to be like Boris Sabarsky, and Boris Sabarsky was one of Mozilla's three distinguished engineers. And so that was really what I had my sights on in terms of the level of impact that I wanted to have. And the way that I really aimed at having this impact was by trying to contribute to as many parts of the code as possible and just be prolific. And so after working on Quantum CSS, I went and worked on web render, and I worked on

• 45:44Speaker 00

  mobile, and multi-process, and layout, and all sorts of different parts of the

• 45:51Speaker 00

  Of the code. And I remember my manager at the time, Joe Hildebrand, who ran engineering, told me that he wanted me to spend one day a week when I didn't open my terminal. And I was just shocked. I was like, what? What are you asking me to do here? Right? You're asking me to not work for an entire day. And he was encouraging me to start thinking about it in the impact that I could have in a much broader way, as opposed to direct contributions to the code. And so this was something that I started to think about, but it was really not the natural way that I thought to work.

• 46:27Ryan Peterman?

  When you talk about the individual contributions, because you mentioned that was your way of having a lot of impact, do you have a way of thinking about which contributions are more impactful than others?

• 46:39Speaker 00

  I was very much motivated to work on whatever felt like the biggest problem. And so that was early on. That felt like the DOM bindings and XP connect and all the security stuff. And then at some point, it was the media playback to solve the YouTube problem. And then it was multi-process. And then it was this REST and Servo CSS stuff. And after that, I thought that Web Render, which was the new graphics backend that we were uplifting from Servo, so I went and worked on that, then Mobile. So I was always drawn, I think, to what felt like the most pressing problem. And my model of how to have impact on that was to jump in and learn all of the code

• 47:22Speaker 00

  in the most pressing problem and help work on it. And not just necessarily typing all of the code myself, but working with the other leaders in those areas on figuring out what the right things to do would be and reviewing code and setting direction and setting priorities. So there were aspects of leadership, but it was all very much oriented around the code that needed the most help.

• 47:44Ryan Peterman?

  So you were prolific, but it was a guided search specifically on the biggest problems for the organization?

• 47:52Speaker 00

  That was how I thought about it, yes.

• 47:54Ryan Peterman?

  One thing that I'm curious about, because in the industry, when it comes to career growth, I often hear two methodologies. One is where people, they focus on the promo first and foremost. And as a byproduct, you get the promo. They, you know, grow and level our skills and various other things, behaviors. And in the other school of thought, it's ignore the promos, but focus on the behaviors, focus on the skills, and then the promos will come. I'm curious your thought between the two methodologies and which one you think is better for structuring an engineering career?

• 48:34Speaker 00

  I have pretty strong opinions about this. I am very much of the mind that is important to focus on impact and not the title. And I think there are a number of reasons for this. But that is certainly what I look for now that I'm in a situation when I am helping people with their careers. And I am often in the decision of, you know, figuring out, you know, who's the right person to advance. And what I am looking for is I am looking for the impact. And my general philosophy is that if you are having that level of impact, you don't need to sell yourself in that way.

• 49:15Speaker 00

  Or sell yourself is not quite the right word. But there is a particular philosophy where you want to promo hack, right? You want to pin your management chain down on what are the things that I need to do to receive this promotion? And that, just speaking frankly, is very annoying, right? There are aspects to this that I think are just intuitive to how I, at least as a person, perceive things. You know, like I'm a father of kids. And when my daughter asks me, like, when, you know, I ask her to do something and she's like, uh, you know, will you give me this if I do that, right? It's, it just does not land well with me. But I think that there's also a substantive aspect to it,

• 49:57Speaker 00

  which is that if you're really focused on the impact, that is how you're really going to achieve things, right? Whereas if you're focused on checking some boxes that somebody has presented to you, then you're in this awkward situation where it's like, well, I did technically say that if you did this, you know, that would maybe make a case for this next level. But the way in which you did it and was very sort of focused on checking the box as opposed to having the impact. And at the end of the day, impact is what matters and impact is what needs to be recognized and rewarded.

• 50:27Ryan Peterman?

  One thing that I hear often, because I think this is a failure mode for engineers that are a little more soft spoken, is they're great at writing code and landing it and having impact, but they're a little weaker on the, you know, selling yourself or kind of marketing your work. And so that kind of bites people. And so sometimes I hear career advice that is focused on, you know, hey, make sure you create a BRAG document and you share the wins when you land the thing. And, and I think a lot of those people don't have an ulterior motive of, let me, you know, kind of connive my way into the promo.

• 51:11Ryan Peterman?

  But I think there is this, this real thing of promos are decided by people. And there's some value in, I guess, a balance. Because obviously, if you were completely conniving, you say, hey, I'm gonna talk to my manager and sell my work to him. That's kind of not ideal. But, you know, making posts or sharing it and thinking about the levels and stuff seems to be a good way for those little quieter or less proactive engineers and structuring their careers. So, you know, how would you kind of balance those two frames of thinking?

• 51:47Speaker 00

  I think there are often multiple ways to frame similar activities. But that the framing of how you think about it is going to have a big impact on how you carry it out and therefore what it actually achieves. And so I think a good example of this is this notion of, that you brought up, selling yourself or BRAG documents or something like that.

• 52:12Speaker 00

  I think it is important that people have visibility and visibility around the work that they're doing. And this is just a very, like, a basic fact about impact. Because if you want to have impact, you need to be connected to people who understand what are the biggest challenges, and you need to be coordinated. And you also need to make sure that people understand what you are capable of, such that they can put you in a position of having more impact. But if you think of it as I am trying to brag more about me versus I am trying to

• 52:55Speaker 00

  communicate and coordinate and create visibility for work that I think that is important, you will have very different outcomes.

• 53:02Ryan Peterman?

  Yeah, it's interesting because it's subtle. I feel like you could have two people with different incentives. One goes into it saying, I'm gonna create this update so I get visibility. And the other person comes in to say, I'm gonna create this update so the right people know the right things. And they get to the same path at the end of the day, but it's like this subtle...

• 53:23Speaker 00

  It's worded slightly differently, and people can tell. A lot of it is the degree to which, you know, how much you make it about you versus how much you make it about other people and the work and the team. And everybody knows, you know, that the person sending the update likely had a significant hand in the work. And a very important thing to do as part of those updates is to make sure that you give visibility to other people and to thank the team and not make it so much about you. Because also, that is a way to work well with people, to have people feel like you are a, you know...

• 54:04Speaker 00

  Humble leader and somebody that they want to work with. And that is an important part of leadership is making sure that you create recognition for other people's contributions and don't make it all about you.

• 54:19Ryan Peterman?

  It is an ideal world that people have impact and they get recognized exactly as much as their impact. One thing that I can imagine happening, though, is, you know, there's that idiom that the squeaky wheel gets the grease. Basically, if you had a loud person that is really vying for their promo and, you know, aggressively trying to make the posts and, you know, constantly bothering their manager, "Hey, I want to get promoted," that oftentimes, assuming they have impact, if they don't have impact, then it's not gonna, nothing's gonna happen. But assuming if they're also a high performer and they're doing that, compared to the high

• 55:00Ryan Peterman?

  performer that's, you know, reasonable, quiet and solid, not selfish, that actually that loud person, unfortunately, I think in most organizations will get rewarded first or faster. What do you think about that?

• 55:17Speaker 00

  I think it does depend on the organization and it depends on the technical culture. You know, one of the things that we try really hard to do in our culture is to not reward that. But I think there is a balance to be struck because my message is not, "Don't be ambitious." But rather, think about your ambition in terms of the impact that you want to have as opposed to the status you want to achieve. And if you have the impact and create visibility around the impact, the rest will follow. And yes, there are edge cases and there are probably some situations where people lobbying for themselves will kind of grudgingly get themselves a little more consideration just because it's top of mind.

• 56:02Speaker 00

  But the best kinds of conversations that I like to have with people are not conversations where people come and ask, "How do I get promoted?" It is, "How do I grow my career and have more impact?" Right? Like, "Am I working on the most important thing? Like, what could I do to grow? What could I do to have more impact?" Right? And I think that those are important conversations to have with your leadership to... Make sure that you're focused on the right things because the worst and saddest thing is when somebody is just working so hard at something that is just not that important and not going to have the type of impact that they ultimately want to do. And they might do a great job at it, but it's just not the right thing. And so I do think that some coordination of like, this is the type of impact that I want to have, and not everybody wants to have that kind of impact. And making it known that you want to have that impact and you're looking for swings

• 56:47Speaker 00

  at the bat and you're looking for opportunities to do more, I think that that is helpful. But trying to pin people down of like, what are the boxes that I can check? And, you know, I did this, like, are you going to promote me now? Is just honestly extremely annoying from my perspective. And I just think zooming out at a level of life, you don't want the people in leadership to find you to be annoying.

• 57:14Ryan Peterman?

  So in your case, you were focusing on impact. You had these role models of these DEs that you wanted to be like. And I'm kind of curious, what eventually did the growth look like? And how did your impact grow?

• 57:28Speaker 00

  I was really focused on continuing to work on like, what was the hardest and gnarliest challenge in the browser and moving on from team to team. And that was really the model that I had in my head. And I did have in my head this idea that I eventually, you know, wanted to have the level of impact that Boris had. And I assumed that that would eventually mean becoming a distinguished engineer. And then at some point, the CTO at the time, Eric Riscorla, sat me down because at this point, you know, Boris and David and this guy, Robert O'Callaghan, had been distinguished engineers kind of since the beginning, you know, since maybe like 2010 or something. And we hadn't made any new ones since. And the CTO sat me down and he said, look, I want to let you know that for the first

• 58:11Speaker 00

  time in, you know, almost a decade, we're going to make some new distinguished engineers. And I thought you should know that you're not going to be one of them. And this was, I think, a little bit surprising to me because I do think that, you know, I thought and that I think anybody at the time would say that at that time, like probably maybe not quite Boris level, but that in terms of knowledge of all of the Firefox code, that was something that I had more than almost anyone else. But what he explained to me was that that was a more traditional model of distinguished engineer inside of Mozilla, but that he was looking for something different now and that he was looking

• 58:55Speaker 00

  for something that was more oriented around industry impact. And that was very much how he saw the world.

• 59:04Ryan Peterman?

  The definition of impact seems to have shifted while you were thinking about your DE journey. Did you ever debate the definition of impact? Or what was your thinking when you saw that the criteria was updated?

• 59:20Speaker 00

  Honestly, it was just kind of inspiring for me. I think there was, you know, some people were concerned that I would be upset, but I think it was more like, you know, you work out and you think that you're really strong and you gotta go do some Pilates move or something, you're like, oh, my gosh, like this muscle that I have, I didn't even know that I have, but it is extremely weak.

• 59:36Speaker 00

  And the truth is, is that there are lots of different ways to define DE and it's what matters for an organization. Right? The thing that mattered early on in Mozilla and what was the make or break for Firefox's success was just the management of this incredibly complex and sprawling codebase by very few people. And the thing that was needed most was a kind of prolific contributor and enabler of that. And that was the archetype of distinguished engineer at Mozilla. There are other companies where distinguished engineer means something else, right? I know that some companies, generally it means you built a billion dollar business. But that is not what Mozilla is about. And what Mozilla is about is changing the world. And so the criteria for DE are really focused around that.

• 1:00:22Speaker 00

  And this was, I think, an inspiring thing for me. And I still think that I, you know, didn't ultimately, you know, I still probably have not had the level of industry impact of some of our other distinguished engineers. And so probably in the end, when I got to DE, it was maybe a combination or a hybrid of the old and the new definitions. But understanding that model of thinking about the world and that model of impact, I think was very profound for me and really oriented how I directed my efforts.

• 1:00:56Ryan Peterman?

  I'd love to hear some examples of what a DE level of impact looks like. Can you talk about some of the people who have done really impressive engineering work?

• 1:01:06Speaker 00

  Sure, yeah. So at the time, there were, you know, when we had this conversation, there were three new distinguished engineers that were promoted. So the first one was Martin Thompson, and Martin still is at Mozilla today. And Martin is an international standards extraordinaire in terms of the development of the technical standards that power the Internet. And so his name is, you know, in the author line of protocols like HTTP2 and Quick and Web Push. And he was heavily involved in the creation of WebRTC and geolocation.

• 1:01:51Speaker 00

  He authored other protocols like Oblivious HTTP. Like, I don't know, I'm honestly not super familiar, but it wouldn't surprise me if there is really nobody else who has had that level of prolific impact on all of the core technical standards of the Internet. And this isn't just about, you know, doing some editorial work on making sure that all of the drafts are formatted correctly. This is about core design challenges of making these protocols better in a way that advances Mozilla's objectives. And so, similar to the situation around TLS, there were generally problems around the privacy and security

• 1:02:33Speaker 00

  and performance of the legacy HTTP protocols. And by designing new ones that made it faster, while at the same time making it more private and having less metadata leaked and having fewer security issues, meant that there was a business incentive for the entire industry to move on to these protocols because they were faster. And by designing them that way, he managed to really increase the privacy and security of the Internet. So that was one archetype.

• 1:03:03Speaker 00

  The second was Luke Wagner. Luke was many people. Luke was probably, if there was anyone in the world, the most single-handedly responsible for the development of WebAssembly. So historically, people wanted to run code in browsers that was not written in JavaScript. Maybe you wanted to run AutoCAD and they didn't want to rewrite the entire codebase in JS. And the original way of doing that was through plugins, where there was this basically native binary blob that got shunted into the browser window. And it was awful, had lots of terrible security properties and bad user experience. And so we tried to move away from that.

• 1:03:47Speaker 00

  And the first idea was actually from, there was an engineer who worked on it as a side project, Alon Skai, who made this tool at Mozilla called Mscriptin. And the idea of Mscriptin was, we just take the C++ and then we compile it to some very naive JavaScript and then run the JavaScript, which technically worked, but it was extremely slow. And Luke was one of the lead engineers of the SpiderMonkey JavaScript engine in Firefox at the time. And he wanted to make this something that was actually viable. So the first thing they did was, there was some back and forth of, well, how do we optimize the JavaScript engine for what Mscriptin is emitting? And that was obviously not a sustainable approach.

• 1:04:29Speaker 00

  But then, they had this idea of ASMJS. And the idea was that Mscriptin would output a subset of JS, which would be detected by the JavaScript engine. And the JS engine would then verify that it was from this restricted subset, which was sort of a proto bytecode format, and then run it in a more natively unoptimized way. But, and so this was what got the flywheel going, of people actually being able to say, hey, we can deploy these amazing native experiences that are compiled and that they're running in the browser. Because the ultimate goal was that we wanted basically everything on the desktop to be running in the browser, or most things on the desktop to be running in the browser. And there were some stuff that just couldn't.

• 1:05:13Speaker 00

  And that flywheel then created enough industry momentum to say, why don't we actually create a new bytecode format, which was WebAssembly, where you could have code that was native, but that was also sandboxed and portable and running inside the browser?

• 1:05:32Speaker 00

  And this was at a time when Google was pushing a very different technology that was more akin to plugins. They had this thing called Native Client or Nackle, and then they had this thing called Pinnacle. And there were a lot of very complicated industry dynamics of getting all of the browsers to get onboard with this WebAssembly train. And so Luke managed the technical pieces and the political pieces to eventually forge a browser consensus on WebAssembly, which has been a very transformative technology in terms of what browsers can do and even what can be done outside of browsers, because WebAssembly was built with such great technical properties, is now being used all over the place on the backend, things like serverless. So this is a bunch of the work that the BIC Code Alliance has involved in. So that is another archetype of impact of the creation

• 1:06:17Speaker 00

  of WebAssembly. The third distinguished engineer at the time was Tim Teraberry, who was our codec lead. And so, one major problem that Mozilla, and honestly, the web had back in the early days, was that all of the video playback happened through proprietary plugins like Adobe Flash. And you would think that the obvious answer, which is the obvious answer, is to build native video playback into the browser. But the problem was that the formats that are used to compress video are heavily patent-encumbered. And there are lots of reasons for this, but it's just the way the world works, where in this particular

• 1:07:01Speaker 00

  space, everybody is fighting over patents. And this meant that an open-source browser like Mozilla Firefox couldn't ship native support for the H264 codec, which was really what was powering most of the video on the internet at the time. It was the most efficient codec, and that was what YouTube was using. And this was a pretty significant problem for us. And so Mozilla worked over a number of years to develop truly open and free and non-patent-encumbered and royalty-free codecs.

• 1:07:41Speaker 00

  The team led by Tim first developed an audio codec called Opus, and then they worked to develop a video codec called DALLA. And DALLA was then incorporated into AV1, which became an industry standard

• 1:07:59Speaker 00

  video format that is now developed by the Alliance for Open Media, which is a consortium across many companies like Mozilla and Google and Apple and Netflix. And so this is basically what powers video today. And a lot of that vision around what would it mean to actually develop truly patent encumbered video codecs came from Tim. So those, I think, are three archetypes of the types of world-changing impacts that you can have as an engineer in ICE at a place like Mozilla.

• 1:08:30Ryan Peterman?

  Yeah, the achievement of these engineers is absurd. I mean, you mentioned Live Assembly. I mean, I think of a business like Figma as possible because of a lot of that work. And that's a tens of billions of dollars industry, or I mean, a company just by its public market cap. And so the value that's being created by these these engineers is absurd. What's interesting to me is, I guess, because it's a more open source company, the value capture is very different.

• 1:09:10Ryan Peterman?

  I'm curious, is that something that, I don't know, people in at Mozilla are ever not feeling great about? Because they've created so much value, but the capture is happening by these capitalist other companies.

• 1:09:25Speaker 00

  Yeah, I think from Mozilla's perspective, we don't have shareholders, right? We're not trying to make gobs and gobs of money. The Mozilla mission and what we are here to do at the end of the day is to create a better internet and one that works for people and it's open accessible for all. Right? And sure, it is very convenient for big tech companies to have a royalty-free codec for video streaming, so they don't have to pay anybody, but they totally could pay, right? What Mozilla's angle here is to make these formats and make these changes in a way that really empowers everybody. And obviously, that empower, that creates economic value for everybody, and some people capture that more than others.

• 1:10:08Speaker 00

  I think from Mozilla's perspective, we want to be able to continue to exist, and that does mean that we have to think about revenue. And at the moment, you know, a lot of Mozilla's revenue comes from one of the dominant advertising players on the internet, which is Google. And so we would like to change that, but it's hard. And it's particularly hard because it's not necessarily where our heart is, right? Our heart is around having impact on the internet and making that better. And so it is a tension that I think does exist between, within Mozilla of, you know, how much to focus on one and how much to focus on the other. But I think that Mozilla at its best, you know, we have some necessary business stuff that we have to manage, but the end goal of what Mozilla is about is having this impact that benefits everyone.

• 1:10:54Ryan Peterman?

  So then, after these examples of distinguished engineers, I'm curious, what advice you would give to someone who wants to become a distinguished engineer?

• 1:11:04Speaker 00

  In terms of one thing that I think that basically everybody can work on at whatever level they are, is writing. This is a particular hobby horse for me because I think that this was an area where Mozilla, and myself included, did not have a lot of developed capability. And I think it is one of the most powerful things. We have this internal memo, which is actually, you know, public on the internet, directed at Mozilla engineers called "Writing Things Down." And one of the things that this memo does is it makes the case that writing is both about the artifact that you produce, which is useful for coordination and wide review and getting feedback, but also...

• 1:11:48Speaker 00

  Useful in the process itself. And that is because the process of writing is the process of thinking and refining your ideas and articulating them. And this is just an infinite game that you can always get better at. And I think it is one of the most powerful ways to create clarity of thinking for yourself, because clarity of thinking is, in my view, one of the most important things. And you only can really get that by forcing yourself to write down your ideas. And so I suppose one pertinent or timely piece of advice is, if you care about this, do not use an LLM to produce text for you.

• 1:12:30Speaker 00

  Because, first of all, if you're putting your name on this thing, at least today, the LLM's like, it's pretty easy to tell when somebody does that. But more importantly, you are losing the opportunity to do the thinking yourself. And that thinking that happens when you write is one of the most important types of thinking for growth.

• 1:12:50Ryan Peterman?

  Do you have tips on how to write well?

• 1:12:53Speaker 00

  Yes. I would say, one of the most important things is that it doesn't need to be any longer than what you have to say. I think I always prefer shorter documents. And I think there's some notion where people run out of things to say. And at that point, you either need to refine your ideas more, or you're just done. And it's never useful to anybody to add fluff. And dealing with documents that are fluffy and overly overwrought is one of my least favorite things.

• 1:13:37Speaker 00

  So I think that's one piece. Related to that, I think that it's important to focus on substance rather than rhetoric, which is sort of related to fluff. But I think that some people say, oh, well, I'm not a native English speaker, and therefore I'm never going to be able to write as beautifully as others. And I think that's the wrong way to think about it. Like, there is some basic baseline that you need to work on in terms of articulation of the ideas, but... It is not about using the most flowery words. It is about clarity of thought. And clarity of thought, once you have it, is relatively easy to express in written form if you practice.

• 1:14:18Ryan Peterman?

  Yeah, I think every single person I've ever asked this question to has always said, be concise. It's shocking to me how unanimous that advice is. Okay, so going back to the browser wars, I think we all know how things have played out. And I see that there's this new era of browser wars that's, things are heating up again. There's a lot of competition for an AI browser or bringing AI into existing browsers. And I'm kind of curious for your take on what the future of the browser wars is going to look like as the CTO of Firefox.

• 1:14:54Speaker 00

  So I think one thing that's really worth clearing up is that you hear that there's, you know, every month you hear about a new browser from this AI company or that. And what I think is really important to understand is that these are not browsers as I think of browsers. These are different front ends, different window dressing and monetization keys on top of Chrome. Because much of what powers a browser is the browser engine. And browser engines are also extremely complicated and difficult to build. And there have been fewer and fewer browser engines as time goes on.

• 1:15:36Speaker 00

  And today Firefox is the only independent browser that has no dependency on browser engine from somebody else. So you've got Safari, which is pretty vertically integrated with WebKit. Nobody else really uses WebKit directly.

• 1:15:52Speaker 00

  Then you've got Firefox, where we build Firefox with Gecko underneath it. And then you've got Chrome on top of Chromium and all these other browsers like Opera and Perplexity and even Edge. They all use Chromium under the hood. Because what they're seeking to do is not necessarily to evolve the web platform and the web experience. They are trying to control the touch point with the consumer in

• 1:16:19Speaker 00

  This world that is coming with AI. Because on desktop, the way consumers experience the internet is through their browser. And everybody is jockeying over what this experience is going to look like. And people want to control that experience. So I am very much of the opinion that AI is coming to consumer technology like water running downhill. And that this is not something that is going to be stopped or reversed, because there is so much disruptive potential of the technology. But where that water goes and how it ends up, it can end up over here in somebody's reservoir, or it can end up in the ocean, or it can end up as a river.

• 1:17:02Speaker 00

  There are lots of opportunities now to steer how it plays out. And there is also a lot of desire for people to capture it for themselves. And Chrome, as an example, is making a very aggressive play to control the AI stack by leveraging its dominance in Chrome. And they're doing this at multiple levels. They're obviously doing it as part of the Chrome browser, where there's 5,000 different integrations with Gemini as part of Chrome now. But they're also doing it as part of the web platform. They are advancing some proprietary APIs that we do not support to allow websites to have local, to access to an LLM inside of the browser.

• 1:17:47Speaker 00

  And this is attractive to web authors, because it allows you to offload your compute, and you don't have to pay for the inference, and you can just ask the browser to do the inference. But the thing that is different about this from other web standards is that there is no way to specify an LLM or to have it have deterministic behavior that is interoperable. And so the API calls that you have into this LLM are fundamentally unspecified. But As the owner of the world's dominant web browser, they have a lot of market power to get websites to tune their usage of the LLM to what Gemini expects.

• 1:18:29Speaker 00

  And there is obviously a co-evolution of AI experiences and the models underneath them. And creating a world where more and more AI experiences are built on top of Gemini is part of Google's strategy to achieve and dominate control of the space. And we at Mozilla are not building our own foundation model. We don't have any particular horse in the race. But what we are trying to do is to make sure that the space remains open in the way that the web has been open and that users have choice of how they integrate with these things and which ones they use. And also that the web remains important and relevant because there is a big risk

• 1:19:10Speaker 00

  that the web becomes a backend detail that's used by RAG from your ChatGPT query or that's used by Operator on the backend, where it is no longer the focal point of the user's experience. But if this were to happen, there would be much that would be lost because the web is a very unique system in the sense that it provides users agency and control that they would not necessarily achieve elsewhere. Right? It is not a direct experience that is produced and delivered in a way that can't be modified. It is reinterpretable content that the browser can modify and render as the user chooses. Right? And there are lots of ways that this matters in practice.

• 1:19:53Speaker 00

  You know, things like ad blockers are a great example of how the reinterpretable nature of the web allows users to control their experiences in important ways. And that the web is a safe environment where it's safe to open a link even from an untrusted source and that all of this is not created and controlled by any one company, but that it is built as an interoperable system where anybody who produces code that implements the right protocols can show up and be part of the system. And this leads to a multi-stakeholder environment where no single entity controls the evolution of the web platform. And that is not how most platforms happen, right? It's not how the platforms happened before the web,

• 1:20:37Speaker 00

  and it's not how the native mobile platforms happened after the web. It is, in some sense, a very historical accident that we got something that is this well-aligned with the vision that Mozilla has for how the internet should work. And it's very important to us to make sure that the changes that happen with AI do not sacrifice these properties.

• 1:20:58Ryan Peterman?

  You mentioned all the components of the browser, and there's the web engine and a lot of those AI browsers and also other ones like Opera, et cetera. They're actually built on Chromium at the end. I'm curious, what do you think would happen if Firefox was built on Chromium? Like, what is the advantage of Firefox building its own web engine?

• 1:21:23Speaker 00

  So it's a lot of work, and we like doing hard things. But the strategic advantage from the perspective of Mozilla's mission is that it gives Mozilla a seat at the table when web standards and the future of the web is decided. And it used to be the case that standards bodies were wide groups of many different vendors. You had Opera back in there with its own engine. You had Internet Explorer. You had Safari. You had Mozilla. And as time has gone on, engines have dropped off the map. Because if you look at it from a bottom-line perspective, there's basically no justification for doing it. It is a giant cost center from a business perspective. But from a mission perspective,

• 1:22:05Speaker 00

  the benefits of being part of the conversations that shape the web are profound. And that is why Mozilla is the last independent player that is doing it, because we are not driven by that bottom line. We're driven by the impact. And so imagine if there was a world where there was no Firefox at the standards table. You would have Apple running WebKit, and you would have Google in charge of everything else. And Apple and Google, very capable and innovative companies, do not get along particularly well and are very entrenched in their own interests. And in a bilateral environment, as you can even see in the mobile space,

• 1:22:49Speaker 00

  there is a sense of Entrenchment that can prevent progress. And one unique thing that Mozilla brings to the table, even though we have relatively smaller market share compared to these other browsers, is that we bring a fresh perspective and the ability to tilt the conversation back towards what is best for the user and what is best for the web. One example that I'm particularly proud of is not necessarily one of web standards discussions per se. These happen all the time in working groups all over. The editor of the HTML specification is Mozilla employee who works with everybody in continuing to refine HTML.

• 1:23:33Speaker 00

  But the example that I would go to is actually one about performance. So it's long been the case that competition is what drives the advancement of browsers getting faster. And it is just this amazing achievement that browsers continually get faster, right? It's like the semiconductor supply chain. It's something that people don't even think about. And just like every year, the stuff underneath that is powering all the experiences is getting faster and faster. And the thing, if you look at the history, the thing that makes browsers get faster is competition from other browsers. And the way this has historically been measured is the different browsers would produce a

• 1:24:15Speaker 00

  benchmark that they would design in a way that was particularly well-suited for their engine, and then they would optimize it. And then other browsers would say, well, yes, that's your private cooked benchmark. And there was an initiative a couple of years ago around this benchmark called Spodometer 3 to try and produce a shared benchmark that we could all agree on that would be representative not of just what worked well in any particular browser, but would be representative of the web and the experiences that we wanted to get faster. And this is a type of engagement where if you just have the bilateral relationship between Google and Apple, would predictably go nowhere because there is just too, there

• 1:24:58Speaker 00

  are too many opportunities for somebody to tilt it in their own interests in a way that the other side says, aha, I see what you're doing there. But I think that Mozilla's involvement was really critical in keeping everybody focused on the stated mission of the operation, which was not to make any particular browser look good, but to provide us a shared incentive to make browser engines faster together in order to make the web faster. And in Firefox, the impact has been pretty incredible of what has been driven by this benchmark.

• 1:25:41Speaker 00

  Over the last couple of years, I think that really across all of the browsers, at least Firefox and Chrome, both Firefox and Chrome have improved their score on the benchmark by a factor of two, so have gotten twice as fast. And we have actually measured in Firefox the impact that this has translated to on real-world performance on the same machines in terms of page load and time to first paint and all these things, and it's on the order of 30%. So I think it is reasonable to say that having this benchmark has made the web 30% faster, and that is something that benefits everyone, and I don't think it's something that would have happened without Mozilla in the room.

• 1:26:27Ryan Peterman?

  Coming to the end of the interview, I just wanted to ask you a few questions, kind of reflecting over your career, because I feel like you have a lot of experience. When you look back on your career, and you made a lot of contributions, you were an IC through and through for almost all of your time at Mozilla. I'm curious, is there any technical decision that in hindsight you wish you had changed?

• 1:26:49Speaker 00

  One thing that comes to mind is a situation where I think the outcome, the technical outcome was good, but I think in some sense, the process wasn't worth it. And this was around WebRender, which was the graphics backend in Servo. When I was working on Quantum CSS to take the Servo CSS engine and put it in Firefox, I was very excited about the momentum of uplifting other parts of the rendering pipeline of Servo into Firefox. And I advocated very heavily that the graphics team should replace its rendering and rasterization pipeline with the one from Servo. And

• 1:27:33Speaker 00

  This was not based on technical expertise on my part around it. It was very much a vibes thing of this is where it's going. And that vibes thing was infectious and I was not necessarily strictly in a decision-making point at this time. But there were, I think that point of view really caught on with management. And I remember there was a, the, the basic issue was that the team, many engineers on the team had very strong doubts about this. They thought that it was not ready, that it was too much, that it was a very risky project. And they tried to express some of these concerns. And there was a engineering leader at the time who heard them and responded with C++

• 1:28:17Speaker 00

  is not the future. And that vibes-based enthusiasm for a risky technical direction over the concerns of the people who knew it the best, ended up being very toxic. And a number of our most senior and most impactful graphics engineers ended up leaving. Now, the rest of the team ended up rallying and WebRender did turn out to be a major and difficult and expensive and many-year project. The actual technical result is awesome. It is a fantastic engine and it is really performant and safe and does all these great things. And I'm really proud of the team for what they built.

• 1:29:00Speaker 00

  But I think that pushing it on the team when the team wasn't ready and wasn't aligned on it, had collateral damage that I regret.

• 1:29:11Ryan Peterman?

  Is there a top book that made an impact on your career?

• 1:29:14Speaker 00

  I would consider myself to be an avid reader, but the truth is, I don't read a lot of books. I do read some, but honestly, you know, the, the books that I can think of that probably have had the most profound impact on my thinking are not books that I've actually read. And that's just a personal style of mine. I tend to be very wide in terms of what I consume and reading a whole book on a topic is a depth-first approach. And I certainly wouldn't want anybody to take this as encouragement not to read books. But, and I think that there's a lot of value that can be gained from that and it's something that many people do and do very successfully. But it's not the only way.

• 1:30:00Speaker 00

  I think, you know, there was this book that I just read a book review of called Good Strategy, Bad Strategy, and I just read three pages, and that has just stuck with me as a description of like what is wrong of many strategy, what is wrong with many strategy documents that I read, just a very clear and crisp articulation of the model that gets created and what the right thing looks like. And that's something, I never actually read the book, I just read a three-page summary. Other, I think, ideas that have had a profound impact on me from books, this book Turn the Ship Around, which I also haven't read, which is about empowering teams on the subject we just discussed, Innovator's Dilemma, Clayton Christensen, I think it's

• 1:30:42Speaker 00

  a very powerful analysis of how the industry works. I definitely do like to consume content from a lot of different areas. I really encourage people to look wide, not just within tech. Within tech, you know, I do really appreciate Ben Thompson's writing, I think is really good for understanding the tech industry, but I like to also spend a lot of time consuming writing from other intellectual disciplines. One example is I've found that the whole geopolitics, national security space is an area where there's actually a lot of thoughtful writing

• 1:31:25Speaker 00

  that I had never been exposed to. And there are people who basically spend their lives really thinking about what happens on a battlefield and what goes well and what doesn't. And a lot of those ideas actually translate very similarly, or, you know, it certainly rhymes with situations, you know, in battles between big tech and how do you manage and motivate teams. And so I would definitely encourage people to take a wide view of the thought that is out there, because there are many things that can be learned from things that are outside of just tech and business.

• 1:32:09Ryan Peterman?

  Last question for you is if you could give yourself some advice when you had just entered the industry, is there anything you'd say, knowing everything you know now?

• 1:32:17Speaker 00

  At the end of the day, to be humble, is the most important. I think that I had a particular attitude, not strictly about myself, but about engineering and the role of engineers and how important engineers were. And there was definitely this attitude of that was the center of the universe. And as I've gone on, I've started to realize that there are many other important things that go and, you know, that you don't necessarily understand when you're first starting out at an organization of why there are certain processes and why there are certain functions and what do we need a legal team for and

• 1:32:57Speaker 00

  all these kinds of things. And as I've progressed and found myself more in a leadership role, I have really managed to develop an appreciation and empathy for all of the different players within the organization and within the industry. And realizing that everybody has a point of view and a place that they're coming from, and that working to understand where people are coming from is the way to make progress.

• 1:33:27Ryan Peterman?

  All right. Well, thank you, Bobby, for your time. I really appreciate it. With that being said, is there any last words you want to say for the audience?

• 1:33:35Speaker 00

  Yeah, I think the most important message I want to send is that the web very much matters. And the web is something that underpins so much of what we do. And I think for a lot of people, it is invisible and something that they take for granted, right? It is like water flowing through the taps or the semiconductor supply chain of this amazing resource that continues to get better over time and continues to deliver us more value that we take for granted. But it doesn't have to be that way, and it easily could not. And we could easily backslide into a situation of unilateral control by big tech over the experiences that people have.

• 1:34:18Speaker 00

  I think that simply losing focus on something because we've had it so good for so long would be a tragedy. And I encourage people to think about what they gain from the existence of an open and free web and to participate in trying, doing your best to keep it open. And, you know, from a selfish perspective and from the perspective of the web, a great thing that you can do if you are building websites and experiences is making sure that they work in all browsers and that you don't do the lazy path and build something that only works in Google Chrome, because that is the first step to ending us back in a situation where we were 25 years ago.

• 1:34:57Ryan Peterman?

  Thanks for listening to the podcast. I don't sell anything or do sponsorships, but if you want to help out with the podcast, you can support by engaging with the content on YouTube or on Spotify. If you want to drop a review, that'll be super helpful. And if there's any guests that you want to bring on to, please let me know. I feel like sourcing very senior ICs. There's no well-studied list out there on Google that I can just search this up. So if there's someone in your org or at your company who you really look up to and you want to hear their career story, let me know and I'll reach out to them.