Pod Insights Software Engineering Daily FreeBSD with John Baldwin About this episode FreeBSD https://freebsdfoundation.org/ is one of the longest-running and most influential open-source operating systems in the world. It was born from the Berkeley Software Distribution in the early 1990s, it has powered everything from high-performance networking infrastructure to game consoles and content delivery networks. Over three decades, it has evolved through major architectural shifts, from symmetric multiprocessing and kernel scalability to modern storage systems and predictable release engineering.
John Baldwin http://linkedin.com/in/john-baldwin-86241a10a has spent more than 25 years working on FreeBSD as a developer, contributor, and consultant. In this episode, John joins Gregor Vand to discuss the origins of FreeBSD, how its governance model differs from other open-source projects, its role inside systems like Netflix’s CDN and the PlayStation 4, the challenges of maintaining a 30-year-old codebase, and much more.
https://i0.wp.com/softwareengineeringdaily.com/wp-content/uploads/2023/11/Gregor-Vand.jpegGregor Vand is a security-focused technologist, having previously been a CTO across cybersecurity, cyber insurance and general software engineering companies. He is based in Singapore and can be found via his profile at vand.hk or on LinkedIn.
Please click here to see the transcript of this episode. http://softwareengineeringdaily.com/wp-content/uploads/2026/03/SED1914-FreeBSD.txt
Sponsorship inquiries: [email protected] https://cms.megaphone.fm/organizations/619b680e-d813-11ea-9750-e7ecac406436/podcasts/e60108fe-e328-11ea-b2b6-1348249f900a/episodes/3913370a-bc44-11ed-ace6-8b1f96d82bba/[email protected]
The post FreeBSD with John Baldwin https://softwareengineeringdaily.com/podcasts/freebsd-with-john-baldwin/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.
Transcript 12,681 words · 3 speakers
0:00 Speaker 00 Free BSD is one of the longest-running and most influential open-source operating systems in the world. It was born from the Berkeley Software Distribution in the early 1990s, and it has powered everything from high-performance networking infrastructure to game consoles and content-delivery networks. Over three decades, it has evolved through major architectural shifts, from symmetric multi-processing and kernel scalability to modern storage systems and predictable-release engineering. John Baldwin has spent more than 25 years working on Free BSD as a developer, contributor, and consultant. In this episode, John joins Gregor Van to discuss the origins of Free BSD, how its
0:43 Speaker 00 governance model differs from other open-source projects, its role inside systems like Netflix's CDN and the PlayStation 4, the challenges of maintaining a 30-year-old codebase, and much more. Gregor Van is a security-focused technologist, having previously been a CTO across cybersecurity, cyber insurance, and general software engineering companies. He is based in Singapore and can be found via his profile at van.hk or on LinkedIn.
1:26 Speaker 01 Hello and welcome to Software Engineering Daily. My guest today is John Baldwin. Thanks so much for being here with us today, John.
1:34 Speaker 02 Thanks for having me.
1:35 Speaker 01 So yeah, today we're going to be talking about all things Free BSD. Some of our listener base will already know exactly what that is, others will perhaps have some inkling or have read that's somewhere to do with something they use in their daily life, which we'll get on to. But I think the first part to go through, which is what we like to do on Software Engineering Daily, is just to kind of understand your background, John. You've had a very interesting technical career. So yeah, I'd love to just step through that before we get into what is Free BSD.
2:07 Speaker 02 Sure. I guess I have enjoyed working with computers and working with software and kind of how you build things from kind of a young age. I first started programming when I was around 12 or so on a Comver64 with BASIC in high school. I started programming in Pascal and learning some assembly and had interest in low-level details like operating systems. A good friend of mine in high school and I, we actually wanted to write our own operating system from scratch, which was a bit overly ambitious on our part. But we were already interested in kind of that level and kind of doing systems-level programming. When I was an undergrad, I was first exposed to FreeBSD. This would be in the mid-1990s when free and open-source Unixes were coming onto the scene. And that's the one that I
2:51 Speaker 02 first started using in college as undergrad. I got really interested in using it both as a user myself. I kind of forced myself to use that as my daily driver during school and for my classes. But I also ended up being a sysadmin at my university and kind of managing our undergraduate lab. And as part of that, I kept working with FreeBSD more and more. And by the end of my undergrad, I actually started contributing enough patches to join the project as a member, even as a senior in university. So then, when I graduated from school, I had the opportunity to actually go work for a FreeBSD company doing the things that I really enjoyed doing. And that was my first job out of college. And so I've been active in FreeBSD and working on various parts of the system since around 2000 or
3:33 Speaker 02 so. And I've gone through various jobs and various employers, but the one constant at all of them has been that in some way, each job I've gone through, or now is a contractor, each client that I have that use FreeBSD in some form. So I've had the wonderful pleasure and opportunity to work on this project that I really enjoy. And it's that many people only get to work on as a hobby, and I get to work on it paid in some fashion and get to spend my time working on this project. And even though my employers have changed over the years, I now have this kind of, it's crazy, a 25-year career of hacking on an operating system kernel and various bits of user land and other things that go along with that. So that's been really neat, a lot of fun. I feel very privileged and honored that I get the chance to do this. And that's the way I approach all of my work, is that
4:18 Speaker 02 it's really enjoyable to do it.
4:19 Speaker 01 That's awesome. And I think, yeah, we obviously have quite a few listeners that are probably at the beginning of their software journeys. And I think that's like a really sort of interesting way to frame it, that actually you can get so deep into a specific technology that actually employer, not that this employer doesn't matter, so to speak, but if you're able to weave that thread of the technology that you just enjoy through it, then that maybe makes life more enjoyable as well.
4:45 Speaker 02 It's fun to work on, I'll say that. One benefit, I guess, of, I guess it's a trade off. In the one case, you would love to have this nice, stable job that lasts forever, that is the same thing all the time in some ways. But by moving between different employers at different times, I got to work on different types of projects and different workloads and find out different areas of stuff. So one of my jobs is at the Weather Channel, working on local TV insertion, where you insert local content that's relative to a geographic region, like the local weather forecast, instead of the national weather forecast. And so that was exposure to TV and video. And I could tell my parents, hey, that thing you see on TV, I would help work with that, versus other jobs that involve much more low level performance and low latency
5:28 Speaker 02 networking and all sorts of different things. And it's, one benefit of that for me is I've kind of, I've done a tour through different parts of the kernel and had to learn different parts of the operating system over time, motivated by the different projects I was working on. So that's also been neat to have that chance to wander around in FreeBSD sort of tree, and kind of in parallel to wandering around both the US as a country and the different jobs I've had.
5:50 Speaker 01 Yeah, that's awesome. So let's get into FreeBSD. What is it? Why did it come about? Let's start there. So for those that just aren't fully in the know, what is FreeBSD?
6:03 Speaker 02 FreeBSD is a general-purpose Unix-like operating system, so very similar to Linux or macOS, for example. FreeBSD started in the mid 90s. So it's actually descended from the kind of variant, you might call it, or flavor of Unix that was developed at UC Berkeley in the 80s, where like the first kind of widely used version of TCP/IP came from. And Berkeley kind of wound down their research stuff in the early 90s, but they released what they had done as open source. And so various communities popped up trying to use those bits. And in particular, there was a gentleman, I think his name is Bill Joelitz, and he wrote a series of articles in Dr. Dobbs talking about porting 4.3 BSD, I believe it was,
6:46 Speaker 02 or maybe it was a version of 4.4, to run on x86 PCs. It's like a 3.6 at the time. And overUSENET, and this is before my time, a community developed around collecting patches that would run on top of 3.6 BSD to fix various bugs and issues that came up. And there were some personality issues. And eventually, people kind of, like, Bill Joelitz would vanish for a while, and then come back and then vanish for a while. People got tired of waiting for the next release of 3.6 BSD to actually come out. So eventually, the people, the community that had collected around 3.6 BSD, they split off into two groups. One of them was NetBSD, they split off first, and the second was FreeBSD.
7:26 Speaker 02 And they said, "Let's take this foundation we have of BSD from UC Berkeley, and the patches that Bill had initially developed as 3.6 BSD, and then a whole bunch of patches from various people all over the world overUSENET, and make a release out of it." And then that started them rolling. And that's kind of how the FreeBSD project got started.
7:45 Speaker 01 Yeah, and I think it's helpful to sort of, I guess, slightly compare and contrast very, very high level with Linux in the sense like, these are two lineages of Unix, is that right?
7:56 Speaker 02 Yeah, so Linux was written from scratch, but influenced by the design of the way Unix works. So it aimed to be POSIX-compatible. So in particular, GNU already existed as a project, backed by the Free Software Foundation, and was trying to make an alternative OS, an alternative to commercial Unixes at the time. And one of the things GNU didn't have, well, they had a kernel called HERD, but I think HERD was perhaps not as well along. And Linus came along and wrote a kernel that would sit on top of the rest of the GNU system. And Linux worked really well, and it became a community that people could go and work on it, get a free Unix-like system, something that was comparable to maybe boxes they had used at work in a Unix environment, but this is one they could install on their home PC and run.
8:40 Speaker 02 And so FreeBSD and Linux were both kind of operated in that same kind of space.
8:45 Speaker 01 Yeah, and to kind of, I guess, those like trying to understand why use one over the other, just at this high level, Linux being this kind of fast to change, community driven, like very, very large community driven, FreeBSD perhaps being sort of more of a workhorse that's more like, quote, "stable" for those that need to use something.
9:08 Speaker 02 Well, I think sometimes things happen by accident in history. And one of the things that happened in the 90s is in addition to there being people who took BSD from UC Berkeley and tried to make an open source distribution, there were also a group of folks, some of the people who had been at UC Berkeley, who formed a company called BSDI that was going to sell a commercial version of BSD US. And they decided that one of their marketing strategies would be to pick the telephone number of 1-800 its Unix, even though AT&T owned the Unix trademark. And that made the lawyers at AT&T very unhappy and resulted in a pretty big lawsuit between AT&T and UC Berkeley. And when that happened, that kind of put a bit of a kibosh around many folks in regards to the BSD community, because there was a lot of uncertainty about, well, what was going to happen to this project?
9:53 Speaker 02 And the long web, NetBSD and FreeBSD, like in this whole universe, what's going to be the outcome of the lawsuit? Is the source code still going to exist? Is this like a safe platform I could build something on top of? Or is it going to get yanked out because the source code will, you know, the judge will decide it's encumbered? And so I think one side effect of that is that people who weren't certain found Linux as an alternative where that uncertainty wasn't happening. And so you had a shift of mindshare of developers because they were kind of weighing both options early on. And the BSD world had this obstruction in the way. And by the time I got there, the obstruction had been lifted. I didn't start working with FreeBSD till a couple of years later in the mid 90s when I was in college. And that time, the lawsuit had been resolved. But in terms of early developer
10:36 Speaker 02 mindshare, like damage had been done, if that makes sense. So in some ways, it wasn't even a technical, it was just things outside of the technical realm that individual developers don't have control over. And it's just kind of the way, it's luck that kind of sometimes decides how things go.
10:50 Speaker 01 Wow. Yeah, I had not picked that up. So that's like really interesting history. And as you call out, it is sometimes something that is completely untechnical that has ended up driving a direction for. Decisions and just sort of, I guess, we'll get into some of the use cases in more detail later. But just to get to set the scene before we dive in, things that FreeBSD might have touched that you're using as a listener. So at a very high level, PS4 OS runs on FreeBSD, for example. Netflix, their CDN servers, for example, also a huge use case. And macOS, so most of the, well, I say most of the listener base, that's incredibly biased of me. I'm on a Mac, but a lot of our listener base are on macOS and so it's in there as well.
11:35 Speaker 01 But we're going to get into those use cases a little bit later. I think looking at how FreeBSD has actually worked as a project, because we've just talked about sort of, well, there was a sort of mindshare split, I guess, as we've just learned between Linux and FreeBSD. As projects, they're set up quite differently. And yeah, I guess it'd be really interesting to understand like, how does the FreeBSD project work? Let's go from there.
12:01 Speaker 02 Sure. So one common model in a lot of open source projects is you have, we even have an acronym for it, a benevolent dictator for life, or BDFL. So you have a person who kind of has a vision for what the project should be and kind of how it should go forward. And that's a model that works quite well in a lot of open source projects. FreeBSD has never really had that model. Earlier, when I said that FreeBSD came out of this community of folks around this 36 BSD patch kit, well, there was already a group and there wasn't like a single person. And if anything, the single person they're waiting on proved to be someone who was absent. So from its initial start, FreeBSD was already a group of several folks together and no one single architect or no one single kind of benevolent dictator.
12:45 Speaker 02 And that group over time started expanding. Initially, they had a kind of a leadership team. We still have the same name for the core team, but the first group of people that were on the core team were basically the people who had root on the box where the source tree lived. And FreeBSD from its inception used source code controls. So we had CVS in the very early days and the box had the CVS repository. If you have root on that, you are part of the core team. And over time, you would add more. Folks who were either committers, so they were people who could had commit access. So you could SSH or you CBS, at least SSH, you can push commits to the repository. As well as folks who were in this core team and they were all self-selected or invited. That's still true for committers today, kind of, or for the most part.
13:29 Speaker 02 But for Core, it was originally some kind of self-selected group. And it kind of slowly grew over time and wasn't as good about maybe removing people who are no longer active. For example, as it could have been. And around 2000 or so, there was a bit of friction between the developer community and Core. And not thinking that Core was quite responsive to some of what the developer wanted to do. So we had a little kind of mini internal revolution by the developer community and crafted a set of bylaws. And instituted elections and started electing our governing board. And we've held regular elections every two years since 2000. And have a different rotating group of folks who are on our core team. And that's, we have different trade-offs, I guess, compared to the benevolent dictator model.
14:14 Speaker 02 One of the advantages that we have had is that we've, many of our senior developers from that kind of first generation. Well, they worked on FreeBSD at that point, but then some of them found other things to do and they left and went off to other projects. And we were able to have younger folks who had come up through the ranks. Grow into leadership positions in the core team and kind of survive having multiple generations of leadership in the project. So that we can live without, if we don't have a dictator that if they leave suddenly or they unfortunately hit by a bus. We can survive that. We've already kind of survived that multiple times in our history in that. So we have the ability to have a structure that will sustain beyond the last of any one individual. On the other hand, when you have committees and groups, things aren't always as efficient.
14:58 Speaker 02 And also, the way that technical direction has historically worked in the project is that developers as individuals. Work on kind of, the phrase we have for it is you scratch the itch that you have. Sometimes it's an itch your employer has, is often the case. Or sometimes it's just something that you have. You went and bought a laptop and it doesn't, some driver on it doesn't work. Or suspended resume doesn't work quite right. And so you want to go get that thing to work on your. Their own time and that's where some of the patches come from. Or if it's employer motivated and employer like Netflix as a use case for something and so they spend their own resources employing somebody to work on fixing bugs or adding features that they need that they then contribute back upstream. And so technical direction arises from these individual developers working on different tasks.
15:43 Speaker 02 And not always, there's not always a single unified vision or direction. And then sometimes that's not the best. Sometimes we could benefit, I think, from having a bit more direction. That's one of the things internally we've talked about in our leadership as we're kind of evolving and changing over the years is do we at some point need to have a bit more active direction and trying to have a roadmap as a project of where we're going. And part of that is talking to our corporate consumers who are active in our community and our individual developers and trying to figure out, well, where are things where you guys are aligned? And there's common things that if somebody wants to volunteer in doing something and they would like to know, they don't have a particular itch to scratch. They just want to know where's the best place to spend my time. That's, I think, the value of a roadmap could be is those folks would know how best to contribute their time.
16:26 Speaker 02 So that's currently one thing we're looking at as a community is maybe trying to play with developing a project roadmap that will update over time. But that's still something to be developed and it's a lot of non-coding work, which is not always the most fun work to do in open source.
16:41 Speaker 01 Yeah. And just roughly like percentage wise, what would you say in terms of those that contribute? Those that are within sort of fairly major companies that rely on FreeBSD versus the kernel hackers and the person who's at some point picked up a laptop and wants to hack away on it.
16:58 Speaker 02 So the FreeBSD project is kind of a couple of different open source projects, at least two big open source projects within a single open source project. At least as compared to, say, the Linux landscape. In FreeBSD, we have a lot of development of source code for our kernel and our base system utilities. And then we have this system called ports that allows us to import third-party code or rather build packages of third-party code. Things like KDE or GNOME or Wayland, X Windows, other, all sorts of stuff like that, like tens of thousands of those packages. And together that allows you to build a distribution. And in the Linux world, these things are kind of split up. So you have the Linux kernel folks work on just the kernel piece. And you have a different group of people who might work on the C runtime library, like glibc or muscle.
17:43 Speaker 02 And then you have folks over in Debian or Ubuntu or other places who then kind of assemble different bits from these different sources and glue them together to make a distribution. And in FreeBSD, we do all of that in one place. But it does mean in particular that the work model and a lot of the workflow that happens inside the like develop the kernel and the base system part is very different from working with third-party packages like KDE. And that shows up in various different ways. So one way it shows up is a lot of our work on the source side tends to be funded. I would say maybe at least 80% or so commits that go in probably have a tag that they're sponsored by somebody. Or if you look at the lines of code, at least 80% probably are paid for in some fashion by some kind of employer
18:27 Speaker 02 or a client for a consultant or something to that effect. Whereas on the port side, where we're dealing with how we manage patches to third-party software and keep it building and working on current versions of FreeBSD, a lot more of that work is volunteer basis, almost inverted. I would say like 90% of that work is probably volunteer instead of funded. So you have this different mix depending on what kind of work is going on inside of our project.
18:50 Speaker 01 Got it. And just before we kind of move on to some of these key use cases, if you if you like, that we've seen FreeBSD end up in, would you say that having this government's model as opposed to call it benevolent dictator model, does that create a degree of trust? And that's why FreeBSD can end up in these really critical, large projects? Like, how is that sort of decision made, do you think, when someone's coming along that is like the size of a Netflix and trying to figure out which path to take?
19:22 Speaker 02 I honestly don't know if our governance model is a factor in anybody's decision. When I'm aware of people who have decided to use FreeBSD in a product, often it comes down to the fact that an engineer who's participated in the design of the architecture of the product is familiar with FreeBSD in some fashion. Either they used it before, or I have a prior job, I won't name them to protect the innocent, but they were trying to pick a platform for a product. And at the time, Linux was having some turmoil and swapping their VM system it felt like to them every three months. They looked at that and said, "We don't want that. We want to use FreeBSD." And that was the basis of their decision. And it was kind of a gut feeling on the part of their system architect.
20:02 Speaker 02 So I think in many cases, many of these things are a lot more serendipity than they are very well thought out in terms of deciding what kind of technologies you want to use or what things do I know that the engineers who are designing or picking the system design, what are they familiar with? That's what they'll choose.
20:20 Speaker 01 Yeah.
20:21 Speaker 02 You can't always just say that it's on a very clear technical basis as much as, "I'm just comfortable with this."
20:25 Speaker 01 That makes a lot of sense. I'm probably speaking purely from a personal perspective where a lot of my job is basically de-risking projects from a technical perspective. So I'm often looking at these maybe slightly more macro areas as well. But the reality is often, as I know, a developer has picked up a technology and that's why it is. And then you have to kind of come back later and address some things based on that choice. But that's the way time goes, or history goes rather. So.
20:51 Speaker 02 It happens.
20:54 Speaker 01 So I mean, like, let's go into some of these examples of where FreeBSD has ended up. I think it'd be interesting just to touch very briefly on PlayStation, but I know that we're going to talk at more length about Netflix. That's the one that you had a lot more of a hand in. So yeah, just for example, why do you think PlayStation ended up with FreeBSD? And then we can jump into Netflix as well.
21:16 Speaker 02 As a project, we haven't talked a lot with Sony, but earlier on, when they were working on the PS4, some of their engineers did talk with us a bit and gave a few talks at some conferences. And from what I recall from when we were talking with them, that before the PS4, when Sony would build a platform for their new PlayStation console, they had to figure out what software they were going to run on it. They would maybe pick bits and pieces of software from different places on the internet, maybe a bit of their C library from one open source project and a TCP IP stack from somebody else, and various shared libraries and things, and kind of assemble what was effectively a homegrown operating system for each kind of version of the PlayStation. And this meant that they were in the kind of operating system support and development business,
22:00 Speaker 02 as well as building a gaming development platform and hardware business. And I think with the PS4, they decided we would rather not be in the operating system business any more than we have to be. And we would rather use something that was off the shelf as our starting point. And when they're evaluating alternatives, one of the things that also happened around the time of the developer of the PS4 was the FFF released version 3 of the GPL license, which included some clauses related to granting of patent rights and so forth, that for various, not all, but some companies, that gives their lawyers a bit more heartburn than previous versions of the GPL did, for example. And I believe that was true for the case for Sony, in that when they were evaluating what platform to use, one of the big reasons they chose 3BSD
22:44 Speaker 02 was the 3BSD was BSD license. And that they did not have to worry about dealing with GPL v3 or what possible implications that might have if they were to use GPL v3 software in the PlayStation OS. That's my understanding of how they ended up landing on 3BSD for the PS4. We have gotten some contributions from Sony on the PS that came out of the PlayStation effort. Some of it is direct to us. For example, some support for things like AVX and our kernel, I think, came from Sony originally. But they've also contributed in other ways that have really benefited the project. Another thing that was happening kind of in the, I guess that's about the right timeframe, mid 2010s or 2000s, was the rise of LVM, that's another alternative open source compiler suite. And LVM was very attractive.
23:26 Speaker 02 To Sony and they spent a lot of work contributing to the linker side of the story, which at the time wasn't nearly as well developed as Clang as a C compiler. So they contributed a lot of effort into the LD linker. And one of the ways FreeBSD has benefited from that is that now we use Clang and LD as our default toolchain for all our platforms. So that I think with FreeBSD 15, for example, I believe we have one binary left in the base system that is GPL, which is like diff 3 or something. But the rest of our system now is fully BSD licensed. So that's been one big benefit that we've gotten from effort that Sony has done.
24:00 Speaker 01 Nice. So then Netflix is also quite a big story for FreeBSD and I think for you personally as well, the work that you've done. So let's get into that. Where does FreeBSD turn up in Netflix's infrastructure? And then what kind of happened there?
24:16 Speaker 02 Sure. So to be clear, I've done some contract work for Netflix, but I'm not an employee of Netflix.
24:21 Speaker 01 Yes, yes, I should have clarified that.
24:22 Speaker 02 Flamer of there may be bits that I don't know. I'll say it that way. But Netflix uses FreeBSD and their CDN. So if you're watching a movie from Netflix and the bits are being streamed to your device, probably it is coming from a FreeBSD box. That's at your ISP. And typically they're trying to avoid sending lots of bits across the actual internet and only sending the bits locally within your kind of local WAN for your ISP or so forth. And they do that with a distributed set of boxes to build a CDN. And the boxes in the CDN are running FreeBSD. And that's where they're playing with pushing a lot of bits out of boxes, hundreds of gigabits of TLS encrypted traffic to all sorts of customers. And so that workload is a very high performance workload in terms of the raw throughput.
25:06 Speaker 02 The individual connections, my understanding is they're not aiming for like 100 gigabits on a single connection. Instead, you need thousands of clients connected to a box, many of them at the end of very slow links with maybe turbo latency and packet loss and drop. And so being efficient about pushing out as many bits as you can while tolerating a very mixed quality of the links that you have, that's kind of what they are focused on and working on. And the things, the changes they make in particular to FreeBSD around a lot of work in the TCP stack and dealing with TLS offload and things like that.
25:39 Speaker 01 Yeah, so there was some collaboration with Chelseao, I believe, and that's who you were contracting for.
25:46 Speaker 02 So one of the things that Netflix was very interested in early on was the ability to encrypt all the traffic that was going by. And they had a couple of different reasons for doing this that I probably need to get into. But this presented a bit of a technical problem. Traditionally, when you were just sending web traffic, because that's effectively what serving movies is. It's just web requests, like fetching a couple of megabytes at a time of these backend movie files. Traditionally, OS's, both Linux and Freestyle, had an optimization for our web server. That if you're sending a chunk of a file over a socket, you can make a single system call called send file. And the kernel would take care of running the state machinery between as interrupts are coming in from the network device. And interrupts are coming in from the storage device. And when a block shows up, you can send it straight out the socket to the neck and never involve user land at all.
26:31 Speaker 02 And the kernel can do this all asynchronously and have driven it's very efficient. Well, the minute that you throw TLS into the equation, all that goes out the window because somebody has to encrypt the traffic. And the old way of doing this was you did all the encryption in user land. So now you have to go get bits from the disk using a blocking system call, wait for the bits to come from the disk into user land. Copy them out to user land, do the encryption, copy that data back into the kernel into a separate buffer to be sent out on the neck, on the socket. And unlike send file, where if you're not using TLS and you're sending the same file to like 20 different clients. You only need one copy of that file's data in memory. Inside the kernel, we can share the pages that hold that data among multiple open network connections. That's really easy to do.
27:15 Speaker 02 You just share the same page and send it down multiple network connections. With TLS, it's not the same data because every different connection has its own session keys. You have to encrypt the data differently for every session. And so no longer can you share the data, you use much more memory. You start using more bandwidth on the PCI bus and so forth, and it kind of all cascades down into horribleness. So one of the solutions that Netflix pursued to help address this was to move the TLS processing out of user spaces into the kernel. It allows you to get back to using send file and have the kernel manage all the workflow. They did this was to you still end up with multiple copies of the data, but you're able to do the bulk encryption like the AES, GCM, what would most commonly be used nowadays to actually encrypt the data and send it out on each connection.
28:00 Speaker 02 And they had an internal version of that. One of my first projects that I worked on with Netflix was helping to clean it up a bit so that it can be upstreamed into free B S D. Because they didn't consider this to be their secret sauce. Their secret sauce is making movies. The secret sauce is not TLS encryption. So they were happy to have that be something that we could put into stock free B S D so that other people could use it and also contribute to it. Then there's a follow on project to that. Another of my clients is a company called Chelseao that makes some smart NICS that have more than just kind of Ethernet inside their NIC. They can do various things like T speed offload and whatnot. And they have the ability to do the actual encryption of individual TLS packets on the NIC. So then I extended the framework that had come initially from Netflix to allow for some network connections.
28:44 Speaker 02 We may not need to do the encryption and software in the kernel, but we can send the raw unencrypted data all the way down to the NIC and it can encrypt it on the wire on the way out. And one of the advantages of that approach is you no longer need separate copies of the data. You're back completely to the original SIM file version where you have one copy of the data for the raw file on disk of what the movie is. And that's the only copy you need inside memory on the host. And only the NIC is actually dealing with encrypted data on the fly. So that was kind of an interesting project because I got to work with, it ended up being effectively a joint project of what does Netflix need as a customer? What can help Chelseao kind of sell their NICS? And what's a design that works for both of those to allow us to glue things together?
29:25 Speaker 01 Yeah, as you call it, it was a sort of collaboration where Chelseao, they had to open up their hardware specs effectively to make this possible. Netflix kind of funded the whole thing. Is that sort of, I mean, just in broad brush terms, how it worked?
29:39 Speaker 02 So I'm a contractor for Chelseao. So I can access their docs under the NDA that I have as a contractor. So the Chelsea specific bits I wrote as on the Chelseao side, but I collaborated with Netflix on the design of how this can plug in and kind of how the framework would work.
29:55 Speaker 01 Ah, okay. Yeah. No, that's very interesting. I think again for our listeners. So understanding how these things come together is often helpful as well. Equally, I work on projects that have these kind of same dynamics where sort of it's three parties and party A doesn't necessarily directly talk to party C, but because you've got B in the middle, the whole thing can work, basically. So yeah, makes a lot of sense. And then maybe just touching briefly on Mac OS again, like what was maybe the history there for why Free BSD ended up in Mac OS?
30:23 Speaker 02 So I think my understanding is after Mac OS nine, I think is what it would be, Apple started working on Mac OS X. They borrowed a bunch from, I believe, Next. That's kind of where the Maki bits came from because there's bits of Mac OS that are very mock derived, but there's also a fair bit that is BSD derived. In particular, I believe the initial network stack in Mac OS X dot zero. This is largely came from Free BSD or at least Net BSD, but BSD bits in general. And a lot of their lib C, their equivalent of Glib C in user space is largely derived from Free BSD. And in particular, one of the things that also happened is several of the folks that I originally worked with in my first job out of college
31:03 Speaker 02 ended up going to Apple and kind of taking part of the Free BSD culture, I guess you might say, or at least like the mind share a little bit. And working on whether it was user space bits or kernel A bits and all sorts of Apple products that came down the line. So there's a lot of kind of friendship between the projects in part because there's a lot of friends between the different communities.
31:23 Speaker 01 Yeah, okay, makes sense. So let's move beyond where Free BSD has sort of ended up and actually a little bit to maybe a key part of the journey of Free BSD itself, I guess is what we call SMP, so Symmetric Multiprocessing. I think this was something that sort of had to evolve over a few years even. But yeah, could you maybe just talk us through like, what was the problem? And why did this have to be solved and how was it solved?
31:54 Speaker 02 Problem is physics at its root, but.
31:58 Speaker 02 I mean, SMP is the problem that at some point CPUs, we can no longer scale them individually to get performance. We had to scale horizontally instead of vertically. So we started having to add multiple cores into systems. And dealing with multi-threading is a complex problem in any environment, much less an OS kernel. FreeBSD first started supporting SMP systems, like dual Pentium systems or dual Pentium Pro back in, you know, that kind of era, very late 90s, a lot of parallelism, and that was kind of the point when I got this first job working on FreeBSD stuff. And I was supposed to, fresh out of college, work on like a bootloader for Itanium, I think some changes to the installer to have a better user interface
32:43 Speaker 02 in the installer was one of the things I was supposed to work on. But one of the things that happened is, right around this time of 2000, the company I was at was called Wanocreek City ROM, and it merged with the company who had the 1800, it's Unix telephone number, to form a newer company called BSDI, who had this commercial BSD operating system. And to help bootstrap FreeBSD's effort at having a more mature support for multi-processors, they kind of gave us a code dump and said, hey, you can borrow stuff from us. Like an initial version of kind of their way of doing locking and so forth. And so we had some developers in the community who are working on that. And I started helping out on IRC during the nights and during the days at my new job. And somehow, a couple of months in, I'm actually
33:26 Speaker 02 doing a lot of this work, even though I'm fresh out of school and went behind the years and probably shouldn't be doing this work, and having to learn a lot of things about atomics and so forth, which is actually the, Itanium manuals are very useful for me because I learned about things like acquire and release semantics of memory that's now like how you talk about concurrency in C and so forth. But I started working on this project, which internally in FreeBSD, we called it SMP next generation. So SMP NG is what we called it. And it's a long-running project. Initially, FreeBSD, when they were just trying to get this to work on plain x86, like the dual Pentiums, they did the most naive thing you can do, which is you have one giant spinlock around the entire kernel and it needs to A user process would go into the kernel that would have to wait for this one giant spinlock.
34:12 Speaker 02 There's a wonderful book I read when I got out of college, or maybe it was my senior year or so, called Unix Systems for Modern Architectures, which modern meant late 90s. And it talked about the way that systems like System 5 release 4 and just like other, and a few other systems, how they had done SMP. And the model that previously had is the one that I believe the book describes is about the worst possible thing you could do. And in terms of it doesn't scale at all, once you start having more numbers, of course. So the effort around SMP and G was how do we not have one giant spinlock? And how do we have a system where multiple things can happen, not just in user space, but in the kernel concurrently on different CPUs? And previously chose to go with a design that modeled more of the way that Solaris and perhaps IREX kind of,
34:57 Speaker 02 and other commercial Unixes did it, where we did things like we created dedicated threads in the kernel where interrupt handlers would run instead of running interrupt handlers on your kind of borrowed context. Linux still actually does the kind of borrowed context thing. I believe Windows does too as well, in effect. Although a DPC in Windows is kind of like kicking things off to a thread, sorta, that we do in FreeBSD. But we had this, that was one of the first big changes, was how we move our interrupt handlers into threads. That was actually the first thing that was kind of helping to stabilize and get landed into the tree. And then from there, I started working on things like, well, we've got this code dump from these nice generous folks, but it's kind of very x86 specific. And we had some kind of early patches for an alpha port that we had to try to bring up SMP on alpha.
35:40 Speaker 02 And I kind of atom smashed those things together and tried to clean up the SMP code to not, to be more portable and not very x86 specific, which meant refactoring the mutex code we'd inherited and defining extensions to our atomic operations to deal with memory barriers. And various things to kind of get us to the point where we're a little bit cleaner and kind of can expand over time. In terms of like the project itself, though, multi-threading a kernel and like, you might think of it just like performance scaling across multiple cores is a never-ending project. Like you're constantly just finding new things, like Netflix continues to push the boundary in certain areas of the system. We've gotten rid of kind of this legacy giant lock that we had as our holdover. And I think it's now effectively around the keyboard driver and a few other things that
36:25 Speaker 02 just kind of aren't worth fixing almost at this point. So it's mostly done, even though like the variable still exists. But even so that we have a multi-threaded kernel, like performance scaling continues to be a thing, right? We continue to grow sideways. When we were first doing S and P and G, we were worried about scaling well on four core or maybe like six core, eight core systems. And now we have to scale on 512 core systems. So it's kind of a never ending problem because physics, like we just don't get to have 40 gigahertz processes. So, oh well.
36:57 Speaker 01 I mean, just briefly going back to kind of how you described the project being set up overall. Is that still almost like a certain set of the contributors that work on like that scaling side? Or is that again, just like a distributed problem across all of the pieces? Yeah.
37:13 Speaker 02 Much more distributed problem. You know, developers scratching individual itches, right? For example, they're very worried about network scalability or virtual memory system scalability. So when they are working, looking at scaling problems and like in things like CIN file, that's the areas they look at. They're not necessarily trying to make sure that some little timer device driver can scale across cores for some reason. That's not the problem they're trying to solve. So they focus on the part that they're trying to solve. And that's still how things in general work is people are working on what's relevant to them.
37:43 Speaker 01 Yeah. So as FreeBSD has evolved, modernized, storage, I believe, has sort of been an interesting era that things have had to develop. You know, as I think a lot of our listeners will understand, just this evolution of how storage operates, going from HDDs to SSDs. And then now we've got NVMe's. We can probably do a whole episode on that in the future. I think it'd be interesting just to understand how FreeBSD has had to, you know, adapt to this evolution. And then we're also going to just look at how like a major release came out, really just at the end of last year, version 15. Be very interesting just to hear what leads up to that. And then again, perhaps like what is 15 as well? So that's a lot of questions. Let's go back to storage. Like, how has FreeBSD and storage had to evolve?
38:28 Speaker 02 Well, I think in general, storage has evolved in the industry. If you look at old spinning rust and the command sets we had with things like ATA, Reskazi, and Reskazi in particular, has very complex state machines. And like, if you go look at the spec for, I say the spec, there's actually several specs. If you look at the specs for Reskazi. It's very, very complicated. And if you go now to look at the spec for NVMe, NVMe is very simple. And part of what's happened is as they've moved away from spinning rust, they moved to flash memory, it turns out that a lot of the complexity you needed to deal with controller timeouts or, you know, seek latency and so forth, those are no longer relevant in a flash storage world.
39:09 Speaker 02 And so NVMe arose a bit out of, well, can we make something that is a lot simpler and cuts away a lot of the complexity? And that complexity, though, gets mirrored in the software stack too, right? So in previously, we have a most unified kind of storage framework, something that I think was a standard developed outside of previously, but previously adopted called CAM, like Common Access Method, I think is what it stands for, which is how we deal with scsi and ATA and how we deal with NVMe. But if you look at the bits, like the state machine handling for NVMe inside of CAM is much simpler than the state machine handling for scsi and ATA and so forth. We've been able to make that model work fine, but certainly moving forward, life gets a little less complicated. Like you don't have to worry about disk scheduling in the same way. But now
39:53 Speaker 02 with NVMe or flash storage, you do have to think about things like trim, which you didn't have to think about before in the same way on spinning rust. But you don't have to think about like the elevator algorithm and trying to sort and minimize seek head, the amount of time you're moving the head around and what the seek time of moving your heads is, that's gone. So seek latency is basically gone, but you do have to kind of think about when do I trim? How much do I trim? That's kind of my big bottleneck of how much I O can I schedule between trims? And so it's a different kind of problem, I guess, to think about. And then there's also network attached storage, things like ice gysi and NVMe over fabrics, which is you want to prevent, present the same command set but over a network connection. And there, you don't like, definitely seek latency
40:37 Speaker 02 is not quite the same thing at all, because it's all this kind of virtualized notion of a storage device. And that all kind of hooks up in the same way, all hooks up under our storage layer in the same way.
40:46 Speaker 01 The APIs had to modernize as well alongside storage, not necessarily because of storage, but you know, in terms of the modernization.
40:54 Speaker 02 Well, I guess one way to look at it is that as a long-running source project, gosh, I'm 30 years old. It's crazy to think I've been around it for 25 of those 30 years. Over time, you accrete a bunch of technical debt and you have APIs that looked good in the past. And then as you've worked with them for several years, suddenly they're not quite as useful, maybe, or they're a little crusty. And so one of the things that I also do, this is more my volunteer time, is I will try to find some things that are kind of crusty and clean them up. It is a balance. You don't want to do churn for churn's sake, because that doesn't do anybody any good. People who are carrying downstream forks like Netflix, you don't want to cause them undue suffering by making them merge a bunch of conflicting changes.
41:38 Speaker 02 But in particular, in our device driver frameworks, I had found a couple of things over the years that annoyed me, where due to some legacy reasons, like in every device driver, we required you to declare a global variable that was used in a macro that no one ever used. And so I did some work over the past couple of years to transition as to where you could use the macro both with and without the variable. And then once all the tree been converted, then I kind of dropped the compatibility shims. I've done another one this year. The compatibility shims are in free B S T 15, but I haven't finished the transition. The finish will be in 16, where when you're dealing with I O resources in a device driver, things like remapped I O or an I O port for like a P C I bar, but also other things like P C I bus numbers and so forth that a driver might need to kind of allocate
42:23 Speaker 02 and hold on to or map and then use it to read write registers. The way our API worked is there was various functions you had to call, both to allocate the resource when you wanted to use it, maybe to map it, and then to release it when you were done. You had to pass several redundant arguments kind of to every step along the path. And so even though like in a true object oriented design, like the very first thing you call to allocate a resource, it gives you an opaque object back. And the opaque object like knows things about itself. And it should know all the parameters you pass to it. So one of the changes I made was to make it like learn the one missing parameter that it didn't already know about itself, so that for all the following calls, like releasing a resource or Mapping ones into a like a CPU address space.
43:07 Speaker 02 You just pass the resource now instead of having to say, is it memory versus I O? And what identifier did I use? Like which PCI bar was it? So that it just makes that a little bit simpler and less for the programmer to think about. And those seem like worthy cleanups. And they're ones where using either evil or magic, depending on your perspective, and the C P processor, you can allow both forms to work for a while so that you can, across multiple versions, you can allow there to be compatibility so that device driver developers don't have to worry about it. If they adopt the new one, they can still use it fine when they merge changes to older branches. And that's kind of the strategy I've taken in that case. And those are, those are nice cleanups to do. It's healthy, I think, in code bases to clean up things that get crusty after a while. Because if you, if you avoid dealing with technical debt, it just gets bigger.
43:52 Speaker 02 And at some point it becomes a bigger mountain to have to shovel through.
43:55 Speaker 01 Yeah, absolutely. I really like that framing because at the end of the day, this is helping developers use it. And yeah, I think often other projects, usually just because of time of the maintainers, it's just almost impossible for them to come back to these things, even though they fully admit or agree that something is crusty or just out of date by the standards of how a developer might want to interact with that framework.
44:19 Speaker 02 And also, one of the things that, especially early on, we would kind of talk about in the previous C community is that in a corporate environment, when you're writing software, you're under a time crunch and a schedule. And it's ship it fast and not ship it well. Right? And you often, you write code that you know is kind of crap or prototype, and that's going out the door. And it's like, there's nothing you can do about it at the engineer level. And one of the things that I think some people really enjoy about working in open source in general, and in previous in particular, is we have a place where we can do things right. And you can take time and not, you're not under a time crunch of the schedule in the same way to push something out the door. And you can sit down and think, but what is a good design? And work through the good design and like take your time to do it well and like maintain it and keep it clean kind of in this platform.
45:04 Speaker 02 And that's one of the reasons I think people are attracted to doing this kind of work in an open source space. Is without that time pressure of an employer, you get to do it right and kind of do it well. And they're like to have peace about what you did instead of knowing that some product shipped and it's you wrote it and you would rather disown it.
45:21 Speaker 01 Yeah. Well, talking about shipping version 15. Came out at the end of last year. I believe that was roughly a kind of two year cycle versus when fourteen came out. Is that kind of a normal cycle? And like, I guess, what contributes to the, at the stage of thought around a major version bump on free BSD?
45:42 Speaker 02 So it is now kind of a normal cycle. Historically, we've aimed for a kind of cadence of roughly two and a half years or so between major releases. Or before that, it was a bit more up in the air. One of the changes that's happened in the last couple of years is we had a different person become our lead release engineer. And this individual, he's a really great guy, his name's Colin Percival, very smart. He wanted to have a very fixed kind of release schedule, which is what some folks in the community have been calling out for years. But he sat down and done it. And so he's got us on a pretty fixed schedule where we're now, and fifteen was our first major release that was kind of on this new schedule that he proposed. We're doing a major release now in Q4 every other year. So every two years, we'll have a new, that's like, so sixteen dot O will be
46:25 Speaker 02 two years from last December. Like, that's already kind of set in stone for us. And our minor releases are now once a quarter. The only difference being when we're doing a major release, we take a whole half year to focus on that. So we don't have a Q3 release on those years. But, you know, so now we have, I think, fourteen four is going to be Q1 this year and fifteen dot one will be Q2. So now we're, we're actually on a better cadence, I think is healthy for us, of having releases at a fixed interval. And this means they're not gated on a feature. And so developers still got to develop. That's the sauce that goes into releases is the itches they scratch, the things they work on. And so it's a core notion of developers working on stuff, but then being respectful of the timelines too. I think one of the things sometimes we would see in the past when we didn't have a, a good fixed schedule is
47:11 Speaker 02 when you would announce a release, because there was uncertainty about when the next minor release would be. Then you get this last minute flurry of like, a ton of stuff that would land in the tree all at once. Not always the most stable of things being merged last minute into the tree. Whereas having a fixed schedule means that we can, and our release engineer is empowered to say no, if he needs to say no, or she needs to say no, and Developers know, well, if I miss this one, well, there'll be another one on this branch in six months. So it's not the end of the world. I know it's coming. I know my bits will get out and it takes a little bit of the stress off. It also allows companies who support drivers in a tree, for example, they have wanted to have a schedule to know how to schedule their internal resourcing. And now they have that. And so they can figure
47:53 Speaker 02 out when they want to put bits into the stable branches in advance enough of when a release will come out.
47:59 Speaker 01 Yeah, that makes a lot of sense. And yeah, that predictable cadence, I think, as you say, sort of helps literally everybody who's part of the process and the end consumer, if you like, i.e. the company's taking which bits they would like to as well. So as we kind of start cruising towards the end of the episode, I do want to touch on Cheri, that's C-H-E-R-I, for those looking this one up. What is Cheri? So it's Cheri BSD, I believe. And like, what is that project? And yeah, that's a sort of interesting evolution.
48:28 Speaker 02 Cheri is a research project that's kind of come, been developed at the University of Cambridge in the UK. That's where most of the effort is centered. There are a few of us, like myself, who are contractors who can help with the project, who are scattered in the US and other places. But it's mostly a team of folks in the UK at Cambridge. And Cheri is trying to leverage some ideas from an older set of computer systems called capability systems. And I won't dive too much into that, but folks who are a bit more gray hair than I do might be familiar with capability systems, which were a different way of thinking about software. But Cheri aims to take some of the ideas, not exactly the same capability systems from the past, but some of the ideas, and see if we can use them to make existing
49:12 Speaker 02 real-world C and C++ code more memory safe. Because existing C and C++ code is very memory unsafe today. So if you're familiar with buffer overflows, the reason we have them is that at a core level, at the ISAs of contemporary CPUs like x86 and ARM and RISC-V, the way we think about regions of memory is we have a pointer that has the address, the starting address, and it knows nothing else. It doesn't know how big the region is. It doesn't know if you've moved partway into the region or partway out, or if you've kind of jumped in front of the region. The pointer has no idea. It's just a number. It's just the address. And so this means that it is up to software and software engineers to maintain all the metadata correctly about how big it, when I called malloc to get memory,
49:56 Speaker 02 how much did I ask for, and keeping track of how much I asked for and making sure I don't get that wrong. And if history has shown us anything over the last many several decades, it's that we as engineers get that wrong all the time. So kind of one of the big changes Cherry makes is that Cherry proposes a different register type down in the hardware that is something called a capability. And this part is kind of similar to capability systems, where you can have a register that has not just an address, kind of in the low 64 bits or 32 bits, but you have another word of data that goes along with that that includes other information about a region of memory. So it includes information like bounds and permissions. And the bounds are encoded using kind of a floating point scheme to avoid having really big pointers.
50:40 Speaker 02 They're only twice as big, not like four times as big, which was the starting point. But we're able to store this information about pointers down inside the ISA. And in addition to having this metadata, we also have this one bit tag on the side that we use to maintain whether or not pointers are valid. And part of the reason the tag is important is it allows us to constrain the operations you can do on a capability. So you can reduce permissions on a capability by reducing the bounds or handing out some of the permissions you already have, but you can never increase your permissions. You can't gain broader rights to the capability that you already have. And in particular, if you try to do an operation that might do that, what will happen is maybe we'll actually modify like the metadata word of the capability, but also clear the tag to say that the thing you got as a result,
51:24 Speaker 02 you can no longer use. Then the other change we make in the ISA is we make load and store operations now use these capability registers as your base address register instead of a plain integer. So now all your loads and stores of memory accesses have to be authorized by a valid capability. And we can verify when your access to memory isn't in bounds. Are you doing a load against a capability that has read permission? Are you doing instruction fetch against a capability in your program counter that has execute permission? And if you break those rules, you get an exception, instead of wandering off into undefined magic machine land where bad things happen. Then on top of that, so we've modified the ISA to now have this new type, and we modify all the general purpose registers in your CPU to use this new register type. And we have modified the memory subsystem
52:09 Speaker 02 to allow us to store tags. Tags in memory and make sure it's all coherent. So for example, if you do atomics, well, like make sure the compare and swap does the right thing with tags all the way through. Or if you do just random memory writes to something, you'll actually clear tags. Like you can only preserve tags if you go out of your way to do the right thing correctly. Then on top of that, we have extended L L V M to support Cherry architectures. And when you're compiling for a Cherry architecture, all the pointers in your C and C++ application now become these minor capabilities of metadata. And that includes both explicit pointers, so things you declare in your source code, but also there's lots of things that your runtime creates kind of inside your process that you as a developer never see.
52:51 Speaker 02 Things like GOT tables, which are how we kind of indirect to get to global variables inside of other parts of your program. Or like in C++ if you have a class with virtual methods and you have an array of these pointers to all your function pointers, a V table, and all those things become capabilities in Cherry C++. And it turns out that for some code, things like operating system kernels, or maybe your C library or C runtime that implements malloc, you do have to make some changes because for example, in malloc, you need to make sure that if someone asks for 16 bytes, that you get a chunk of memory from the OS. You need to narrow the bounds to make sure that the pointer you return to the caller can only access 16 bytes. So you have to make some changes in places like that. But in a lot of application level code, you turns out you don't have to make so many changes.
53:33 Speaker 02 So for example, we've been able to run most of KDE and Qt on top of a system I'll talk about in a second, without having to make hardly any changes inside of KDE or Qt itself. It's pretty clean, well-disciplined C++ code, and that just compiles for this alternate ABI out of the box. In terms of like places you can run Cherry. When I started with Cherry, it was kind of definitely a very, and still is a research system. There's all sorts of research we're still doing, but it was focused on MIPS. And so we had an extension to MIPS, which is a lovely little architecture, one way to describe it. But other folks have also shown interest, in particular ARM has found Cherry very appealing. And they actually built a custom CPU in SOC called Morello around that implements kind of Cherry extensions on top of the ARM architecture.
54:18 Speaker 02 And it's like I have one sitting over here, it's like a Quad-core 2.4 gigahertz processor, and that's the thing that like it can run KDE, it can run kind of a full-blown Unix. Cherry BSD is a port that we have a free BSD to run on top of these type of systems and to support these architectures. And that's so that's my box is running Cherry BSD, it can run KDE, it can run a web browser. We actually even have an early port of Chrome, which is an incredibly amount of work for people to do. But it is a real thing and it can run real software. There's also an interest in the RISC-V community. There's currently an ongoing effort to standardize a new extension called RVY. Hopefully later this year, perhaps, will finally be ratified as an alternate base ISA to RISC-V to support cherry systems.
55:02 Speaker 02 So it's a very interesting bit of work and it's kind of complements other work going on, the kind of memory safety space. So if you look at other things like Rust or other languages like that, Rust still needs bits of itself. And it's runtime and so forth that are written in C. And so cherry gives you the opportunity to enforce the memory model all the way down into even the bits of C to interact. We had an earlier research project that dealt with this in Java and dealing with JNI, where you could enforce the memory model from Java down inside the JNI code that was running because it had to use cherry capabilities to access anything inside of Java's heap. And so all the restrictions that Java wanted to enforce. Where things like unsafe Rust code could still have the same restrictions because it's in the ISA.
55:44 Speaker 02 It's not something that can be bypassed, it's not purely maintained kind of in software.
55:49 Speaker 01 Yeah, I was going to touch on the Rust piece because we do have a lot of Rust listeners, I believe. And so probably some of them are saying, but why not just use Rust? So yeah, and you've given very good explanation of, at the end of the day, having a much lower level implementation of this memory safety can be very advantageous.
56:06 Speaker 02 Well, and I think memory safety is a big issue, right? I think Microsoft did a study that something like 70% of the critical vulnerabilities where they had to ship a patch for boiled down to memory safety in some form or fashion. And we need different toolkits to address different parts of the problem. It doesn't seem to me realistic that we're going to rewrite everything that's currently in C in some other language. There's billions, if not trillions, of existing C code in the world, for better or for worse. And, you know, we're kind of stuck with it. And some of that stuff will get rewritten in Rust, but some of it may not. So it may be very hard to rewrite. So. We need different tools for different options. And so I think Chery is a good alternative to give us more tools to attack the problem.
56:46 Speaker 01 So we're coming to the end here. You've imparted a lot of knowledge and wisdom today. I think a lot of our listeners will find that they've learned, I'm not going to say something today. I've learned a lot today. I sometimes ask this question to guests, just to sort of round out, really just what is something that you know now that you would maybe like to tell yourself coming out of college or high school, whichever, what do you know now? I mean, related to perhaps software development or a career in software development, like what would you tell yourself then that you know now?
57:17 Speaker 02 One of the things I've had the privilege of doing a few years ago was teaching on the grad course for four semesters, OSs, as you can imagine. And so I'd actually did get to talk to my students a few times along these lines. I guess some of the things I would tell them are like, have fun. That's definitely true. Things like when you're doing a job interview, recognize an interview is a two way street. It's not solely you as the candidate who are trying to do something. You should be paying attention to the culture of where you're interviewing and decide, do I like these people? Can I work with these people? Or, you know, like, that's a valid thing. You should not purely think about that it's only a view of being serviced as a company. I always recommended a couple of books, some books that really I enjoyed when I first got out of college were The Mythical Man Must.
57:58 Speaker 02 It's one of my favorite books on development and sadly far too relevant today. And I think Frederick Brooks said that on the 25th anniversary, which is what I was reading in the 90s, that it was still sadly, it's irrelevant and it's still very relevant today. And another book I really enjoyed is something called Peopleware, which talked about some of the, neither one of these is about like fixing a bug in a, neither one of these is about like writing forlets or anything. They're about thinking about the art side of software engineering and what you're doing. I guess one of the things I would tell my students is that your job is not to just grab random things off of Stack Overflow and throw them in because You won't earn your salary if that's all you're doing.
58:40 Speaker 02 Like, your job is to engage your brain and think about solving the unique problems that your employer has. And lots of times, that does mean assembling things from other pieces that you find, things you find on the internet. But there's going to be some wrinkle where they just can't use something off the shelf. Because the off-the-shelf thing will be a lot cheaper. So to justify your existence, there's got to be some wrinkle to it and lean into that. Like, in this case, I usually tell at the beginning, do your homework assignments. Not because I need copies of your homework assignments, because you need to practice. You need to get your 10,000 hours in to get competent, so.
59:12 Speaker 01 Yeah, and I think that's more relevant than ever. We've managed to really not even talk about AI whatsoever on this episode, which is great. I think we seem to touch on it almost in every episode. But I think all the things you've just called out, I was talking to one of the other presenters the other day on a more, we have a regular SED News episode where we just talk about kind of more just very current. And we were talking about what does the education landscape look like for people? Today, I argued that a CS degree, and disclaimer, I don't have a CS degree, but a CS degree is still very powerful because it's about how you think about solving problems. And everything you've just said, John, I think holds true. It is just about leaning into solving problems and thinking about how to solve problems and so forth. Yeah.
59:57 Speaker 02 Someone's got to make the AI machines work.
59:59 Speaker 01 That's also true, yeah.
1:00:01 Speaker 02 For the record, my son is a first year, he's a freshman in CS. So, and that's actually a lot of fun because I'm perhaps assisting his teaching and making him learn the painful, low-level bits that maybe he's not getting in class, so.
1:00:12 Speaker 01 Well, I think you'll have probably a very good career as a result, assuming he wants to continue with software development. So, yeah. Well, John, again, thank you so much for the time. I've learned a lot, I'm sure the audience has learned a lot, and managed to sort of peek into the world of free BSD, which is very interesting. And as we've learned, power's a lot of what we basically use daily without even realizing. So, thank you very much.
1:00:35 Speaker 02 Thank you for having me.
Previous Cilium, eBPF, and Modern Kubernetes Networking with Bill Mulligan Next SED News: OpenCode, AI Code vs. Shipped Code, and the LiteLLM Breach Page updated November 6, 2026 · transcript and summary are machine-generated