PyTorch Eng Director: Promo Hacking, Industry Shifts, Regrets | John Myles White

John Myles White recently left his role as a director of engineering at Meta Superintelligence Labs (MSL) so we spoke freely about promo culture, how big tech has changed, and how his career grew. 𝗣𝗼𝗱𝗰𝗮𝘀𝘁 𝗹𝗶𝗻𝗸𝘀: • YouTube: https://youtu.be/aPfnP4iAIH8 • Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835 • Transcript: https://www.developing.dev/p/msl-eng-director-promo-hacking-industry 𝗕𝗿𝗼𝘂𝗴𝗵𝘁 𝘁𝗼 𝘆𝗼𝘂 𝗯𝘆: • Cursor 3: a unified workspace for building software with agents, check it out at https://cursor.com/ • My ergonomic keyboard project, you can follow along here: https://read.compose.llc/ 𝗧𝗶𝗺𝗲𝘀𝘁𝗮𝗺𝗽𝘀: 0:00 - Intro 0:54 - Is he bullish on MSL 5:23 - Running promotions at Meta 15:15 - Growing at Meta 22:22 - Julia core language contributor 29:24 - Academics failing into industry 31:48 - Stats book recommendations 38:02 - Biggest career regret 41:05 - Advice for his younger self 42:46 - Outro 𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗝𝗼𝗵𝗻: • LinkedIn: https://www.linkedin.com/in/john-myles-white-115697180/ • X/Twitter: https://x.com/johnmyleswhite • Personal Website: https://www.johnmyleswhite.com/ • Github: https://github.com/johnmyleswhite 𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗥𝘆𝗮𝗻: • Newsletter: https://www.developing.dev/ • X/Twitter: https://x.com/ryanlpeterman • LinkedIn: https://www.linkedin.com/in/ryanlpeterman/ • Threads: https://www.threads.com/@ryanlpeterman • Instagram: https://www.instagram.com/ryanlpeterman • TikTok: https://www.tiktok.com/@ryanlpeterman 𝗥𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲𝗱 𝗶𝗻 𝘁𝗵𝗶𝘀 𝗲𝗽𝗶𝘀𝗼𝗱𝗲: • Evaluating the design of the R language - https://www.researchgate.net/publication/240040602_Evaluating_the_Design_of_the_R_Language • Stats book he mentioned (not affiliate link) - https://www.amazon.com/Foundations-Agnostic-Statistics-Peter-Aronow/dp/1316631141 • Stats book he mentioned (not affiliate link) - https://www.amazon.com/All-Statistics-Statistical-Inference-Springer/dp/0387402721

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  You have to play the game. It's totally irrational not to play the game.

• 0:03Ryan Peterman?

  This is John Myles White. He was a director of engineering on PyTorch in MSL. And since he quit recently, we talked freely.

• 0:11Speaker 00

  I feel like our goals should not be written in a way where shipping a thing we intend to delete is a success. The general perception is the supply of engineers is like way oversupplied.

• 0:23Ryan Peterman?

  We also talked about the incentives in big tech. You ran promotions in AI infra for a long time.

• 0:29Speaker 00

  You know, the only reason you do anything is because there's a clear story about how it's going to get you a promotion.

• 0:35Ryan Peterman?

  You saw this, I saw this. What could you do, though? Here's the full episode.

• 0:46Ryan Peterman?

  I'm curious if you're bullish on MSL, like after you were working there for a bit and kind of seeing it from the inside.

• 0:54Speaker 00

  Oof, this is a, I guess, like a more radical question, but I guess I will be honest. I am very bullish on Meta as a company that I am now a stockholder of, but not an employee. I am very bearish on Meta if you're an employee who's not a stockholder, which just turns out to not be really anyone, but like, you know, I think you can think of yourself as mostly employee. If you're mostly getting cash and mostly not equity, the more senior folks are the ones who are more mostly stockholder than employee.

• 1:27Speaker 00

  And I think that's the tension is I do think like, my sense is that it's unfortunate. I don't get the perception as unique to MSL. My perception is that I think just Meta as a place to be an employee is less enjoyable than it used to be, but it actually is being run very effectively. If what you care about is the bottom line of the business. And so like I continue to invest in Meta and I suspect I will continue to invest because I do think it's actually from a business perspective run quite well. But I do think pretty uniformly, I think it's not unique to MSL. I do think it's a much more stressful time to be an employee there than before.

• 2:03Ryan Peterman?

  What's the part that makes it where you're saying employees might not enjoy working there as much as they used to?

• 2:11Speaker 00

  I think that, I mean, I think this is true of all of Silicon Valley. I think in general, like, there was a lot of sense that, like, it was incredibly easy to lose your good employees. And so you had to do everything impossible to sacrifice to retain them. And you were constantly constrained by an undersupply of employees. And I think basically everyone in Silicon Valley's view, as far as I can tell, is that that's mostly not true. Outside of like a small set of like AI researchers and frontier labs, where I think people do still behave this way. In fact, maybe behave this way more than ever before. But I think for everyone else, the general perception is the supply of engineers is like way oversupplied. And especially, I think, actually, people's concern is with AI, maybe they're really oversupplied.

• 2:52Speaker 00

  I think what happens is new market dynamic, which is like, I think as an employer, you're thinking, I don't actually have to make so many sacrifices to acquire and retain talents. And so therefore I'm gonna make fewer of them. And this can play out in a million ways, but it can be like compensation. But it can also be things of like, do we do things that upset the employees or do we not? And how hesitant are we? Um, how much do we give them a voice and how much do we not give them a voice? I think that stuff has changed a lot in the years I was at Meta. But my impression is Meta is not in any way unique here. This is just a general property of all of Silicon Valley.

• 3:30Speaker 00

  But I do think, you know, as an employee, I think the like labor supply and demand situation is super different from when I started. And I think it is a thing that is going to make generally being an employee rougher periods.

• 3:44Ryan Peterman?

  It it feels like it can be subtle as well, like what you described of, I guess, employees having less leverage. You could see that affecting, um, maybe even like reorg decisions or things like that. Where it's like, you know, if people leave, that is part of the calculus of the reorg. And that is less of a downside in now and maybe in the future. So I can see there being a lot of downstream effects where, uh, things just don't go as well for for employees.

• 4:13Speaker 00

  Yeah, I mean, there's just so much stuff that in the previous versions of Meta was done that arguably was a bad decision for the business, but made sense from the context of retaining talent. You know, and I think that the company is just doing less of that. But on the flip side, I mean, I guess to be clear, I didn't say, as I said previously before, I do think for most people, like fundamentally, just like the possibility you might get laid off is the number one emotional thing that causes people stress. And living in a world where you know that is both happened recently and may happen again in the future. I think sort of like is probably beats out all the other questions or sort of lifts and like food and stuff. I think I think what prior to the layoffs, when those things got taken away, people freaked out and were like, this is unforgivable.

• 4:54Speaker 00

  But then we're like, once they're like, oh, wait, uh, actually you guys might just fire me. So now I'm much more tolerant of you taking away my food away. Um, but I do think that like that that fundamentally like it probably drives for most people, like the vast majority of it. And I actually think for us, well, this probably is one of the things that makes say MSL more stressful than the other frontier labs. It's the sense that it might have layoffs in the sense that I think, you know, so far there have been fewer rounds or at least perceived to be fewer rounds. It's one of the smaller startups like OpenAI and on the RMAC.

• 5:25Ryan Peterman?

  I I think another thing that is oftentimes used for retention is the promise of growth or I guess career growth for an employee. And, you know, I know you you ran promotions in AI infra for a long time. I was curious like if you saw things change over the years.

• 5:41Speaker 00

  I observed a lot of divisions of Meta would really like stop talking about like, well, your comp is why you're here. And we pay you good money and you enjoy the work and that's why you stay. And a lot of people were like, you know, the only reason you do anything is because there's a clear story about how it's going to get you a promotion. And you would like, especially, and this was really striking when I moved from data infra, whereas one of the places I was in earlier in infra, moving to AI infra and I was in AI infra for a while before I came into sort of pipeline part of AI infra and then later MSL. One of the things that blew my mind is like, there was not a single person I would meet on any team in AI infra, whose first and foremost goal wasn't promotion.

• 6:22Speaker 00

  This was a thing that sort of came up in data info, but was not like 90% of people's attention. And when I started at Meta, it was just like not ever a thing. Uh, actually, like, you know, I used to manage one of your previous guests, Adrian, and like Adrian and I were talking about career growth back in the day. And one of the things that was really amazing was like, we started this org growth that actually had never had anyone above IC seven ever in history. On the sweet side, to be clear, there were some other rules that had it. But like, at least this is what we were told. I actually don't know for a fact it was objectively true, but we were told this by several people.

• 6:59Speaker 00

  I think in that world, you weren't focused on promotion, so it wasn't a big thing. But then it became a thing where it was like, this is the thing. Like, you will, one, get more money, more promoted, but also you'll have a title that you can use for your next job. And like, everything is driven by promotion dynamics. Actually, I would say that the org I found this by far most strong in was monetization. Which is like, when monetization would hire people out of AI infra, it would always literally be like, here is a very detailed plan of work you will do in order to get promoted. And that was a thing that worked very effectively. But it also, my experience, like, wrecked tons of teams. And I managed a bunch of those teams now to fix them, but they were teams where like,

• 7:44Speaker 00

  basically, everyone agreed the thing we were working on was bad. No one thought it would succeed, but people were like, oh, but I have to ship it because that's my promo bar. And you were getting to the state where it's sort of just impossible to even make good decisions about software anymore, because the promotions were so important. And then I think what happened as the market became less pro employee is that people were like, oh, no, no, no, you should be afraid of being laid off. You should not be worried about the positive chance of a promotion. But I don't know that that was a, I don't think that was a good cultural fix, but I do think it's been a bit of like an attempt to undo some of the damage from before. But I do think that like sort of promotion mindset was incredibly intense everywhere. And ironically, it led to this thing which I found really troubling, which was

• 8:28Speaker 00

  the thing that would actually let you develop real skills that would do better in your career going forward, increasingly got decoupled from the promotions. You know, and I've seen a lot of people on Twitter over the years say this, and I find it very compelling, which was people like, the main thing that's wrong with the engineering cultures at the big tech companies is the promo culture. And I really agree. I actually think that ironically, my perception of this seems to be doing better at this than many of the other companies that have even more formal processes and more anonymous parties involved. But I do think they're like, Oh, what really matters is that you like ran a rollout that affected ten other systems. This causes people to like not build clean systems.

• 9:09Speaker 00

  It causes them to build systems that maximally are coupled to the other systems in order to be able to hit the promo bar. And that they think, actually, I met so many people who didn't even like the work they were doing because they're like, well, this is what will get me promoted. And so do you think that stuff was really unfortunate? And especially, I think, actually, in the AI world has been especially unfortunate because it means like, unless someone is clear how using the AI tooling is going to get them promoted, they may not do it. But it is without doubt in my mind, the only thing that's going to matter for actual professional development and growth and actually being able to do more interesting work in the future.

• 9:45Ryan Peterman?

  Yeah, I definitely saw some unusual behaviors, but we're all trying to play the game because it helps people get promoted and you retain people and all that, so.

• 9:56Speaker 00

  Yeah, I mean, unless your VP is ready to fix that and totally shut it down, like, you have to play the game. It's totally irrational not to play the game. And I think, you know, you either, if you don't want to be in that, you have to leave the org. If you're there, you've got to play the game. You cannot be the one who sort of unilaterally disarms. But it is, I mean, it was, it was mind blowing to me to see this culture, I think, was unique to both monetization AI infra, but it was completely dominant in AI infra. And it was really, it was not great for any parties. And funny is, it was like, it wasn't even good for the people who were in it. Like, they themselves had gotten to this rat race that seemed to be demoralizing to them.

• 10:32Ryan Peterman?

  You saw this, I saw this. What, what could you do, though? When I was in it too, the people who were talking about it, they were not necessarily saying, yeah, this is, this is good. They were saying, yep, I'm doing this that I don't believe in, but we both know that I need to do this for this reason, so I'm doing it.

• 10:50Speaker 00

  I think there are sort of two high sources of uncertainty that compete, and I don't quite know how they balance out. But I think one is like,

• 10:58Speaker 00

  I think people have a ton of agency in choosing the culture of the team they want to be on. And if they're in a situation like that and they don't love it, Meta has a lot of teams that don't have that culture. There are a lot of teams, like PyTorch was one of them, where people were just like genuinely in it for the love of the craft. And like people loved engineering as engineering in PyTorch in a way that I think was not present to a bunch of other parts of Meta. I think people wanted to come to PyTorch for that reason. And I think people came and were happy for that reason. So I do think people just have agency in that sense, which is like, yeah, within the context of that machine, you gotta follow the rules, but it's not like you're forced to be in that machine. There were other rules. Not everyone would get them. And PyTorch was very choosy, but I think many people could and it was worth doing.

• 11:42Speaker 00

  I think the alternative there was also was like, even within a team, managers can differ a lot in how much they push on this. And I have personally been a person who I think mostly benefited from actually trying to bet on, be on the stable team that will gradually succeed over time. And we'll have a healthy culture and not collapse and not do things like over level ourselves. But it does mean that you get promoted slower. And this is where I think I do struggle with this a bit, which is I think like, if at the end of the day, what you really want to do is do something like compute your total sum of earnings over your entire lifetime. It may be better to be in the orgs that get promoted really fast and get fired really fast. But there are a lot of those orgs at Meta where people are like, well, they're the stars.

• 12:27Speaker 00

  Oh, actually, no, it turns out they're terrible and lied to us. So we got rid of all of them. This happened a bunch of times in the years.

• 12:36Speaker 00

  It might be that that is actually economically rational. I'm not sure, but certainly I think from myself about emotionally rational and feeling like I enjoy the craft of stuff. Being on the teams and PyTorch was like this. I mean, one of the challenges actually PyTorch had with hiring was actually the perception that PyTorch held a higher bar. People would be like, oh, I've heard you guys are actually like much tougher bet promotions. And I'd be like, yeah, honestly, we are. Like honestly, like our aides are like as good as you're gonna get anywhere in this whole world. And you know, if you want to be the best engineer you're ever gonna be in your entire life, you should work with them. But like, Our rates probably are better than the 10s in a couple of other teams. If you want to be a 10, you maybe should be in that team instead. And I think for some people that would really turn them off.

• 13:21Speaker 00

  But I think part of this was again, the sort of like agency and selection mechanism. I think PyTorch selected people who actually just loved engineering as engineering. And then there were people who were willing to tolerate slightly fewer promotions. And ironically, one thing that was interesting though, was that it became a bit of like magic thing that kind of turned out well, which is because people perceived PyTorch to hold such a high bar, it was much easier for us to convince people outside of PyTorch that actually our people were ready for eight or nine or 10. Whereas other teams, when they tried to make this argument, they're like, oh, but you you guys are not well known for holding a high bar. Maybe you guys are just overselling these people. But in PyTorch, people were like, oh yeah, like you you guys hired our guy and thought he was pretty bad. So actually, we think you probably are credible.

• 14:06Speaker 00

  And I think that that like, again, it's like, in the short term, doesn't actually accumulate as fast, but I think in the long term, does have a ton of benefits. Whether it is the absolute like compensation maximizing algorithm, I'm not sure. And that's where I have a little bit like torn torn. But for me, I was also like, I was willing to get 20% less compensation to be in a place that I was more proud of.

• 14:30Ryan Peterman?

  Actually, that's funny because I remember someone from PyTorch would join one of our collaborations, for instance, you know, someone would join and go, oh, he's a five, but really, he's like a seven, like he's, he's better than all of our engineers, but we don't know why he's a five, but he's a five. That was not uncommon, working with that org.

• 14:49Speaker 00

  Yeah, I mean, I mean, yeah, this was like, I mean, it would, it would come out like flat out in like opportunity chats where we tried to hire someone and they're like, hey, you know, PyTorch is cool, but like, aren't you guys like really underleveled? That would be like the first question people would ask and have to be like. Maybe, or maybe we're holding the right levels. You know, you gotta decide where you wanna take a chance on us. But you know, but I think the flip side is, you know, that, you know, we really did train people. Like, a lot of people who were in PyTorch really learned to be remarkably good.

• 15:15Ryan Peterman?

  I know before PyTorch, you, you worked on some of the data and experimentation tools at Meta. How was that work? And like, how was your experience growing in, in early Facebook before all this promo craziness?

• 15:28Speaker 00

  Well, I came into a wild ride, uh, which was actually honestly an amazing experience. And, you know, some of my happiest years of my life were like my first two years of Meta, but also some of the most fearsome and scary years were also there. But like, um, I joined the team that I signed up for. And when I signed up, it was supposed to be called Data Science. Before I arrived, because I asked for a six-month leave to work on Julia, the programming language I was one of the core contributors to. I asked for a six-month leave between finishing grad school and going to Facebook at that time. And during that six-month period, the guy who hired me quit. And then the team that was called Data Science that I was hired into got split into two teams. One called Core Data Science and one called Data Science Infrastructure.

• 16:11Speaker 00

  And then that itself began really tricky because I wound up joining Core Data Science, but really loving collaborating with the Data Science Infrastructure people, who were the ones who owned the experimentation tools.

• 16:23Speaker 00

  Working in the experimentation tools was like, honestly, like, I think to this day, the most, most people who know me from Meta are like, oh yeah, John was really helpful for that stuff. I actually think like, almost nothing I worked on as an IC ever went anywhere close to being as valuable to the business as the experimentation stuff.

• 16:40Speaker 00

  And I don't think I was ever as good at any of the other stuff. I really loved being in that space. And I think it was like, really influential to the business. That said, I was on this Core Data Science team that was like, an insane ball of stress. You know, I joined, it was radically reorged, it had new managers. I actually wound up really liking the new managers, but that was still a source of churn. But then a few months into it, someone who actually was on the Data Science Infrastructure team, which is particularly what's amusing, but attributed his team to being Core Data Science, because he perceived that to be sort of the team he really was on when he wrote this paper, published this paper in PNAS, the Proceedings of National Academy of Science, called something like Emotional Contagion in Social Networks.

• 17:24Speaker 00

  That wound up just becoming like the absolute singular worst piece of PR for Meta as a business that year. Just absolute disaster. I was like really, really trying to prevent this from happening, but I didn't have the authority to prevent it. But like, you know, at least people perceive I was on the side who was not happy with this decision. But it meant that I, you know, was on this team where I was doing the data science infrastructure where it was sort of very interfacing and very safe. But I was affiliated with a more researchy division that was publishing these papers. That went from becoming sort of a PR win to a PR nightmare very rapidly. And that team, I think, was one of the most my formative experiences at Meta because it really was like, well, what happens if this team just gets fully disbanded? And this was in a world where there weren't layoffs. It was like, well,

• 18:09Speaker 00

  Meta doesn't do layoffs, but this team maybe has got itself to a state where it's going to get laid off.

• 18:15Speaker 00

  You know, and the guy who wrote this paper wound up having to do like a company-wide Q and A where effectively he sort of apologized to the entire business as his Q and A.

• 18:24Speaker 00

  It was really just like a mind-blowing experience. And it was especially it was like one where it's like, you know, I came as an IC4 and all of a sudden we were like in these meetings, we're like, you know, the head of legal, being like, well, why did you guys do this? And, you know, having to have these discussions with them on a regular basis and really trying to figure through like, what was the future of our team? But it was great. That blew over for what it's worth. And then I spent several years just working on our experimentation tools. You know, I was like one of the main developers of Deltoid 3, which is I think now just called Deltoid because I think it's been Deltoid for so long, they just referred to it as Deltoid. But at one point it was like the third iteration. And I worked a ton on that. And then especially it was a real example of how career growth can actually happen, which is like,

• 19:07Speaker 00

  I was on the team and then all of a sudden, almost all the senior engineers left the team in the span of like six months. And then I went from being like, and at the point, maybe I was already in IC5 when they all left. I'm not sure. But I went from being like one of the people on the team building experimentation tools to the only one who remembered how anything worked, left. And suddenly I went from being sort of brain-damned IC5 to like de facto TL for a bunch of stuff, which was actually an amazing opportunity for growth. And I think people understate how often these things can happen in tech. But it meant that I wound up like really being like able to drive a bunch of the vision for the A B testing tools for years, which were, you know, hugely successful at Meta. And it really was like some of the most fulfilling work I ever did at Meta.

• 19:51Ryan Peterman?

  And just to give people context, Deltoide is the AV testing framework, or the thing that you opt in, you know, one code path to A, one code path to B, and you measure all the downstream benefits of ideally your test, right?

• 20:05Speaker 00

  Yeah, I mean, I think one of the things that's actually kind of mind blowing to me is like, you know, actually, it's one of these decisions where maybe I did make bad career decisions, which is like, as far as I can tell, every time I've ever looked at it, the StatsSig product, which I now don't know what its status is after they got put open AI, is just like, is Deltoide. This is one of the things where like, you know, stuff in Meta, when you've been at Meta, you work on these things like, oh, I'm into DataSwarm. And when people are like, what's DataSwarm? I'm like, it is literally Airflow. And it's not like metaphorically Airflow. Like, the guy who wrote Airflow built DataSwarm, quit, and like, a week later, open sourced Airflow.

• 20:40Speaker 00

  And Deltoide is like, not quite the same, because it's not the same people left and built StatsSig. But my perception is that for most people who will be watching this, if they've ever seen StatsSig, my perception is that like, almost the entire UI is the same, almost all the functionality is the same. It's one of these things where I like, you know, probably should have built a Deltoide startup many years earlier, and I did not have the wisdom to do that.

• 21:01Ryan Peterman?

  I think there is another one called Optimizely as well. Yeah, so I've seen many companies. It seems like a very repeatable playbook, where you just take something that people take for granted, that's state of the art from a big tech company, and you just give it to everyone in the industry. And it actually creates like, a billion dollar company is pretty... I, it's hard, but at least the product market fit and idea part are relatively solved, since it's creating so much value for these big companies. For this podcast, I produced transcripts for every episode for convenient skimming, and I built a custom tool to automate that. Recently, I noticed in the Barbara Liskoff transcript, my simple speech to text tool was getting a lot of things wrong.

• 21:44Ryan Peterman?

  For instance, the CLU programming language is spelled, all caps, C L U, not clue. So to fix this, I used Cursor 3, picked the strongest version of Opus 4.7 Extra High, and had an agent make a plan to fix that. And while I was waiting, I figured I'd trigger a few more agents for code cleanups and front-end improvements. It generated a reasonable plan with rich system diagrams. It applied all the changes within minutes and worked on the first try. So if you want to build something with the flexibility of sending off a bunch of agents with frontier models of your choice, you can go to cursor.com to try out Cursor 3. You know, I saw before you worked at Meta, you were working on the Julia programming

• 22:26Ryan Peterman?

  language, and I actually wasn't familiar about it, so I read into it a little bit. It looks like it was part of these data science language wars, basically, where there was R versus Julia versus Python. What is Julia and what is the context on that war there?

• 22:45Speaker 00

  Yeah, well, certainly, I think I made it more of a part of the war. I don't think it had to necessarily be part of it, although you think also, like, the simple fact of the reality is, like, programming languages are products, and products exist in an ecosystem where they're in zero-sum competition, and claiming that they're not in zero-sum competition is like a very cute thing that people say is appropriate, but is clearly false, and I think just makes everyone worse by misleading them. But like, I mean, so Julia, for me, and it's actually sort of why did Julia so appealing? I mean, for me, what Julia's pitch was like, we should be able to write code in a high-level language that looks like Python or like MATLAB, which is really the language it was originally designed to destroy. It was really designed to get rid of MATLAB. It was made by MIT math people who wanted

• 23:29Speaker 00

  to get rid of MATLAB, and it really like targeted that market much more than data science at start. And sort of, I think I was involved in pushing it towards data science. But you know, to me, the thing I always do when I give talks about Julia is be like, listen, let's look at the R function for distance, like compute a distance matrix between a bunch of vectors. So you're like, you know, pairs of vectors, and you get all the distance matrix. If you look at like, that function, and then you actually try to figure out how it's implemented in R, what you find is like C code that is very reasonable C code, that is just a bunch of for loops, like, you know, loop through all the rows and all the columns, and then compute the distance at that row, and you're done. If you basically take that code verbatim and just translate it naively into R,

• 24:15Speaker 00

  you're gonna take some type information away, you're gonna get rid of some ints and float signatures. But otherwise, you're gonna basically write for loops that look exactly the same. The R code is gonna be like somewhere between a thousand to 10,000 times slower than C. And this to me was the thing that just like drove me insane, where I'm just like, wait, what? Like, these two programs are like 80% the same. Why is one not as fast? And Julia really was all about this notion that like that was unacceptable. And that's what made it so appealing when the first post by the original founders went out. I was like, oh, you guys are doing the thing I wanted people to do, which is like,

• 24:53Speaker 00

  not claim that it is impossible to make high-level languages fast, which is like so much of actually how the Python R community sometimes behave is to be like, oh, well, we can't be fast, but also fast isn't important. And to me, like that that double hit of like, well, we can't be it and it's not important really did work for me. So Julia really resonated. I think I probably is the guilty party of trying to make it more part of the day of science wars, because I was myself a heavy user of R and was just so disappointed in R. Just so incredibly disappointed in how often I would try to do a project and R just like fought me at every step of the way.

• 25:32Speaker 00

  But Python is also like this. And if you look at all the really great libraries like PyTorch, like, you know, deep down, at the end of the day, you're gonna look at C++ code. You're maybe even looking at like handwritten assembly or handwritten like kernels for GPUs. Or at least you're looking at something written in a much lower level language. And so Julia was really like about trying to solve that. And I don't think it totally won, which I think is probably why you didn't know about it. I think it was very hip at one point and has become less hip. But it's actually doing okay. Like, I think it's in the top 25 programming languages by users in the world. So I think it's a real, a real language that's really out there. But for me, the thing that really matters, even though I don't know that it's killing it, Julia is like the only people still actually fighting that fight.

• 26:15Ryan Peterman?

  Well, what's the intuition behind why R is 10,000 times slower when the code is, you know, the symbols are relatively similar to the C?

• 26:23Speaker 00

  Fundamentally, any code that's slow is slow because it's doing stuff it doesn't need to do. Like, that's just sort of the most basic fact about slow code is that the reason you're slow is because you could have done something else and you did something slower instead. And something like R is doing this pretty easy. This in Python, it's not quite as dire, but it's still there, is you wind up paying an enormous amount of overhead cost for the possibility that someone might do something more dynamic. And because they might do it, and to give you an example, which is really astonishing about R is, in R, for instance, the brace that you use to define a block is an operator that can be overridden and the user can redefine. So they can make braces mean something else.

• 27:05Speaker 00

  But so that means when you see a brace in code, you can't just be like, I know what this is. I can move on. You have to be like, no, I need to look up and check did the user redefine this. I think, I think it's brace and not parenthesis, but it's been a while, so I haven't dug in. But it may also be parenthesis, or it's possible I flipped them. We can like, you know, check offline and see whether my memory is good. But like, you just wind up with so much stuff like this that is so like, maybe changed and you don't know whether it changed. So you need to go check whether it changed. And the checks are very expensive, especially if you're doing something like adding to 60 bit, 64 bit integers. That's like one machine cycle, like, it is one machine cycle. But a check, like, does addition still mean what I think it is?

• 27:49Speaker 00

  Could be hundreds to thousands of machine cycles. And so you wind up like swapping in things that are very inefficient, places that you don't need. And this is particularly, R is amazing. Like, there's this amazing paper by a couple of students and a senior professor named Jan Vitek. But it's about the design, called something like Evaluating Design of the R Programming Language. And one of the things they look at is, especially R has an especially tricky thing, which is unlike Python, R is also a lazily evaluated language, where the arguments to functions are not evaluated before you start the function body. They wait until the function kicks off and they just are passed as promise objects. And what they look at is they look at like, well,

• 28:32Speaker 00

  how often are these promises could have been effect like eagerly evaluated? And how often is the overhead of these promises worth it? And their conclusion is like 70% or maybe more, maybe it's 90%. I forget the numbers. You basically have no reason you needed to do this. Like almost never do you need this? But you actually pay like an enormous overhead cost for having agreed to do this. And a good example, like saying in Python also, it's like in Python, you can like manipulate the symbol table using functions in the inspect module. And so what that means is like, you can never be sure of what something's bound to. You always have to be afraid and check. And just sort of general, the lack of invariance, that's what makes a language fast. It's like you have lots of invariance.

• 29:15Speaker 00

  What makes your language slow is you have lots of stuff you might have to go confirm at runtime. And R is just incredibly pervasively like this.

• 29:24Ryan Peterman?

  I looked at some of your popular past tweets and I thought maybe we could discuss some of them. So one of them, this is the most popular tweet that I think you ever wrote. And you said that you're continually continually disappointed by how many grad students and post docs get the impression that industry is a safe position of last resort. They can always fall back on if things sour in their academic careers. And I thought that was interesting because I thought the I thought the opposite was also very commonly true, where people might, you know, want to avoid industry so they go and get higher education. So I curious your, you know, your thought on this and what made you think this?

• 30:07Speaker 00

  Really what drove me nuts was there were just a ton of people fundamentally wanted to be professors or post docs and were in a PhD program. And they were like, well, if I fail out, I'll go into industry. And this like, one is that you would interact with people during interviews who like clearly didn't want to be there, just like so unambiguously did not want to be there and clearly viewed this as like a failure that they were interviewing. And you're like, well, that's not really like a positive sign that we want to hire someone who like, doesn't seem like they're going to enjoy the job. But in addition, a bunch of people, and this is what drove me so insane, because I think it's like all parties involved in the academic system hurt students doing this,

• 30:51Speaker 00

  is that I'm like, so many people just assumed that when they finally decided to get an industry position, it was going to be trivial. And then they didn't find it trivial. I think a lot of academics who are like, well, smart people are in academia and the dumb people are in industry. So if I need to go compete with the dumb people, it will be easy. And I think there was a lot of that. But you know, there was a person, I'll give you an example, there was a person who was like effectively a CS professor who I interviewed. And this person like could not figure out how to pass values between the various functions that they were calling in the interview. Like literally, they were like, what I would do is I would call this function, it would print out in the REPL. And they would read it as a human and then I would go like type it into this other piece of code.

• 31:33Speaker 00

  And I was like, oh, you are better at programming than this, right? Cause like you're a professor of computer science and they're like, no, no, this is how I work. And I was like, oh, this is not going to set you up for success if we actually have to get you writing code in broad here.

• 31:48Ryan Peterman?

  I saw a few other popular tweets that you had. They were about like favorite statistics papers and favorite recommendations of statistical books that you're saying, oh, everyone's got to read these. How come you have such strong recommendations on statistical literature? And then also, what are those recommendations?

• 32:07Speaker 00

  I love statistics. I think I'll never not love it, but I think it's the craziest field. And what I mean by craziest, it's a field that fundamentally sells people the idea that they can use statistical methods in real life. But in reality, what they do is do pure mathematics and study how statistical methods work in an idealized theoretical world. And in pure math, like, you know, as an undergrad, I did pure math and I loved things like number theory. In pure math, you're just period. It's pure. You prove it, it's internally coherent. There's no attempt to like reconcile with reality. Reality doesn't even matter. You're just like, there's rules, we follow the rules, we're in this internally consistent system. And then in super applied fields like software engineering, you're just like, well, the thing runs, like the code runs.

• 32:52Speaker 00

  I don't know what to tell you. Like I can't prove this code runs, but like we ran it in broad and it had like eight nines of reliability, which makes it better than like most of the software, everything we humans were good. Statistics is the super crazy field where you reason about mathematics, but then make all these claims about how it's going to be useful to people in practice. And this, I think, is where opinions come in so strongly is that I think some people just like are very, very honest and hold themselves to a super high bar. And some people are super careful here about stuff and are roughly just like, well, I said it's true. And then you're like, well, is it true? And they're like, uh, and you're like, let's really dig into the proof. And they're like, fine.

• 33:35Speaker 00

  You know, one of the four people I love, love, love, love more than anybody is Larry Wasserman. I've never met the guy, uh, so I don't know what he's like as a human, but like his books are like, to me, the embodiment of like, hyper intense honesty. It just seems like a person just like, I cannot tell a lie. And just like, therefore, like, everything you get from Wasserman is exactly true. Like, he tells you exactly what he's assuming. He tells you exactly what to imply. And he's also extremely clear about being like, I actually don't claim these other things that you might want me to claim because they're not true. Um, and I think a ton of statistics books are not like that. A ton of statistics were like, use our methods, they're great. Um, and so I think Wasserman is really at the, at the top.

• 34:17Speaker 00

  Another book that someone recommended to me sort of halfway through my career, a meta that I loved, um, I think it's called something like Introduction to Agnostic Statistics is a book by a guy named Peter Aronow. And he's, I think, another person like, sort of just like incredibly concerned with whether the things he says are true or false. And he's hyper rigorous and hyper careful. And again, I think a lot of people in statistics are not hyper rigorous and hyper careful. So I love the book. And so the recommendations I gave Wasserman, like literally anything you can get by Larry Wasserman, buy and read. If you want to learn statistics, like, I don't think any book has ever been better than the books he's written in my entire life. I've never seen anything come close to being as good. The Peter Aronow book, I think, is probably as good as like a thing you could be ever

• 34:59Speaker 00

  read if you're like a social scientist or someone working more practically. It's a little less math heavy than the Wasserman books.

• 35:07Ryan Peterman?

  And I think there is a lot of value in big technology and understanding some statistics or doing it rigorously, because I've been in so many AV test review meetings. And I think a lot of people who kind of just enter the industry and, you know, they, they see the UI and they go, I got a green bar here. Please give me the approval to ship my, my code path. But actually, if you kind of dig in, you ask some whys and you're like, wait, there was, it was red yesterday. Why is it, you know, what's going on here? The, the understanding is very superficial and people are just trying to, they're just trying to move forward, whether or not it's actually statistically, you know, beneficial across the user base. So.

• 35:52Speaker 00

  I mean, that's a, I mean, ironically, it's a great example of sort of everything we talked about today summed up as like a story happened when we were trying to get Deltoid 3 to ship out. You know, at that point, Deltoid 1 was still the default. And there was a person who came to us and this person was actually like, otherwise great. And I loved interacting with them, but this interaction was really, I was like, oof, this reflects a lot of cultural pathology at our company where they're like, hey, I can't let you ship Deltoid 3. And I'm like, why? Why can't, why can't we ship it? And they're like, our holdout goes from being statistically significant win for the company to being not statistically significant in Deltoid 3. And I think it's a regression. And I was like, all right, it might be regression, but maybe it's also the truth. They're like, I actually don't know which it is, but you guys are going to wait until

• 36:34Speaker 00

  the half is over and we've decided we hit our goal and then you can ship Deltoid 3. And I was like, oh, but wait. I was like, your, your win is so close to the border between you did nothing and won that literally like mild tweaks in our code have turned it off. I'm like, I feel like that's not a win people be so concerned about. And especially this notion that like, you're almost significant, so therefore you failed or you're just barely not significant, you know, your win or fail. That I think is this super dangerous culture. I mean, one of the worst things I ever saw was a team that I manage of Sweets. And one of the things they told me was they're like, hey, you know, we have to do this thing because we made a goal.

• 37:18Speaker 00

  And I was like, okay, but we're going to do it next half. So they're like, oh, definitely on July 2nd, we're going to delete this code. And I was like, wait, then why are we shipping it? They're like, well, because we can't miss our goals. And I was like, I feel like our goals should not be written in a way where shipping a thing we intend to delete literally a day later is a success. And they're like, yeah, that's fair. And I was like, you're freaking, I mean, like, don't do this. And eventually, you know, I convinced them not to do it. It's like, I'm the manager, I can decide the ratings. We don't have to just like, do this. But people really like firmly believe this. And I think the statistics thing with the promise, like, a lack of understanding of statistics bleeds into other weird pathologies of how people are evaluated. And the two together become like extra dangerous.

• 38:02Ryan Peterman?

  Coming to an end, just like a few questions, kind of like reflecting on your career so far. Do you have a regret that maybe other people could learn from?

• 38:11Speaker 00

  Oh, yeah. People ask me this so much as over the years, especially as it became like a more senior manager and then a director. But I like, you know, have a keen answer because I've been asked this a million times. I for like my first several years had Javi as my skip. Um, for people who don't know him, he's currently, I think, the COO of Meta. But you know, he was first the head of growth and then the head of growth and ads. And then I think now he runs like roughly 50% of Meta. Um, but like, Javi was my skip. And every time my actual manager, this guy named Danny, would be like, Javi wants people to come to his office hours. No one shows up and he feels like it's a waste of his time, so I should go. And I basically just, I was like, well, I don't have anything like actually that valuable to say to Javi. So I'm just going to waste his time and I don't want to be the person who wastes his time.

• 38:56Speaker 00

  So I never went. And I look back and I'm like, because especially as I started running office hours, as I had a larger work where I couldn't do like one on ones with everyone and had to do office hours and people wouldn't co. And I was like, man, I wish people would come to my office hours. And like, I, Javi was also feeling this way and he's like, man, I wish I would give anything for one of these people to show up. But like, I just never showed up. And I look back and I'm like, first of all, this was an amazing opportunity that I wasted. But in addition, it's not just that I wasted it for myself. I also probably just like made his life worse off by not actually getting him to be able to take advantage of this thing he was offering us. And so I was like, this is a decision that basically harmed both parties that I did out of fear.

• 39:40Speaker 00

  I'm sure there's probably a million other examples of me doing this that are not as clear in my head. But this is one where like, you know, Danny probably came to like our team once a week and told us to sign up and none of us ever did. And I think of this every time where I'm like, listen, if like you know something important about the future of this org or this team, or even if you're just like honestly have any questions at all. If you have a leader who's showing up and saying, I want to hear from you folks, go. I think especially like if you're a more junior IC watching this, like I think it is hard to understand how painful it is a director and above to actually know what it's like on the ground. You're just like so removed from being an IC3 and IC4 anymore, just so removed.

• 40:24Speaker 00

  And not just from their place in life, but also like what's happening to them, what's true in the code base, what the team dynamics are. You know, like, you know, there were points where I had like a manager managing managers, managing managers, you know, there's just so much interaction. I think people way, way should more often, if a leader says like, please show up and talk to me, do it. Um, aside, obviously it's like you come and say something weird, you know, that can be bad, you know, don't show up and be like, I want you to tell me how to get promoted. Uh, it's probably like not your greatest first conversation, but if you're like, hey, I think this part of our org could be better. Could you help me? Leaders love that. That's what they want. And like so many people are afraid to do it. And I think it basically makes all parties for ourselves.

• 41:05Ryan Peterman?

  And then last question for you, with all the experience that you have now, if you could go back to the beginning of your career and give yourself some advice, what would you say?

• 41:14Speaker 00

  It's tricky because I think I am a person with very strong opinions, but I also think I can be more self-conscious than I should. And do you think that like have more confidence that you can do bigger stuff? And that as long as you hold yourself to a high level of discipline, the bigger stuff is really possible? I think like when I was younger, you know, basically at every step of the way, like from high school students to college students all the way, I think I tended to way too often like cast doubt about what I could do. Um, and I think, you know, that is the single like biggest set of mistakes I've made. It's just this repeated pattern of like not being ambitious enough or not believing something was tractable. And I think I've gotten a lot better. I see why I don't like the defeatism in other programming language communities.

• 41:58Speaker 00

  But like, um, I think it for me is like this thing that sort of I think, you know, really could have been better. And I think it's true for a lot of people. I think it's a lot of people like they, they over convince themselves of the greatness of other people and under convince themselves of how much they can achieve. And that combination means that they just like way less try to do risky things than they could have. Um, and I think they like, especially I think until we interacted with enough people who have succeeded, I think it's hard to realize like how often they're like not actually doing that much better than you. They just tried and you didn't try. Um, so I think that is probably like, you know, the, the biggest thing that if I could go back and tell like a 20 year old me, um, probably is that.

• 42:40Ryan Peterman?

  Thank you so much for your time. I, I really appreciate it, John.

• 42:43Speaker 00

  It's been a real pleasure and thanks for having me.

• 42:46Ryan Peterman?

  Thank you for listening to the podcast. It's a passion project of mine that I've really enjoyed building. Another passion project that I've been working on, kind of in secret, is building an ergonomic keyboard that I wish existed. And I finally have a prototype, so I'd love to show you what we've built. It's ultra low profile and ergonomic, and I couldn't find anything like it on the market. So that's why we built it. I'll put a link to the keyboard in the description. You can take a look and learn more about the project there. We could definitely use your support. Also, if you have any feedback from me about the show, I'd love to hear it. Comments on YouTube have led to guests coming on like Ilya Gregorik and David Fowler. I wasn't aware of them until someone dropped a comment. Also, feedback in the comments helped me learn to reduce the number of cliffhangers

• 43:31Ryan Peterman?

  in the intros. So your comments definitely make a difference. Please keep letting me know what you'd like to see more of in the show and I'll see you in the next episode.