Meta Senior Staff (IC7) Eng's Honest Demotion Story

In this episode, I talked to Igor, a senior staff engineer who has worked at Meta, Google, and Cruise. We discussed his experience of wanting a demotion at Meta and the challenges he faced in that process. 𝗣𝗼𝗱𝗰𝗮𝘀𝘁 𝗹𝗶𝗻𝗸𝘀: • YouTube: https://youtu.be/i1iBweuOQI4 • Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835 • Transcript: https://www.developing.dev/p/meta-senior-staff-ic7-engs-honest 𝗘𝗽𝗶𝘀𝗼𝗱𝗲 𝗹𝗶𝗻𝗸𝘀: • Igor's post: https://www.linkedin.com/feed/update/urn:li:activity:7401415295409700864/ 𝗧𝗶𝗺𝗲𝘀𝘁𝗮𝗺𝗽𝘀: 00:00:00 - Intro 00:00:37 - Why he wanted a demotion 00:07:32 - Why Senior Staff at Meta was different 00:16:01 - Meta vs Google culture 00:19:09 - Downleveling at Google 00:23:17 - Why he's willing to be transparent 00:25:11 - Best quality of life eng level 00:30:42 - Senior Staff promo at Google 00:42:27 - Mentorship stories 00:43:11 - Biggest career regret 00:46:46 - Advice for younger self 00:49:06 - Outro 𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗜𝗴𝗼𝗿: • LinkedIn: https://www.linkedin.com/in/igorts/ • ML basics youtube videos he made: https://www.youtube.com/playlist?list=PLVcptlT8D7DgN5FtLMFUdHb5pJXW1g0YL 𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗥𝘆𝗮𝗻: • Newsletter: https://www.developing.dev/ • X/Twitter: https://x.com/ryanlpeterman • LinkedIn: https://www.linkedin.com/in/ryanlpeterman/ • Threads: https://www.threads.com/@ryanlpeterman • Instagram: https://www.instagram.com/ryanlpeterman • TikTok: https://www.tiktok.com/@ryanlpeterman

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Then I asked the management, can I actually drop a level?

• 0:04Ryan Peterman?

  This is Igor, a senior staff engineer at Meta, Google, and Cruise, who wanted a demotion.

• 0:09Speaker 00

  You cannot still justify my level with that project being completed.

• 0:14Ryan Peterman?

  He failed to get the demotion at Meta, and going back to Google at a lower level was harder than expected.

• 0:20Speaker 00

  They don't have a process for bringing people at a level down, but they made it possible for me.

• 0:25Ryan Peterman?

  We went over that experience and some transparent discussion across levels. Which engineering level has the best quality of life?

• 0:33Speaker 00

  There are many things that can go wrong when you share stuff like this.

• 0:37Ryan Peterman?

  What makes you willing to share? Here's the full episode.

• 0:46Ryan Peterman?

  You mentioned in a post, a pretty famous post that I'll link so people can see, that Meta didn't have a process for demotion, and you were looking for demotion. Can you talk about what led you to wanting a demotion at Meta?

• 1:02Speaker 00

  Yeah, so when you join a big company, Meta is not the exception here, but you're joining a big company at a very high level, like senior staff. There are certain expectations from your performance, right? Come performance review, like at Meta it's called PSC. You're being judged against other senior staff engineers in the organization, right? And Meta, as of a year ago, they started to lay off people, like, you know, they're starting doing the Amazon thing essentially, like, let's lay off the 10% lowest performers. So what it means is that you have at most one year to ramp up

• 1:43Speaker 00

  to be comparable in performance to other old timers in the company. And also, what does it even mean to be in, you know, E7 in the company like this, right? You need to be an expert. A big expert in the field. You need to know a lot of people. You need to understand the infrastructure that you're working with very deeply. You probably need to be easily familiar with like any piece of code that you, your team is working with. And also you need to know all the surrounding, like what all the surrounding teams are doing. And you need to know all these people, like all core people around you.

• 2:26Speaker 00

  And they need to know you and they need to trust you. Right? That's a very difficult thing to achieve within like a relatively short period of time. I initially saw that, okay, when you were so again, joining Meta day one, I know less than an intern sitting next to me, you know, who spent maybe two weeks, who was hired two weeks prior to me, right? So like I, I start from like level zero. And then hopefully like you, you be able to get to something that like L3 would perform. So like you, you can take some very easy task that under supervision from other folks, you can kind of get promote yourself to like a junior engineer, right?

• 3:12Speaker 00

  Then, then you do some, some more stuff, you learn additional things, you can get promoted to like a less junior engineer, like a E4. And then slowly you ramp up, you know, to more and most of like higher levels essentially. Like you start from zero and you climb the ladder as fast as you can. Yeah, so you, you ramp up yourself to, you know, the level that you're supposed to achieve. And you know, the more senior you are, the more levels you need to jump through. And I believe that within this year and two months that I spent in Meta, maybe if I'm generous with myself, I maybe achieved like E6. So I, I don't feel like I reached E7.

• 3:56Ryan Peterman?

  What did you see that made you think you weren't living up to the expectations?

• 4:00Speaker 00

  It's self judgment, mostly, like, you know, I, this scale where we, we call numbers, it's not like quantum states, right? There are many things in the middle, right? It's a, it's a continuous scale, so.

• 4:16Speaker 00

  It's very hard to say where you are on the ladder. But again, comparing myself to other senior staff engineers, the work that I knew, I felt like if anybody gonna be on the chopping board, that's gonna be me. Plus, so that was one thing about this ramp up process. And the second thing, I also noticed that I really enjoyed doing the coding stuff, like just sitting down debugging things. Like that's what I really loved about the job. So during my ramp up, when I was around E5, E6 territory,

• 5:01Speaker 00

  that's what I really wanted to do, right? I felt like, as a, you know, as someone who was programming from age of 12, I really liked coding, debugging, designing, mentoring more junior people. But like a senior staff engineer, it's more than that. It's like a leader who, you know, spends most of the time in meetings and design docs and touches much less code. And it's a slightly different type of work. And just by going through this process, I realized that actually I want to be,

• 5:41Speaker 00

  I want to be in the E5, E6 territory. And then I asked the management, can I actually drop a level? And I understand that for the company, it's a difficult thing to implement because I already have some granted stock, I have a certain compensation package. And so how do you, like, you know, how do you execute that? It's not a well polished process. I know that they can switch levels when you go from like one title to another. So like, let's say you were a director and you want to become an AC. So they do have some of that, like, demotion process. And frankly, I don't know how they do it.

• 6:26Speaker 00

  But yeah, like staying in the same job category and dropping level just...

• 6:34Speaker 00

  Like they said it's not possible.

• 6:37Ryan Peterman?

  I see. So when you asked if it's possible for demotion, your manager went to HR and kind of the the result was just it's just an impossible thing to do.

• 6:47Speaker 00

  You know, I asked the question and I got the answer, no, maybe if I pushed harder, if I like went to talk to the, you know, VP or something, they would have made it possible. But at the same time, I,

• 7:02Speaker 00

  you know, I also felt like I'm too far outside of my comfort zone. And before coming to Meta, I worked a little bit at Cruise and before that I worked at Google for fourteen and a half years and that was my comfort zone. I just had this option that like the easy way out essentially, right? It's had the recruiter who was contacting me periodically, like I already to come back to Google. So this time I just said, yeah, let's talk.

• 7:32Ryan Peterman?

  When you were at Google and at Cruise, you were at these senior levels as a senior staff engineer? What's the difference that makes it so that you can perform there, but not at Meta?

• 7:45Speaker 00

  So when I came to Google, I started from the lowest E3, like L3 level, so junior software engineer. And slowly, over the many years, I raced through like promotions, many times failing. Like they could promote it and then, you know, a year or two later, asking again and then climbing the ladder. My last promotion to senior staff happened because I was able to accomplish something big in the project that I worked, I worked in that project for several years, so I was already like an expert.

• 8:23Speaker 00

  I knew everybody, right? Everybody knew me. And I was able to build something that I can be proud of. But again, coming then, then leaving Google and going to Cruise, it was also very difficult for me to ramp up there because you're coming, starting from zero. And the first year was very difficult for me in Cruise, but over time I found like a relatively safe, like, you know, my my found some comfort zone there until Cruise as a company.

• 9:00Speaker 00

  Went bad, like they had the accident and then they had layoffs and then like people were just quitting all the time, like losing people. So I decided that I want to leave as well. But before, before the accident at Cruise, I actually was quite happy. I was able to do some coding. I was able to do like mentoring of other folks and I was able to also like act as an Uber TL a little bit, like, you know, guiding the team on things that, you know, they need to do.

• 9:35Ryan Peterman?

  You successfully ramped up the senior staff at Cruise and then at Meta it was a bit harder. What, what were the differences between those two ramp ups in your opinion?

• 9:45Speaker 00

  Cruise was a smaller company, so less infrastructure to learn. The team is like less crowded space, I would say, like you, you can easily carve out space for you to grow into, like, you know, like there is lack of people and you can say, okay, well, this project needs someone to work on and, and you can just go and work on, on the stuff. At Meta, I felt like it's quite crowded, like especially among the senior folks.

• 10:20Speaker 00

  It felt a little bit like they have too many, frankly.

• 10:25Speaker 00

  At least in the work where I worked. So I don't know about all of Meta, but it, it felt like the space is a little bit crowded. You need, you need to find scope for yourself.

• 10:38Ryan Peterman?

  Is there anything that you, you would have changed in your onboarding process that could have made things different?

• 10:45Speaker 00

  Probably. Again, I didn't switch companies that many times in my career. I started at Google, spent so many years there, then I only switched Cruise and then Meta. So I'm, I'm not very well experienced in switching companies. And essentially, you know, this whole ramp up process is new to me and I also felt like The management at Meta also doesn't know how to import senior folks, because most of the senior folks in the company, they grew within the company to those senior levels. Even if they came to this specific org, you know, from another org within Meta,

• 11:30Speaker 00

  but at least they already have the Meta knowledge, you know, like they know how to run a job in the cluster, right? I think I spent a lot of time just reading docs and trying to like build this foundational knowledge. I think I spent too much time doing that. Like it is much better to just get your hands dirty and just try doing stuff, like, you know, and then like doing those things in parallel. Like you build something, accomplish some tasks, and also you're learning in parallel. And I did more of the learning part, less of the...

• 12:12Ryan Peterman?

  The building?

• 12:12Speaker 00

  Yeah.

• 12:13Ryan Peterman?

  Is there something that you wish was done or maybe something that would have helped you from the manager's side?

• 12:20Speaker 00

  Just guiding me, right? If they say like, this other person just ramped up, like, you know, three quarters ago, and here's what they did, I would have followed that recipe. We just didn't have the recipe. Like, I'm not, I don't have any complaints against my management. Like, they, you know, they tried to support me the best. They just also lacked this expertise of like ramping up with senior folks, you know? So it was a learning experience for all of us.

• 12:53Ryan Peterman?

  I've heard in conversations with other people as well that, I guess, job mobility as a senior IC actually becomes progressively riskier or maybe scarier, I guess, because you get so used to your existing org and all of that. Is that something that you've seen in other people or like peers and as well? And that senior IC job mobility is a lot tougher?

• 13:21Speaker 00

  I've seen folks at Google that left Google and then came back a year or two later saying that it didn't work out well for them.

• 13:33Speaker 00

  I, yeah, it's, you know, people don't share things like that openly usually. So it's, it's much harder to, to hear those stories. You, you hear the success stories, you don't hear the failures. Um, also, I think that if I kept trying, like instead of going back to Google, I went to, I don't know, some other company. Um, I would be much like, you know, more experienced in ramping up. And then maybe, maybe if you do this often enough, uh, it becomes a habit and, and then, and then you can do it easily.

• 14:08Ryan Peterman?

  You mentioned that you, you thought you weren't meeting expectations based on your own judgments. Um, I'm curious, did you ever get any feedback from your manager or anyone saying, hey, you need to do more for your levels expectations?

• 14:22Speaker 00

  Yeah, I did get feedback, I just, you know, don't necessarily want to openly share all of that. Uh, but, uh, yeah, I, I was so in the part like just the few months leading to my living meta. I was working on a project that, uh, initially I thought it will take me like two weeks to accomplish. It was like a small thing that, uh, I thought would be easy to do. It turned out to be actually a lot more involved and a lot more complicated. And I felt like, uh, even if I finish it, uh, which I almost did, like I, I almost brought it to completion. Um, even if I finish it and launch it, everything's successful, it is still not an E7 level project.

• 15:08Speaker 00

  It, it, you cannot still just justify my level with that project being completed. Probably it's my fault in like which projects I pick, uh, you know, or how I underestimated the complexity of the thing.

• 15:24Ryan Peterman?

  Cause you're going back to Google, um, are you going back to an org where you have all the existing context and, and relationships and all that?

• 15:34Speaker 00

  No, no, everything will be new to me, new people, new infrastructure, new, new everything. Um, but at least I know, you know, I know how Google operates, I know the. You know, the culture, there are differences in cultures also between the companies. And I think that Meta also like in terms of internal culture at Meta, I didn't feel it like it fits me the best.

• 16:01Ryan Peterman?

  What's the biggest cultural differences between Meta and Google?

• 16:05Speaker 00

  Meta tends to to set up very ambitious goals. They give you like oftentimes they will give you very arbitrary deadline. Saying like, okay, this project, you need to finish it within like, you know, by September fifteenth, whatever, like one month from now. And then everybody works hard, like there is a lot of pressure, like you need to constantly send the updates to the leadership, how the project progresses. Comes the date, the deadline, the project is still unfinished and it just keeps dragging on. And so like, and everybody is fine with that. Again, I don't know, I haven't seen all of Meta, I've seen the specific work where I work.

• 16:45Speaker 00

  And it feels like, so then what was the purpose of setting this aggressive deadline? Right, then like, you know, are we like, you know, you can do it once, but like after five times. Going through this project, through this like artificial pressure with like non-realistic goals, then the people just say like, okay, you know what? Five P M, I'm going home. Like, I'm not, I'm not gonna try to to work hard because I know that there is like. This this whole pressure is artificial. Right? I think Google is much more reasonable in that regard. Like if there is a deadline, it's probably for good reasons and.

• 17:28Speaker 00

  People would like would work hard, but usually there will not be pressure. Again, talking about Google like as of ten years ago, I'm not sure that it's still true today, but. You would be pressured to to fix something or to accomplish something when there is really.

• 17:46Speaker 00

  You know, exceptional case for that. Like you wouldn't be pressured to to work under pressure.

• 17:55Speaker 00

  You know, for years.

• 17:57Ryan Peterman?

  Yeah, I mean, I get the sense that the industry as a whole is kind of becoming a little bit more intense when it comes to execution and deadlines. I've heard some people saying that Google as well has felt a little bit of pressure too. But hard to say, it depends on the org, I'm sure.

• 18:15Speaker 00

  Yeah, could be. And again, I didn't feel personally much pressure at Meta, but just talking to other folks, seeing how they work and operate. And as I said, there is this cultural thing where the leadership wants like updates and everything. And then, and then the people on the ground, they actually kind of dismiss it essentially. Like it's, it's like the, you know, it almost feels like a elementary school, like where the teacher is yelling, but kids are still playing. Like if you're yelling too much, it's, it's like, it, it's, it stops working.

• 18:56Speaker 00

  If you, if you constantly put pressure on your people, it just doesn't work anymore. Like you say, so what do you do next? You start laying off people? Yeah, it might work for to some degree, but again, people adjust to everything.

• 19:09Ryan Peterman?

  That's crazy. You mentioned that you, when you reached out to Google too, you, you explicitly asked for kind of a demotion or going back as a L6 when before you were in L7. Was that a challenge or was that just a very straightforward process with the recruiter?

• 19:25Speaker 00

  It was a challenge for the recruiter to also make this happen because they have a process for bringing people in back at the same level or even like level up. They don't have a process for bringing people at a level down, but they made it possible for me. Again, I can totally see how like a person comes back and says like, I want to be a level down. It's like, there might be some red flags, there may be, you know, things that I'm not telling. You know, it's a gamble for a manager to hire a person like that. Also, just by giving me offer at the level down, they understand that it will be, they will not be able to match my compensation, you know, to previous place.

• 20:11Speaker 00

  So, so like, would they even accept it? Right? And so, so I had to like. And assure my recruiter that, yes, I'm happy to accept the offer at the lower compensation. Like, you know, a lot of people will not do something like I did because it's like a significant drop in compensation.

• 20:33Ryan Peterman?

  You mentioned stuff that people might be hiding from the recruiter. What comes to mind when you you mention that?

• 20:41Speaker 00

  I don't know, maybe you maybe you did something in the company that, you know, you like a fireball offense or something like, like, I I don't know, like, if if you're a manager and somebody comes to you with like, you know, saying like, yeah, previously I was this level, now I want to be level below, please hire me. It does sound fishy, like, why don't you, like, what's wrong with you? Like, you, like, you know, there are so many companies out there, like, why, why would you come back to to this company? Like, why don't you try something elsewhere? Like, I I don't know what questions popped up in my hiring manager's head when he saw this.

• 21:25Ryan Peterman?

  Google's current process, do they do host matching? Or what was the, like, did you meet up with the hiring manager beforehand or just through the recruiter?

• 21:34Speaker 00

  The recruiter sent me a few openings and asked me which ones sound interesting. I spoke to a few hiring managers.

• 21:45Speaker 00

  One was sound like sounded like the best match for me. And then I spoke to a few engineers on the team, then I spoke to the manager's manager. And that was it. So another nice thing for me is that they didn't require me to re-interview.

• 22:03Ryan Peterman?

  Oh, interesting, but it had been like three years, maybe?

• 22:08Speaker 00

  Over, yeah, three and a half years. So, yeah, they didn't ask me to re-interview. Uh, actually, that was another interesting thing that they said like if if you're coming back as E7, uh, you definitely don't need an interview because you were already L7. But if you're going to be like L6, then you might need to do a coding interview because who knows if you wrote code that, you know, while you were out.

• 22:35Ryan Peterman?

  That's kind of funny.

• 22:36Speaker 00

  Yeah, I got an exception from coding.

• 22:39Ryan Peterman?

  I see. That's funny. You got promoted out of competence in the lower levels.

• 22:46Speaker 00

  Yes, but it's it's understandable when people like go to higher levels, they write less code. And maybe they get rusty. Who knows?

• 22:56Ryan Peterman?

  It's kind of crazy that you could boomerang after had been three years or more.

• 23:01Speaker 00

  Yeah, to be fair, I don't know if that's a general Google's policy or it was just a special case for me. I don't I don't have any friends that did this recently to to to understand if if like they did it for anybody else.

• 23:17Ryan Peterman?

  You mentioned earlier that, you know, a lot of people don't share this kind of stuff about emotions and these types of things. And I'm curious what makes you willing to share?

• 23:29Speaker 00

  I'm generally a very open person, like I like sharing my my life experiences, positive and negative with friends and the, uh, you know, I, yeah, I generally quite open. And also I'm quite confident. That's another thing like you wouldn't share if you worried about your career, if you worried about, you know, there are many things that can go wrong when you share stuff like this, right? So, uh, I'm I'm very privileged, essentially. I don't need the work visa, so I can live in this country uh, without being attached to a certain employer.

• 24:12Speaker 00

  Uh, my spouse has a health insurance. My kids are already grown up and the off to college and I don't have mortgage on on my house. So like I I can afford not to work for a few years. I can, you know, I don't, I I I'm fine with like, if let's say I I said something in this chat right now and and Google says we don't want you for some reason, like I'm totally fine. I will just.

• 24:44Speaker 00

  So that's a position of privilege, right?

• 24:47Ryan Peterman?

  Definitely. I mean, you're you're free. Yeah, you're entirely free. You're your own person, which is awesome.

• 24:53Speaker 00

  Yeah, so that's a that's a position of privilege that many other people wouldn't have. So I if if another person asked me, like, should I post about my challenges at work, I would probably say no.

• 25:08Speaker 00

  Unless you feel so so safe and secure.

• 25:11Ryan Peterman?

  And you mentioned at Google, you were asking for promos, you went from their lowest level up and you were pushing and pushing for promos. And that's a very different mindset from now, which you're kind of pushing for the motion. I'm curious. Is there did something major change that made you not motivated to go for promos anymore in your career?

• 25:36Speaker 00

  Essentially, if you're in the senior engineer, right, to E5 or E4, you don't even know, like, what's the life of like two levels above you looks like. Right? You you probably you can imagine what what happens at like one level above you, but but it's it's very hard to see like what what does a principal engineer do? Like, most people don't know. And I don't know. By going through this, like, I realized that my happiest time was when I was like working in a relatively small team, doing a lot of coding, debugging, designing, mentoring more junior folks. That's that's where I felt the happiest.

• 26:22Ryan Peterman?

  Let's say money is not a constraint at all, and it was just which engineering level has the best quality of life? In your opinion, which one would you say?

• 26:36Speaker 00

  Senior engineer like E5, L5. Yeah, that's probably least pressure. You're still shielded by, you know, probably like if you're working on a team, you probably have an ATL who is E6 on the team, who, you know, sends all the updates to the upper levels and shields the lower, like the team from all this stuff. You have the management, also like the lower level managers who shield you from all this stuff. So you can just do the stuff that, hopefully you enjoy what you're doing. Like, not everybody enjoys doing work. Like, a lot of people do it just for money.

• 27:20Speaker 00

  But I, personally, I was programming since I was 12 years old. So I really like this stuff. I got into this job because I love it and it happens to pay well. So, but I would do it even if it wasn't paying well. Like, a lot of people come to software engineering because, just because that's the money thing.

• 27:41Ryan Peterman?

  So then, if it was easy, as easy to do, would your ideal situation be two levels of the motion?

• 27:50Speaker 00

  That's too extreme. I think I can, I think I can still enjoy being like E6. I'm quite comfortable with. And maybe I haven't been E7 long enough to, to like, to get comfortable in the, in the role. Like, maybe if I just kept doing it for some time longer,

• 28:12Speaker 00

  I would have started enjoying it. Yeah. Like, you know, one thing is like, was I, was I really, like, you know, imposter syndrome is something that everybody has, regardless of what your level is. I'm sure Elon Musk has imposter syndrome. So, you know, I'm questioning myself, like, was I really? You know, qualified to do, to be E7 and my answer is like, partially yes, but not fully. It's like a multi-dimensional thing and in some dimensions it probably was good and then some dimensions was not as good.

• 28:49Ryan Peterman?

  You mentioned working with other peer ICs that were also really high level. Um. Is there skills that you saw that they had which would have closed the gap for you personally or um something that you thought made them so strong?

• 29:04Speaker 00

  It's often a mistake to compare yourself against a group of people. Like that's what uh gives you a lot of uh imposter syndrome where you're saying like, oh, those people around me, they're so smart. They are so good at talking. They're so good at doing presentations. They're so good at communicating. They're so good at leading. But turns out that it's like one person is good at this, the other person is good at this and like you, you're comparing yourself against like a team of people. Uh and so that's uh, that's always uh, you need to be careful uh not to make that mistake. So yeah, I, I, I, I think I, I, you know, given enough ramp up time, I would have been able to reach like full productivity and the

• 29:47Speaker 00

  being useful to the company at E7. I just felt like I, like I, I was not enjoying the ride essentially. And I didn't have to like, uh, and, and, and I debated a lot before, before making the decision to leave, right? I was talking to friends. I was talking to family and pretty much everybody was telling me like, are you crazy? Like you're not laid off yet. You're not, nobody showed you the door. Why would you do something crazy like that? And, and even if you're leaving, like you don't enjoy it, why, why won't you try E7 elsewhere? Like, why do you need to go level down? That's, there is a lot of peer pressure. And my answer to that was just like, yes, I can try that. I can do that, but I like, I don't want that.

• 30:31Speaker 00

  Right? It's like, why, why, I, I'm capable of running a marathon, but I don't like running a marathon. I don't like running. And so why, why would I do that?

• 30:42Ryan Peterman?

  You mentioned that you kind of rose through the ranks at Google. I'm kind of curious, what was the project that got you promoted to senior staff at Google?

• 30:52Speaker 00

  It was a project in ads.

• 30:55Speaker 00

  Ads builds a lot of machine learning infrastructure because, you know, the ranking of ads is essentially a recommendation system. And I was working on this machine learning infra. And when I came to that team, they were doing training on CPUs, not on GPUs, but on CPUs. And TPU was a new thing, new hardware that Google was developing. It was about to be released like maybe a year or two, you know, when I joined the team.

• 31:35Speaker 00

  And my role was to make sure that this ads training infrastructure can run on those TPUs. And additionally, when you train machine learning models, the scale matters and the utilization matters. And there are lots of nuances that can, like, you know, your model may be training, but things go wrong.

• 32:07Speaker 00

  It's online training, which is also like something that most people don't know. Like LLMs don't usually train in online training mode. So there are lots of experience on the team. Like the whole infrastructure that existed within that team was built for the CPU training. And it was polished over, I don't know, over 10 years. It was polished to be super reliable, super like you well monitor, the unit tested. Everything is like very rigid, very like polished, right? And here I am coming and building a completely new piece of infrastructure to run on TPUs.

• 32:48Speaker 00

  And it it has to be as good as as they all, you know, as polished, as monitored, as tested, as reliable. So it was a big project and, you know, it succeeded. Yeah, it's something to be like, I was proud of what I built. So that got me promoted.

• 33:08Ryan Peterman?

  What was the operating model of that project? Were you TL or were you kind of writing a lot of it yourself? Or how would you describe the execution?

• 33:18Speaker 00

  Yeah, initially it was like a quite exploratory project. So it was just two pure people. It was the TL plus one person to building like a prototype. Plus a lot of people from other teams helping like the TPU team, the compiler team, they like, there are lots of other people who were helping us succeed, like the TensorFlow team, we were writing TensorFlow back then. You know, there's so it's also like where you need credibility, like that's, that's why knowing people and them knowing you is very important. Like trusting you. If if you don't have that, you cannot succeed.

• 33:59Speaker 00

  So it's a really like cross team collaboration, a very big project. And again, hardware like this, you need to decide how many of these chips you want to order. And they will be delivered like 18 months from now.

• 34:17Speaker 00

  Before and right now you don't have any, it's not like you can like try out and and see. So like, and those chips are super expensive, right? So it's a, it's a very risky thing to do for the company. And yeah, it's like

• 34:36Speaker 00

  a lot of risk taking, a lot of like leadership skills that they needed for for doing stuff like that.

• 34:44Ryan Peterman?

  So you were on the, well, kind of the product team, not actually product, you were machine learning infra for the ads org. And then all the underlying infra teams were helping collaborate with you.

• 34:58Speaker 00

  Yeah.

• 34:58Ryan Peterman?

  You mentioned compilers, maybe some training infra, other infrastructure teams.

• 35:04Speaker 00

  Yeah, and once the thing started working a little bit, the team grew quite a bit more. A lot more people joined the effort and like started like working polishing. And then again, like a first generation TPUs comes in. And then they already tell you like, we already designed the next generation who will be coming, you know, next year. So you're already rushing to, you know, adjust your infrastructure to the next thing. And that's like pretty much every year, you're like upgrading your infrastructure.

• 35:34Ryan Peterman?

  How did you test the initial builds of this infrastructure? If you didn't have the chips to begin with?

• 35:41Speaker 00

  A lot of things were just about like input, reading and processing, which you can do without having the chips like the, the difference between CPU training where you have, you have the CPU. And Intel CPU, they can do floating point arithmetic and integer arithmetic in parallel. So, and, and the floating point arithmetic is relatively slow. So input reading and processing was free on the same CPU.

• 36:16Speaker 00

  Going from that mode of operation to you have a relatively

• 36:22Speaker 00

  weak computer with eight TPU chips. Each one of those TPU chips is like order of magnitude faster than the CPU that you had before, if several orders, I think it's like two orders of magnitude faster. Now there is no chance you can do the input reading on the host of the, of those TPUs. So you need to like build the, the input processing pipelines and everything. But that's something you can test outside of, like without having TPUs. Actually, we did, many times we did this test where people say, next TPU will be like two times more performant than the previous one.

• 37:08Speaker 00

  And a nice test to test your infrastructures to say like, okay, we will just remove all computation from our models. Just let's see how fast we can, you know, feed the data in, take the outputs out, and process this whole thing, like how fast your system can run if the CPU was infinitely fast. And that's a nice test to do, to see that, you know, you don't have bottlenecks, and you do find a lot of bottlenecks all the time. Like, things that previously were like, like, things that schedules like which data to train on,

• 37:47Speaker 00

  it's, you know, previously was never like an issue, and then suddenly this is your bottleneck in this model. And many, many things like that.

• 37:58Ryan Peterman?

  So if I'm understanding correctly, the, the TPU consumes data orders of magnitude faster than the CPU one. So everything around the processing unit, like the data loading and maybe, I don't know, the scheduling and all the other things that are around the TPU needed to be scaled and tested. And that's a lot of what you did to get promoted, is that right?

• 38:22Speaker 00

  Yeah. Plus, you know, there are always like funny things that you get where like certain resources are more available than other resources. For example, so you, you know, you're training on some insane amount of data, petabytes of data. This data is stored on spinning disks, because there is like not enough SSDs in the world. So spinning disks, if you look at the history of spinning disks, they're getting bigger in terms of storage space. But the speed at which they rotate is staying constant. And the speed at which the head moves is staying constant.

• 39:01Speaker 00

  So if I gave you previously, let's say like we go back in year 2010, a typical disk would be like maybe 200 gigabytes. And it has a certain throughput of how much it can read. Now, fast forward to today, the typical disk is like 6 terabytes. So your data center has a lot fewer disks to store the same amount of data. But this throughput is like an order of magnitude less.

• 39:32Speaker 00

  So and and and you're coming to the people responsible like on building data centers. And you say, I need more disks, not for storage, I need more disks for throughput. And you usually would hear an absurd thing that says, disks are very cheap. Like compared to everything else in the data center, super cheap, but you cannot get them. You know, you cannot easily go and buy like a few thousand disks and and and easily install them. Like you need racks, you need power, you need, you know. Oh, yeah, it it it just, it it's similar to how like during COVID, we all ran out of toilet paper. Right? Like the stores just couldn't couldn't like, like, why would you run out of toilet paper?

• 40:17Speaker 00

  Right? It's just because it's bulky item that the stores cannot easily store on the shelves. Right? So that's that's roughly.

• 40:25Ryan Peterman?

  You you mentioned having to predict eighteen months out, how many TPUs you'd need. I'm curious, when you look back on that prediction of how many you needed, did you over order or under order?

• 40:37Speaker 00

  Usually under. We try to be relatively conservative because the main goal is to make money. Right? It it like with L L M's, if you look at this current world of L L M's, they just order more like we want more gigawatts of data centers. They don't even really like none, none of these companies profitable with L L M's, right? They're losing money and they all fine with that. But when you have a mature business like ads, you you need to be making money on that. So if you if you wasted, you know, if you over provisioned and this goes to waste, then it's not good. But then it constrains what kind of models you can train, like also like models don't stay the same.

• 41:21Speaker 00

  So when you're saying eighteen months from now, we'll not be training what we're training today. But you don't know what you're going to be training, like, you know, the attention mechanism wasn't in use. Ten years ago, like, and suddenly it's it's it's needed. So like, you you your chip might not be even capable of doing that. Uh, so TPUs are quite specialized hardware. So if they are not as generic as GPUs. Um, and there were certain things that we had to work around a lot of times, like, uh, where certain functionality was very difficult to implement.

• 42:01Ryan Peterman?

  Like what?

• 42:02Speaker 00

  Yeah, embedding lookups, uh, in certain types of embeddings where, essentially, the the chip is very good at doing matrix multiplication. And accessing memory by big chunks of memory. And then embeddings are things where you need sparse uh, access, like, you know, random access, essentially. And that's that's that's difficult to do in a chip.

• 42:27Ryan Peterman?

  Yeah, one thing I want to ask, because you said you enjoy mentoring others. And I was curious, do you have favorite, maybe stories of mentorship or favorite advice that you like to give when you're mentoring other people?

• 42:40Speaker 00

  Well, I once had an intern on the team. I was also not very senior back then. I was like L L four, maybe. And I had an intern that I hosted. And then later that intern converted to full timer. And a few years later, he became the manager of the team that he interned in.

• 43:01Ryan Peterman?

  Oh, that's funny. Were you were you reporting to him eventually?

• 43:04Speaker 00

  No, by that time, I already switched projects. But don't underestimate your interns. They can be really, really good.

• 43:11Ryan Peterman?

  When you look back on your career, is there any regret that you have that some people could learn from or could help people avoid?

• 43:20Speaker 00

  Not everything was very smooth in my career, but uh, I think it was still like a learning experience, like I wouldn't be who I am today if I didn't go through those periods of time. I was very idealistic when I was younger, like I believed that like Google is a really positive force in the world, that it's really like, you know, I would work here just because it's Google, not because it pays more, like, you know, not, not, not because of some, like, you know, I really wanted to be in Google because of my personal values and how the company operated.

• 43:59Speaker 00

  And then I got disillusioned over the years, that, yes, it's just a corporate, like, maybe in the, as of seventeen years ago, it was really a more different company, but today it's, it is a corporation,

• 44:17Speaker 00

  just like any, any other corporation, has positive things, it has negative things, but what matters to the company at the end of the day is the bottom line on the financial reports and, you know, they will do whatever it takes to, to get there, to increase that number.

• 44:34Ryan Peterman?

  Is there something that led to that disillusionment?

• 44:37Speaker 00

  You know, Google had a lot of remote offices, for example, and I was working in the Pittsburgh office, which was a relatively remote office and they, at the same time, they shut down an office in Atlanta. And it was roughly the same size. And it was quite shocking to everybody, like, why would you shut down an office with a few hundred people working there? And the answer we got from the leadership, they said, well, we have those big senior vice presidents of the company who decide how to allocate the headcount and where to invest, like, you know, I want to hire in Bay Area, I want to hire in Seattle, I want to hire in this place.

• 45:22Speaker 00

  And just so happened that the big lead who was sponsoring that Atlanta office decided to pull out and nobody else was willing to take over the headcount of that office. So it was just like, you, you feel like, yeah, at the end of the day, I'm just a cell in a spreadsheet. You know, it's like.

• 45:45Speaker 00

  There was very little empathy that company showed to those people. They said like, yeah, we can help you relocate if you want to to other places. But if not, here at your exit package and good luck. And that was like many, many years ago, before all the layoffs that happened after COVID. And you know, nowadays, it's people are much less. It's much more understandable that the company can lay off anybody just because this this project here doesn't make sense anymore. We'll just like shut it down and let people go like this is now a common scene. It was not the case as of 10 years ago.

• 46:27Ryan Peterman?

  Yeah, I remember, I think when Google was a lot earlier, it was, um, don't be evil. And the culture was very, very set on that.

• 46:37Speaker 00

  Yeah, I think they still like, they still trying not to be evil. But again, the bottom line often drives the overrides that decision.

• 46:46Ryan Peterman?

  I guess it's true for all public companies. Yeah, and then the last question I'd like to ask is if you could go back to when you just entered the industry or you were working at Google and give yourself some advice, knowing everything you know now, what would you say?

• 47:02Speaker 00

  I worked on some projects that didn't make sense to me. And essentially, when I especially when I was a more junior, like I was working on some projects that like, why, why are we building this in the first place? Who, who needs this? And then a year later, the leadership realizes the same thing and they just shut down the project.

• 47:26Speaker 00

  So, so maybe work on what matters, like ask yourself like, does my company really benefit from what I'm doing? Like, and, and if not, then maybe you shouldn't be there.

• 47:39Ryan Peterman?

  In that situation, let's say you recognized it. And then your, your org's going in that direction and you think it's useless. But you're a junior engineer. What, what could you do to kind of adjust your direction?

• 47:53Speaker 00

  I mean, talk to other managers, find another project.

• 47:58Speaker 00

  There are different, different companies have different treatment of people who want to switch projects. Before Google, I actually briefly worked at Intel. Their switching project was almost impossible. Like it's, it's much easier to just quit and then reapply. Versus Google was was very fluid like you could easily switch projects within Google. So this advice doesn't work for any company like I know some companies like IBM is another example I've heard where switching projects is impossible. Yeah, there's some companies have certain reputations. I don't know. I never worked at IBM. So I don't know, but that's that's the reputation they got. Within Google, it was easy to switch projects. Within Meta, I believe it's easy to switch projects.

• 48:44Speaker 00

  Yeah, I don't know about other companies.

• 48:47Ryan Peterman?

  Yeah, I think most modern day Silicon Valley companies are inspired by Google, Meta, those types of companies. So similar culture on team switches. So awesome. Well, yeah, thanks so much for your time, Igor. I really appreciate it.

• 49:01Speaker 00

  Yeah, thank you very much. And yeah, I hope somebody finds it useful.

• 49:06Ryan Peterman?

  Thanks for listening to the podcast. I don't sell anything or do sponsorships, but if you want to help out with the podcast, you can support by engaging with the content on YouTube or on Spotify. If you want to drop a review, that'll be super helpful. And if there's any guests that you want to bring on to, please let me know. I feel like sourcing very senior ICs. There's no well studied list out there on Google that I can just search this up. So if there's someone in your org or at your company who you really look up to and you want to hear their career story, let me know and I'll reach out to them.