Productionizing AI at LinkedIn

Suju Rajan from LinkedIn joined us to talk about how they are operationalizing state-of-the-art AI at LinkedIn. She sheds light on how AI can and is being used in recruiting, and she weaves in some great explanations of how graph-structured data, personalization, and representation learning can be applied to LinkedIn’s candidate search problem. Suju is passionate about helping people deal with machine learning technical debt, and that gives this episode a good dose of practicality. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean’s developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.99% uptime SLA, and 24/7/365 world-class support to back that up. Get your $100 credit at do.co/changelog https://do.co/changelog. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Suju Rajan – LinkedIn https://www.linkedin.com/in/suju-rajanChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: The AI behind LinkedIn’s recruiter search https://engineering.linkedin.com/blog/2019/04/ai-behind-linkedin-recruiter-search-and-recommendation-systemsTensorFlow Extended (TFX) https://www.tensorflow.org/tfx/Paper: “Hidden Technical Debt in Machine Learning Systems” https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdfUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  A lot of us get fascinated by technological advances in AI, right? The fact that we can ask questions out of an AI agent and express a meaningful, almost human-like response. When it comes to actually putting these systems into production, I want to say that AI in some sense is an engineering discipline. But I think we are very much far away and it's for a reason, right? It's not because we are unwilling to do so. It's just that the inputs are not as deterministic as some of these other systems, right? So whenever you have a change in your data distribution or something happens, then how do you put in quote the right unit and regression test to make sure that you're going to be able to catch some of these errors? How do you keep track of the number of people?

• 0:44Speaker 00

  And I can easily tell you the number of instances where, you know, a team deploys a feature and then there are 20 other teams with unbeknownst to you who are starting to use this feature. And the next time you make the change, all hell breaks through.

• 0:58Speaker 01

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com and we're hosted on Linode cloud servers. Head to linode.com/changelog. This episode is brought to you by Digital Ocean. Droplets, managed Kubernetes, managed databases, spaces, object storage, volume block storage, advanced networking like virtual private clouds and cloud firewalls, developer tooling with a robust API and CLI to make sure you can interact with your infrastructure the way you want to. Digital Ocean is designed for developers and built for businesses.

• 1:38Speaker 01

  Join over 150,000 businesses that develop, manage and scale their applications with Digital Ocean. Head to do.co/changelog to get started with a $100 credit. Again, do.co/changelog.

• 2:09Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We're at Practical AI FM.

• 2:36Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist at SIL International, and I'm joined as always by my co-host, Chris Benson, who is a principal emerging technology strategist at Lockheed Martin. How are you doing, Chris? I am doing very well today, Daniel. How's it going? It's going great. It's a nice, cool day here, and so no complaints. Over the weekend, the internet's been crashing at my wife's business, and it's because there's been too many, the access points couldn't handle the number of devices. It wasn't so much of like a bandwidth issue, it was like an access point. So we just went full in and like I ordered,

• 3:17Daniel Whitenack?

  some people might be familiar with Ubiquiti or Unifi, network appliances and that sort of thing. So I got like a new gateway plus 48 port switch, plus a network video recorder, and four cameras, and a bunch of cable and all that stuff, and two new access points. I think there's something ridiculous, you can have like 500 people or something on these access points. So, might have gone a little bit overboard, but you know. Maybe you know.

• 3:48Daniel Whitenack?

  It was, it was really fun. It was like a nice, different sort of tech thing that I'm not qualified to be interacting with at all, but was fun. Yeah.

• 3:58Chris Benson?

  Yeah, the weather's been really nice. I'm enjoying the fall weather. Things are cooling down. Spent the weekend outside as much as I possibly could doing that. And then we're into this week. And just as an incidental note, this is the GPU Technology Conference Week for Nvidia. So been kind of watching the announcements and some of the talks on that. So lots of cool new things for us to talk about going forward as people adjust to this year's announcements. But I'll leave it there for a point where we're actually going to dive into that.

• 4:28Daniel Whitenack?

  Yeah, definitely. And people should check out our episode number 106, where we had Will Ramey on talking about the GTC conference and some of the things related to that. So if you're at GTC, that would be a good one to listen to as well.

• 4:45Chris Benson?

  There you go. It's all virtual this year, though, you know, it's a virtual GTC.

• 4:48Daniel Whitenack?

  Yeah, it is all virtual. It's good. Yeah. But I'm sure at GTC, people are virtually exchanging some LinkedIn requests to connect back and forth. And of course, we're all used to that at this point, and that's part of our lives. And they're doing a lot of cool machine learning and AI stuff at LinkedIn. And we're really excited today to have Sujit Rajan with us, who is Senior Director of Enterprise AI at LinkedIn. Welcome, Sujit.

• 5:19Speaker 00

  Hello. Hey, Daniel. Hey, Chris. Thanks for having me here.

• 5:22Daniel Whitenack?

  Yeah, great to have you. I know I could look on your LinkedIn and look at your whole background. But for the sake of those listening at home, could you kind of give us a sense of your background, how you got interested in AI and machine learning, and eventually ended up where you're at right now?

• 5:41Speaker 00

  Wow, goes back a very long way, right back to my undergrad, where I had a professor, and I was taking a class on image and signal processing with him. And one of my undergrad senior year projects was on using bidirectional associative memories for recognizing digits. And it actually worked very well. And but I remember, like, you know, compared to these days, when it's the scale of the data is huge, it was like this tiny computer in which I had my MATLAB code. Running out all of these um uh different PAM and EPM architectures and whatnot. Um so that resulted in my getting admitted to UT Austin um under Professor Joy D. Koosh,

• 6:24Speaker 00

  uh whose lab was doing data mining, kinda dates me now, but data mining was what it was.

• 6:31Daniel Whitenack?

  Do people still uh uh label themselves as da- data mi- or doing data mining?

• 6:36Speaker 00

  Is it cool anymore?

• 6:38Daniel Whitenack?

  I I don't know, maybe.

• 6:39Chris Benson?

  No, sadly, I I was there with you. It's not cool anymore.

• 6:43Speaker 00

  It's not at all cool.

• 6:46Daniel Whitenack?

  They they make fun of us. Still data mine, yeah.

• 6:50Speaker 00

  But I don't think that'll ever go away, but yes, let's not call us as a data miner.

• 6:54Daniel Whitenack?

  Okay.

• 6:56Chris Benson?

  I just want the listeners to know that once upon a time, data mining was in fact cool, and we wanted to do that, but that's.

• 7:02Speaker 00

  Yes, and we had uh NSF funding.

• 7:05Daniel Whitenack?

  Exactly.

• 7:08Speaker 00

  So at UT Austin, again, was a very interesting application. It was to analyze satellite images, um and it was very hard to get labeled data for these images. So you would have like these um marshy lands in Botswana and all of those places, and you had to identify what was the sort of vegetation that grows. And uh and uh we were we were collaborating with the Center for Space Research, and they'd actually fly out people to go walk and record that, hey, this is the sort of vegetation that is growing and whatnot.

• 7:38Daniel Whitenack?

  Yeah, talk about annotation.

• 7:40Speaker 00

  I'm seriously, right? No mechanical markers to fly out to Botswana. But over there, um my thesis was on knowledge transfer, uh because you kind of wanted to learn from uh seasonal data to some extent, so you can extrapolate to satellite images in the future, also from related geographic regions as well. So you don't, basically to bring down the cost of annotation uh while maintaining good performance. So that was uh what I did, had a very fun time. Converted into an internship at Yahoo. Yahoo Labs, Yahoo Research, lots of different names. But was a fantastic nine years as an individual contributor of light machine learning scientists,

• 8:21Speaker 00

  call it what you will, but got to work on a whole bunch of different machine learning applications. And I think it was right at the point where ML was kind of taking off, big data was taking off, map reduce was sort of starting to come in. But then I started, we didn't have it yet. In the beginning, it was still all of these single models that you would build to build classifiers, better keyword clustering for sponsored search, targeting for advertisements and whatnot, right? So had a lot of fun. Eventually, it became larger and larger data. Now we had to do news clustering within a few hundred milliseconds and so on. So for we had to have all of these models that were tens of twenties milliseconds, if I may, to be able to figure out if someone's going to

• 9:06Speaker 00

  convert for an ad or read a news article and so on. So had a lot of fun doing that. And eventually switched into managing a group that worked on again personalization across a whole bunch of Yahoo products. This was on the apps, on video recommendation, news recommendation, really got a good sense for what would work as a consumer facing machine learning product. How do we measure interactions? Following which I went to Critio, where I headed the Critio AI Lab. This was my full deep dive into computational advertising. And now if you talk to anyone about machine learning for advertising, they immediately think of CTR prediction and hey, it's kind of done, right? It's sort of already there.

• 9:51Speaker 00

  There's nothing interesting going on in that space. But given the amount of money that is spent on advertising, there is a lot more that's got to do with the causality of the models, right? Because the people who are funding these campaigns want to know that something useful is coming for that money that they are spending. So how do we show that our models actually work was an interesting phenomenon? And more than anything else, I was working at this company, which was sort of like a demand site platform. So you're kind of listening to all of these publishers who want to put ads on their different pages and you have all these advertisers that you're synchronizing with. Uh, but if you've ever like, uh, had a deep dive into that world, there are so many, many,

• 10:35Speaker 00

  many players, there's this waterfall model that they have, right? Like the first party publishers want the first dibs. If they cannot do it, then it goes to the, uh, exchanges and so on, so forth. But all of that entails a very, very, very, very tiny window in which you need to sort of make that bid and then assemble the ad unit to be shown and so on, so forth. So it was a very interesting time where you got, where I learnt quite a bit about, um, the real world constraints, right? It's not as much about the, uh, technical complexity of the model, but hey, can it work in this specific constraints in terms of, uh, the latency in which you have to operate. Okay. So that was Gero AI Lab. Um, also got to

• 11:17Speaker 00

  show that there were lots of cool problems in computational advertising. So we established the lab as a pretty strong presence in Europe and, uh, off late, uh, a little over a year now, um, I've been at LinkedIn where I head what we call enterprise AI. So think of all the business facing applications LinkedIn has, one of which is talent solutions. This is where recruiters come to LinkedIn to look for candidates and we also have all of our members who are trying to find the right jobs. So how do we, uh, make sure that this, uh, marketplace works well, right? So that's one aspect of it. Of course, there's the advertising aspect of it, the sales navigator notion where, uh, folks who want to pitch their cool new products and services to LinkedIn members, how do we make sure that

• 12:01Speaker 00

  it's relevant and there is value on both sides? So these are some of the, and AI for all of these different business facing applications is what I'm working on right now.

• 12:10Daniel Whitenack?

  Super interesting. So you've definitely always been at the forefront of these different fields as they're really, you know, have so much momentum. Does it seem like the sort of current hype and influx of people into machine learning and AI? Does it seem like, you know, faster paced or, or more momentum than in the past to you? Or does it just seem like a sort of new area that, you know, people are, are putting focus on and switching from, from other things?

• 12:37Speaker 00

  For better or worse, we became a cool area to be in, right? There's.

• 12:44Chris Benson?

  That's funny.

• 12:45Speaker 00

  A lot of media coverage on what AI can unlock. Uh but it is true, right? Uh maybe uh ten years ago, it was sort of behind the scenes. Nobody really talked about it. Hey, why do you see the results that you do? Why are you being shown the ads that you have? Was not really a part of the discourse, but uh um I think all of these uh interesting innovations in terms of. I'm I wanna call it the PR events at some of the big companies sort of set up. Uh helped the field get the recognition uh that it perhaps deserves to have, right? Given how integrated some of these models are into our daily lives. You know, the fact that whether I get a home loan or not is perhaps has a machine learning model behind it, right? And it's important for folks to be aware of it.

• 13:29Speaker 00

  I would say it's a combination of both. Uh it feels like a new space because there is a lot more recognition of the fact that it is very integrated, right? And data-driven decisions and AI models are never going to go away uh from uh now on. So how do you wanna think about this field? From a societal standpoint, I think deserves conversation. And that's how we make sure we don't make the wrong choices. And at the same time, open sourcing has become a big deal, right? With TensorFlow and PyTorch and all of these uh larger companies caught democratizing it as they call it. Like, hey, get onto AWS and you have the SageMaker and the whole uh lot of it. Just give us your data, right?

• 14:11Speaker 00

  And the fact that it is becoming a lot more accessible means that a lot more folks are also interested. Uh without having to, you know, in the past, it used to be, hey, you had to go to this lab. You had to get a PhD. You had to publish papers before you can even think about machine learning. And you know, the barrier for entry is a lot lower these days. But given the ubiquity of the application, I think it's important for everyone to understand uh what AI can do and the implications that it has. So uh for the better, but uh I wish we didn't have as much media attention.

• 14:46Chris Benson?

  So I guess, and I wanted, you sort of answered the thing I was about to ask you, and that is, is kind of, is since you did your PhD and you did one in machine learning, and just to put things in context, you know, it was unusual, I would argue, for lots of people to have machine learning PhDs at the point where you did in 2006 and you were done with that program and stuff. And obviously, you've seen this change. Did you ever have any inkling where this might go in terms of the evolution? You've talked a little bit about the change over time, but just your perception of machine learning as you came out of your program versus your perception of it now, what really strikes you? Like, what were the big things that when you do look back, you're kind of like going, wow, I never saw that coming, or I told you so?

• 15:31Speaker 00

  I told you so. So the fact that a lot of problems, I don't think it's fully there yet, right? I think the biggest thing, folks somehow forget, is for any machine learning or AI to work on, at least this is my personal take, you have to have enough data to make it be meaningful, right? And otherwise, it's handcrafted rules, you do some tiny things here and there. Of course, you could overfit like heck the N, but this is, but I think what changed, I don't know if it is necessarily a change, the rise of all of these companies like Facebook and Amazon and, you know, if I may, the Microsofts of the world, right?

• 16:10Speaker 00

  The Googles and whatnot, that managed to get as much user data and hence were required to deploy machine learning models. Did I predict the rise of these platforms as much and that they would aggregate these massive amounts of user engagement that it would be required? Perhaps not. But Google was sort of taking off at that point when I was doing my PhD as well. And Yahoo was one of these forefront companies. But within Yahoo, for instance, the fact that, hey, from everything, from news recommendation to figuring out what tags to put on your email to what ads to show was driven by a machine learning model. So that aspect was always the case. But that there would be these few centers where it's just going to explode.

• 16:54Speaker 00

  And the democratization aspect of it, that it is also good for these companies to open source code there, and not necessarily what is running in production. I know for a fact it cannot be the same models because we are struggling to adopt some of these frameworks when you're outside of Google and whatnot. But in that sense, opening that up, I think, was a change that I didn't necessarily see coming. Like, you would think that, hey, this would be a secret sauce. This is something that we kind of want to do, keep internally in our company, because this is the thing. But that there is so much learning and sharing of what machine learning is, that is, as a field, the accessibility of it and the opening up of it to more and more people, that I didn't see coming.

• 17:36Chris Benson?

  You know, just to make a point that you just said there, if you put that in context, as you were describing that, I was thinking back, about the time roughly that you were getting out of your PhD program, we were doing Myspace.

• 17:49Speaker 00

  Ah.

• 17:49Chris Benson?

  And if you think about the evolution of social media, from the days of Myspace as the first major platform that did not survive in a meaningful way, to where we are today with LinkedIn and Twitter and Facebook and stuff, it's quite a change. And in terms of the data aggregation, as you said, it's, it's a completely different world.

• 18:07Speaker 00

  And even how we shop, how we get information, how we look at videos, right? This much acceleration in how humanity lives packed into such a short period of time, sometimes I think I'm lucky to be a part of this and you're always excited to see what it looks like. But on the other side, I guess with anything that gets accelerated, then comes all the challenges, right? Are we thinking about this, right? And it's good that we've started asking these questions, but there are no clear answers. That's why we are here, right? So the answers are also interesting to think and ponder and make sure we are doing the right things on.

• 18:58Speaker 02

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 19:49Daniel Whitenack?

  So Suju, we've talked a lot about the AI industry in general and, you know, interesting shifts that have happened and all of that. And I'm sure some of those are even visible, you know, within LinkedIn. I was thinking while we were talking, like, you know, how many people have switched their title from like data mining person or data scientist or like data analyst to like AI engineer or AI, you know, whatever. So that's super interesting, but I think more to the things that you're working on personally. I'd love to hear, particularly you brought up the, some ideas around recruiting and that sort of thing. Maybe before we jump into the specific things within LinkedIn,

• 20:33Daniel Whitenack?

  could you give us a sense of how machine learning and AI are kind of starting to influence recruiting in a more meaningful way?

• 20:42Speaker 00

  Uh. Sure. So historically, how would recruiters uh hire, right? Uh. It came down to your network. Of course, it was a physical network. You probably had your rolodex, you'd go through uh contacts to say, I'm hiring for these uh candidates and so on and so forth, though you would uh engage in some external job sites and whatnot, right? Um. But the burden um then is, um, I think this is a sort of an underappreciated to some extent. There are a lot of noisy applicants that might come through and you do not want to, as a recruiter, send the noisy applicants up to your hiring manager. That's not going to set you up for success. It also limits the number of candidates that you can give your bandwidth to, right?

• 21:24Speaker 00

  You can't talk to a hundred people in two weeks in any meaningful way to close them. So for sure, as uh there are more and more jobs that are not, not at this current point in our existence, but in general, as more and more jobs.

• 21:40Daniel Whitenack?

  I would say we're living in an anomaly. Yeah, that's probably true.

• 21:45Speaker 00

  Um. Taking that outlier out of the, hopefully outlier out of the picture, how do we help our recruiters be? Have to spend their time on, let's say, those candidates that are necessarily worth spending their time on, right? Because it's not just, just because you call up someone, they're going to immediately, you know, drop their job and move over, right? You still need to work on the storytelling, as our recruiters call it, to convince them that this company is worth it. So if we could uh in some form or fashion uh make the hiring of our uh members be that much more efficient. I think that is the change that we are trying to drive as well.

• 22:24Speaker 00

  There is also a lot, these days for sure, um and LinkedIn is also trying out some of these uh interesting elements where, you know, maybe it is not all about going on site, right? Improving over and over again, going through the same lead code exercises over and over again to show that, hey, I can do this job, right? So what if you could have your credentials in some sense assessed by a platform, right? So we have all of these skill assessments that you can take, uh which I'm sure you've seen in some of these other places like HackerRank, Tcode and whatnot, right? But think of it as your credentials that sort of travel with you. Right, and if you could then not have to prove yourself on these same elements over and over again.

• 23:05Speaker 00

  But what really matters, what is the differentiation for this recruitment company or this particular job that would make you that much of an ideal fit? So it's a lot more about, given the enormous number of candidates that have their profiles online. How can we make it that much faster for the recruiters to hone down that smaller set? And then spend their time in converting them. The same is true on the seeker's side as well. There are lots of non-traditional roles now, there are lots of options to switch careers, right? You're not like a career employee anymore. People take these very interesting paths. And so how do we sort of help our seekers understand here are all these career trajectories one can have? What are the skills that you require?

• 23:50Speaker 00

  Say if I wanted to switch to becoming a P M instead of being in the A I field or design or some such thing, right? So what would that path look like? So helping our members also be aware of here are the opportunities, here are the skills that are required. And in some ways, even tying it up with LinkedIn Learning to say here are the courses that you could take if you wanted to switch careers. I think that is where we want to think about recruitment moving forward, right? Technology is interesting, it keeps changing. So can we automate the reskilling and the up-leveling of our members? And continue to show to them all the opportunities that come with the new workforce, new marketplace? So I think that's where it's headed. It's also going to be a lot more data-driven.

• 24:35Speaker 00

  A lot of our hiring managers and whatnot want to understand the landscape, right? Like, yeah, I'm trying to hire this particular profile in, say, Lafayette, right? So what is my probability of getting someone with these skill sets? Do I need to broaden my search? What is the typical salary band that is being paid in some of these other places that I'll have to compete with? And maybe even this world after we all come out of this COVID situation, right? What does the future of remote working look like, right? What are the pay scales that you'll all have to think about at that point? So it's also a lot more of trying to understand from the broader trends that are happening in the industry. So it's not as niche as it used to be, right? These tools are being made available in a data-driven way.

• 25:19Speaker 00

  So yeah, hopefully that answered some of it.

• 25:22Daniel Whitenack?

  It's really interesting to think about, like you were saying, the path that someone takes to a certain position and the changes that go along with that. Yeah, because there's so many unique trajectories into particularly into machine learning and AI into those positions. You've got people coming from science, people coming from engineering, people coming from even other places, philosophy, liberal arts, economics, business. And so there are so many sort of unique trajectories. And I often get asked when I'm like doing a workshop or just talking to someone doing like mentoring or something like that, like what are the steps I need to take like between where I'm at now? And where like this position that I want or have in my mind?

• 26:07Daniel Whitenack?

  What's the sort of pathway there? And I of course have my own bias opinion about that. But it is really interesting to think about, you know, mapping out the trajectories and helping a person understand from where I'm at now to where I want to be. You know, how do people typically get there? Where do they jump off from? And two, that's super interesting to me. And it also makes me think sort of like a graph structure sort of coming up. I think somewhere in I heard or I read about, you know, LinkedIn's knowledge graph. And I was looking through a couple of the links in one of your blog posts with the papers and I saw some graphs. So yeah, that seems like a very interesting relationship where you've got like

• 26:52Daniel Whitenack?

  people who work for companies who are in places, who are in industries. And there's of course time in there and all sorts of things. So it seems like there's a definitely even a unique data structure that's going on with the LinkedIn data. Is that correct?

• 27:09Speaker 00

  Yeah, um, I think LinkedIn has been very open about calling it the economic graph, right? So how do we see the whole job space evolving? And to your point, yes, it starts with industries and companies. What are the skills that our members are acquiring? What are the sorts of educational qualifications people are getting? What are the titles that we have? And sort of standardizing all of this and normalizing it, if I may, into an internal representation. So that, you know, I can look at, say, Chris's trajectory and then map it down.

• 27:42Chris Benson?

  Oh, God. Oh, don't do that to me.

• 27:44Speaker 00

  Or figure out, hey, how can we learn broadly that, hey, this is the movement in mass? In terms of it's no longer, I don't know, things that used to be in MATLAB anymore, right? It's more of these other sorts of tools that are becoming that much more popular. And being able to make sense of all of this data, right? As you can imagine, not everybody writes the same things in their job postings or on their profiles. So how do we, if I may, project it down, even though that's not exactly what we're doing, but into a common vocabulary that we can drive products and machine learning systems across LinkedIn? Is what we do. And besides internal LinkedIn, the economic graph is also used in lots of different forums, right?

• 28:28Speaker 00

  To give insights into what are the fields that are hiring. Here are the top things that candidates or members are interested in. Here is how skills are changing across the world and so on. So then sometimes you also get very interesting insights in terms of behavior, how women behave, how men behave when they approach a job and whatnot, right? So it unlocks a whole bunch of value. And I think that is the most remarkable asset that we have. Purely members sharing our journeys and career journeys. And of course, from the marketplace of hires, how is the job landscape changing? How are the skill demands changing and so on and so forth? Yeah.

• 29:05Chris Benson?

  So that's super cool. I guess to continue building on kind of what you're doing, because I love where this conversation is going, is you've kind of laid out how you're looking at the problems that you're trying to solve going forward. If you start thinking about where AI technology is today and these types of problems, where are you matching up various AI architectures or even approach in general to solving problems that LinkedIn and LinkedIn's customers care about? Where does AI fit into that, that large set of tasks that you guys are addressing?

• 29:36Speaker 00

  Well, so. I want to say in almost all product services that we have, uh just given the scale of the data, right? So we are talking seven hundred million plus members, uh any given day fourteen million to twenty million job postings. Even in my small domain, right? So there are things like hey, do we show job opportunities to people? How do we rank things on the feed, given the number of users who show up and so on and so forth? But maybe just talking about the talent space itself, right? Now, making sense of the these member trajectories as an example. Uh and again, uh while one view could have been, hey, let's just pass and code all of these features and try to build some sort of a simple model which says, hey, am I going to apply to this job posting? And of course, that is restrictive.

• 30:20Speaker 00

  Then how can we uh not have to do with these sorts of standardized categorical features? If I were to look at the entirety of the text that the job posting has, can we leverage some of these recent? I don't know, fine tuned versions of BERT, say the internal BERT option to give us some of these semantic meanings? Because not all job postings are created exactly the same, even though they might be talking about exactly the same jobs, right? So putting all of these uh into play, basically to, at this point, we are leveraging, if I may, more deeper understanding of member career trajectories, if I may.

• 30:57Speaker 00

  Uh plus whatever is available on the job posting sites to be able to do more than, am I just going to click on that particular job posting? Uh it's even more of, am I going to apply? If a recruiter is going to reach out to me as an example, am I going to accept that recruiter's connection? And to get to that notion of understanding uh what the job posting is really about. And we've taken it a little step further in our uh recruiter facing projects, where as a recruiter you um perhaps uh don't need to give very, very explicit signals in terms of, you know, here are all these Boolean operations I want to do on the member places to get me that narrow set. But just identify a few of the candidates that you're interested in.

• 31:41Speaker 00

  And behind the scenes, we are able to then say, hey, this is the role that the recruiter is uh looking for, given their. interests that have shown. So let's organically start pushing some of these matches, if I may, which we call recommended matches down to our recruiters. So that's something that we're working on. Um. Underlying all of these, um, I'm not going to say are incredibly complex models. At some point, we could even set this up as an RL problem, right? Like say, hey, Suju's uh benefit or whatever. The thing that she wants to optimize in her life is maximize her career potential, whatever it is, right? Or economic potential or some title potential or whatnot. Now, given that is my long-term strategic play, what are the series of short-term actions that I need to take?

• 32:24Speaker 00

  To be able to almost be a coach to a member's career trajectory is my personal dream of where I want to take this uh uh thing. But the interesting part is LinkedIn is just one view of it, right? In the sense that uh I can see that, hey, uh, say Suju's applied to this particular job, post which it sort of moves off of LinkedIn and we get very delayed feedback, if I may. So maybe after four months, if I'm an active LinkedIn user, I change my profile and then we, hey, hey, this actually worked, right? And now we are able to get these sorts of positive signals. So coming up with all of these proxies and trying to make our AI models be smart enough where at least we are efficient, right? From a matching process is where the current focus is on.

• 33:07Speaker 00

  But is there scope to do twenty times more? Definitely, yes. And this is where we want to take the thing.

• 33:13Chris Benson?

  So I guess as you look at all this possibility and all the things that you can apply different AI architectures to, uh, it made me wonder as you were talking, how much crossover is there? I I realize that you guys are owned fully by Microsoft and yet LinkedIn remains its own brand.

• 33:28Speaker 00

  Thank you.

• 33:28Chris Benson?

  Um. How much crossover is there between the the AI teams at LinkedIn as a subset of Microsoft and and the micro, the various Microsoft AI teams as a whole? And and just to just to ask, you probably don't have anything you can announce yet, but we're all, you know, interested in the fact that GPT three just uh came under license uh from OpenAI. And and do you foresee uh some interesting use cases going forward with GPT three and LinkedIn?

• 33:57Speaker 00

  Well, I I'm not going to comment on the.

• 33:59Chris Benson?

  I I I that's why I I was giving you the out right there. So I I recognize that. But but I I just got to say, I I can certainly think of some interesting uh interesting things to do there.

• 34:10Speaker 00

  Exactly, right. Um. I think the biggest benefit of all of these models, especially that in terms of how we want to represent, I don't know, the text that humans generate, right? Or even images, or even how we want to think about knowledge overall in some senses. But the cost of doing that, of course, is non-trivial, right? So in these cases, can we go this fine-tuned approach? Sort of a route where we start with these pre-trained models, then get it into the LinkedIn specific context and then see, hey, end-to-end, if you were to optimize for an outcome of an hiring, are we able to see more benefits? This is an active piece of work that is constantly ongoing in LinkedIn.

• 34:51Speaker 00

  So trying to make sense of these glove embeddings at some point, of course, Lee glove embeddings, if I may, then there is Bert, and then there is Lee Bert that we are trying to do. And of course, as this space evolves and the heavy lifting is done maybe once or twice, and then it's more about adapting these architectures to your internal use cases and seeing what the benefit is, I think it makes a ton of sense. And it seems also wasteful in some senses for all of us to be doing these massive trainings over and over again, that piggybacking on some of these methodologies to see what is the value that it unlocks for sure. Right? I think the dream case is for us to show up at LinkedIn and say, hey, I want to get to be X in the next two years. Help me.

• 35:35Speaker 00

  Right now, can we make use of everything that we understand about a member's journeys, about the skills that they have acquired? You know, in some interesting world, if they could also, if the companies, for instance, are willing to participate, right? Like, I want to up-level my talent to this kind of a place. Like, I want to upskill all of these folks. Now, how do we have these personalized recommendations for folks in the career trajectory space? Can we learn globally and say, hey, this is what a career transformation looks for someone in the automobile industry, right? I think it's definitely doable in some ways.

• 36:15Chris Benson?

  I need some help here, you know?

• 36:17Speaker 00

  Okay.

• 36:34Daniel Whitenack?

  So I would love to get into a few of the practicalities of some of this work at LinkedIn. I've read a bit through some of the information about the recruiter systems and all of that. It seems like there's kind of maybe a couple of different areas that have been drawn out in what I've read. One of those is kind of the search side of things and representing candidates. And then the second side is more of a personalization side of things. I was wondering if you could touch on those two? And if the representation side of things, like if you have a sort of,

• 37:15Daniel Whitenack?

  are you developing kind of embeddings of candidates within the graph of LinkedIn? And that's what you're leveraging in part to maybe pre-compute that and get search done? How does that tie in?

• 37:27Speaker 00

  Sort of building off of the standardized version of us looking at the data that we have, right? So Daniel, exactly to your point, we do have a notion of what a member embedding needs to look like. And the data for LinkedIn comes from our LinkedIn profiles, right? So in this sense, understanding that, hey, this particular member has a, in that space, that sort of looks like people who are in this technical industry, in this location, and here are the sorts of jobs that they kind of apply to, as much as we can represent them in that member embedding space. Likewise, we have things that are on the job embedding side as well. And someone asking for a data miner, maybe because they were under a rock,

• 38:12Speaker 00

  is also still asking for an AI scientist. So how can you sort of bring that job embedding's view into it? And in some senses, be able to scale this because if you think about where uh there was an earlier question of what are the different services services in which we have AI, right? There is the search aspect of it, then we also organically recommend jobs to people. There is something that we call instant jobs where you want to be the first to apply. So members actually sign up for alerts to say, hey, the moment you see a job come into our system, then send me an alert right away. So you need this to be high, high, high precision. As an example, then on the flip side, on the recruiter search as an example, right? It's all about matching members to get jobs. So we sort of try to

• 38:57Speaker 00

  build this notion of what we call these two target embeddings, which represent the member side and the job side and bring that together along with more near line and real time features to sort of then personalize it to what that particular member is also looking for. Uh to do, if I may, are matching, right? Of course, if it's on the search angle, then you also have the query context that comes into play. On the recommendation side, it's a lot more just based off of your uh activity signals and so on and so forth, that we try to infer intent uh to be able to make those recommendations. But to your point, yes, on the underlying surface, we do have these embeddings. Uh and from what, yeah, we are going to be writing a blog post pretty soon

• 39:40Speaker 00

  about uh at least this uh particular data layer uh in our talent solution process. Yeah.

• 39:45Daniel Whitenack?

  Awesome. Yeah. Thank you for explaining that. And and for for listeners that are maybe new in the space as well. Um when we're talking about embeddings, we're we're talking about, um you know, you could think about representing a candidate or member in in this case as a as a series of numbers, you could encode them and you could define how those numbers should be yourself, you know, that you could encode that in a certain way. But you could also say, well, I may not be able to, you know. Pick the right encoding for this candidate for my task. I'll let my model learn how to encode or how to represent this certain entity or the set of data, in this case, a set of data about a candidate. And that sort of learned representation or learned set of numbers

• 40:29Daniel Whitenack?

  that represents, in this case, the job or candidate, we would call an embedding. So, thank you for mentioning that.

• 40:36Chris Benson?

  Kind of curious, as we've been talking about this, and within the context of LinkedIn is very good. If you, if you look at its, at its long evolution, you know, from the very beginnings, when it was basically at just an online resume. And over the years, you've developed that into really finding the person in that resume and showing those different aspects. Do you have any thoughts on, you know, and I don't, I'm not pushing you in an exact direction, but any thoughts on kind of how you find that personality, those soft skills beyond just, you know, good at presentations kind of thing, you know, that are not strictly like a hard skill, like doing Python, but finding the person and trying to fit that personality type into different types of cultures and jobs?

• 41:20Chris Benson?

  Because I would imagine that that's pretty complicated and that maybe, any thoughts on how AI might be able to be used or anything that y'all might be able to share on that?

• 41:29Speaker 00

  So LinkedIn per se tries to, what we do at LinkedIn tries to learn from career transitions even, right? So that's something that we have as a signal that goes into our recommend the models as recently to say, hey, this is the career trajectory that this person has went from here to here to here. In some senses, you could start thinking of how long did it take for this person to get there to say, hey, this person is like an outlier superstar, or this is like an average sort of a career path? That, you know, could be something that the recruiters may be interested in. But you're asking an excellent question in terms of, hey, how can we factor in more latent aspects of what a member brings to a table, right?

• 42:11Speaker 00

  Not just that, hey, look at my Python proficiency, but this person is really, really motivated, executor of tasks, kind of a thing. Not yet, this is not something that we are. Able to leverage right now. But again, what we are trying to do in some of our recruiter facing projects, we'll call this our Recruit Assistance Connect. This gives LinkedIn an inside view into how the candidate is sort of progressing through their different interview funnels as an example, right? And now maybe from understanding and at some point, if all the interviews are moving to video interviews, we already have debuted a notion of what we called a video introduction.

• 42:51Speaker 00

  And this, to your earlier question, also makes use of Microsoft technology to sort of try to figure out, hey, how is the, today it is with the intention of giving feedback to the candidate to say, hey, speak with more confidence, or you know, you had all these ums and ahs and so on, so forth, so to enable the candidate to prepare further, right? Now, with the permission of the candidate and of course, the company which is gathering all of this data, can be at some point then learn from all of these behavioral signals. But I think it has to be more than just your resume at that point, right? You need to have all of these other data points flowing into the system. So we are making smaller steps, but still within the world of what is being collected for the purposes of recruiting.

• 43:36Speaker 00

  But I can visualize a case for someone wanting to say, hey, then, you know, in some very ambitious world, if we could then say, hey, how is this person interfacing within the company, right? How quickly are you able to start collaborating with your teammates? How quickly are you able to be up and running in being able to get something shipped to production and so on, so forth, right? Does this data exist? Of course it does exist, right? Now, are we, do we think we'll be able to drive the right decisions? Do we think we'll be able to build the right models to tease apart the Latin signals? So make sure, again, the idea here should be to want to do something to propel everybody forward, right? And at the expense of nobody else in some, in some way, right?

• 44:21Speaker 00

  Would we do it? Well, I I need to think about this a little bit more. I want to say we want to get there and more so more than anything else. There is inefficiency in the interview process, right? Like, hey, I've spent twenty years working. And how are you going to judge whether I'm going to be good for my job? Of course, there is some skill and art to it. But it's also exhausting that you have to keep taking and talking about these credentials, establishing this credibility over and over and over again every time you want to switch something else. So how can we sort of package that into something more than assessments? Uh, to say, hey, here is this candidate. In my mind, I call it a credit score of sorts, if I may.

• 45:00Speaker 00

  Uh, to that you can sort of carry along with you, but we are ways from there. But I think the building blocks are being put in place. So we take the pain out of recruiting. Sorry, it was a little bit of fiction and a little bit of reality.

• 45:14Chris Benson?

  I I liked it. No, it's good. I I think I appreciate the the answer there. As I was asking it, I thought, I hope they're they're thinking. I was imagining if anybody was thinking about this. It was gonna be you guys. And I imagine you guys are gonna be leading on this, uh, given your your position. So thank you for tackling a tough question for me there.

• 45:31Daniel Whitenack?

  And I have kind of a selfish question. I think from your perspective, it seems like you've you've spent a lot of time. Transitioning a lot of bleeding edge AI technology into the enterprise, into practical usage, into production usage. Even at LinkedIn, you know, I'm looking at some of these blog posts about the. Search and recommendation and recruiter product and there's links to, you know, research papers and. You know, to some degree, you're you're tasked with bringing that, you know, bleeding edge research stuff into actual practical usage. And I'm just curious about your perspective. Or any recommendations that you could provide to other people out there who are wanting to use some of these.

• 46:17Daniel Whitenack?

  You know, maybe new AI technologies or new models that are published. Um, in terms of like the tech debt that comes with them and the things that you need to like think about as you're transitioning some of this technology into an actual practical use case.

• 46:33Speaker 00

  Thank you for asking this question. I think while a lot of us sort of get fascinated by, if I may, the technological advances in AI, right? The fact that we can ask questions out of an AI agent and express a meaningful and maybe almost human-like response. When it comes to actually putting these systems into production, I want to say that AI in some sense i- is an engineering discipline. But I think we are very much far away and it comes, it's for a reason, right? It's not because we are unwilling to do so. It's just that the inputs are not as deterministic as some of these other systems, right?

• 47:17Speaker 00

  So whenever you have a change in your data distribution or something happens, then how do you put in quote the right unit and regression test to make sure that you're going to be able to catch some of these errors? How do you keep track of the number of people? And I can easily tell you the number of instances where, you know, a team deploys a feature and then there are twenty other teams with unbeknownst to you who are starting to use this feature. And the next time you make the change, all hell breaks through, right? So how do you sort of have that discipline? And of course, many, many simple things that we possibly haven't spent enough time on in terms of feature lineage, when something goes hairwire, how are we, are we able to quickly monitor, alert, track, roll back to a previously

• 48:00Speaker 00

  versioned system? How do we recover from that particular point of view? Do we keep going back to reduce, if I may, feature bloat, right? Because it's very interesting as a machine learning researcher to keep on adding, here's some set of new features, here's some set of new features. But at some point, you want to sort of take a step back and say, hey, holistically, how much have I improved this particular platform, right? Have some of the changes that we have done? And reversed some of the other changes that we would have pushed out otherwise. And even within a field of metrics, right, there is one layer of things that maybe we are pushing in the right direction, but then when you take a step back and say, hey, but what about long term engagement? Is this something that we have been able to qualistically measure?

• 48:44Speaker 00

  So the more the field progresses, I wish a lot of us were a lot more diligent and cognizant of what it takes to maintain a machine learning system in production, especially if you're supporting a large organization, right? Uh, there, there are lots of customers and the outputs of your model could be used in many interesting ways. So how do you have the right contracts in place, right? And say, um, at least in some machine learning teams, the ideas, hey, here is this nice thing that the P M comes up with. Then you quickly work on it, things work well, everyone celebrates, and then you move on to the next thing. Then now who maintains, understands, or even figures out, hey, has this model drifted with time?

• 49:25Speaker 00

  And as much as I want to say A I has been democratized, uh, people who are not familiar with what to expect out of the system, don't even know what a rollback for this model should look like, or how do you even diagnose this thing? Um, so there's been some good work from folks like in TensorFlow with TensorFlow X platform and these sorts of paradigms where we talk about A I metadata, which we need to be very religious about, which we need to start having health assurance and model monitoring and these sorts of things on. But as a discipline, as much as we want to train our folks in terms of all the latest and greatest in D N and tech, I wish we also had courses which talk about, uh, you know, machine learning, model maintenance

• 50:06Speaker 00

  and tech debt in schools as well, uh, because as this is going to just grow larger, it's going to become that much more important.

• 50:14Daniel Whitenack?

  Yeah, and it sounds like, you know, maybe at least at this point, hopefully that will change in the future. I certainly hope so, but it sounds like maybe in the interim, it's up to us who are maybe leaders on, on teams or leading certain projects to really insist and, and ask hard questions around this and anticipate into the future what we might need to monitor and, you know, where sources of, of bias are and what happens over time when we need to trace back what happened with a certain model. Do you have any other practical sort of, uh, tips for maybe team leaders who are trying to push their team in this direction? And are maybe struggling?

• 50:53Speaker 00

  Well, I want to maybe plug a paper. Google has definitely uh the machine learning depth of machine learning depth, sorry, is the uh paper that came out a few years ago. But more recently, they also released a framework of machine learning test scores. Which basically is a very simple thing which looks at, you know, the minimum score you could have get gotten along. Either your uh fe- feature uh monitoring, feature health about your model monitoring and so on so forth, which sort of gives you a simple rubric, if I may, to figure out. How good is your system, right? Is it a house of cards which is going to collapse the first uh time something goes on? Or is it a robust thing that can be deployed in production? So. I think even adopting such a framework and no- most of these things perhaps are not even automated.

• 51:35Speaker 00

  But if you have been in the weeds and you understand how your model has been set up, you should be able to come up with this. Simple checklist to say, hey, yes, uh I think we are going to be robust to something uh catastrophic happening because it will. And it has no reason to think that um eh anyone's going to be an exception. That would be a first start to even figuring out uh with respect to uh if I may, the health of something that is being deployed in production. Now, this is like one on one, if I may, like the basics of what you need to keep in mind. But Daniel, you also mentioned something even more fantastic, which is about. Uh it there is like model biases and data biases in the data and then there is all the choices that our. Models make, right? Like because we were going after a particular product objective.

• 52:20Speaker 00

  Now, what is the longer term impact that this may have? Overall in our user base, is it going to be unfair to some segments of our population? That's an even bigger ethical fairness question that I think is becoming more and more pertinent these days. Uh but it's interesting, right? In one hand, we are talking about like your health and just monitoring of the system and on the other hand. We are in in the crossroads where we are even having to think about societal impacts, uh which I think speaks to the. Rocket ship speed at which this field has grown, but something for everyone getting into AI to be aware of.

• 52:57Daniel Whitenack?

  Yeah, I think that's a really good perspective as we kind of come here to a close. I really appreciate you kind of bringing us in on a couple of those practical things because obviously you've developed a lot of great systems that have made an impact over time and it's great to get that perspective. We certainly appreciate you joining us on the podcast and taking time out of your busy schedule to join us and of course, you know, all you people out there on LinkedIn right now, make sure and post this episode when it's coming live on LinkedIn and, you know, get that in the LinkedIn knowledge graph for all time. But yeah, thank you so much, Suju, for joining us.

• 53:38Daniel Whitenack?

  It's been a real pleasure and I hope we do get to meet at some type of networking event in the future in the real world. So thank you so much.

• 53:49Speaker 00

  Of course, thank you, Chris and Daniel, for having me. It was very nice talking about the future and the lessons of everything in people. Thank you.

• 54:00Speaker 05

  If you enjoy Practical AI, we would enjoy a five-star review on Apple Podcasts, a blog post in response to something said on the show, and/or a recommendation to a friend or colleague. Those word-of-mouth recommendations really do make a difference. Practical AI is hosted by Chris Benson and Daniel Whitehead. It is produced by Jared Santo, with music by the mysterious Breakmaster Cylinder. Thanks again to our partners who support this show's existence. Shout out to Fastly, Linode, and Robar. That's all we have for you today. We'll talk to you again next week.