Fighting bias in AI (and in hiring)

Lindsey Zuloaga joins us to discuss bias in hiring, bias in AI, and how we can fight bias in hiring with AI. Lindsey tells us about her experiences fighting bias at HireVue, where she is director of data science, and she gives some practical advice to AI practitioners about fairness in models and data. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean is simplicity at scale. Whether your business is running one virtual machine or ten thousand, DigitalOcean gets out of your way so your team can build, deploy, and scale faster and more efficiently. New accounts get $100 in credit to use in your first 60 days. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We catch our errors before our users do because of Rollbar. Resolve errors in minutes, and deploy your code with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Linode https://linode.com/changelog – Our cloud server of choice. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2018. Start your server - head to linode.com/changelog https://linode.com/changelogFeaturing: Lindsey Zuloaga – Website https://www.linkedin.com/in/ljeanderson/, X https://x.com/LindseyZuloagaChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Compas recidivism model https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithmHireVue https://www.hirevue.com/IBM fairness 360 https://www.ibm.com/blogs/research/2018/09/ai-fairness-360/Attacking discrimination with smarter machine learning https://research.google.com/bigpicture/attacking-discrimination-in-ml/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Beta for change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at change law because of roll bar. Check them out at rollbar.com and we're hosted on Lino servers. Head to linode.com slash change log. This episode is brought to you by Digital Ocean. They now have CPU optimized droplets with dedicated hyper threads from best in class. Intel CPUs for all your machine learning and batch processing needs. You can easily spin up their one click machine learning and AI application image. This gives you immediate access to Python three, R, Jupiter notebook, TensorFlow, side kit and PyTorch. Use our special link to get a hundred dollar credit for Digital Ocean and try it today for free.

• 0:45Speaker 00

  At the do.co slash change log. Once again, do.co slash change log.

• 1:03Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show at change.com slash community, follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:29Daniel Whitenack?

  Welcome to another Practical AI. Chris, I know that you've had a number of jobs throughout your career. Was the hiring process always super smooth for you?

• 1:39Chris Benson?

  Anything but. I've been hired more than a few times and I've hired lots of people over the years. And no, for me at least way, way more art than science. So I'm looking forward to, uh, to maybe learning something here.

• 1:52Daniel Whitenack?

  Yeah, we've got Lindsay Zulaga with us. Welcome, Lindsay.

• 1:55Speaker 03

  Hi, nice to be here.

• 1:56Daniel Whitenack?

  Yeah, did I get the name right?

• 1:58Speaker 03

  Yeah, you did good.

• 1:59Daniel Whitenack?

  Okay, perfect. Well, I'm excited to have you on the show today. I know me as well, kind of with Chris. I've had some awkward experiences in the hiring process. I've done well at interviewing. I've crashed in the interviewing process. I've done well and bad at assessments and coding things. And we're just excited to have you because we're going to be talking today about your work with AI and hiring, and also bias in AI. So, super great to have you here. It'd be great if we could just hear a little bit about your background. I know you started out in academia and then eventually moved into industry. So, give us a little bit of your story.

• 2:35Speaker 03

  Sure. I studied physics. So, I did my undergrad here in Utah, the University of Utah. And then I did a master's and PhD at Rice University in Houston, Texas, and a postdoc in Germany as well. So, during that time, I was in the field of nanophotonics. So, I was studying and doing experiments on how nanoparticles interact with light. So, building laser setups and kind of a pretty different world than what I'm in now. When I went into graduate school, I really wanted to work with my hands. I thought I didn't want to sit at a computer all day. But to my surprise, what I actually enjoyed the most about my work was writing code to analyze data.

• 3:18Speaker 03

  So, when I did transition into industry, data science ended up being a really good fit. Kind of relies on a lot of those similar problem-solving skills that I learned. Obviously, having a strong math background is useful. And analyzing data, writing code to analyze data. So, it was kind of a good fit, right place at the right time. My transition, and you guys talked about job interviewing. I'll say, I've written a blog post about this, but my transition from academia to industry was a lot more difficult than I expected. I was doing well in academia, so I kind of thought it would be easy for me to transition into industry. And really kind of was naive about the importance of connections. You know, I had

• 4:03Speaker 03

  A CV, not a resume, but a CV with publications on it and things that people in the industry don't really care about. So I came into the whole industry job world, a little naive, and ended up applying for a lot of jobs online. And going through this process, and many people have probably been through it, where you apply for a job through what's called an applicant tracking system, an ATS, and you enter all your information in, you upload your resume, and then you have to re-enter all the information in. And then all your information kind of gets parsed into plain text, and you finally submit, you know, and you've, you've spent all this time kind of trying to personalize your cover letter, and you submit, and you just never hear

• 4:47Speaker 03

  anything again. So it's kind of this black hole.

• 4:50Chris Benson?

  I hate those systems, whether I'm an applicant or a hiring manager, either way, they're terrible.

• 4:55Daniel Whitenack?

  Yeah, I hate how you format your resume perfectly, and you get it all flashy-looking, and then you go through the system, and then you realize that you just have to like, put it in as plain text or something, and all of that work is for naught.

• 5:08Speaker 03

  Exactly. And there's a lot of gaming in the system, which I didn't know, and maybe I would have benefited from knowing this at the time. But there's actually, like, there's a website, can't remember what it's called, but you can go and see, you put in a job posting, and then you put in your cover letter, your resume, and it will tell you the likelihood of getting past these ATS filters. And it's literally just, a lot of times, it's just like a keyword match. And I always felt like that was a little weird to just like, put the exact keywords that are in the job posting in my, in my application. But it turns out that does help you get past these filters. A lot of times, these filters are pretty simple. They're looking for certain keywords, or they're looking for certain school you

• 5:51Speaker 03

  went to, or GPA, which is silly, because we found a lot of times that doesn't really tie to job performance very strongly at all. But the bottom line is, you know, companies just have so many applicants, they need some way of filtering through people. And I went away from the experience feeling like, something's wrong with this system, when me and so many people that I know that had PhDs, and once they got a job, they did really well, they were just passed up by so many companies. The companies are losing out as well. That story is kind of my motivation of, of why I care about hiring and kind of fixing this broken, broken system.

• 6:28Daniel Whitenack?

  Now you're Director of Data Science at HireVue, is that correct?

• 6:31Speaker 03

  Yes.

• 6:32Daniel Whitenack?

  Yeah, and what does HireVue do? Is it one of these systems? Or what do you work on and what does the company do?

• 6:38Speaker 03

  No, yeah, we're not an applicant tracking system, although we do a lot of times have to work with them, integrate into them. We are a video interviewing company. So our philosophy is that a resume and a cover letter are not a very good representation of a person. So, you know, we started years ago with a video interviewing platform and our most popular product is called an on-demand interview, which is asynchronous. Our customers are companies that create interviews. They can have any different types of questions. They can have professional football player ask the question. They can do all sorts of interesting things. They can send the same interview out to many different people

• 7:21Speaker 03

  and the people record themselves answering questions on their own time. And then the companies can review those interviews on their own time. So that's a very popular product. And that was, you know, kind of our main product for a long time because it kind of replaces this resume phone screening, like initial stage of the funnel. Because we've all experienced, you know, looking at resumes and they all look the same and it's really hard to differentiate people. But once you hear them talk about what they're interested in and how they communicate, you can get a better feel for who they are. So we had a lot of success with that product, but we still had this issue of volume. Like I said before, like companies are just getting so many applicants that it's impossible for them to actually look at all of them.

• 8:05Speaker 03

  And the way it ends up going today is that a lot of people are just randomly ignored. So we started building our AI product a few years back where we said, we have all this, this rich data from job interviews. And if our customers can tell us who ended up being good at a job and who was bad at a job and that what that is depends totally on the job. So we have some performance metrics around, you know, this person was a really good salesperson. They sold a lot and this person wasn't. Can we train algorithms to notice patterns between, you know, people who are top performers in a job and others? So that is the assessments product that I work on.

• 8:46Chris Benson?

  So would it be fair to say that you're kind of focusing on using machine learning to take the bias out of the process of hiring? And if so, how does that work? How does that manifest itself? How do you train to get rid of that?

• 9:00Speaker 03

  Yeah, so it is a common question that we get pretty immediately when people hear about what they do, what we do, is a little bit of like, oh, this is creepy. And how do you know what the algorithm's doing? How do you know it's not biased? So, you know, algorithms are really good at stereotyping. And that can be an issue anywhere where AI is used. If there's any bias in the training data, or just even under-representation in the training data of certain groups, the algorithm could mirror that bias.

• 9:30Daniel Whitenack?

  So do you mean kind of like if there's only a representation of certain type of candidates, let's say, then your algorithm might behave differently when it's trained on that data according to, when it sees those candidates versus, you know, candidates that weren't in the, in the pool, in the training pool? Is that kind of a fair statement?

• 9:51Speaker 03

  Sure. And I think an even bigger issue is if there's a small number of, like, say there's only one female software engineer and she wasn't very good, then the algorithm takes that and says, oh, every time I've seen someone act like this or talk like this, they were bad. So if there's no one, the algorithm doesn't learn as strong of patterns, although it could, and it's something you want to look out for. But a lot of times, under-representation or just explicit, you know, bias, like, in the data, which, which we do sometimes see, and depending on how subjective that performance metric is, that can be strong. Depending on the country as well, we've seen it vary.

• 10:34Speaker 03

  And kind of like, like manager ratings and things that are subjective like that. So we definitely prefer objective metrics, like sales numbers, call handle time, you know, kind of productivity measures, things like that.

• 10:46Daniel Whitenack?

  I'm curious, have you had more of a challenge on this front in certain industries? I'm not sure which industries, you know, HigherView is, is working with. You mentioned sales a little bit, maybe software engineering. Do you have to kind of approach this as far as your models go differently in different industries? Or is this something that's kind of a problem across the board?

• 11:06Speaker 03

  Yeah, I would say it's probably more on a company level or a cultural level that we notice differences. So a lot of what is important in trying to level the playing field is, you know, these interviews ask people very consistent questions and that's something that's been done in hiring over the past several decades because, you know, hiring is very much about gut feelings. So we've proved it by trying to treat all candidates in a consistent way, but it's pretty much impossible for humans to actually do that. Humans have this implicit bias that we don't even know we have. So there's also a big

• 11:47Speaker 03

  culture recently of like this concept of cultural fit, which is very popular. And companies say they want to hire someone who they like and that can communicate well with them and work well with their teams, but this often results in like a similarity bias where I don't know why I just like that person. Well, you like them because they're a lot like you or they are a lot like your team already. So you get this homogeneity in your team.

• 12:12Chris Benson?

  So to some degree, would it be fair to say that when a company is looking for cultural fit, are they almost acknowledging their bias and saying we're gonna, we're going to accept that as part of the process? Or am I misreading that?

• 12:25Speaker 03

  I mean, I think some people have made that argument. There's, you know, there's articles written about the issues with cultural fit, which is just, are you just opening the door for bias? I wouldn't go that far to necessarily say that that's exactly what's going on. I mean, I do understand the concept, but it is, it is very tricky and, and, you know, humans are probably going to be a part of the hiring process for a long time. So it's something that we need to try to deal with.

• 12:53Daniel Whitenack?

  So I'm kind of thinking in my mind right now in terms of like, okay, we know that humans are biased in terms of these ways that we've mentioned. We know that we can kind of subtly introduce bias into our machine learning and AI models via representation in the data set and other ways. I'm just wondering, as kind of human AI developers, you know, what chance do we have of kind of fighting this bias? And how can we have hope to actually do something better?

• 13:24Speaker 03

  Yeah, I think a big part of it is just becoming aware. So as data scientists, I think we've, we spend a lot of time just trying to optimize the accuracy of our algorithms and kind of not thinking about bias or fairness at all. As I've studied algorithmic fairness more and more, I found that it's a, it's a more nuanced, tricky topic than you might assume. So if you look up, there's a recidivism model. This is kind of a, started a whole conversation. It's called Compass and it was this recidivism model in Florida where they tried to predict the chances that someone would reoffend after they were released from prison. When you looked at the data, actually blacks had a higher false positive rate.

• 14:07Speaker 03

  So they were marked as being at risk when they actually didn't reoffend in the training data at a, at a higher rate than whites. That algorithm was trained to optimize accuracy, but because of different base rates in the data, this was a side effect. So this, this whole thing spurred a really interesting conversation around fairness and how to define it. And the upshot is that basically there's, there's many different notions of what makes it an algorithm fair. And with most real world problems, it's impossible to satisfy all of them. So it makes things tricky for data scientists and we actually need to consider what notions of fairness matter the most for our particular problem. Another example, and I think marketing is a really interesting space because it, it relies a lot on demographics.

• 14:52Speaker 03

  So an example of a situation to think about is if you're trying to predict who would click on a data science job posting, like an ad for a data science job, the algorithm could look at a bunch of browser data and say users who look at female type things online are less likely to, to click on that ad and end up making an algorithm that doesn't show it to any females. It's a really strict notion of fairness to say, we need this to be shown to the same percentage of men and women. That's obviously pretty strict because there are more men that are interested in the ad and would click on the ad. So, so you, the marketing company would lose money, but it's maybe realistic to aim for something else like we just want the same true positive rate.

• 15:34Speaker 03

  So out of the people that are interested, same percentage of men and percentage of women saw the ad, for example. So those are the kinds of things, um, and, and there's a lot more detail beneath that, but, um, those are the kinds of different notions of fairness that I think you need to take into consideration when you're building an algorithm.

• 15:52Chris Benson?

  So we've kind of dived right into doing it from the algorithm. And I guess I'd like to see if we can differentiate a little bit between what a traditional job assessment process looks like and how HireVue is approaching it algorithmically at this point. And what are the things that might be the same for companies going from one to the other? And what are some of the things that might change for them? And how do they prepare for that?

• 16:16Speaker 03

  Sure. So yeah, a lot of people are familiar with this traditional job assessment, which is often like multiple choice tests. And they've been around for a long time. They are the result of trying to make the process more consistent. Some of the drawbacks are that they're closed ended. So you have maybe multiple choice, but none of those choices describe you. And they also can be kind of a bad candidate experience. So companies care a lot about that. Like, they want people to come in and have a good experience. Even if they didn't get the job, they don't want to damage their brand by having this awful experience. So those assessments can be long and make that experience negative. And they also give results like personality traits.

• 16:59Speaker 03

  And the connection between personality traits and actual job performance is loose, or it's maybe kind of made up by a person. So assuming, you know, we want a salesperson to have these exact personality traits is sometimes not validated. In our process, we actually, like I said, we train straight to performance. Like I mentioned before, we try to get objective performance metrics, and that could depend on the job, what exactly that means.

• 17:28Chris Benson?

  So like in the example of the salesman that you talked about, there's a stereotype that people have about what is a salesman? What's that natural born salesperson look like, personality wise? And that usually has a picture that is the stereotype in our head. Are you essentially trying to take those stereotypes out of the process by validating which of the metrics are applicable for that job versus what we can see from the data is not?

• 17:53Speaker 03

  Yeah, sure. And I think sometimes that does happen, that humans have an assumption about what is going to make the perfect person for this job versus what is actually in the data. And so I think a lot of times those notions are overturned by looking at actual performance data.

• 18:08Daniel Whitenack?

  And one thing that I'm thinking about here is, you know, it might be like you already mentioned the example where you only have the one example of a female software engineer who went through and maybe performed one way or another. Is it hard for you to, as you're thinking about, you know, being objective in these ways? I imagine in some cases, it might be hard for you to actually get the data that you need to be objective. Like maybe, you know, when you're first working with a company, you don't know the performance information of how the people that they've hired in the past have performed in this objective way. How do you go about kind of establishing that data that you need as the foundation?

• 18:52Speaker 03

  Yeah, a lot of times that's a process. So a lot of companies don't have really strong performance metrics. And so we have a team of I O psychologists or industrial organizational psychologists who go in from the very beginning and help our customers kind of get set up. If they're existing customers, they might already have their own interview and their own questions. But ideally, we kind of start with them from the beginning. What is important in this job? We do a whole job analysis, right? So what do you want to measure? What are you looking at? And our I O psychologists have a lot of experience with knowing which questions to ask to actually tease out that information.

• 19:35Speaker 03

  So it's kind of interesting, like there's questions like, "Tell me about yourself," which are good warm-up questions that don't actually differentiate people very well at all. Whereas questions that are about a situation, like, "What would you do if this happened? You have this difficult customer and, you know, some detailed scenario. How would you act in that situation?" Those questions tend to be better at differentiating top and bottom performers. So the hope is we go in from the beginning and kind of design the interview. We design the process of, you know, how we're going to collect performance data. As you guys know, machine learning algorithms do rely on our training data being kind of representative of who's coming in the funnel. So we want to see a distribution of people.

• 20:20Speaker 03

  Sometimes gathering enough data is a challenge, though, so we have continuous monitoring of our algorithms. I can say a little bit more about that. After we release an algorithm, we're always watching for how it scores different groups of people. And making sure that it's not treating different groups of people in a statistically significantly different way. That makes sense.

• 20:42Daniel Whitenack?

  That that was something, Chris. I know we we talked about in our last news updates thing is, you know, Google recommending through their their AI, I forget what they called it, AI guidelines, to always be continuously monitoring for those those biases and everything. Yep.

• 20:58Speaker 03

  Yeah. So for us, I mean, I mentioned before, you know, when I when I've done research on fairness and AI and bias and AI, there's a lot of problems that are really difficult to solve because the features that you're looking at, the inputs to your model, that actually do matter for the thing you're trying to predict, have different base rates in the data. So an example would be like, if you want to predict who should be given a loan or not, you'd need to look at credit score and income, but credit score have different and income have different distributions among different age, race, gender groups. So it's really hard to get away from that coming into your model. And away, we're really lucky because we are only looking at this job interview. We

• 21:43Speaker 03

  don't do any kind of facial recognition. We don't find out who this person is and try to like scrape the internet for more information about them. We're not throwing in a bunch of data that we don't understand. We know exactly what we're dealing with. And the way we take our video interview data and structure it is intentionally made to kind of obscure some of the things that we don't want to know. Like, we don't want to know your age, race, gender, attractiveness. We want to know the content of what you said, how you said it, like tone of voice, pauses, things like that, and your facial expressions. So those are kind of the three types of features we pull out and structure. So we're already kind of blinding

• 22:25Speaker 03

  the algorithm to demographic traits. But one thing to be aware of is that, you know, if there's bias in the training data, sometimes those traits can leak through somehow. So, for example, maybe you have an algorithm that was trained to be sexist, and it will notice some little difference in how men and women speak in the data set. So if that's the case, this this continuous monitoring is really important to see how the algorithm is behaving in the wild. And if it does have any issues, like it's scoring men and women differently, we can go back and say, What are the features that are even telling the algorithm who's a man and who's a woman and then remove some of those features? So we'll do a mitigation process.

• 23:07Speaker 03

  We are in the situation where we have a lot of features, so we can afford to throw some out. If they're contributing to bias, we simply remove them. In doing that, we might lose a little bit of predictive power, but we mitigate that adverse impact. We're also lucky in the sense that our rules are very well-defined by the EEOC or the Equal Employment Opportunity Commission. So there's federal laws about how assessments can need to behave, and so we follow those very closely. And basically, the rules say that whatever, if you have some kind of a cutoff, like people who score above this score continue on to the next steps, and if you score below, you're out of the running.

• 23:50Speaker 03

  At that cutoff, no group can be scoring less than 80% or four-fifths of the top-scoring group. So we have to follow those rules. That's US law, and making sure that our algorithms are not treating people differently. And if we ever see anything, you know, we can go through this mitigation process.

• 24:08Chris Benson?

  Okay, so coming back out of break, I have a question for you, Lindsay. What types of things cannot be covered well algorithmically? And, you know, starting with that, and then kind of where do humans fit into the equation? You noted at the beginning that you thought humans would be in the equation for a while to go, for a long time potentially. And I'd like to understand kind of where they fit in and how the human and the algorithm work together.

• 24:36Speaker 03

  Yeah, so definitely we're not taking humans out of the loop anytime soon. I always kind of laugh when I try to talk to Siri, and she does a terrible job of understanding what I'm saying.

• 24:48Chris Benson?

  No kidding.

• 24:49Speaker 03

  And I think, like, oh, my God, we're worried about these robots taking over. There's still so many things that humans are a lot better at. I think the important things that AI will be taking over are the mundane, boring things that AI can do well, while humans still really need to be a part of making personal connections, making final decisions, and taking in... Other information that might not be available to the AI for hiring. That does, you know, on the other side of the coin mean that bias will still be a part of hiring. But we've found that even removing bias from a chunk of that hiring funnel can help people get through to later stages that they they might not have originally.

• 25:31Speaker 03

  We've had, you know, we've had customers say they've increased their diversity by 16% or give us these great metrics around, you know, kind of if this initial stage of the funnel is open to more people, they tend to get further along in the funnel. So definitely for the slice that AI is taking over, we we hope to remove that bias.

• 25:53Daniel Whitenack?

  And one of the things you mentioned is kind of monitoring around fairness. And I was wondering, you know, it seems like you have to kind of develop a certain culture as data scientists and as a data science or AI team to really make that like a core part of a goal on each one of your products to kind of monitor for fairness and all of that. I was wondering if you could kind of briefly talk about, you know, how how you went about developing that culture on your team? And, you know, maybe make some recommendations for those out there that are kind of thinking about, oh, well, this is something I'd really like to do on our team, but I maybe don't know where to get started or or how to develop that culture.

• 26:32Speaker 03

  Yeah, definitely. I think for us, a lot of it came from our I O psychology team and being in the assessment space. So starting from there, we kind of had, you know, like I said, we have laws around what our assessment, how our assessment scores people. Our particular assessment happens to include AI. We were coming into this space that the job assessment space that had been around for decades. So we got a lot of those those ideas started there. And then it's kind of blossomed more and more as we've studied. There's a lot of academic study going on around this and we and we collaborate pretty closely with some researchers here at the University of Utah. Who study algorithmic fairness. Like I said, it's, you know, what constitutes fair is not well defined.

• 27:15Speaker 03

  So it's usually something that needs to be discussed and refined for every individual problem. I would suggest a great place to start IBM just released. It's called AI Fairness 360. You can go on their website and just play. I played with it a little bit with just some Kaggle data and they show, you know, a lot of, a lot of these metrics that I talked about, you know, these kind of definitions of fairness. And you can kind of see how those things are related to each other and how you can possibly mitigate bias. Another recommendation I have, just to kind of illustrate the concept is Google did some research. I think if you just, if you look for attacking discrimination with smarter machine learning,

• 27:56Speaker 03

  there's an article with an interactive portion where you can play with this fake data where it's credit score. And you're trying to predict who would repay a loan. And this is something I mentioned earlier, but it's a great thing to play with and kind of see how there's trade offs. So there's in real world situations, there's really not one way to do things where you could satisfy all notions of fairness. So you're always dealing with these trade offs. And I think that's something that's good to look at. And again, this really varies from problem to problem, depending on your inputs and how different your base rates are and how much you rely on inputs with different base rates to predict your outcome.

• 28:37Daniel Whitenack?

  So, you know, keeping things practical, because this is, is practical AI. I'm finding all of this really, really fascinating. And I was wondering if you could just kind of walk through. So do you establish like maybe based on looking at some of this Google work or IBM work, kind of figure out some metrics that at least makes sense to track first. And then how are you tracking them? So you're, you're making predictions with your, your model. And then are you, are you running those metrics on the, on the predictions? Are you running them kind of on the, the training data that you're, that you're feeding in? What exactly are you, you monitoring and what's kind of the process?

• 29:16Daniel Whitenack?

  Like you put the metrics in place and then you kind of send notifications to people to, to review them and who reviews them? I'm kind of interested in those sorts of details.

• 29:25Speaker 03

  Yeah, so like I said, the notions of fairness that we look at are tightly tied to law, like employment law, but we also do, we look at other things as well and we're always kind of interested in being ahead of it. I think it's kind of common that people assume data scientists don't care about this and we've really given it a lot of thought. And we're always looking for different ways of looking at it and seeing how we can improve certain notions. But again, we kind of always come back to the regulations in the employment space as being kind of our most important base to cover. So I mentioned the forfeits rule or the 80% rule for us, which is something we closely monitor. And you, and like you

• 30:06Speaker 03

  did ask before about training data versus kind of how the algorithm is behaving in the wild. So we're always watching that, like here's the customer's cutoff score. They are watching job interviews for everyone who has scored above this and maybe first or maybe they're not watching the lower scorer, the lower scorers at all. So what are, what are those ratios at that cutoff? You know, how are men scoring compared to women? How are the different races scoring? If we ever have an issue there, continuous monitoring is really important because we start off, you know, with a training set of maybe hundreds and hundreds of interviews and there wasn't a lot of diversity. Possibly there's, there's

• 30:48Speaker 03

  groups that were small and it was kind of hard to see with all the noise, how the, how the algorithm's treating those groups. So watching how the algorithm actually behaves in the wild is very important as well. So we're, we're always watching those numbers and being proactive about coming to our customers and saying, hey, we need to mitigate your algorithm. Obviously, we also mitigate at the beginning, but if we ever see that, that we need to mitigate after the algorithm's been out in the wild for a while, we will do that.

• 31:17Daniel Whitenack?

  Have you seen kind of have have certain things surprise you as you've done this sort of monitoring like biases or things pop up where you you thought you did a really good job preparing the algorithm, but it turns out like you didn't in some way or another?

• 31:31Speaker 03

  Yeah, most of that probably just come if if there's any bias that comes in later on, a lot of that is because your training group just wasn't very diverse. So that is something that, you know, we see when we have, maybe there were very few people of color in this data set, or maybe there are very few women. So like I said, it was really hard to tell with just the training data that there was some some feature that was allowing the the algorithm to mimic bias in the data, but it becomes apparent later on. And we have seen that usually, usually not too badly. I mean, usually our we're pretty on top of our monitoring. We don't see anything too drastically different than we expected.

• 32:14Daniel Whitenack?

  Cool. Yeah. So do you see I'm thinking, you know, maybe we can transition a little bit here to kind of the machine learning and AI community in general, maybe out outside of hiring. Are there things in like trends in the community around how we're developing AI that that concern you around like the the topic of fairness? And then are there maybe other things that are encouraging? Maybe these these these projects from IBM and Google, for example?

• 32:45Speaker 03

  Yeah, I think the conversation like IBM's 360 Toolkit is an awesome example of how this is kind of coming into the conversation and people are talking about it. For the last few years, I've sometimes been frustrated by the alarmism that goes on in the media, kind of calling out situations where data scientists did behave really irresponsibly, or just absolutely didn't think about repercussions. And it's hard as a as a data scientist who does care about this and works on it a lot, to not get a little defensive when you're stereotyped. But I think there are some legitimate concerns. And there are a lot of books and articles about algorithms gone wrong. And kind of showcasing these kinds of examples.

• 33:29Speaker 03

  I think it's good that that conversation is out there. In some ways, it scares people and they kind of make assumptions that all algorithms are bad, which can be frustrating. From the hiring point of view, you know, I talked about how broken hiring is. And I really feel like we've made huge improvements where with an algorithm, we can actually look inside the algorithm and say, okay, what features are causing this bias? You know, you really quantitatively see how the algorithm is treating different people, where it's a lot harder to do that with human beings. Human beings don't even know why they made the decisions they made. You can't open up their brain and figure out, oh yeah, you're a little racist and that's why you're doing that. Let's just tweak your brain and account for that.

• 34:11Speaker 03

  So, so we have like these tools that are amazing. But you know, like any powerful tool, they could be good or bad. And so I think it's, we're reaching a point where people are having these really important conversations about using them responsibly.

• 34:27Chris Benson?

  Talking about bias in these ways, we've had various conversations across different episodes with, with people doing all sorts of different types of work. And, and it's, it really seems that you have a great process now on how you're approaching it from with the monitoring and with the feature selection and trying to make sure your data fairly represents where you want to go. In a broader sense, beyond just the topic of hiring, we have so many people that listen that are faced with similar challenges. Do you have any, any more generalized recommendations that you would make to a data science team that is trying to get the bias out of their own situations, out of their own circumstances? Or, or something where rules of thumb that you utilize on that? That is kind of broad based and simple for them to follow?

• 35:10Daniel Whitenack?

  Yeah, I know I've seen like, for example, like checklists come out. I don't know if those are useful or anything around like, you know, your data and your process and all of that.

• 35:19Speaker 03

  Yeah, I think, like I said, it's, it's, it's hard to define what fair is. And I think you have to kind of sit down and have a conversation with a lot of input about, you know, what you care about in this problem and, and, and being transparent about it. You know, are, are we? If you're not just trying to get a higher prediction accuracy, be clear that we care about these notions of fairness and this is what we're doing. This is what we're measuring and this is what we're doing to mitigate. That's something that's just been really useful for us because we were doing this for a long time and not really talking that much about it. We were getting criticized and you know, when people assumed that we were being careless.

• 36:02Speaker 03

  So I think now this conversation started and people are really transparent to be really open about it and say, hey, what we're trying to do is difficult. These are the notions of fairness that we care about and that we're trying to optimize. And we're open to have conversations about that and we're open to changing that. I think everybody understands that machine learning can be very powerful. And if there isn't clear answers, we want to have a conversation about what we're trying to do with it.

• 36:31Chris Benson?

  One of the things that we've noted before is we're still in the very early days in data science, especially if you compare it to software engineering who has been maturing for decades now. And I'm kind of talking about the AI space specifically. But do you think that this period right now where we're all grappling with bias is a kind of growing pains that we're going through? Or do you think this is going to be inherent from now on? Is it always something that we're going to contend with? Or do you think we'll have better tools going forward to tackle it?

• 37:00Speaker 03

  I think kind of both. I mean, I do think it's a growing pain. I think in five to 10 years, way more data scientists will be well-versed in fairness and understand that it's a part of their job and it's something they need to think about. But at the end of the day, it's like any complex topic. There's always going to be different opinions. So because there's not one clear answer, I think there will always be debate about what an algorithm should be doing. And this is a great example with the compass model, the recidivism model that I mentioned. At the end of the day, there's no agreed upon way it should behave because different notions of fairness, to satisfy them, you sacrifice another. And there will always be people that have their opinions about what the most important notions are.

• 37:45Speaker 03

  So I think it will be something that's controversial going forward.

• 37:49Daniel Whitenack?

  I know that I have definitely appreciated your perspective on this, Lindsay. It's been super enlightening to me. So thank you so much for being on the show. Are there any places where you'd like to point to people to to find you online? Or certain resources or blog posts that you'd like to highlight?

• 38:08Speaker 03

  Sure, I'm mostly just on LinkedIn, Lindsay with an E Y Zulaga, Z U L O A G A. That's where I'm probably the most active.

• 38:18Daniel Whitenack?

  Awesome. Well, thank you so much for for being on the show. And I know I'm really looking forward to seeing more of the great content that you put out and the great work that you and your team are doing. So thank you so much.

• 38:32Speaker 03

  Thanks for having me.

• 38:33Daniel Whitenack?

  Thanks a lot.

• 38:36Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly, learn more at fastly.com. And we catch our ears before users do here at changelog because of Robar. Check them out at rollbar.com/changelog. And we're hosted on Leno cloud servers. Head to leno.com/changelog, check them out, support this show. This episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address.

• 39:20Speaker 00

  Get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

• 39:35Speaker 04

  I'm Nick Neese.

• 39:36Speaker 05

  This is K-Ball.

• 39:37Speaker 06

  And I'm Rachel White.

• 39:39Speaker 05

  We're panelists on JS Party, a community celebration of JavaScript and the web.

• 39:43Speaker 04

  Every Thursday at noon Central, a few of us get together and chat about JavaScript, Node, and topics ranging from practical accessibility to weird web APIs.

• 39:51Daniel Whitenack?

  You could just eval the text that you're given and then, and that's basically what it's doing. What could go wrong?

• 39:58Speaker 05

  Yeah, exactly. This is not legal advice to eval text as it comes in. Join us live on Thursdays at noon Central. Listen and Slack with us in real time or wait for the recording to hit. New episodes come out each Friday.

• 40:10Speaker 06

  Find the show at changelog.com/jsparty or wherever you listen to podcasts.

• 40:25Speaker 07

  I'm Tim Smith, and my show Away from Keyboard explores the human side of creative work. You'll hear stories, sometimes deeply personal, about the triumphs and struggles of doing what you love. I got really depressed last year. And the reason it was so hard is because basically everything culminated at once. All these things I'd been avoiding, all these things I'd swept under the rug, they all came out at once. New episodes premiere every other Wednesday. Find the show at changelog.com/afk or wherever you listen to podcasts.