Pod Insights a16z Podcast a16z Podcast | It's Not What You Say, It's How You Say It -- When Language Meets Big Data About this episode When most people think of big data they think of numbers, but it turns out that a lot of big data -- a lot of the output of our work and activity as humans in fact -- is in the form of words. So what can we learn when we apply machine learning and natural language processing techniques to text?
The findings may surprise you. For example, did you know that you can predict whether a Kickstarter project will be funded or not based on textual elements alone ... before it's even published? Other findings are not so surprising; e.g., hopefully we all know by now that a word like "synergy" can sink a job description! But what words DO appeal in tech job descriptions when you're trying to draw the most qualified, diverse candidates? And speaking of diversity: What's up with those findings about differences in how men and women describe themselves on their resumes -- or are described by others in their performance reviews?
On this episode of the a16z Podcast, Textio co-founder and CEO Kieran Snyder (who has a PhD in linguistics and formerly led product and design in roles at Microsoft and Amazon) shares her findings, answers to some of these questions, and other insights based on several studies they've conducted on language, technology, and document bias.
Transcript 0:00 Speaker 1 Hi everyone, welcome to the A6C and Z podcast. I'm Sonal, and I'm here today with Michael, and we are talking to Kieran Snyder, who is a CEO and co-founder of Texio, a company that analyzes job listings to predict how well they're going to perform and can help optimize them to get more qualified diverse candidates. And interestingly, they've been able to figure out besides what what doesn't work very well in job descriptions, words like synergize, they've been able to figure out what does work well.
0:27 Speaker 0 Language like in tech, people love to talk about hard problems and uh tough challenges.
0:33 Speaker 1 But it's a lot bigger than just about jobs. The ability to understand the words we use and how we use them is pretty important because even though we're completely immersed in a world of tech where a lot of the conversation is around big data as numbers, a lot of the data that we produce of the output of our work is actually taking place in the form of words. And those words matter.
0:54 Speaker 0 Sometimes how you say things is more influential than what you're actually saying, right? And it's counterintuitive to any of us who've built products before because you like to think you're leading with a strong vision.
1:06 Speaker 1 Clearly words matter. And another place that that plays out is with hidden biases that are often revealed in words. Um, for example, Kieran examined a number of resumes to see the differences between how women and men describe themselves as well as in performance reviews to see the ways that women and men were described differently.
1:26 Speaker 0 The word abrasive, which has been talked about uh since then uh ended up, you know, being used in 17 out of a couple hundred women's reviews and zero times in in men's reviews, right? Um the the sort of stereotypical, like aggressive was used in a man's review with an exhortation to be more of it and in women's reviews as a term of some judgment.
1:48 Speaker 1 Okay, let's get started. Kieran, welcome. So the reason we actually invited you to the ACNC podcast today is because you've been writing a lot of interesting work based on the outcomes of your product where you've been analyzing people's use of language in certain contexts as as a way to surface insights. And I think that's really fascinating because I think we have a tendency in our world to focus on big data as if it's just numbers and not other forms of data, because you're really describing I mean what you described your work as doing is applying machine learning to text and natural language. So how does that how did how did you kind of how does that work? And then we can talk a little bit more about um how you got there.
2:25 Speaker 0 Yeah. So how does it work? I mean, language is just an encoding of concepts, right? And anything that can be encoded can be measured. And so I I was uh sharing the story the other day. We were actually originally started out looking at Kickstarter projects, right? So we started out with this question could we just look at the text of a Kickstarter project and some of its you know metadata around the text and predict, you know, before it was ever published whether it was going to raise money. And we didn't look at the quality of the idea. We didn't look at whether a celebrity endorsed it. Turns out uh we got over 90% predictive on minute zero of a project as to whether it was going to hit its fundraising goal based solely on things like how long is the text and what kind of fonts are you using and how many headings do you have.
3:14 Speaker 1 So wait a minute, just to clear unpack that a little bit. So before the project even went live on Kickstarter, just looking at the the that those features of the text, you're able to predict whether it'd be successful or not.
3:26 Speaker 0 Exactly. What
3:26 Speaker 1 were some of the high-level takeaways from that?
3:28 Speaker 0 Yeah. So uh longer is better where Kickstarter is uh is concerned. Um kind of counterintuitive. One thing that broke our hearts um, because my co-founder Jensen Harris and I both have some design background. You would think these cleanly designed projects with this beautiful use of single typography would do best. Not so uh you want it to look like a ransom note, so you want to move you wanna mix and match types, you want lots and lots of headings. Uh you can't do that. Oh my god, that sounds visually painful. Images to be front loaded, kind of makes sense. Um, but a lot of what we found was not intuitive. And so it's just demonstrated for us the value of actually measuring, because the whole Kickstarter carp corpus is out there in the world, right? So you can actually have great training data, you can see how well prior projects have performed.
4:16 Speaker 0 And we saw, hey, we're kind of on to something here just looking at the it's a very painful as a product person, the quality of your idea doesn't matter. Um, just looking at the the content uh aspects we could predict.
4:28 Speaker 2 And how do you account then for all the other sort of outside variables? You know, whether it was at the beginning of the Kickstarter kind of like craze, whether it was a certain time of year for that matter, uh third type of product, even yeah, or geography. Um how do you know that in fact your analysis was correct?
4:46 Speaker 0 I mean, you can look at some of those other factors, right? Because you can see when projects are published, turns out that doesn't make a big difference. You can see the only things that really move the needle in a very short-term way are do you have a celebrity endorsing you? Um, because that can get you a lot of social media attention. Um it doesn't make or break you, but it can it can help uh quite a bit. Um, and generally how good you are at your social media strategy can can tip the balance a bit, but none of those other factors turned out to be as significant as we expected.
5:18 Speaker 2 The ability to really zero in via just the text. Did that surprise you?
5:23 Speaker 0 Um I mean, we started off with the hypothesis that it would be that way. Uh, and that you know, sometimes how you say things is more influential than what you're actually saying, right? And it's counterintuitive to any of us who've built products before because you like to think you're leading with a strong vision. We weren't surprised. Um we were curious as we started to apply the technology to some other verticals, whether it would extend. Um, you know, our first big area has really been in the area of job listings where we've looked to see the first real product application where we've looked at listings now from over 10,000 different companies. We've measured who's applied to which listings, and we do see the content matters. We do see some tailoring by geography. Turns out what works in New York is different than what works in San Francisco.
6:10 Speaker 0 We see a lot of tailoring by industry. So what works to hire in tech is very different than what it looks like to hire a claims adjuster um or someone in retail, right? So you see some differentiation, but in all cases, you know, depending on how you're slicing and dicing the categories, that text leads. You know, we we've looked at real estate a little bit um prior to launching our our uh jobs application, and we've seen the same principles apply.
6:35 Speaker 1 So far you've been talking about the form of the text, like the length and and the fonts and the design. But like, were there particular words that popped out as well in terms of what people said on those Kickstarter descriptions or anything like that? I'm bringing this up because there's just this recent anecdote in the news that I read about someone saying that you can predict um the success or default of loan applications based on words people use like God or using God a lot will actually mean you'll you'll default you're more likely to fall on your loan, for example.
7:04 Speaker 2 Every month I post.
7:06 Speaker 0 In in Kickstarter, we didn't look at that. We started looking at that for real estate listings and then jobs where we've looked at it quite a bit. So we saw when we were prototyping out the real estate stuff that if you say off-street parking, um, that really moves the needle for low-income homes. But for high income homes, in terms of the number of people who go to your open house and then of the eventual sale price of your home, uh, for higher priced homes, it's actually a negative because why would you want to highlight that it has off-street parking? It's just sort of an expectation. Um we saw, you know, um vocabulary matter quite a bit. In jobs, it matters hugely. Um, you know, we've we've identified at this point over 25,000 unique phrases that move the needle on how many people will apply for your job, what demographics how qualified they are.
7:56 Speaker 1 Could you share some of that um insight with us? Because you know, the reason I came across your work is because I read an article about how you analyzed um performance appraisals and job descriptions for insights about what moves the needle and and the differences and how people communicate. Um what are some of the things? I mean, just because we have a huge audience list that does job descriptions, is that you need to hire some of the things.
8:18 Speaker 0 Yeah, so there's sort of a set of language that works really well for everybody. These are not surprising on the face of them, but when you look, you see lots of them. So things like we'd love to hear from you, be really encouraging and positive in your listing. Um, using the right balance of talking to the job seeker. So your background is in science and you really enjoy roller skating in your free time. And talking about the company. So we stand for this. So it turns out the balance between you statements and we statements can matter. Um, you know, language like in tech, people love to talk about hard problems and uh tough challenges. Curiously, we see patterns change over time. So my favorite example of this is the phrase big data.
9:00 Speaker 1 Uh-huh.
9:01 Speaker 0 So a year and a half ago, if you use the phrase big data in a tech job listing, it was positive. Uh, you know, it it was seen as compelling and cutting edge in June of 2015, it's not negative, but it's totally neutral.
9:13 Speaker 2 That's interesting. I I wanted to ask because if everybody sort of gloms on to these best practices, how then does the signal versus the noise shift?
9:23 Speaker 0 Exactly. Mark marketing content, as with any marketing content, the patterns that work change as they get popular and get adopted. And so one of the reasons we believe software is so interesting as a solution here is that it can kind of keep track at broad scale of what's actually happening right now in the market. So you may have published a job listing uh that worked really well a year ago, and probably how a lot of your listeners write their job listings is they go back to that one and then they try to edit it and tweak it a little bit and fix it. That's exactly what
9:52 Speaker 1 happens.
9:53 Speaker 0 Right. But it it actually doesn't necessarily work because the the put the market has changed. And so there's a lot there.
10:00 Speaker 1 Were you ever I mean I'm just curious about this. Were you ever able to find um or study associations between people's intent and outcomes in job listings? So for example, and one of the things that we've seen happen a lot is that people only become real about what they actually want out of a job description when they actually put words to paper and words have that power to sort of help discipline what you're looking for. You might not even know what you're looking for until you write it down. Um have you ever looked at anything around that or found heard interesting anecdotes around that given your work? Aaron
10:29 Speaker 0 Powell We have seen that listings tend to perform better when they are originally authored. So you can see some degradation over time when people patch, you know, I take a little bit from this listing and a little bit from this one, and I sort of stitch them together. And it's probably because when you're originally authoring it, you bring that coherent point of view. It's really
10:48 Speaker 1 interesting.
10:49 Speaker 0 Um a little bit. We're pretty early for us to it to have seen that. Um we also identify phrases that torpedo your listing, right? You know, there are you know, um corporate sort of cliches and jargon is a buzzwords, basically. One of the very common, we call it a gateway term um that kind of torpedoes your listing is the word synergy. Oh my
11:10 Speaker 1 god. That that should torpedo any piece of content. I don't care what it is.
11:13 Speaker 0 But it's a gateway term because when people include synergy, they're also significantly more likely to include, you know, value add and make it pop and kind of silly, but they're all over the place. And and it turns out every candidate of every demographic group hates them. Um so there's a lot of opportunity to improve in business jobs.
11:35 Speaker 2 In in the sort of editorial world, uh we would call that jargon. And it sounds like
11:40 Speaker 0 we also call it jargon. I think
11:41 Speaker 1 we specifically jargon is jargon, but no, totally actually it's interesting because with words like that, they're obviously in use because they're useful words. And it's kind of sad because I mean synergy at some point was probably a useful word. So it's kind of interesting because over time with your corpus of data, you'll be able to sort of map how people's language changes and when you think of dictionaries as like these static instruments for capturing text these days. It is kind of fascinating how language is changing in a way that we're able to track differently now, thanks to online and software and
12:11 Speaker 0 it changes lexicography. Like just as a whole discipline, it changes lexicography for sure. I don't know that you could do it in a static way anymore.
12:18 Speaker 1 Right. I I totally the internet
12:20 Speaker 0 has just exploded that.
12:22 Speaker 1 Right. Exactly.
12:23 Speaker 2 Is there so if big data is kind of neutral now? Is there a kind of job type or job description that's the celebrity of the job search world
12:32 Speaker 1 right now? Yeah. What word is sort of popping out that's really moving the needle for you guys or that you've observed?
12:36 Speaker 0 Um there are there are several. Um most of your listeners are probably in tech. Uh it varies a lot by industry. So at scale right now, out scale is a very popular phrase. That's
12:47 Speaker 1 popular here too. Yeah, you well, it is. You
12:49 Speaker 0 don't you don't want to do things in in uh use methods that are perceived to be manual or perceived to be limited in some way. So at scale is one that that shines in and it started in tech, but it spread to other uh other industries, which is common that we see that. Uh one of my favorite examples, given that we spend a lot of time talking to HR people uh is turns out workforce analytics is no longer a good phrase to use. You want to use people analytics. Um you can get these highly specific, you know, deep in an industry changes that if you're in the industry and you're on the cutting edge, you probably know. But if you're just a startup trying to hire your first analytics person, you probably have no idea you don't have a deep background in in the industry, right?
13:35 Speaker 0 Yeah.
13:36 Speaker 2 So you you've described it for job listings in real estate. And and so this approach you you think can extend in in different directions. You start with Kickstarter, but what is it that it's doing and how do you like it seems a little bit magical, I have to say that like I know that this is a job listing, so therefore um it's gonna have to do this, but a real estate listing has to do something kind of different.
13:57 Speaker 0 That's great. A really good question. So uh, you know, this is this approach is as powerful as the data set that you have. So if you want to understand a document type, the very first thing you need to do is collect a lot of examples of the document type. And that means you need the documents and you also need some information about their outcomes. So you are publishing a Kickstarter project. We want to know, did you make money or not? That's that's signal for us. You're publishing a job listing, we want to know, did you attract a lot of good people? Did you attract only men? Did you attract no one? Um so you know, for each document type that we take on, the first thing we do is we make sure we build out a great training data set. And then we apply really classical natural language processing techniques.
14:43 Speaker 0 So we look for patterns. And so we say, okay, these are the ones that were successful. Where successful is defined as, you know, attracted more applicants than 80% of similar listings, maybe. Um then we start looking for the linguistic patterns and the successes, the ones that aren't as successful, uh, ones that skew in a certain way demographically, uh, and then we play that back. So it's sort of a key thing for us is that you get that feedback in real time as you're typing. Um so as you're working on your document before you ever publish it for you ever pay to publish it somewhere, you can make it good. And so the training set is the sort of core of all of that because without that outcomes data, then it's just someone's opinion.
15:25 Speaker 2 I mean, could you extend that to say like, look, I want to write a screenplay for a blockbuster. Um I mean, does it can could you I mean people have probably tried this? Uh
15:34 Speaker 0 in fact, a very prominent Bay Area CEO proposed to us a couple months ago that we start applying this to uh screenplays
15:42 Speaker 1 to start writing to actually start producing content or just analyzing them.
15:46 Speaker 0 Sell it to Hollywood. Oh wow, that's great. Yeah. So I think any time you're writing content to sell something, this is really interesting technology. And you could be selling your company. You could be selling yourself, you're a job seeker with a resume that you want to have optimized. You could be selling your product in a you know, an e-commerce setup. You could be marketing yourself, you could be marketing blast emails. Anytime you're writing content to get people to take an action, um, this is really useful technology. Aaron Powell
16:12 Speaker 1 Let's talk about where this fits and let's actually go let's purposefully use some jargon here and let's talk about where it fits in the tech trends like where the it fits in that space. So it sounds like you're describing big data techniques applied to natural language or machine learning techniques applied to natural language. But natural language has been around for over three decades, 30 years. I mean, and in the early days, they didn't have this l this kind of corpus to train the algorithms on on, obviously. So they had to use different kinds of techniques. Like where does your work fit and how do you see how it fits in the evolution of natural language? Like how is that how has it been and where we are where are we now kind of? Aaron Powell
16:47 Speaker 0 Yeah. I mean, I think uh in core natural language processing, empirical strategies have always been really important. So when I was a grad student years ago writing a dissertation, collecting data was just a lot more work, right? So I had to go and record people in the field and I had to transcribe things. I mean it feels like ancient now, actually, but I actually finished my PhD 12 years ago. It wasn't that ancient. Um the fact that the internet has codified everything over the last 15 or 20 years, at least in in English and most Western languages, uh means that you have this ready set of corpora available for you. The tricky part is collecting the text and the outcomes that go with it. The outcomes are the part that's hard.
17:34 Speaker 0 Finding the content is easy. So
17:36 Speaker 1 you're describing the difference between just analyzing something and being able to predict something using that text.
17:40 Speaker 0 Exactly. When you analyze something, you can say, oh, cool, this word is really popular now. That's an interesting fact. Um might be valuable to someone to know it, but it's different than saying this word is actually helping your document in some way. Aaron Powell
17:54 Speaker 1 What are some other scenarios where you could use sort of this natural language text analysis to make more predict interesting things?
18:02 Speaker 0 Uh yeah. So people are really starting to think broadly about this. We saw uh a New York City-based company uh helping people optimize the sale of their New York City apartments recently using uh the right phrases. Um we've seen people do things in healthcare that I think are really interesting. It's not a known vertical to me, but uh looking at the kind of notes that doctors take about a patient and predict that patient's likelihood of having a major insurance incident over the next you know 12 to 15 months is really interesting things in actual science. Like I think any time people are producing text, which by the way, in businesses, whatever your business is, text is actually the thing you produce the most of. Right. Um I believe that which any any industry and so people produce a lot of text, it's meant to describe often what they think is going to happen.
18:52 Speaker 0 And so the I mean the field of opportunity is pretty big. Aaron Powell
18:56 Speaker 1 D the techniques you're describing, is it the same underlying technique applied to all different domains, but do you have to also train each corpus on a different um domain? Like there's special like there's inside language in each industry, or are they are there also universals across all of them?
19:10 Speaker 0 Um that's a really good question. Uh you don't know until you train is the short answer to the question. So uh we have a set of NLP libraries that look for common attributes of text, and we always start out any new vertical by turning them on the ver on the documents and seeing what happens. So things like sentence length, almost always interesting, things like the density of verbs and adjectives almost always interesting. Document length almost always interesting, but the specific phrases that matter and what it means to write a job listing is very different than what it means to predict whether a patient is going to become ill, right? And so this the specifics matter. Um the goals matter. So if it's a document that's intended for broad consumption, it really probably shouldn't be longer than 600.
19:59 Speaker 0 700 words. If it's uh a stock prospectus where you're giving a company some information about how their stocks are likely to perform, it's gonna be pages and pages. And so, you know, the the specific benchmarks that you're looking for often vary vertical by vertical, but the principles of the kinds of things you look for um are pretty similar.
20:20 Speaker 1 In the past it seemed like only really big companies could do this because they had like the type of computing hardware and power processing power to pull this off. Like what's changed that. AWS,
20:30 Speaker 0 AWS is what has changed things, right? I mean cloud compute at scale and you know, Google Cloud and Azure, there's a lot of competitors now, but AWS did this for startups, I think. Um I say that not because I worked at Amazon before. But but it actually is like for uh for our team to set up the server infrastructure that we need is trivial. Um, you know, so I think that that's a thing. And it just the fact that there's so much text data encoded on the internet has Google has democratized a lot of access to data. Um so that has helped too.
21:04 Speaker 2 That's great. Yeah. Did you guys I have to ask, did you um kind of put any Kickstarter projects up there yourself just to give it a whirl?
21:13 Speaker 0 We were asked this a lot during our fundraising. We did look at pitch decks, by the way. One of the things I want to hear about that. I will come back to your question. Um of the things that's been fascinating about having the beta out there in the world is the ways people are using it. So of course they're using it for job listings, but people are using it for everything. Like just a couple days ago, I had a material science professor write to me saying, I put all my course syllabi through. And I was like, really? Like how did that work for you? I can't imagine that that was a good result. And he's like, Oh, I threw out all of the job parts. I just looked at gender bias. Wow. That was a component that I needed for what I was doing.
21:47 Speaker 2 When you say put it through, like what happens? I understand like you're sort of my in my head, I have this idea that I'm typing along and you know, suggestions come flying at me, but
21:55 Speaker 0 that's exactly what happened. So there's a website and you paste or type in your content, and as you're typing, it's getting annotated and marked up for you with patterns, suggestions, things you might want to change, scores. So we're and you
22:07 Speaker 2 can in the case of the syllabi, right? You can dial it up or down depending on what you want the outcome to be. So in his case, look, I'm I'm sort of tracking for gender bias or
22:18 Speaker 0 he was looking for a specific aspect of what we provide. And that of course the product isn't tuned for what he wants, but he still found that aspect to be applicable to what he was doing. We're seeing people put marketing content through pitch deck content through. So to your question about did we initiate any Kickstarter campaigns? We didn't because we weren't making it. We might be, yes. We've given a lot of advice to people on Kickstarter projects since then. Um we didn't because we were making an enterprise product, right? And we're if we had followed through on a Kickstarter product and then it got funded, then we'd have to build it.
22:54 Speaker 1 Right.
22:55 Speaker 0 Um what'd you find? We helped friends for sure.
22:57 Speaker 1 That's great. So what'd you find out about the pitch decks? Actually, I'm totally intrigued by that, obviously, given who listens to our podcast.
23:03 Speaker 0 I mean, pitch decks, pitch decks are not always highly text-oriented, right? So great pitch decks don't include just your your text attributes, but there are certainly things like length of your deck that matters, slide titles end up mattering quite a bit because people are looking to see a certain style of content. Um
23:23 Speaker 1 let's face it, we've all seen um any kind of meeting where some one person gets hung up on one word and and in a headline. Yeah, it always happens. We
23:30 Speaker 0 didn't go we didn't go deep on pitch decks, but we looked at as many as we could find as we were building our own pitch deck in our last round of funding and found some patterns in.
23:41 Speaker 2 Like in the synergy line of questioning, were there was there were there words or phrases you should never include in your pitch deck.
23:48 Speaker 0 You know, I don't know. I don't know.
23:50 Speaker 1 I guess there might not even actually be I wonder if there's and there's never, I guess. I bet I bet there
23:55 Speaker 0 are. We didn't identify them.
23:57 Speaker 2 Synergy is probably
23:59 Speaker 1 actually let's talk a little bit more about some of and maybe we should wrap up on this note. Let's talk a little bit more about some of your findings around gender differences. So you said the material science professor tested his own syllabus, which again, I'm not sure that that made sense, like you said, because there wasn't a reference corpus to I I guess I guess it's there wasn't,
24:15 Speaker 0 but when you have, you know, tens of thousands, tens of thousands of phrases that are lighting up and he's writing for a science STEM student population, odds are good that there's going to be some lexical overlap. Right. So he, you know, he found some things there.
24:30 Speaker 1 Just so describe some of your findings around job descriptions, because that's given what your product focuses on right now in terms of gender differences and how people um what things you picked up on that.
24:40 Speaker 0 Yeah. So um prior to us doing this, there was some really strong qualitative research, right? That National Coalition of Women in Technology, the Clayman Institute here at Stanford, they've done some really interesting qualitative work. But the number of phrases that they identified was on the order of a couple hundred. Avoid Rockstar, avoid ninja, you know, we want to hire more
25:03 Speaker 1 women in technology.
25:04 Speaker 0 The interesting things for us, first of all, we've talked to a lot of industries outside of tech. And so while in technology we want to hire more women, when I talk to people who are hiring ICU nurses or elementary school teachers, bias goes the other way. And so it's very important to us that we don't judge. We just forecast and let you make the right choices for your business.
25:25 Speaker 1 Um, whatever you're optimizing for given wherever there's an indifference or imbalance.
25:29 Speaker 0 So I will say we have validated much of the qualitative research, which is good that there's you know some alignment on those points. Um we have found cases where uh things are it's pretty subtle, right? So the difference between fast-paced environment and rapidly moving environment, it's it's almost head scratchingly tiny, but statistically, one of them draws many fewer women to apply.
25:56 Speaker 1 And which one is it by the way? Are you allowed to talk about it? Fast paced. Oh,
25:59 Speaker 0 okay. Interesting. Fast paste. Um you see sometimes these very fine distinctions between terms that you can only kind of play out statistically. The other thing I would say is that most individual terms aren't that egregious one way or the other. Uh we put a lot of effort into making something visual so that you could see patterns. So if you have, you know, one sort of male bias term or female bias term, you're probably not gonna shift your applicant mix that much. But if you have 10, then you're gonna see more substantial uh impact on your applicant set. Aaron Powell
26:32 Speaker 1 Interesting. I feel like this kind of validates the natural language approach because in the past um I think in general people tend to put too much stock only on numbers and not on words, like your the whole point of what your work is. But the second part of it is that even, you know, I was thinking about when I was back in grad school, there was a lot of debates between qualitative and quantitative data and what was more valuable. And obviously at the end of the day, they're both valuable. Exactly. But it's interesting because for the first time, you're really bringing quantitative quantitative techniques to something that was traditionally in the qualitative domain. Just like conversation analysis.
27:07 Speaker 0 Yes, it's true. I mean, so you know, I've looked at in some of my prior research, I've looked at some other document types also. Uh as Texo was getting started, um the piece that actually really brought us into jobs was some work I did on performance reviews. So I collected hundreds of performance reviews from men and women who work in technology. Uh I they were all voluntarily given, which meant they were all good reviews, which I was betting on that I was gonna be comparing strong performers. Regardless. Regardless, because you don't give your review unless it's a good one mostly. Um and I found really striking demographic differences and how men and women who were getting good reviews were described in the language that was used. Um Wow, that's kind of interesting.
27:53 Speaker 0 The the word abrasive, which has been talked about uh since then uh ended up, you know, being used in 17 out of a couple hundred women's reviews and zero times in in men's reviews, right? Um the the sort of stereotypical, like aggressive was used in a man's review with an exhortation to be more of it, and in women's reviews is a term of some judgment. And so that was really interesting. Um I looked more recently at resumes. So I collected 1100 resumes from men and women in technology, about half of each, and found for men and women who have very similar backgrounds, very systematic differences in how they present themselves in a resume, which is really interesting. How
28:33 Speaker 1 how what was the difference? How did that difference play out?
28:36 Speaker 0 So uh men's resumes were shorter. They were much deeper into detail about what they actually produced and worked on.
28:45 Speaker 1 That's kind of counterintuitive because you would think that shorter means you'd be less detailed. But you're saying that they were shorter but more detailed about specific things versus other things.
28:54 Speaker 0 Women's resumes tended to tell a story. They were written in prose. They didn't use bullets nearly as much. They included executive summary summaries, they included detailed statements of their personal interests that were twice as long as what men tended to include. So the women's resumes were stronger on narrative, much later on detail. The men's resumes were generally stronger on detail and lighter on narrative. But one of those kinds of resumes gets flagged as positive much more frequently, right? In tech, especially, we look really for what did they deliver, how quickly and tersely can they communicate. And so as we started looking at some of these documents, we realized that there was just fascinating opportunities on the the job listing front because in these other important business documents, we were seeing demographic differences play out.
29:43 Speaker 0 Aaron Powell
29:44 Speaker 1 That's fascinating. Were there any other sort of takeaways you have for people who are job seekers out there who want to like optimize their resume based on what you discovered? I mean, the length and the narrative is an interesting point. I mean, does it matter by the way for an in a particular industry? I know you said you did.
29:58 Speaker 0 I looked at tech in resumes. Okay. Um I bet some of the same uh findings apply in other STEM fields too. I bet finance you would see some similar patterns. We do see tech and finance pattern together quite a bit in in other document types. That's interesting, by
30:15 Speaker 1 the way, that that those two domains. Quant.
30:17 Speaker 0 It's the quant. Okay. We love numbers. We love data. We love rigor. Those are things that you know we share. Yeah, I don't know. I mean, I'm always very reluctant to tell somebody to change the way they tell their story because I think both of those styles are needed for companies. You need people who can tell a customer story and you need people who can track fine detail. And so I guess I would prefer to tell the story the way you actually are and find the company that values it rather than change the way you tell your story.
30:50 Speaker 1 You're right, because that is actually key. That's a really good point. Aaron Powell
30:53 Speaker 2 My one question though is this whole idea of like optimizing language. How far does that go? Because at some point I do think that an optimized becomes average or bland or you know, not to to point fingers, but I'm thinking of demand media, for example. Like collect all this data on what people like or want or and then spew out on the other side something that nobody likes or nobody wants.
31:16 Speaker 0 I love that question. Um so it turns out that if everybody sounds the same, no one stands out, and the good writers continue to find a way to stand out, and then that changes what works. So the beauty of a learning system, we used the example of big data before, is that if everybody tries to glom on to the same patterns, they're no longer effective. Someone is gonna figure out, as with any marketer, someone is gonna figure out how to do it better and they're gonna introduce the next pattern for success. So that's how how we see it.
31:47 Speaker 2 What are some other ways that people are using what's out there right now that's been surprising to you?
31:52 Speaker 0 So it was initially surprising to us that people were using our tool for anything other than job listings because we trained on job listings. We're data quant oriented company. That's the promise we made is that this is gonna help you with job listings. As people started trying new things, we realized, oh, there's nothing doing quite like what we're doing. So people want to test its limits and see what kinds of content they can put through. Um some of the crazier things we've seen. Um we've seen resumes, which kind of makes sense. We've seen lots and lots of marketing content. We've seen people putting their product descriptions through. Um toy company that recently removed uh gender labels from their children's toys, put toy descriptions through to see if they were still flagging any gender
32:41 Speaker 2 language. Just the ninja toys, I expect just the
32:43 Speaker 0 ninja toys, right. And you know, so it's uh it we're not trained on those documents, but people are continuing to use the system for it because I think there's a hunger for that kind of experience, and there's nothing tailored to what they need. And so for us, it has offered some insight into what people need and what we might do to support that.
33:04 Speaker 1 Will we be able will we ever be able to use your technology to ask ask questions? Like say, hey, I want to know X, Y, or Z based on all the things you've trained in on.
33:13 Speaker 0 I don't know if you'll be able to ask questions, but we do think that you'll be able to use it to generate simple content. So I think there's something fascinating in the hey, answer these 10 questions about yourself. Tell me where you'd like to work, and I will make the resume that is most likely to get you a
33:30 Speaker 1 good screening
33:31 Speaker 0 interview. And from there you're on your own. You've got to be good, right? We're not gonna lie for you. We're gonna tell your story, but we're gonna tell it in the way that is most likely to get you that job at Google that you want. Or
33:41 Speaker 2 get that loan down the road if that's what it is, right?
33:48 Speaker 0 And grant proposals and you know, again, people are trying to write to get a result. So we're seeing quite a bit of variation. It's not the majority of the usage, but it's you know, a few percent.
33:58 Speaker 1 All right. Well, that was Kieran Snyder of Texio and um another episode of the AA6NZ podcast. Thank you, everyone.
Previous Company Building in Crypto: A Conversation between Brian Armstrong and Chris Dixon Next What's Next for Education Startups Page updated August 25, 2026 · transcript and summary are machine-generated