Modern NLP with spaCy

SpaCy is awesome for NLP! It’s easy to use, has widespread adoption, is open source, and integrates the latest language models. Ines Montani and Matthew Honnibal (core developers of spaCy and co-founders of Explosion) join us to discuss the history of the project, its capabilities, and the latest trends in NLP. We also dig into the practicalities of taking NLP workflows to production. You don’t want to miss this episode! Featuring: Ines Montani – GitHub https://github.com/ines, X https://x.com/_inesmontaniMatthew Honnibal – GitHub https://github.com/honnibal, X https://x.com/honnibalChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: SpaCy https://spacy.io/Advanced NLP with SpaCy course https://course.spacy.io/Thinc https://github.com/explosion/thincExplosion AI https://explosion.ai/Prodigy https://prodi.gy/Fast.ai https://www.fast.ai/Machine learning yearning https://www.deeplearning.ai/machine-learning-yearning/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front-end web servers that need to be full duty CPU all day, every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co/changelog. Again, do.co/changelog.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at changelog.com/community. Follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:46Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with S I L International. And I am joined as always by my co-host, Chris Benson, who is a principal AI strategist at Lockheed Martin. How are you doing, Chris?

• 2:03Chris Benson?

  I am doing great. How's it going today, Daniel?

• 2:05Daniel Whitenack?

  It's going really good. It's been a busy week. Had calls with teams in India. And if you're listening, hello earlier in the week. And that kind of got my schedule off, so I'm a little bit tired. We'll see how the conversations go. I think it'll be all right.

• 2:20Chris Benson?

  I think this conversation is going to wake you up because we're talking about what I think of as your favorite topic in AI.

• 2:25Daniel Whitenack?

  It definitely is. As you know, I'm always doing language-related things, natural language-related things, and I'm really excited. I hope you are able to get some questions in today because I have all sorts of questions. I'll pause every once in a while to let you get one in. Yeah, I'm guessing this is the last moment listeners are gonna hear my voice. But today, we're joined by a couple of the core developers of Spacy and co-founders of Explosion. And we're joined by Enes Montani and Matthew Hannibal. Welcome.

• 2:58Speaker 03

  Hi.

• 2:58Speaker 04

  Hey, thanks.

• 3:00Daniel Whitenack?

  Yeah, great to have you both on the podcast. Appreciate you taking time out of, I'm sure, the busy development of Spacy to join us. Really excited to talk about everything.

• 3:10Speaker 04

  No problem. Glad to be here.

• 3:12Daniel Whitenack?

  Yeah, I was telling you before the podcast, I recently got your latest Spacy stickers and have them proudly on my laptop.

• 3:20Speaker 03

  Oh, that's so cool. Like, I'm still, I'm still actually waiting to really see them in the wild. Because we've sent, this round, we've sent like over a thousand sticker packs. And so they're everywhere. And I'm like waiting for the day where, no, I'm like sitting in a cafe and someone has my sticker and I'm like, oh my god.

• 3:37Daniel Whitenack?

  Yeah, so for our listeners who don't know, every once in a while, and I don't know how many times you've done this, but you kind of just put out the call for anybody that wants stickers to send you some info and you'll send them stickers. And I saw it on Twitter and I was like, yeah, that's, I've got to get in on that right away. And they're really great stickers too. There's a couple like related to like Prodigy and data annotation, but then there's some NLP tattoos, I would say. Is that what they're trying to characterize?

• 4:06Speaker 03

  Yeah, kind of like the old-school tattoo style, yeah.

• 4:08Daniel Whitenack?

  Yeah, yeah, really nice design.

• 4:10Speaker 04

  Yeah, sort of like, I think of it as a sailor sort of style, like.

• 4:13Speaker 03

  Yeah, I was kind of, I've been joking that like, I don't know, how many retweets should I ask for to like get it actually tattooed on me? Because I'm at a level where I have enough tattoos that it kind of doesn't matter as much anymore. And I'm like, oh yeah, sure, I totally walk into.

• 4:26Speaker 04

  Yeah, I mean, your threshold would be far fewer retweets than me, like, you know, it'd be the first tattoo for me.

• 4:31Speaker 03

  Yeah, no, please don't get it tattooed. Like, it's so unique to not.

• 4:37Daniel Whitenack?

  On that note, I would love to hear about both of your backgrounds outside of tattoos. Would you want to give a little bit of your background and then maybe Matthew as well?

• 4:47Speaker 03

  Yeah, actually, I mean, we kind of need to start with Matt, actually, because that's kind of a better story. Or should I start?

• 4:54Speaker 04

  Yeah, actually, I guess we're first, yeah. So I've been working on natural language processing for a long time. I started my PhD in 2005 and graduated from that in like, you know, 2009, 2010. And then I was doing research on this for a few years after that as well as a postdoc. So basically, like, as the technologies improved and there was more and more interest in this, I saw there were companies who were trying to use my research code. So, you know, I'd written some blog posts that had gotten some attention. And so, you know, I just had a GitHub repo sitting there. And so people were trying to use this. And I was like, well, it really was just supposed to print results and exit. That was like its mission in life. And, you know, I designed it quite tightly around that, like, core task.

• 5:38Speaker 04

  So I saw that there was actually a need for, in the software ecosystem, for more production-ready stuff and things that could, you know, basically cross that gap. And so I was, you know, I was at about the level where I was supposed to be writing grant proposals, which wasn't really my thing. And so I decided, well, okay, if I leave this, I can have a go at starting a company and starting something with that. And then soon after this, I met Ines and then we started working on, I think it was the Displace the

• 6:05Speaker 03

  Yeah, I think actually, yeah, because we met and I'd always done like, so I actually, I started making websites when I was a teenager. So that's kind of how I got into programming and my degree is also partly linguistics. So I kind of had a good idea of what Matt was doing then, like, yeah, he always wanted to have a visualizer and have like better user experience. And at that time I was working as a freelancer and I remember like, the first thing I actually said was like, ah, look, I totally know what you're looking for there, but it sounds a bit boring. I don't know if I want to work on that. I have other things to do. So that was actually my first reaction, but I ended up doing it.

• 6:36Daniel Whitenack?

  Validation.

• 6:39Speaker 03

  So yeah, we did end up working together and we very quickly saw that there was like, I don't know, a lot we could do with both our skills kind of combined. And I I started working on the core libraries, basically, shortly after that. And that was kind of when it was first released and yeah.

• 6:54Daniel Whitenack?

  Yeah. So when was that?

• 6:55Speaker 03

  Uh, early two thousand fifteen.

• 6:57Daniel Whitenack?

  Okay.

• 6:57Speaker 03

  Wow. So that's like, that's quite a long time ago.

• 7:00Daniel Whitenack?

  Yeah. Yeah. And in terms of the initial ideas, did the company and sort of consulting things and other stuff like that come first? Or was the initial idea really to build the library?

• 7:12Speaker 04

  So when I was just thinking about leaving academia, I had, I guess, a, you know, a range of ideas for exactly what I could do. One of them was actually to work on software to assist uh language learners. So, you know, I thought, well, okay, the tools for, you know, learning another language are kind of primitive. And there's kind of a computational linguistic angle on that. Then I quickly saw, well, okay, it wasn't quite what I wanted to do. But, you know, and I saw, okay, there's, you know, a gap in the software ecosystem for a library like this. So it was very much around like, okay, well, there's a potential for having something that it's going to be useful to people in a commercial context. And I think that the way it would be most useful to people would be if it was open source.

• 7:53Speaker 04

  Because I feel like this type of technology, if it's closed source or if it's like under an API or something, it's just not as useful. And I thought, well, okay, if we can make the software useful to people, then there'll be a range of ways that we can support it commercially as well. Especially if we, you know, keep it relatively small and don't try to build it as like, you know, don't try to necessarily have a story for how it could be the biggest company in the world or something. Um, there would be plenty of like, you know, interest from companies to, you know, make their usage of it a bit better and gain something and have a commercial opportunity around it.

• 8:28Speaker 03

  Yeah. So like, yeah, Spacey was definitely there first. And then when we started the company, that's when we thought about, okay, how are we gonna make money? Or even we had ideas for products we wanted to build. We didn't want to take venture, so we were like, well, okay, we have users who want to use our stuff. And so we put out kind of a call for consulting and we had quite a few companies applying and that we ended up working with. And that's how we initially bootstrapped Explosion when we first started for like the first six months, I think we did consulting.

• 8:56Speaker 04

  Mm hmm. Yeah. And that was enough to get our first product developed, um, Prodigy, which is an annotation tool that, you know, is on a sort of old school software licensing where you pay for it and then you keep it. And, you know, instead of like renting it like most software is these days. And that's been enough to, you know, keep the bills paid and lend some since and the team's been growing slowly since then as well.

• 9:19Chris Benson?

  So I've got a question. I know Daniel has been intimately involved in using Spacy. I'm kind of curious as a newbie, though, why is it called Spacy?

• 9:29Speaker 04

  Um, so initially, the very first idea that I had was around tokenization because I thought that the tools for data weren't really up to production grade. And it's the first thing that anybody ever needs to do in a natural language is split the text into tokens. So I was like, well, it's based on spaces. It was short and it wasn't taken. And, you know, I had been working in Scythin for a while and I liked, you know, basically developing it as a Scythin program with like, you know, basically a Python API. And so that also, you know, emphasized the sort of speed aspect and the Scythin aspect. So, um, you know, the same way, things are like, um, everything Pi, this was like ending in Si. And so I was like, okay, Spacy, it works.

• 10:13Daniel Whitenack?

  Yeah. Yeah, definitely. And just for our listeners, we'll, we'll try to clarify for those that aren't familiar with natural language stuff, some jargon throughout. So tokens and that sort of thing.

• 10:24Chris Benson?

  I'm going to keep you honest there because I'm the only person here who's not an NLP expert. So.

• 10:28Daniel Whitenack?

  Yeah. So like tokenization and, and tokens, you mentioned like spaces. So you've got text and you've got, you know, words in that text and tokenization would be what?

• 10:38Speaker 03

  Yeah. Well, tokenization would be to split the text into words. And it sounds, if you just look at it, it sounds very simple, but it's actually, once you get to things like punctuation, um, more complex, um, ways of phrasing things, contractions, yeah, in English, for example, that gets a lot more complicated. And then there are also different definitions. Okay, what's a token? A token is not always necessarily a word. Okay, in English, you just have lots of like small words that form bigger words. In German, for instance, um, a lot of these would be like one word. Well, that's where all these jokes about the German language come from, that we have these like massive words that express something very, very specific.

• 11:16Speaker 04

  Yeah, well, but you know, in in languages like English, we still have the same things of terminology. It's just the terminology is, you know, written with spaces in it.

• 11:25Speaker 03

  Yeah.

• 11:25Speaker 04

  Uh, you know, natural language processing is a a term that we use and we call it that specific thing. And in German, that would be written without spaces, but it doesn't actually.

• 11:34Speaker 03

  I do think I would, there would be probably one space in it.

• 11:36Speaker 04

  Oh, okay.

• 11:37Speaker 03

  But it's more like income tax returns or something.

• 11:39Speaker 04

  Sure, yeah.

• 11:40Speaker 03

  That's definitely one word in German.

• 11:41Speaker 04

  Yeah, or federal income tax rebate guarantee or something. I don't know.

• 11:45Speaker 03

  Yeah. I'm pretty sure that's one word.

• 11:49Daniel Whitenack?

  Yeah, like you say, creates all of these complexities that you don't realize first, because in certain languages, right, that certain things are said with a certain number of words. In other languages, it's not the same number. So people might, for example, think of translation as, you know, from one word to another, just this sort of word to word thing, but um, it gets much more complicated than that quickly.

• 12:09Speaker 04

  Yes, exactly. And these differences between languages sort of seen in the algorithms that have been developed in the way that people do things. So I often tell people that, you know, you can basically expect that any uh natural language processing technique will work best on a language depending on how similar it is to English. So English being the language that's most similar to English, everything works best on English. And it's not because there's like, I don't know, some magic property of English that makes it easier or more amenable to computation. It's just that, you know, for the last like, you know, fifty years that people have been thinking about these problems, the dominant language that's been the test case that people have been developing towards has been English. And so that's the way that, and so you can really see that bias in the the way that the algorithms have unfolded.

• 12:54Speaker 04

  And so even when algorithms touted as, you know, language independent, it's like, okay, the algorithm doesn't have any, you know, might not have any specific thing where you need to have a, you know, a resource that depends on a particular language. It'll still work better or worse depending on the characteristics of that language and the complexity of like an individual word versus a, you know, how free the word order is, like all of these things will affect that.

• 13:20Daniel Whitenack?

  So one of the things before we go too much further, I was wondering is kind of, I I know, you know, we're talking about Spacy and I know you've mentioned Prodigy. Is there anything else that Explosion AI does or is it really focused on those?

• 13:33Speaker 03

  We mostly are a developer tools company, so we definitely, we have a few other open source projects that are kind of cool. We have like, Some other projects in the pipeline and products that we're working on, but ultimately that's what we're doing. We're not doing consulting anymore. We haven't been doing consulting for a long time and we're building products for developers who are developing machine learning and AI systems.

• 13:54Daniel Whitenack?

  Okay, gotcha. Yeah, and a prodigy, as you mentioned, is a data labeling system. Is that the best way to put it? I know that like it's more than just a user interface. It's a, it actually integrates with models and other things, right?

• 14:08Speaker 03

  Yeah, yeah, it's like a Python library. I mean, we sometimes refer to the whole like concept as machine teaching because it's, um, yeah, it's sometimes, it's often a bit more than just like annotating or labeling, because if you think of labeling, you think of, okay, you just get presented with something and you sign some label and that's it. Whereas, you know, prodigy really lets you script much more complex workflows, try out ideas, iterate on the label schemes, really develop the models and how you want to structure your data. But yeah, essentially it's, you can download it, you can install it, pip install it, just like any other package. And then it runs on your machine, on your data, just like, you know, like back in the old days when you would buy Photoshop and then you download Photoshop and then you have Photoshop and then you can keep using Photoshop.

• 14:51Daniel Whitenack?

  We were chatting a bit before jumping on the podcast and you were mentioning that both of you are really passionate about the sort of workflow and production details of, of actually doing natural language processing, machine learning in, in a practical setting. Is that kind of where prodigy came about, um, that you were seeing that, that slowdown in terms of, I guess, machine teaching and iteration around models and all those things?

• 15:16Speaker 04

  Yeah, so it's definitely informed by that. Uh, so we already had this, you know, pretty strong hunch that annotation tooling was a missing piece in a lot of people's workflows. And when we were doing the consulting, it really confirmed this as well. So we sort of, the data, you know, annotation was really where people were lacking in the terms of the success of their projects. And if we were thinking about why somebody would fail at a natural language processing project, and you know, indeed, many of these projects are going to fail because it is exploratory and stuff, it would be around these data questions and around the questions. Of like, okay, they couldn't get the annotation scheme uh such that they could carve up the the problem correctly to make the models work well, or they didn't have the annotations at all, or they had

• 15:59Speaker 04

  a process that too slow a feedback loop between the annotations and the developers. And all of these things were things we thought wanted to address.

• 16:06Speaker 03

  Yeah, for a long time, I mean, people were just like, okay, even though the data was the core of the application and really what the whole model depended on. They would like to throw this on like Amazon Mechanical Turk uh with some like vague instructions and like I don't know, some survey that looked like early two thousands online survey and then wait to get the data back. Get it back two weeks later and then, yeah, it wasn't that great. That wasn't very surprising.

• 16:29Daniel Whitenack?

  Yeah, a lot of effort for not much, not much useful data.

• 16:33Speaker 04

  Right, exactly. And, you know, it's it's sort of interesting to take a step back and think about the economic irrationalism of some of this stuff because the data science uh salaries are pretty sky high and they've been sky high for a long time. And now even the the machine costs of running the experiments are quite high as well. And if you look at all of the like, you know, efficacy of those ingredients and how many thousands of dollars of investment are going into things. Data should actually be pretty cheap and uh the tooling is pretty cheap as well. So for want of like, you know, a thousand or two thousand dollars of annotation, people would be spending these, you know, tens of thousands of dollars on a project that, you know, basically will be doomed to fail. So you can spend way more computation and try to eke out like a

• 17:18Speaker 04

  longer hyperparameter search and get an extra one percent. Or if you annotated like four times as much data and for like about four thousand dollars, maybe you'd get like an extra twelve percent, um ten percent. Uh so it's pretty starkly far more effective to be optimizing at that end of the scale.

• 17:34Speaker 03

  Yeah, so if that's what you're doing also, very early on, we saw, well, the the problem is not even, not the fact that we have to collect data. That's what people often um how people looked at this a lot, like, okay, we have to find methods like unsupervised learning. How can we like just not label anything anymore? It's like, well, labeling data is actually great because that's how you can program your model, like you you tell it what you want it to predict. Um the problem is the tooling, like if, you know, if the tooling is bad, then yeah, okay, the data is going to be bad. And if we can improve the tooling and we can actually make it much more efficient and better to use, more productive, then we can also end up with better data, better models, a better data science process.

• 18:12Speaker 04

  In particular, we saw that one of the problems that people had were, or one of the reasons why they hit this stumbling block on the annotation was that was often where people would hit the organizational boundaries. So that would be the part where, okay, you'd now needed to request a different type of budget, or you needed to cross over into a different type of, you know, basically organizational thing, or you'd have to have a meeting with an upper level manager because the annotation team was like across an organizational boundary because they had some people who, you know, didn't have much else to do today became the annotation team. And so we saw that actually all of those things where you, you know, now you've got an idea and you want to train a model. And now it's like, okay, but then I have to schedule a meeting for next Wednesday, pitch my idea and get buy-in for this. And

• 18:57Speaker 04

  I'm not at all sure that this idea will actually succeed. So that's really where the innovation or iteration speed goes to die. And that's why we actually set it up to be much more built for rapid prototyping so that data scientists could do some of this annotation themselves and say, all right, well, if I've got an idea, I don't need to ask anybody's permission about this. I can do the annotation to test it out over, you know, between now and lunchtime. And, you know, it's just sort of part of my daily work or like, you know, okay, I experimented with this. And it's not something where you need to, you know, basically taking all of your head brain schemes up to management and pitching them in meetings because that's not a productive way to work.

• 19:50Speaker 05

  This episode is brought to you by Brave. Big news from the Brave team. Version 1.0 is official. That means our favorite open source, privacy focused, blazing fast browser is ready for primetime. Their brand new iOS app landed just in time for the announcement and the Brave team is celebrating by granting 8 million basic attention tokens to the community. That means when you download the iOS app, you get 20 bat absolutely free. Put it to good use by heading to changelog.com, hitting the triangle icon in the upper right hand corner and flipping us a tip.

• 20:37Daniel Whitenack?

  So I'd love to switch directions just a little bit here. I saw one of your recent blog posts, which was talking about state-of-the-art NLP models, and there's a sort of simple four-step formula for a lot of these models. And I think that might be a good way to maybe introduce people to state-of-the-art NLP. And also, I found it really useful myself to kind of have that scaffolding in my mind as I'm approaching these problems. So I was wondering if you could introduce that formula a bit for these state-of-the-art NLP models?

• 21:13Speaker 04

  Sure. So let me see. So I guess one way to think about it is that these neural network models are all these sort of trainable functions that you can plug together. So the details of like exactly how they work sort of differ, and there's lots of different ideas for those individual components. But you can kind of take a step back and think about them as the input outputs of them. So one type of input that you'll have for language is, well, it's always going to kind of be a sequence of these discrete symbols. So at least if it's text data, right? So it'll either be a sequence of like characters, and you know, then you'll have an ID per character, or it'll be a sequence of words, or you could chunk it up into different segments that you've got like different IDs, but it's going to be come in as this stream of like numeric identifiers.

• 21:56Speaker 04

  So the first transform that you want to perform if you want to apply neural networks to this type of input is you need to take somehow take those that sequence of discrete IDs and map it into a dense representation, so vectors. So the simplest way to do this is to just have sort of a lookup table where, you know, let's say you've got a vocabulary of, I don't know, 10,000 words. It'll be a table of like, you know, with 10,000 rows and say 300 dimensions. And then you'll take some word and it might be like, you know, the 50th most frequent word in your vocabulary. So you'll select like row 50 of that table. And then that embedding table will be the parameters of the model, and you'll sort of train this to have representations where similar words are sort of mapped to

• 22:39Speaker 04

  a similar meaning. So you'd hope that dog and puppy would have similar vectors, and I don't know, drink and eat will have similar vectors, and you know, it'll all kind of work out as this sort of vector space. So that's the first embed step.

• 22:51Daniel Whitenack?

  Yeah, so basically neural networks like to act or like to operate on numbers, right? And so when you have these sequence of symbols or characters or words or whatever it is, in some way you have to represent that in numbers, correct?

• 23:05Speaker 04

  Sure. You know, in fact, every machine learning algorithm, neural or otherwise, is going to need to work on numbers in some way. The thing that was like, I guess, a puzzling challenge when I first started using neural networks as opposed to the other models which we were working with is the other models really like having sparse representations, like, you know, you can have an ID that's sort of just a key in a dictionary and it doesn't matter how many keys you have or like doesn't matter what the total space of the keys is, you only care about which ones you happen to see. In neural networks, it's not like that. You want to have a denser representation where you've got like some relationship between those. And that's kind of nice in that, you know, even if you haven't seen many examples of a puppy, I know that it's got a, going to have a similar representation to this other word. So you've kind of got that sort

• 23:48Speaker 04

  of, you know, relationship between things in a denser representation. And that's one of the advantages of neural networks.

• 23:55Speaker 03

  Yeah, I mean, ultimately, I think the main challenge is always there's so much knowledge about the world and the language that we kind of encode in text and we want to be able to encode as much as possible about all of this extra knowledge in our model as well. And that's also if you look at, yeah, the recent developments in LP, that's kind of what it all circles back to. And that's also, you know, the better the representations, the more we can do.

• 24:20Daniel Whitenack?

  Actually, you already said the word I was about to ask about and that was encoding, if you would go there when you're ready.

• 24:26Speaker 04

  Right. So the next step, you know, if you've got this four steps, it's like embed, encode. So after we've got like, you know, a vector representation for these word IDs, well, we're still at the stage where the vectors are kind of, you know, isolated. You've just looked up this word ID and you're going to have the same representation for that word type regardless of its context. So every time I see a word like duck, I'll have the same representation, whether it's a noun or a verb or, you know, so I'm not getting any sense of its like sense and context. So the next step that we want to do is sort of rewrite those vectors based on the words. Around them so that if it were based on the surrounding context, I can get another representation out. So this is a sort of input type that takes like, you know, this sequence of vectors.

• 25:12Speaker 04

  And it outputs a sequence of vectors with, you know, taking into account the linear order of the vectors that were put in. So we take an unordered set and we output like a, an ordered set. Well, wait, that's not quite right. Basically, we take the same sort of dimensions of like input and output, but now we've encoded the context around it. So one way of doing this is to say, okay, I've, I'll just concatenate the, the vectors for each word with the, some of its surrounding context. And I'll use that to recalculate that vector. So this is a convolutional operation. Or you can have a, um, you can read them in turn and at each point, remember that something of the context before and rewrite the vector based on that. So that's something like a recurrent neural network.

• 25:55Speaker 04

  But regardless of which, you know, method you use to encode that context, we probably want something like this. Because if we don't have something like that, then we're not going to be able to see the, the meaning of the sentence together. We'll have to just see the sentences, you know, the sum of its words or something, which is not the way that language actually behaves.

• 26:12Chris Benson?

  Yeah, I noticed in the article, you, you talk about both LSTMs and, and GRU architectures working well for that. Do you have a, a preference or is there one that you would recommend over the other?

• 26:23Speaker 04

  Uh, so these days, the transformer architecture is, you know, better than the LSTM for most situations. Partly because it's just more amenable to the way that GPU performance characteristics work. Um, so that's the one which pretty much everybody is using in place of the LSTM these days. Because we want space to work well in relatively small networks that you can, you know, use it without a GPU. We use a convolutional architecture instead. But I would say that those are, you know, basically the two, uh, architecture types that I use for the encoding step. The LSTMs are still sort of useful, but in most situations, like, you know, especially for larger models, transformers work better.

• 27:01Daniel Whitenack?

  So, uh, you mentioned the embed and encode. Apparently I've heard attention is all you need and you have a step that's attend. So maybe just, yeah, so I think the two steps left are attend and predict. Why is the attend, uh, step important?

• 27:16Speaker 04

  So I should, I should probably have called this reduce rather than attend. Um, I guess I was. Kind of hinting at like, you know, the attention uh layers were getting a lot of attention at the time. And it really would have been more useful to call it reduce. So the input and output types of this, you take like your matrix of things where you've got one row per word. And you basically want to get some sort of summary vector out of that. So, you know, I want a representation for the whole sentence rather than just a representation that includes all of the words of the sentence. So the simplest type of this would be just summing them all up or I'm taking the average of them or something. But you can also take a, you know, basically more parametric few ways of doing this and have one of these attention uh layers or something like that.

• 27:58Speaker 04

  Um. You can also use an LSTM for uh for this as well and just take the output state. So there's again a variety of ways of, you know, framing this, but it's essentially just a reduction operation where we take, you know, a matrix and we output a vector. And then finally, the the predict step is we take a vector and uh we want to get an ID out. So that's kind of the last, you know, type of thing that we want to do. So, you know, if we think of those as the sort of four data types of four, like, you know, signatures that we're going to have in these networks. Well, we're probably going to be, you know, basically composing uh layers that look like that. Um. And then there'll be other details for ones where you've got like, you know, two vectors as the input and then you want to attend over them. But mostly that's what we kind of want to do.

• 28:43Daniel Whitenack?

  And then I guess um in the end you have to predict something, right? Um. Is that uh so you've got the the reduced vector outputs, a a single vector and then the prediction. Is it kind of the opposite of where you started with the embedding?

• 28:59Speaker 04

  Actually, you can kind of think of it that way, but you can also just think of it kind of like a linear model and you're just going to say, all right. Um, uh, take a weighted sum of um a this input vector and some weights and at the end of it all like, you know, do some sort of max operation and say, okay, that's the one with the highest score. So it's just like other types of like, you know, machine learning models.

• 29:20Daniel Whitenack?

  So I'm assuming that, kind of getting back to where we started with Spacy, I'm assuming that Spacy can help you do some of the things that we've just talked about in this formula, but maybe there's other things as well. What's the sort of range of things that you can do with Spacy, I guess?

• 29:39Speaker 03

  Well, first, Spacy is really a library that lets you put together a whole NLP pipeline of the different things you might want to do and extract from your text. So, you know, often that's like, you know, you're not just interested in predicting one thing. You might want to read in your text, you want to find the individual sentences, you want to find out which concepts are mentioned in the text, like which person names, organizations, dates. And then you also maybe want to predict something about like what's in the text. And maybe later you want to relate these things that you've extracted to each other and compute something else. So the idea of Spacy is you have a pipeline, you can plug in functions into your pipeline. Some of them can be these machine learning models, others can just be a function, others where you can write some regular expressions, you can do whatever you need.

• 30:24Speaker 03

  And that's kind of the core principle of Spacy. We always had our own implementations that, you know, usually have a good trade off of accuracy and speed, especially also on CPU. But you can also write your own models, plug them in. And then at the end of it, you have, you can feed in your text and extract things from your text at a very large scale.

• 30:44Daniel Whitenack?

  So I guess I'm curious, like, how would you implement a pre-trained model? How does that fit into Spacy as a component? And, you know, maybe contrast that with if you were going to do it from scratch? How would you do that? What, how does that change the workflow for you?

• 30:58Speaker 04

  So it depends on exactly what you mean by pre-trained. Do you mean a model that's been trained for a particular task? Or do you mean, you know, weights that have been initialized with some language that you can then sort of leverage the knowledge from them?

• 31:12Chris Benson?

  I think from my standpoint, I'm thinking of kind of doing transfer learning and as the like as the newbie in this group, if I was going to dive into it and I'm taking somebody else's model that I want to utilize for a particular task and I'm wanting to stand on the shoulders of giants before me, how would I go about doing that as a newbie versus someone like Daniel who's done tons and tons of work in this space and maybe he's wanting to go in and do one from scratch. How would it be different for me and Daniel?

• 31:41Speaker 04

  Okay, so there's a number of different technologies around this. So, you know, most of your effort is still going to be around like, you know, creating the annotations for the specific problem that you want to do. And I would actually say that, you know, okay, you should sort of mostly be thinking about and focusing on getting the questions around that right, because there's actually a surprising number of choices to be made in how you frame the annotation problems. So for instance, you know, we have a number of users who want to work on medical text, right? And they say, okay, I want to recognize symptoms. And so then naively you'd say, well, okay, if this says, you know, patient suffers from asthma, that should be counted as like, you know, recognized as a symptom

• 32:25Speaker 04

  and that should be highlighted. And then if I have another one where it says, you know, patient used to suffer from asthma, then that shouldn't be recognized as a symptom. And so the sort of immediate intuition is, okay, that's the annotation scheme. That's what I should annotate. But that way framing the problem will be vastly harder to recognize for the models because you're coupling the two pieces of information about, you know, dimension of this thing and whether it actually was exhibited or not. And if you can find ways of framing the problems that you factor out those pieces of information, it's a lot easier. So I would say as a newbie starting out, that would actually be where most of the complexity lies. Then in terms of the actual software of using the pre-training, Spacy has one facility, which is just this command pre-train and you can either download

• 33:10Speaker 04

  the weight, some weights from us or use it yourself and that will, you can use that to initialize in Spacy Train. And then we're also working on, you know, better support for transformer models in our library, Spacy Transformers. Use the Burton Excel net models that have been trained on lots of text.

• 33:26Speaker 03

  Yeah, or even if you're starting just starting out, even just like plain old word vectors, like the common crawl glove vectors have been trained on a lot of a lot of text. Even that can give you like a nice little boost and that's like super easy to use. You don't have to think much about, you know, how it interacts. You just initialize your model with that, then you, you know, write a little script that trains your model on your data and then hopefully you get some nice results out at the end.

• 33:52Daniel Whitenack?

  So I think what you emphasized before in terms of NLP often being like a series of a series of tasks that you want to string together often because there is so much preprocessing and there is like multiple things that you might want to infer from text. And you've mentioned a couple of things like tokenization. You've mentioned finding certain things in the text, which I think you're referring to like finding entities like people or organizations, so like named entities in in text. I was wondering, so those are kind of building blocks that you can put together in these pipelines. I was wondering if you could mention maybe just some of the, some of the most frequently used of those sorts of building blocks in in Spacy.

• 34:33Daniel Whitenack?

  And I'd also be curious, because I have my own perspective from different things that I've done. I was curious, as you view like the community using Spacy, have you been surprised by which ones of those things have been like used most? Or people have found most useful? Or maybe it's the things that you thought they would find useful.

• 34:54Speaker 03

  Yeah.

• 34:55Daniel Whitenack?

  I'd be interested to hear some of that.

• 34:56Speaker 03

  Yeah, so I think definitely what people use the most is named entity recognition, as you just mentioned, and text classification, where you really predict one label over the whole text. We also, our default pipelines will also ship with a dependent with a part of speech tagger. So you can, you know, predict whether a word is a verb or noun and the dependency parser. So you can predict the relationships between the words. And for example, find out what's the subject and what's the object? Because that's also quite important. That's maybe one example where we think, oh, actually the dependency parse can in some cases be quite useful, but of course, you know, that's maybe it's maybe not. As popular as it used to be because, you know, we now have better ways of predicting these things end to end, but definitely it makes total sense to me why people would

• 35:40Speaker 03

  want to use named entity recognition and text classification because that's, you know, the most, yeah, useful information you could extract that also you can then translate into the business problem you're actually trying to solve. And, yeah, and one other thing, the rule-based entity recognition, that was actually in the matching, that's something I'm actually quite happy to see people use more and that's actually very popular. So you can, you can think of it kind of like regular expressions, only that you can write rules that really take advantage of the token attributes and maybe even things that the model predicted. So you can say, I want the word duck, but only if it's a noun and not a verb. And then I want to also extract an adjective if there is an adjective, otherwise not.

• 36:24Speaker 03

  And you can, you can basically, you can write rules like that and then also use that to extract more complex information. And for many, many tasks, this is actually a really, really powerful tool and works much, much better than just trying to predict all of it end to end somehow. So it's nice to see people use these hybrid workflows of statistical models and rule-based systems.

• 36:46Daniel Whitenack?

  Yes, I'm so glad you mentioned that because I think it is often overlooked and especially for maybe cases where you don't have as much data, you know, lower resource languages or something like that, you know, statistical and rule-based approaches can be really powerful. I know recently we were trying to figure out how should we predict if a certain sentence is a question or a statement? And we looked into various things and we tried out various things like, you know, text classification and like a larger model and all these things, but it ended up just some simple rules performed pretty much as good as any model we could train. So it was like, I could see how it could be overlooked a lot and I don't know if you see that.

• 37:30Speaker 03

  Yeah, yeah, I mean, we hear that a lot as well. Also, sometimes it comes from within the organization. Like I often talk to people, I'm like, oh, I need to extract digits from my text. How can I do that with the entity recognizer? And I'm like, actual digits, well, you just write like regx, right? And they're like, no, no, my manager says we need to use like NLP.

• 37:46Daniel Whitenack?

  But that's not as cool.

• 37:48Speaker 03

  Yeah, we need to, we need to train a model. And I'm like, God, that's like, you know, I'm sorry. For your like position in that company because that really sucks, but yeah, stuff like, stuff like that definitely. Or another thing we always try to tell people is this thing about like, okay, build at least some rule based baseline that you're looking to beat. Like for example, for your question um task, like you do want to find out how far do I get if I just check whether the last character is a question mark before you start like predicting predicting.

• 38:16Chris Benson?

  Which is surprisingly a long way. Yeah.

• 38:19Speaker 03

  Yeah, and maybe you're like, whoa, that's eighty percent or eighty-five percent. You're like, well, okay, that's good to know because if your fancy machine learning model only gets eighty-two percent out of context, that would look pretty good. And maybe, you know, you could show this off and it's like, wow, that's great results. But if checking the last character gives you a higher accuracy, then yeah, that's probably not what you want to ship.

• 38:40Speaker 04

  Yeah, this is, it's especially important with um the uh deep neural networks because it used to be that, okay, if the you can pretty much rely on some of the other models and like sort of reverse engineering or getting the accuracy that you would get from the simple rules. But um you can, training a deep neural network, you're you're often sort of running blind and you have no idea whether the score is any good. And you can find that, okay, actually, I'm dramatically underperforming like, you know, a bag of words baseline or something. And so part of the, it's very helpful to have this sort of existence proof of knowing where you should be because then you know, like, you know, okay, what to keep trying or when to keep trying and that sort of thing. So I definitely feel like having that sort of perspective on where you are or where you should be is very helpful.

• 39:24Chris Benson?

  So one of the things that I was wondering was I noticed that you talk about Spacy being designed intentionally to be blazingly fast. So it begs the question for me about, you know, kind of if you're focusing on performance, what are some of your strategies for making Spacy blazingly fast? And also, because we're talking about performance, it also makes me wonder, who are your target users on that? Versus like, what I mean by that is, are you, from a performance standpoint, are you thinking more about the data scientists that are creating the models? Are you thinking more about the engineers and the fact that uh for deployment and such? Because I also noticed that you you talk about it's uh production, you know, really focused on production. Could you speak a little bit toward performance and target users for that performance?

• 40:07Speaker 04

  Sure. So. The things that are important for performance have kind of changed over time as the um uh the technologies have changed. So it used to be that the fact that it was, you know, basically implemented from uh you know, the ground up in Scithen was, you know, very important for some of the performance aspects. Uh because the actual maths that was being done in the model were kind of simple. So uh it was very important that all of the data structures were in, you know, basically memory managed code. Now that it's uh more around neural networks, some of those considerations are a bit different. And there's a bit more forgiving in the Python layer can be a little bit slower because there's kind of more maths that's being done that, you know, is kind of a slower bit that masks the performance of the other parts.

• 40:49Speaker 04

  So I would say in terms of the the sort of target use case, a thing to keep in mind about natural language processing is that the problem sizes constantly get bigger. And this will continue for, you know, a wide range of companies and a wide range of applications. So the working set of like, you know, a problem that you're trying to handle will constantly accelerate. Uh so let's say you're a news site or something, the volume of comments you're processing or the number of articles in your archive. All of that's growing. And in many cases, it's actually growing faster than Moore's law. So the sort of standard approach that people have for computational efficiency of well, I'll not worry and it'll just kind of stop being a problem. It'll just kind of inflate away.

• 41:31Speaker 04

  Doesn't actually work so well for a lot of the problems that we want to do with natural language processing. If you want to work on the Twitter fire hose or other social media monitoring, again, the problem size gets bigger faster than computation gets cheaper. So we need to actually worry about the models being quick enough to work on those things. The other consideration is that if you make the model slow, then deploying them over a a very large cluster is just, it's a hassle that's never going to get easy. Like the more instances you have to spin up to just harder the problem gets. You get failures of the nodes, you get, it's just hard to be marshalling work over a very large number of workers. And so if we can make the models like, you know, fifty or a hundred times faster than the

• 42:15Speaker 04

  just operator expense of running uh things in practice gets a lot easier. Um and then finally, there's latency. So there's a lot of applications where you care a lot about the time to response of one or two things because you want to have the model in the loop of uh some sort of user facing decision. And there again, you need the models to be reasonably fast.

• 42:34Daniel Whitenack?

  So I'm going to switch directions a little bit here, maybe. Being that I'm working for an organization that, you know, whose vision it is to see people flourish in community with the languages that they use most. I would probably get fired if I didn't ask about language support. So there's a lot of languages and I'm guessing that, you know, various of these building blocks that you've discussed have support in certain languages and not other languages. I was wondering if you could speak a little bit to, I guess, first, the current language support, but also what people can do to expand the language support of Spacy, how they can contribute, what's involved in that? What are the challenges of that?

• 43:15Speaker 03

  Yeah, so I don't actually know the number now. Like, I don't know, how many languages do you support? Off the top of my head, I don't know. I don't know the number, but like it's in the docs. But we actually, the base support for a lot of languages is there. And that usually just includes like, okay, just some tokenization rules, getting like basics right. But ultimately, if you do anything for language and you want to train a model, that's kind of where the bottleneck is. And yeah, you can optimize algorithms for different languages, but ultimately it comes down to the data. And being like a library that's used in production, we are somewhat limited to like, okay, we need to source data. We need to source data sets that like can be used commercially, that exist in the first place and that we can maybe pre-train models with or that our users can use. And that's kind of what we're currently seeing is like the biggest problem.

• 44:00Speaker 03

  And that's kind of, that's not like anyone's fault directly, but like that's kind of something we have to work with. And that's also why it's not so easy to, if people are like, oh, why don't you support? Yeah, why don't you give me a perfect pre-trained model for insert language here? Or like, why does, why can't you support this? So why is this language so bad? And it's like, well, we have to work with what's there. We have to, you know, we can run our own annotation projects. We can run our own data collection processes, but like that's. That's really the main thing it comes down to.

• 44:29Daniel Whitenack?

  So have you seen contributions from various language communities around the world that really, you know, take ownership and contribute some of those models and the rules and all of that stuff?

• 44:40Speaker 04

  Um, yeah, so we've had a variety of contributions. So one that was particularly end to end and very effective was Yana Staris did a Google Summer of Code project where he contributed Greek support. We've had a number of people working on Indonesian. I think we had people working on Tamil.

• 45:00Speaker 03

  Yeah, we had like, you know, with some more custom, yeah, especially custom work. We definitely had a few users who went and used Prodigy to create their own NER annotations because that's also, you know, something that's usually lacking. We can have tree banks for dependency parsing and part of speech tagging, but like named entity recognition is like much more important to many users and also there's not enough data. We've had, what else do we have? So actually, yeah, I think some of the Nordic languages, they actually, I think Norwegian, they have like good, I think government sponsored initiatives and publish, have published good corpora under public domain. And that's of course like incredibly helpful and also high quality data. And then we've had users from the community who saw that and ran some experiments on it, got like pretty good results,

• 45:46Speaker 03

  shared that with the community and that meant we were able to ship a Norwegian Spacy model, for example, like a base model that people can build on top of.

• 45:54Daniel Whitenack?

  That's awesome.

• 45:55Speaker 04

  Another one that's like, you know, very notable is, so the Japanese support for a long time has been driven by Paula Leary-McCon, who's now doing freelance work. So if anybody's listening and they, you know, want to work on Japanese projects and they need assistance with this, he'll be, he's a great person to get in touch with and, you know, you can email me for the contact for him.

• 46:19Speaker 03

  Yeah, he's been really driving all the, yeah, all the Japanese, Spacy and Japanese.

• 46:23Speaker 04

  There's, there's another group in Japan who've been working with Spacy and they've got their own library for this as well. And so, you know, there's a number of contributions around that. I would say for something like, you know, the perspective of As still, if you're thinking about the vast number of languages in the world, well, tools like Spacy are kind of designed around different use cases. Like Spacy is actually designed um for, you know, written text processing. And so for a great number of languages, it's, you know, it's a little bit putting the cart before the horse because you say, okay, I can make this tool that can process lots of text, but what written text do I actually have to process? So if you're in a situation where the language actually doesn't have much written text, then, you know, okay, it'll be quite difficult to get Spacy running with

• 47:08Speaker 04

  things. But on the other hand, also Spacy wouldn't really be solving a useful problem for you either. And so I'd say that's actually the position of, you know, most languages, right? The other thing is that for those languages which, you know, sort of less typically written, well, at least the writing system tends to have been designed by linguists. And so they tend to be easier to tokenize than languages with, you know, slightly more complex histories around their writing system.

• 47:31Daniel Whitenack?

  Sure, yeah. And uh, yeah, I mean, there uh, there's a lot of um, great efforts. I I'm, I know the uh, Masacane project right now in in Africa is working on um, NLP tools for a lot of African languages and, you know, trying to, I think it's also the Zendi effort there is trying to, you know, gather a bunch of data that would be relevant to this so that, you know, you would be able to to start out and and build something useful. But um, yeah, it's it's great to hear that that you've had community contributions around that. And there's kind of thriving communities of people that are that are wanting to uh, help build in, build in that support. Um.

• 48:12Speaker 03

  Yeah, and it's also, I mean, this is also part of the reason we really want to focus on like the tooling around like creating the data. Because, you know, it's one thing to talk about like, oh, we don't have enough data. But like, if you, if we can have more efficient ways to script workflows that even maybe a researcher can say, hey, um, I don't have any labeled examples, but I'll just create my own little set so I can run experiments and like move, get this moving forward. That's like, that's pretty good. And I think can have a big impact if, you know, you don't see annotation as this like huge crowdsourced effort and something focused that like is actually not, actually quite easy to achieve.

• 48:45Chris Benson?

  So it's interesting for me, as Daniel just made those comments about some of those efforts, and I'm kind of wondering, as someone relatively new to this aspect, new to NLP compared to you guys, can you give us some perspective on general trends in natural language processing? And what are the exciting things that you see happening going forward? What are you excited about over the next couple of years?

• 49:08Speaker 04

  So I think it's definitely exciting to see the field developing, to see so many more developers get skilled up with this. So I would say that, okay, one way to answer this is, if you went back a few years, what were the questions which people were asking about how trends would develop, or like what would happen? And then if you compare that to now, I think you see an interesting perspective on that. So one of the things that people were wondering a few years ago was whether people would be running the models themselves on their computers, or whether everybody would just use an API. So would everybody use the one API that was like, you know, the Google Understand Language API or something? And that would just be what people would use for this? Or would it be

• 49:50Speaker 04

  the case that, you know, as we've seen now, actually, more and more people, you know, are building the models, and more and more people are involved with, you know, quite detailed libraries, and have even switched over from a library like TensorFlow to something more flexible like PyTorch? And so I think that's been answered decisively in the direction of, okay, people want it to be programmable, and they want to understand the workings of the model. They don't want a black box that, you know, where everything's done for them. And I think that the reason for that is that there is no sort of one answer for a lot of these things. You need to script the problem yourself. You need to, you know, have it recognize exactly what you want it to recognize, and the model to work the way that you want it to work with, the features that you want it to work with. And so I would say that that's definitely been a trend that we expect to continue,

• 50:34Speaker 04

  and we expect the, you know, general like, sort of savviness and knowledge of people, and, you know, they'll want to work with basically the most effective ways of doing this, rather than the ways which are like, you know, superficially the simplest.

• 50:47Speaker 03

  Yeah. Yeah, and I think also, even if you look at like, new developments like transfer learning, that's of course very exciting, because, you know, it means we can reuse knowledge better, and transfer it between different things with training. And so we do think that actually, you know, there's some trend that It moves away from this idea of like, oh, we need this big, big, big data, huge operation. And we can actually, you know, work on like kind of a medium scale, try out a lot more things in the workflows. And also I do think, yeah, we see a lot of systems that like just work end to end. And, um, you know, people are like, well, cool. If I can just like throw like bird at it, um, it'll just like magically work.

• 51:27Speaker 03

  But I think as the field develops and also as like the problem develops, um, I do think there's still, you know, a lot of challenges are much more on a level of, okay, I can predict all kinds of things. I can do it very quickly without needing like too much data. But how do I really translate these predictions into my very, very specific, domain specific problem? That I have to solve for my business use case. And I don't think there's like an easy answer to this on the technology level. That's like something you, you as an expert, you need to know what questions do I ask? What can I train? What will work? If, once it works, how do I interpret the results and how do I all put it all together to answer the questions? And yeah, and that's something I think, yeah, you can't really predict end to end.

• 52:10Daniel Whitenack?

  So let's say that, and I hope that we have, um, but let's say that we've inspired some listeners out there to get hands on with NLP with Spacy and with Prodigy and with the other, the other tools that you're releasing. I know that you've actually built a course for NLP with Spacy. Is that right?

• 52:29Speaker 03

  Yep. Yeah, it's free. It's, it's online. Um, and it's, uh, yeah, it's available at course.spacy.io. And it's kind of interactive. You can, you know, you get some little prompts. You can enter code. Uh, you can check it, but it's like all, yeah, intended for self-study, but it's been very popular and well received. So, yeah.

• 52:44Daniel Whitenack?

  Yeah. Would you, would you recommend that as kind of a good place for someone who's maybe toyed around, uh, at least with, with Python and done maybe some, some site get learned stuff or something like that? And then want to do something NLP related? Um, would that be a good place to start or there, there are better places?

• 53:01Speaker 03

  I think the course, like I really tried to, um, design it in a way that it also explains all the concepts, um, about NLP. So even if you've, if you've never worked with like NLP or machine learning before, um, I think it's still, um, you know, gives a good perspective and gets you productive. But also if you have done a lot of machine learning and no NLP, you know, I think there's still enough in there that like, you know, it feels valuable.

• 53:23Daniel Whitenack?

  Yep. Awesome.

• 53:24Speaker 04

  I would say that in terms of other resources, for like more general machine learning stuff or understanding neural networks a bit better, the Fast AI course is very good.

• 53:34Speaker 03

  Yeah.

• 53:34Speaker 04

  And then, actually, a little book that I recommend to some people is "Machine Learning Yearning" by Andrew Ng. So I don't like all of his talking points, but that little book is actually quite a useful primer in setting up machine learning projects. And it has some advice that, you know, it sort of gets lost along the way about, you know, what's an evaluation said, how do we do these things, like that sort of stuff. And it's a pretty short read, so that's also a nice background as well.

• 54:02Daniel Whitenack?

  That's great. We'll definitely put the link to the course and to the book, Fast AI. Again, I think we've linked Fast AI, I don't know how many times at this point, Chris, but we're big fans, yeah, we're big fans as well. And actually, they have another course that's specifically NLP focused now.

• 54:21Speaker 03

  Yeah, no, that's actually, and I really liked, I kind of liked the curriculum there because it also starts with the basics. I think it looks at like very basic rule-based approaches, the whole history, and it's not just like, oh, here's like, you know, the hippest thing. And I think also it does cover a lot of ethical aspects too, and bias in models, which, yeah, is also something you don't typically get from like your average programming course.

• 54:44Daniel Whitenack?

  Yeah, sure. Yeah, really appreciate what that community is doing and what it's, all the tooling and the courses and everything has meant through the recent years. Well, thank you both for taking time to talk with us. We'll definitely link everything that we've talked about in our show notes. There's a lot to explore. There's a lot of questions that I'm sure people might have. Feel free to join us on our Slack community. You can find that at changelog.com/community, or LinkedIn, or Twitter, or wherever you find yourself, and get plugged in with the Spacy community, try out some things, and really appreciate you both being here and looking forward to great things from Explosion and Spacy in the future.

• 55:24Speaker 03

  Thanks.

• 55:24Speaker 04

  Thanks.

• 55:27Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go on your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for change law is provided by Fastly. Learn more at fastly.com. And we catch our errors before our users do hear change law because of Rollbar. Check them out at rollbar.com/changelaw. And we're hosted on Linode cloud servers. Head to linode.com/changelaw. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address. Get our weekly email, keeping you up to date with the news

• 56:12Speaker 00

  and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.