🤗 All things transformers with Hugging Face

Sash Rush, of Cornell Tech and Hugging Face, catches us up on all the things happening with Hugging Face and transformers. Last time we had Clem from Hugging Face on the show (episode 35), their transformers library wasn’t even a thing yet. Oh how things have changed! This time Sasha tells us all about Hugging Face’s open source NLP work, gives us an intro to the key components of transformers, and shares his perspective on the future of AI research conferences. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean’s developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.99% uptime SLA, and 24/7/365 world-class support to back that up. Get your $100 credit at do.co/changelog https://do.co/changelog. The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Sasha Rush – Website http://rush-nlp.com, X https://x.com/srush_nlpChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Giveaway details!! Check this blog post https://changelog.com/posts/giveaway-celebrating-zeno-rochas-new-book for all the details to win a free copy of Dracula PRO && 14 Habits of Highly Productive Developers Hugging Face https://huggingface.co/Transformers library https://github.com/huggingface/transformersTokenizers library https://github.com/huggingface/tokenizersNLP (data and evaluation metrics) library https://github.com/huggingface/nlpPrevious Practical AI episode with Hugging Face https://changelog.com/practicalai/35TechCrunch announcement about Hugging Face’s recent fundraising https://techcrunch.com/2019/12/17/hugging-face-raises-15-million-to-build-the-definitive-natural-language-processing-library/The annotated transformer https://twitter.com/srush_nlp/status/9815840731886673942000+ models in Hugging Face’s model hub https://twitter.com/julien_c/status/1278336685730869248Attention is all you need paper https://arxiv.org/abs/1706.03762Mini Conf tools http://www.mini-conf.org/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  One thing that we think is really interesting is that unlike generic model hubs like TensorFlow's hub or PyTorch's hub, because our models are all of the same form, we can build a lot of tools and machinery around using them. So for instance, we have a visualizer that works for all of our models. You can just upload your own model and get really interesting visualization of its internal structure. Or this open source project called Text Attack built an adversarial attack system. And it's able to generically build attacks to any of our models in our hub. So because they all have the same interface, it allows people to do these really longitudinal research projects across everything that's going on in the hub itself. And then I should mention that now we have a kind of an inference API.

• 0:41Speaker 00

  On any of the pages, you can just type in some text and it will run against that model. And you can even call that from your own code directly without ever running anything on your machine. Just run it on one of these servers. And we even have a Twitter bot that we just put up last week where you can tweet at it and it will run a model against your tweet.

• 1:01Speaker 01

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at ChangeLaw because of Rollbar. Check them out at rollbar.com and we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 1:19Speaker 01

  This episode is brought to you by Digital Ocean. Droplets, managed Kubernetes, managed databases, spaces, object storage, volume block storage, advanced networking like virtual private clouds and cloud firewalls, developer tool like the robust API and CLI to make sure you can interact with your infrastructure the way you want to. Digital Ocean is designed for developers and built for businesses. Join over 150,000 businesses that develop, manage and scale their applications with Digital Ocean. Head to do.co/changelog to get started with a $100 credit. Again, do.co/changelog.

• 2:06Chris Benson?

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:32Daniel Whitenack?

  Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International, and I'm joined, as always, by my co-host, Chris Benson, who is a Principal AI Strategist at Lockheed Martin. How are you doing, Chris?

• 2:50Chris Benson?

  I'm hanging in there. How are you doing, Daniel?

• 2:52Daniel Whitenack?

  Doing pretty good. As we talked about the last couple of weeks, I've been ordering parts for a AI workstation computer, and it's sitting next to me and it's running.

• 3:03Chris Benson?

  Oh, nice.

• 3:04Daniel Whitenack?

  I am successfully, or at least it appears that I'm successfully overfitting a model on the GPU. So I'll have to deal with that, you know, after recording, but it's running, and it's not overheating yet. It's kind of stable at a, I think, reasonable temperature. So I'm happy on that front.

• 3:23Chris Benson?

  So it's funny, because, you know, as we're on this call, we're on Zoom, and in the video, you have the data center in the background. So I just find it funny.

• 3:31Daniel Whitenack?

  Yeah, a bunch of DGX Nvidia machines or whatever. Yeah.

• 3:35Chris Benson?

  But that's not what you're using.

• 3:36Daniel Whitenack?

  No, mine is much smaller, although it's bigger than I thought, because after I put the GPU in, the case would not close. So I guess that just is like airflow.

• 3:47Chris Benson?

  Perfect. There you go. It solved itself.

• 3:50Daniel Whitenack?

  Yeah.

• 3:50Chris Benson?

  I'm just doing okay. I did something really stupid this morning. I was reminded that I am a klutz. I fell when I was running, and it looks like I broke a rib. And you'd think that I'd do something about that, but I'm lucky. I have a fourth-year med student for a stepdaughter. So I called her up, and we agreed because COVID is running rampant, we were not going to have me go to the emergency room. So she said the treatment would be the same either way. So I'm just kind of cranking through the day, doing my thing. And now we're recording. We're having fun, man.

• 4:20Daniel Whitenack?

  You're really pushing through the pain for AI podcasts.

• 4:24Chris Benson?

  There you go. You got to be practical.

• 4:25Daniel Whitenack?

  Yeah, that's something.

• 4:27Chris Benson?

  Okay.

• 4:27Daniel Whitenack?

  Yeah, feel free to

• 4:29Chris Benson?

  Scream at any

• 4:30Daniel Whitenack?

  mute yourself and scream a couple of times or whatever you need to do.

• 4:33Chris Benson?

  Okay, will do.

• 4:35Daniel Whitenack?

  Well, going from that note to something completely different, as the show might say. We're really excited today because we have a follow up on a show that we did quite a while ago. Actually, this was episode 35. So quite a while ago, we had Clem DeLong on from Hugging Face to talk about what they were doing. And now we're very excited to have Sasha Rush joining us, who is an associate professor at Cornell Tech and is also working at Hugging Face on a bunch of different things and involved in the Transformers library. And so we're really excited to have you Sasha to hear more about Hugging Face.

• 5:12Speaker 00

  Oh, thanks. Thanks for having me on.

• 5:14Daniel Whitenack?

  Yeah. Before we jump into all of that, could you just give us a little bit of a sense of your background? And how you came to into the field of AI and eventually into NLP and what you're doing now?

• 5:27Speaker 00

  Sure. Yeah. So I've been at Cornell Tech for the last year. And if you don't know about Cornell Tech, it's a new university. It's about seven years old, but we've had buildings for the last two years. Our buildings are right in the center of New York City on an island in the middle of the East River. So every day we kind of take a little gondola over to the island and teach courses there.

• 5:52Daniel Whitenack?

  Ah, how romantic. Yeah.

• 5:53Speaker 00

  It's a pretty fun place. And yes, I've been a professor here for the last year. Before that, I was a professor at Harvard for about four and a half years. And before that, I was a postdoc at Facebook AI Research, also in New York. So that's my background. These days I have a lab here at Cornell Tech and I work with the team at Hugging Face who are in Brooklyn. So it's nice. Everything's kind of centered in New York City. Lots of interesting AI and machine learning going around here these days. So my background, after graduating college, I worked as a software engineer for about three years. I'm kind of a person who very much enjoys coding and I kind of

• 6:40Speaker 00

  Have that as kind of the first part of my personality. I then went to graduate school to study natural language processing. When I got into natural language processing, I think I really got into it because I was very interested in language. And particularly kind of the algorithms and data structures involved in studying and understanding how language works. At that time, I did a lot of machine learning, but machine learning wasn't kind of the primary way we studied language. There were all sorts of other aspects about kind of how computers and language interacted. And actually, my dissertation was much more about, say, the optimization aspects of language in a discrete sense. Kind of how you construct trees that represent different linguistic phenomena.

• 7:24Speaker 00

  And how these interact with kind of classical computer science algorithms. And when I graduated my PhD, I kind of graduated right into the beginning of really kind of intense deep learning for language. And doing my postdoc at Facebook, everyone was kind of intensely interested in how we could do translation, how we could do question answering, kind of completely from data, using deep learning based systems. So I kind of dived right into that world. I sat next to the folks who were working on Torch at the time. And then it was written in Lua, and a couple of years later, they converted it to Python and it became PyTorch.

• 8:08Speaker 00

  So I've always been very fascinated by kind of the tools and structures that make it possible to do these sort of systems in a kind of open source way. Some other things I've worked on in the past, I worked on a library called Open N M T, which was an open source translation library, written in PyTorch and TensorFlow. And we worked with a lot of translation companies, particularly in Europe.

• 8:33Speaker 00

  To try to build open source tools to let them build their own kind of custom Google Translate like services. And that was a really fun project and it kind of tied together the research we were doing in my lab, which was on kind of questions of how to improve translation, how to speed it up, how to make it work on devices with questions of how in an open source world, these were used.

• 8:55Daniel Whitenack?

  So I'm kind of curious, since you kind of alluded a little bit to one thing that's kind of happened in recent years in terms of how I guess people maybe used to think about NLP and still do for many tasks as far as like computational linguists have been thinking about these things for a very long time. But now there's been all of this focus on kind of extending these tasks to maybe generalized machine learning type problems. Could you give your perspective on kind of how that shift has happened and like what that's meant, both in terms of momentum in the field and people getting involved in the field and all of that? What are your thoughts on that?

• 9:38Speaker 00

  Yeah, let's see. So I think there's a couple of different perspectives. I don't want to make it seem like kind of data-driven or machine learning systems were kind of new to NLP. There's a long history of use of learning, both in NLP, but also kind of learning systems developed in NLP being used in other areas. So I think it's a field that's always kind of interacted with these methods in a kind of open dialogue. I think the phenomenon we're seeing now is kind of more extreme. And it's extreme for a couple of reasons. I mean, one is the sheer growth of all these fields. We're seeing kind of exponential growth in conference sizes and paper submissions and kind of usage of this technology,

• 10:22Speaker 00

  which I honestly think is a great problem to have, but it obviously brings with it a lot of challenges. So there are kind of organizational questions of kind of running communities or kind of trying to kind of make progress in this world. I think the other question is, what does it mean in terms of methods? And we're seeing lots of interesting things along those lines. I think that people in the field are adapting to the challenges that kind of come kind of from the world around, like as researchers, we're interested in solving the problems that exist now. And a lot of the problems in NLP are suddenly kind of dataset problems.

• 11:04Speaker 00

  How do we construct interesting, novel and difficult datasets? How do we analyze models to understand what they're doing and how they're structured and what they're learning? Kind of societal questions of how do we understand what biases they might have or what issues they might bring? Or even how they might learn, like from what signals are they picking up on? And so there's no shortage of interesting research going on. It's just that what's interesting is maybe less so the kind of how do you make the benchmark problem go up X number of points?

• 11:42Chris Benson?

  So I'm kind of curious. I've been thinking about listening to this and, you know, we had Clem back on the show back, I think it was episode 35.

• 11:52Daniel Whitenack?

  35.

• 11:53Chris Benson?

  Yeah, going way back.

• 11:54Daniel Whitenack?

  It was before the Transformers library came out, which we'll definitely talk about later.

• 12:00Chris Benson?

  Yeah, totally. I think that was what I was thinking about was the fact that when we were talking to Clem, we were really kind of focused on like social AI and chatbots and similar tools and approaches. And then in that time between talking to you today and talking to Clem, you know, Transformers came out and you guys really created the definitive Transformer library. And, you know, we've been talking about Hugging Face in the context of Transformers since then. And I guess, how did Hugging Face make that transition? What caused that? And it's an interesting turn for the, you know, for the history of the company.

• 12:34Speaker 00

  Yeah, so I mean, I guess I should give some perspective. So I've actually only worked at Hugging Face for about eight months now. And honestly, I ended up working there because I was such a fan. I observed them in the same way that you did, which was as an external observer, seeing them make this transition. So impressively from kind of working on chatbots to being this kind of open source powerhouse. And I guess as someone who, I guess, I mean, who knows what it means in open source. But as a competitor, as someone building his own libraries in this space, they were just doing it so much better than I was. And so I think that that always impressed me.

• 13:18Speaker 00

  Now, I should say, even before Transformers came out as an official library, I have memories of, well, I guess now we're getting into some of the technical terminology. When BERT came out as a paper, there was a kind of rush to port BERT to a PyTorch version. And I was working a little bit on this at my own pace. And Hugging Face very, very quickly put out their own version of this, maybe part of their chatbot library, maybe it was a separate thing. And I think it was really useful just to have that immediately, right after the research came. And so I was really appreciative of that even at the time.

• 13:52Daniel Whitenack?

  What's the state of Hugging Face now in terms of, I know that they raised a round of funding. It seems like, from what I'm picking up on Twitter, that the team is growing a little bit. But from chatting with you before, it seems like it's still also very distributed. There might be some kind of creative relationships, like, of course, you're in academia, but you're also with Hugging Face. So sort of what's the state of the Hugging Face team now? And how's it growing to support this really rich ecosystem of tools?

• 14:20Speaker 00

  Yeah, so we have about 15 to 20 people, depending on how you count. We're mainly focused or entirely focused on these open source tool development. The main library is Transformers, which we've talked about, and kind of is this the center of what we're developing. But now there's also several other really interesting open source projects going on. So we have a project based on NLP data sets that now has almost 150 different open data sets that you can easily browse and download and use in a very efficient and Kind of easy to extend way. We also have a library of tokenizers that's written in Rust, a kind of low level library that lets you do very fast tokenization and training.

• 15:06Speaker 00

  And then all this is kind of joined together by a kind of hub of different models and structures that people have uploaded. And if you go to the website, you can kind of see this kind of really rich ecosystem of different models, of different data sets, and the different tokenizers that kind of build this all together. Practically, it is an interesting question of what the company is like. I mentioned earlier that I've been there longer now, probably in COVID than not in COVID.

• 15:34Daniel Whitenack?

  Yeah, that's, that's true.

• 15:36Speaker 00

  But it's always been a distributed company. There are, there's a team in Paris and a team in New York. It's about half and half. But now we also have interns in California and some interns in China, some people in different places. So we mostly kind of communicate through Slack and through other distributed means.

• 16:08Speaker 04

  Change log news is the best way to keep up with the fast moving software world. We track, blog, and contextualize the coolest projects, the best practices, and the biggest stories each and every week. Make changelog.com your daily destination, or hit the snooze button and subscribe to our weekly newsletter that hits inboxes on Sunday mornings. Join more than 15,000 enthusiastic readers. It'll cost you exactly $0 and you can subscribe right now at changelog.com/weekly.

• 16:45Chris Benson?

  So I guess, you know, we've alluded to transformers several times now and kind of talked around it a little bit. For those who are new to the topic, could you kind of define what is a transformer? I mean, it's been a big, big deal in recent months and has really changed NLP, but a lot of people may not be familiar with it or have not kept up to date. Could you kind of just give us a basic run through from the way you see it?

• 17:07Speaker 00

  Sure. So I think the term transformer really kind of implicitly applies to different innovations.

• 17:14Speaker 00

  And both of these were actually connected to each other, but both pretty transformative in their own right. So I'll start with the first. So the first is the transformer as an architecture. So this is the particular kind of development of a very specific type of architecture that came out. And the dominant architecture in natural language processing for about five years had been recurrent neural networks, particularly the LSTM network, and was used basically for everything that we did in the field. And the transformer proposed a different, and in fact, kind of simpler, architecture that instead of kind of reliant on these recurrent connections, kind of connections

• 17:56Speaker 00

  over time, instead used a kind of random addressing style of architecture based on a mechanism called attention. And the way it works is that you basically have everything you've seen in the past ready to access at every point in time. And the main kind of neural network step that you take is the kind of soft random addressing over all your previous history. And you use that in order to compute the next stage in your sequence. So instead of kind of keeping a fixed length vector that gets transformed over time, you keep around everything and you basically search through it at every stage in the process. And this architecture wasn't kind of new on its own right, but kind of demonstrating

• 18:39Speaker 00

  that it was more effective than recurrent neural networks and that it particularly could scale to both kind of fast training and also very, very large models, better than recurrent neural networks, was kind of a big breakthrough in the field. And the first results showed kind of large improvements on translation accuracy.

• 18:59Chris Benson?

  Just a quick question. You mentioned attention and you sort of defined it in the larger thing, but just because it's a kind of a key aspect of that, could you talk about what part of that was attention? Just to differentiate it from the larger process?

• 19:13Speaker 00

  Sure. Yeah, I should say the original transformer paper has the title, "Attention is All You Need." It's kind of the key aspect of what makes a transformer. Attention itself is actually quite simple and it's actually kind of very intuitively appealing idea. So imagine you have a set of objects, say five different objects, and you want to have a neural network decide which one of those objects you want to use. You might have a softmax layer where the softmax gives you a distribution, a probability distribution over which aspect you want to pick. So which of the five things you should choose? You could just end there. And if you ended there, we would just call it multi-class classification.

• 19:55Speaker 00

  What attention does is it uses that distribution, the probability of each of the five things, and feeds that probability back into the model itself. So it would give a weight to each of the five items and then feed them back in with that weight. So imagine I have a sentence like, "The man walked the dog." And I want to predict the next word in that sentence. Those previous five words would be the five items I'd want to choose from. And attention would say, "How much weight should I give to each of those previous five words when trying to decide on the next word?" So maybe I'll give it 80% to "man," 5% to "the," etc.,

• 20:37Speaker 00

  and use those in the next step of the process. All the transformer is, is a kind of repeated version of that game for, say, six to 24 different rounds, where each time you look back at what you've previously decided, use it to feed it back into your network and then use that to try to predict the next step along the line.

• 20:57Daniel Whitenack?

  So you mentioned that this architecture, also in addition to kind of having this new structure, also allowed some performance benefits and scaling as well. I was wondering if you could just give a sense of, because I know this is something people see out there. And in particular, I think there was a thread on Twitter about how many parameters are in the Hugging Face model hub and all of that.

• 21:22Speaker 00

  Yeah.

• 21:22Daniel Whitenack?

  So I was wondering if you could just give us a sense of, you know, what are the sort of scale of models that are out there? People hear about like... BERT and GPT-2, and now, of course, we're getting flooded with GPT-3 things. What are the sort of scale of these models, both in terms of like parameters and also like the data needed to actually train them?

• 21:44Speaker 00

  It's a good question. I never know these numbers offhand. Models range from 50 million parameters to tens of billions of parameters at the top end. In practice, some of the larger models, it's unclear how you would even use them, say, a standard GPU hardware. But scale has been a big kind of main aspect of kind of transformers usage. But actually, maybe let me pick that question to talk a little bit about the second main innovation of the transformer. I talked about the architecture, but I think it's important to also get a sense of the second innovation, because I think it actually matters even more. This is a kind of innovation that started around the use of a model called ELMO. There were a couple other variants, one called COV.

• 22:29Speaker 00

  And then this all kind of peaked with the release of a model called BERT. And the kind of idea behind these models is to take a neural network, in the case of BERT, a transformer, and to train it on a very simple task at a very, very massive data scale. So in the case of BERT, the task is similar to the one I described previously. You're given a bunch of words and you randomly remove some of the words and try to predict them back. It's a game that you can play yourself and try to get a sense of how easy or hard it is to do. Sometimes it's really easy, sometimes it's really challenging. But the point isn't the task itself. The point is to give the model a task that would require it to know something about

• 23:11Speaker 00

  language in order to complete, and then train it at as big a scale as you can. So it's hard to give you a sense of this. I mean, one thing that's nice about language is you can store a ton of it in very little space. So if you have all of Wikipedia, just you can basically fit it on your computer. And companies like Google basically have a non-trivial amount of all the text that's ever been produced. And so you can kind of take all that text, throw it into one of these models, and then train it on this simple task. And it turns out that in the process of trying to complete this task, the model learns a lot about how language works. We say it learns very good features for language. So once you've done that, once you've kind of trained it on all the language that you have,

• 23:55Speaker 00

  you can then apply it to a much smaller task that you maybe have a small amount of supervised data for. So this idea, which people call pre-training, is kind of central to how a lot of NLP works these days and also to how the transformers library is designed.

• 24:12Daniel Whitenack?

  So yeah, I think that's such a great and important point is that people kind of get hung up on the size of these models and it's kind of cool to talk about those things and in some cases annoying to work with them because they're so large and in some cases hard to perform inference with. But yeah, I guess what you're saying is, you know, that the task that they're trained on is just intended to help them learn good features and then the task that you actually want to use them for involves some like fine tuning or transfer learning. Is that right?

• 24:49Speaker 00

  Yeah, I think, I mean, I don't want to claim that this is finished as an idea.

• 24:54Daniel Whitenack?

  Right.

• 24:54Speaker 00

  I think a lot of the tasks we work on now will have a kind of fine tuning stage where you take the model and learn it for a given task. OpenAI has a slightly different model of what they're trying to achieve, which is they're not super interested in fine tuning. They want to kind of just use the model directly and kind of feed it some more sentences and try to directly kind of predict tasks.

• 25:14Daniel Whitenack?

  Yeah, so there is this like, because I've seen, and maybe you could kind of help us through some of this jargon. It seems like people talk about some of these models, they just like they have so much knowledge that you can perform a task that they just right off the bat, like I don't know if it's question answering or information retrieval or whatever it is without really much fine tuning. Is that what you're kind of getting at in that other model?

• 25:39Speaker 00

  Well, I do want to distinguish kind of two aspects. I think that all the kind of state of art models on kind of standard benchmark tasks all use some sort of fine tuning. That's like become a very standard procedure and we kind of understand how that works. But to do fine tuning, you still need some amount of supervised data. I guess we would say it's a small to medium amounts, but you need something in domain for the task you're interested in. I think there's a lot of recent excitement for a kind of a crazier idea, which is this kind of zero-shot or one-shot idea of just the model should know how to do your task immediately, right off the bat.

• 26:17Daniel Whitenack?

  Yeah, I think that's where I was going because they throw around this idea of zero-shot and to some degree, it seems sort of like magical in many, in many ways to people, I think.

• 26:26Speaker 00

  Yeah, I don't want to say anything on record. It's on the research frontier.

• 26:30Daniel Whitenack?

  Yeah.

• 26:31Speaker 00

  It might turn out that's the way to do lots of language tasks, but I think still an open question.

• 26:35Chris Benson?

  So, turning to the transformer library itself, I'm kind of curious, so in recognizing that you've only been there at the company for a limited amount of time.

• 26:44Speaker 00

  Yeah.

• 26:45Chris Benson?

  Do you have any insight into kind of the motivation that moved the company into this transformer library itself? Was it supporting the other operations or was it just something that was an opportunity that came up? What kind of took the company there as far as you're familiar?

• 26:58Speaker 00

  That's a good question. The graph of the usage of this library on GitHub kind of blows me away. Like it went from

• 27:07Daniel Whitenack?

  It's pretty insane.

• 27:08Speaker 00

  No users to about 30,000. So I think they just hit on something that was like, I guess when you have a hit, maybe that changes the mode of thinking.

• 27:16Daniel Whitenack?

  Yeah, so maybe you could describe like, along with that, what is the sort of main usage pattern that people are kind of grabbing on to transformers for? I know that there are multiple, of course, like quite a few of different things that you could use the library for, but what do you see as the sort of like the main thrust of what people are grabbing transformers for? What is that and you know, how is that being supported, I guess?

• 27:45Speaker 00

  Yeah, this is a great question and I think in some ways you guys maybe have insight into this that I would be also interested to hear about. Let me start at the high level. One thing that fascinates me about kind of current usage of deep learning is that you have people who approach it from many different angles. And in one of our papers, we kind of broke this down into three different classes. So we talked about there being architects, there being trainers, and then there being end users. And I think within the ecosystem, transformers kind of has different meanings to all three of those people. So if you're a company like OpenAI or like Allen AI, kind of companies at the kind of cutting edge of research training, you use transformers or kind of related libraries

• 28:29Speaker 00

  to try to build the next architecture or the next pre-trained model. And that often means running these very large training jobs on multi-GPUs over many days. And then using transformers as a way to distribute your model through our hub and make it easy for people to use it or to adapt it for their tasks. If you're like an expert, but maybe not kind of at the kind of front end of the like frontier of research, another common use case is this kind of fine tuning use case where you have data for your company or for a given problem that you want to solve. And you bring that data into the library, use it in training mode to fine tune on your data set.

• 29:13Speaker 00

  It may take a couple of hours and require some GPUs, but out of that you get a really accurate model for the task you're interested in. But then at the other end, you have just end users who want to use the library as a way of kind of performing kind of standard NLP tasks. You might want to use it as a way to do summarization or translation or named entity recognition or question answering. And you could often just use it completely in inference mode, maybe not even using Python, just kind of taking a pre-trained model, using it directly for your task in that kind of setting. So I think all of these people are within the machine learning ecosystem, but they kind of have different end goals or different use cases.

• 29:56Speaker 00

  And I think we're kind of trying to aim to support any of those those kind of outcomes.

• 30:00Chris Benson?

  So I know you have a model hub and was wondering if you could kind of talk about, you know, what users can find there and start incorporating into their own projects. What does the growth of that hub look like? You know, just what kind of ecosystem has developed around it?

• 30:15Speaker 00

  Yeah, so the model hub is kind of part of the open source library. If you want to use a model in the library, you say model dot load and you pull up, you just pull it directly down from the model hub. And you can do that with any of the models that are there. We have kind of a set of models that kind of have brand names that are very often used. So those include models like GPT-2, or a variants of BERT, or Roberta, or new models like this model called BART, or a model called T5. But then it also includes a long tail of other models from the community. So this includes models that are pre-trained to target, say, biomedical text, or extraction from scientific documents, or models that are trained in many different languages,

• 31:00Speaker 00

  kind of by the communities interested in those languages themselves, or models that are experimental, or try to do other things. Or one popular aspect is models that are very small, models that you could run on your phone. So the idea of the model hub is to kind of have all of those have the same API, and have the same easy way to use them. And one thing that we think is really interesting is that, unlike kind of generic model hubs, like TensorFlow's hub, or PyTorch's hub, because our models are all of the same form, we can build a lot of kind of tools and machinery around using them. So for instance, we have a visualizer that works for all of our models. You can just upload your own model and get really interesting visualization of its internal structure.

• 31:44Speaker 00

  Or this open source project called, I think it's called Text Attack, built adversarial attack system. And it's able to kind of generically build attacks to any of our models in our hub. So because they all have the same interface, it allows people to do these really kind of longitudinal research projects across everything that's going on in the hub itself. And then I should mention that now we have a kind of an inference API. On any of the pages, you can just type in some text, and it will run against that model. And you can even call that from your own code directly, without ever running anything on your machine, just run it on one of these servers. And we even have a Twitter bot that we just put up last week, where you can kind of tweet at it, and it will run a model against your tweet.

• 32:29Daniel Whitenack?

  Yeah, that's great. I was wondering, before we leave the topic of the open source projects, you also mentioned these other libraries, tokenizers and NLP, which includes the data sets and evaluation metrics. How do those fit into the puzzle and maybe interact and influence one another?

• 32:46Speaker 00

  Yeah, I mean, at the end of the day, our interest is in building open source NLP. And I think there will continue to be kind of new variants of transformers and new pre-trained models. But kind of, as I mentioned earlier, an increasing area of innovation in NLP is to try to find the right data sets to kind of challenge these models in interesting ways. And so there's a lot of energy in data set construction these days, and a proliferation of really interesting data sets of different sizes and scopes. And so Tom Wolf, who's our main open source engineer, got very passionate about building up open source data sets

• 33:28Speaker 00

  and build a library that makes it very easy to use these models in Python and really makes it extremely efficient to use kind of complex data sets directly within your code across kind of many different aspects of NLP. And so you can, we have a website that you can go to where you can kind of browse through any of these data sets and kind of use them in various tasks. And one nice aspect of this is that we have a lot of examples of how to use transformers. And they had a lot of kind of custom data set code just to run the examples. But now that code is all kind of been factored out. You can just kind of pull it in from NLP and then run the examples, kind of focusing on the machine learning parts.

• 34:23Speaker 04

  We deserve a better internet and the brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default. Quick access to the Tor network for true private browsing and an opt-in reward system so you can get paid to view privacy respecting ads then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

• 35:05Chris Benson?

  So to take the conversation in a slightly different direction for a moment, I know from talking before the show that you put together ICLR and you kind of managed that process this morning and for which is a research conference. And I'm really interested at this point, you know, we're in the time of COVID-19 and so much has changed across all of work, but particularly conferences, many of them are going online, becoming virtual like that. Really interested in what that was like and what, you know, what your experience doing it this way was and you know, what worked, what didn't, that kind of, I'm just curious because I think a lot of people are kind of waiting to see what conferences are turning into and you know, do they want to continue to go down that route or some?

• 35:48Speaker 00

  Yeah, this year I was the general chair of ICLR, the International Conference of Learning Representations. It's a big machine learning conference and really the only one focused completely on deep learning. And it was interesting, I had the chance of being the program chair for the conference last year where we had the conference in New Orleans. And then this year I was the general chair and by about December, we were getting prepped and then by February, March, it became increasingly clear that we weren't going to be able to have this conference live. And so I think we were the first AI conference to really have to be completely virtual. We had about a month and a half before the conference to really come up with something new.

• 36:33Speaker 00

  And we had this wonderful team led by the program chair this year, Shakir Mohammed. And we wanted to do something that kind of fit the spirit of the conference. And so we sat down and wrote a website for the conference from scratch. And we built a website that was based around this idea that everyone in the conference would be in kind of a Zoom-like, sorry, a Slack-like chatroom. And we used an open source platform for that. And that every paper would have its own page with a video of the work and a chatroom for that paper. So people would be able to kind of talk about it or discuss it within that setting itself.

• 37:12Speaker 00

  And in addition, we built out a kind of a bunch of social gatherings that people could have and a kind of calendar for the whole event. And the kind of main challenge is how do you run a conference asynchronous in this way? We didn't really think it was possible to have everyone in the same place at the same time. And so we wanted to kind of use things like chatrooms that kind of feel more asynchronous, particularly with kind of international audience. And the conference itself actually was really fun. We had actually a pretty large increase in attendance over past years. We had people from all over the world, particularly from some places that would have been difficult to attend a conference in other years.

• 37:57Speaker 00

  And a ton of engagement, a lot of the posters were viewed tremendous amount of times and maybe about 100,000 messages over the chat system over a couple of days. I think there were challenges. I think it's hard to get the same kind of spirit of having coffee or kind of just chatting informally in this sort of event. Things like Twitter are helpful, but don't have the same kind of intimacy. But there were also kind of nice things. We ran these kind of mentorship sessions where one person was able to chat with 10 to 20 folks who are interested in mentorship in a kind of one-to-many model that actually I think might have been difficult at a conference. So but kind of works actually pretty nicely over Zoom.

• 38:41Speaker 00

  Anyway, it was an experimental setup. Since then, we open sourced all the tools that we built for the conference. You can get it online. It's if you search for Mini Comp. And the software has been used for about five or six other major conferences since then, including ACL this year, which is the big NLP conference, and ICML, which is another machine learning conference venue. I don't think it's something we've cracked it, but in the meantime, it's nice to have something we built as a community.

• 39:09Daniel Whitenack?

  Yeah, I attended the conference I clear and I was super impressed with everything that was put together, especially given the timeframe. I know you you must have had some very late nights fueled by very much coffee. So congratulations on on in such a short time period, putting together something that was so good. I know one of the things that I appreciated, you know, I've been to other research conferences in person and, you know, posters or talks or something like that. There's just so much going on that it is hard to kind of do that. Like, you can't go to this talk at the same time as this talk, and it's hard to find that person afterwards and ask them some questions about their work.

• 39:54Daniel Whitenack?

  Maybe you walk by their poster or something. So it was kind of nice to just scroll through and look at the different videos, especially given the time zone differences, and, you know, shoot the authors a message that they could respond to asynchronously so that that question didn't get lost or something like that. I found that extremely useful. What are your thoughts on assuming maybe that at some point in the future, research conferences will have an in-person component again? Do you see a sort of hybrid scenario developing? Because I know one of the things that like with NERIPs and all of that was a struggle for so many years was were people getting visas as well, which was just such a shame is like so many people from Africa

• 40:38Daniel Whitenack?

  or from Asia that were doing amazing work, but couldn't actually be at the conference because of visa issues or cost issues or whatever it is. So how do you see that that future happening?

• 40:49Speaker 00

  Yeah, it's a question we're talking actually a lot about in at ICLR right now. I don't think we have an answer, and I think a lot of it will depend on. Kind of what the world looks like in a couple of years. So one thing we're committed to at ICLR is having the conference at venues in other locations or kind of locations that have kind of not been visited as much in the past. So one thing that was very disappointing was that this year's conference for ICLR was supposed to be in Ethiopia, in Anasababa, and we were all really disappointed that we couldn't make it out there. It would have been a really interesting event. So hopefully we will continue to kind of have conferences in kind of a wider range of locations.

• 41:32Speaker 00

  But I think, as I was saying earlier, all these areas are experiencing such kind of hypergrowth that kind of ways of kind of dealing with scale that doesn't lose a kind of sense of interaction is a kind of major challenge for the community. And so I think we need to kind of be creative about ways to handle that problem and ways of kind of maybe giving people the same experience that I think, at least I feel like I had when I was first a graduate student that kind of inspired me to continue in the field. So I don't know what that would look like. Maybe it looks like something more distributed with a virtual component.

• 42:06Chris Benson?

  So kind of wondering, and also turning the corner a little bit on just NLP in general.

• 42:12Speaker 00

  Yeah.

• 42:12Chris Benson?

  And you know, you're doing the work that you're doing. You're right at the center of the NLP world in that way. And it certainly, you know, Daniel and I talk all the time on these episodes about the fact that the last couple of years has felt like, you know, NLP has really come of age. You know, you might say golden age of NLP is how it feels like we're in. And kind of before that, you know, we had seen like CNNs have their moment and stuff. As we've arrived where we are so far in NLP, you know, what does the future look like to you? What kind of big challenges are open and should be focused on? You know, what are your thoughts there on from this point forward?

• 42:48Speaker 00

  Whew, that's a hard question.

• 42:50Chris Benson?

  Big one.

• 42:51Speaker 00

  Yeah. In some ways, as somebody who's been working in NLP for a while, it's been really neat. I mean, I think it's way better than I could have possibly expected. Seeing things like translation get to the point where it's at now is just awe inspiring to me. Like, it's such a useful thing and having it work the way it does is awesome. So what are the challenges now? I think there's a bunch. I think computer vision, for all its successes, has also had a lot of issues and

• 43:19Speaker 00

  There's a lot of conversation in NLP about how to avoid some of the issues or to have those conversations earlier rather than later. Things like what we've seen with facial recognition as a technology and questions about ethnicity there is, I think, a challenging point. And we've somehow managed to solve a lot of the natural language processing questions without solving some of the computational linguistics questions. Like, things work, but we have no real sense of why. And as a scientist, that can be a little bit frustrating. We don't really know what signals these models are using to make predictions.

• 44:00Speaker 00

  And it's very hard to know or to even ask that sort of question in a falsifiable way. Why did this model classify this sentence in this way? Why did it decide to choose this decision? I mean, these models are, at least from a probabilistic sense, completely global. And so it's challenging to do any sort of analysis along those lines. But then more practically, I think there's a lot of practical questions that are not solved yet. You mentioned this idea of dealing with massive, massive models. It's not clear if we're going to need hardware that is 100 times bigger to run these models or whether you can use pruning and distillation to make them super small. Or what does it mean to run it locally?

• 44:45Speaker 00

  Or does it just make us more reliant on cloud systems? I think these all become interesting systems research questions in the short term.

• 44:52Daniel Whitenack?

  Awesome. Well, we appreciate you taking a stab at the future predictions because I know, I think we said on the podcast before, any of the predictions that we make, I feel like are definitely going to be false because it's always something unexpected that happens. But I appreciate you giving your perspective, being part of the center of all of this work. And appreciate you taking time to talk with us and explain a bit about the Transformers library and things that are going on in NLP. Thank you so much for your contributions to the community as well in terms of helping conferences and really pushing forward open source. So appreciate you taking time to join us and looking forward to digging into all of

• 45:36Daniel Whitenack?

  the great things that Hugging Face is releasing and is doing.

• 45:40Speaker 00

  Thanks so much. Thanks for having me on.

• 45:45Speaker 04

  If you're listening to this in the month of July, you've got a shot at some free goodies. We are doing a giveaway in celebration of our friend and open source whiz, Zenoh Roche's new book, 14 Habits of Highly Productive Developers. If you don't know Zenoh by name, you may have heard of his wildly popular Dracula theme. It's an awesome dark mode theme for text editors, terminals, etc. And we have three bundles of Dracula Pro and 14 Habits to give away for absolutely free. That's a $60 value and there are three ways to enter. You can be the reviewer, the socializer, and the recommender. Hit up the link in your show notes to get started. There will be three lucky winners and you could be one of them. Thanks to our longtime sponsors, Fastly, Linode, and Rollbar for their continued support, DoBreakMaster Cylinder

• 46:30Speaker 04

  for our amazing beats, and to you for listening to Practical AI. We appreciate your time and attention. That's all for this week. We'll talk to you next time.