Zero-shot multitask learning

In this Fully-Connected episode, Daniel and Chris ponder whether in-person AI conferences are on the verge of making a post-pandemic comeback. Then on to BigScience from Hugging Face, a year-long research workshop on large multilingual models and datasets. Specifically they dive into the T0, a series of natural language processing (NLP) AI models specifically trained for researching zero-shot multitask learning. Daniel provides a brief tour of the possible with the T0 family. They finish up with a couple of new learning resources. Sponsors: RudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. iMerit ML DataOps Summit https://imerit.net/dataops – A FREE virtual event on December 2nd, 2021! iMerit is excited to gather 1000+ senior leaders at the forefront of deploying data solutions that power machine learning and artificial intelligence. Learn more and register to attend at imerit.net/dataops https://imerit.net/dataopsFastly https://fastly.com/podcast – Compute@Edge free for 3 months — plus up to $100k a month in credit for an additional 6 months. Fastly’s Edge cloud network and modern approach to serverless computing allows you to deploy and run complex logic at the edge with unparalleled security and blazing fast computational speed. Head to fastly.com/podcast https://fastly.com/podcast to take advantage of this limited time promotion! Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: InfoQ: BigScience Research Workshop Releases AI Language Model T0 https://www.infoq.com/news/2021/11/t0-multitask-nlpHugging Face: bigscience / T0pp https://huggingface.co/bigscience/T0ppMultitask Prompted Training Enables Zero-Shot Task Generalization https://arxiv.org/abs/2110.08207IEEE Specturm: How Deep Learning Works https://spectrum.ieee.org/what-is-deep-learning5 Deep Learning Activation Functions You Need to Know https://builtin.com/machine-learning/activation-functions-deep-learningUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Chris Benson?

  There has been more change in the last 30 years than for centuries before that. And then if you add the last couple of millennia, it's for tens of millennia before that. It's accelerating so much that we are in a unique state. If you go forward in time after you and I are gone, they're going to look back on this as pioneering days. We are the, the people setting out across the American prairie. But that's where we're at right now as the world is changing more dramatically than it ever has. It's worth once in a while remembering just how far we've come in just such a short amount of time.

• 0:35Speaker 01

  Big thanks to our partners, Leno, Fastly and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at RudderStack. And we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. RudderStack can be easily integrated into your existing development processes.

• 1:19Speaker 01

  And because they're open source, you can see all their code. So you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free up to 500,000 events and pricing scales transparently from there. Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 1:53Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at change.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:19Daniel Whitenack?

  Welcome to another fully connected episode of the Practical AI Podcast. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss some of the latest AI news and we'll dig into some learning resources to help you level up your machine learning game. I'm Daniel Whitenack. I am a data scientist at S I L International and I'm joined as always by my co-host, Chris Benson, who is a strategist at Lockheed Martin. How you doing, Chris?

• 2:51Chris Benson?

  Doing great, Daniel. Just enjoying the day, tearing through all the AI and deep learning news out there. Having a good time, as always.

• 3:00Daniel Whitenack?

  There's always interesting stuff, right?

• 3:02Chris Benson?

  There is.

• 3:02Daniel Whitenack?

  Yeah, and we're coming up on, here in the States, coming up on US Thanksgiving, for those that are familiar with the holiday. So we'll get a little bit of time off and I'm looking forward to that. It's been a, it's been a long couple of weeks. I submitted a few conference papers to a deadline on, well, the deadline was Monday, but the deadlines for these research conferences are like midnight anywhere on Earth or something like that. So it's actually in my time zone, it was 7:00 AM the next day was the deadline, right? So basically that meant that there wasn't a lot of sleep that was happening from like

• 3:45Daniel Whitenack?

  Monday to Tuesday morning at 7:00 AM. And I think some last minute changes and submissions. So I think I'm still catching up a little bit on sleep after that, but I'm really happy with, you know, with my team and what we submitted. So that's good.

• 4:00Chris Benson?

  Gotcha. Well, as you're looking in, I know our listeners can't see it, but as you're looking at the video feed of me, I'm the one in the center, okay? From the sleep deprivation, the multiple Chris's and stuff.

• 4:11Daniel Whitenack?

  Yeah, yeah. Well, I appreciate that clarification. That'll help throughout. And, you know, if you see me doze off, just send a virtual poke or something. You remember?

• 4:21Chris Benson?

  Pokes? It was in Facebook?

• 4:23Daniel Whitenack?

  That was like one of the first things, right? In Facebook?

• 4:25Chris Benson?

  I never figured out really how to use a poke the right way. It just never made sense. I think I poked one person and then I felt silly and I never did it again. So I'm not sure what it is, you know?

• 4:36Daniel Whitenack?

  Maybe it is worthwhile for our listeners to kind of talk a little bit. I'm sort of taking note that conferences are happening again. And at least in a hybrid sense, you know, of course, virtual conferences happen through COVID times, but normal conferences are happening again. And maybe it would be worth for our listeners to just kind of talk through a bit of the landscape of conferences. Cause occasionally I get questions about, hey, I'm starting my interest in AI or our team is working in AI and we've done some cool things. I'd like to disseminate that somehow to the community. How do I do that? So some teams are getting into this.

• 5:18Daniel Whitenack?

  They don't know like what conferences there are and how to start involving themselves in that community. So yeah, one thing to note is that there's sort of research focused conferences that are primarily involving academics, although industry researchers as well from groups like, you know, Google or other places. And then there are kind of industry conferences. Well, they used to have O'Reilly AI, but.

• 5:45Chris Benson?

  Yeah, they stopped doing it last I heard.

• 5:47Daniel Whitenack?

  I think they stopped doing that one. There's like Open Data Science Conference. Maybe you can think of some others.

• 5:52Chris Benson?

  Nvidia. Nvidia is a big one. They have several, you know, each year, so.

• 5:57Daniel Whitenack?

  Yeah, Nvidia, GTC and other ones. So there is kind of a major difference between these two types of conferences and people might not be aware of this. So some of the things like recently we saw the big science working group come out with the T zero model, which we can talk about that later in the episode, but that's pretty cool. But that was submitted as a research paper to a research conference. And that is actually a sort of peer reviewed process. And that's what I mentioned earlier on that that we were submitting early in the week is, hey, I created a new type of neural network or I applied a neural network in a slightly different way than no one's ever done before. Or, you know, maybe I created a new data set that is in a new language,

• 6:40Daniel Whitenack?

  or it's different than what people had done before, or it includes different types of data. And I want to publish that as a sort of original work and have it peer reviewed by experts, giving feedback, and eventually accepted as a conference paper, which basically means that it's been peer reviewed by other researchers in the field and deemed to be original and sort of scientifically and experimentally valid, at least based on the review. That doesn't happen in an industry conference for the most part, where some talks are maybe invited. And in certain cases, there's a call for proposals for talks,

• 7:23Daniel Whitenack?

  but those talks that are submitted are not peer reviewed in that way. So you might say, hey, I'd like to propose a talk about our AI infrastructure at S I L International, and I could talk about that. It's not like we're doing sort of original things necessarily that others haven't done, but it's useful in a practical sense for people to also hear that information, ask questions, have it be part of the topic in a practical way at a conference. I don't know if there's anything you want to add to that. I just thought maybe the listeners might not be familiar with that whole world.

• 7:55Chris Benson?

  No, I think that's really good in the sense of being able to distinguish between the two and understanding what you're going to. Most of the conferences I've been at were on the commercial slash industry side as opposed to the research side. And they're very different cultures, as you just pointed out. And there's definitely, in the industry side, you know, a desire to kind of say, look what we've done from a, from more of a business standpoint. It may not be trying to sell you something, but they both do prestige, but there's a, they're kind of a different approach to it and different communities that you're, you're selling it. And it makes it interesting to go to both, which I would encourage people to do, because they're very different experiences going to the two.

• 8:33Daniel Whitenack?

  Yeah, and that's not to say, that's a good point, Chris, because you don't even have to present something to go to either one of these types of conferences.

• 8:43Chris Benson?

  Yeah, most people are not presenting. Most are just listening.

• 8:45Daniel Whitenack?

  Right, exactly. So. You know, if maybe you're out there and you're thinking, well, how do I get connected in the AI industry or in AI research? Well, look maybe for a conference on the research side like NERIPS or ACL or EMNLP, something like this. And you know, you could just attend and you know, ask questions, get connected with people, see how it plays out and how papers are presented and what they look like. Or if you're on the industry side and you just want to do that networking, get connected, see practically how people are applying AI or doing infrastructure or something like that, just attend and you know, you can learn a lot.

• 9:27Daniel Whitenack?

  So last week I went to a conference, I gave a, it was just like a six minute presentation. And so I didn't have to prepare that much. And for the other three and a half days, I was just participating as a, as a participant. And that was great. I really enjoyed it.

• 9:42Chris Benson?

  I'll bet. And it's been a while because of the pandemic since I've been to one now. And as the world opens up and, and maybe we don't have a winter spike. If we're really, really lucky, then maybe we can get there sooner. I would also say that it's interesting that there's a different experience if you're a new conference goer. If you, so if this is your first conference or two versus someone who's been to many, then you also, as you go more often, you learn the value of what's available that's not a session. And you end up spending a lot of time making friends and having really interesting hallway conversations. I think most people who go the first time, they feel like they need to get as many sessions in as they can and see as much because they think that that's the best way to get the value.

• 10:27Chris Benson?

  But some of the best takeaways I've ever had were from ad hoc conversations. It's a moment where if you can get over any shyness that you might have, you should just walk up to a group of people and introduce yourself and just be bold and they will welcome you in. And you'll hear things that you can't hear anywhere else often. And people will share in a small group, even a small group of strangers, things that they would not say on stage. So I've had some, some really, really memorable moments that were not in formal sessions per se. And would encourage people to do that. It's one of the reasons to show up in person instead of it being a virtual.

• 11:02Daniel Whitenack?

  You also get a sense sometimes of things that are really kind of hard to pick up online. So online, for example, I could research experiment tracking software for my AI projects. Like I'm going to queue up a job. It has this data. It's going to run. It produces these metrics. I'm going to track all that stuff. Well, I could research around and see like, how are people doing that? What are they doing? Are they rolling their own? There's different types of those kinds of things out there. But if you're at a, you know, a lunch table at a conference with seven to 10 other practitioners, actual practitioners from a variety of organizations.

• 11:44Daniel Whitenack?

  I mean, that is a huge value. And, you know, if that topic comes up, you know, asking the question, hey, what do people use for this? It's not like a complete industry survey, but even just in those few seconds, you get a real good sense about like, you know, how often this is on people's mind and what they're using in terms of specific tooling and how important it is or not important it is in different organizations. And yeah.

• 12:09Chris Benson?

  I actually want to follow up on that because you raised a great point. If you go with other people from your organization and it's time for the lunch break or whatever, don't stay together. Everybody break up.

• 12:21Daniel Whitenack?

  Oh yeah, that's a good tip.

• 12:22Chris Benson?

  Yeah, everybody break up and go to an individual table that has nobody that you know in it and fit in on that and then start the conversation, join in the conversation that's there or start it if everyone's being too shy and then have those rich conversations. And then go back later with your colleagues separately when you're not doing that and share what happened and share what you learned and stuff. And that's a really, really good way to get a solid investment out of those dialogues for your organization is to bring them all back and let everyone benefit from it. Yeah, conferences are not the time to be a shy person. It's, it's the time to, to put on your bold hat and go forward.

• 12:58Daniel Whitenack?

  Yeah, me as an introvert, and introvert in a sort of strange way where-

• 13:03Chris Benson?

  That is a podcast host.

• 13:05Daniel Whitenack?

  Yeah, I'm one of those introverts that genuinely enjoys having conversations with people, but it's incredibly tiring for me.

• 13:11Chris Benson?

  Yeah.

• 13:11Daniel Whitenack?

  Right? Like, if I'm talking, if I have a full day of meetings, I'm just dead by the end of the day. So I do that at conferences. I intentionally go and speak with people. But then I also intentionally have some times where I'm like, you know what? There's these few sessions happening for the next hour, but I'm gonna go outside-

• 13:31Chris Benson?

  Yes.

• 13:32Daniel Whitenack?

  -walk to this place and get a coffee and like, no one's gonna bother me. And I am gonna step away for a second. I'm not gonna miss much, and I need that sort of separate time.

• 13:43Chris Benson?

  I agree. I don't think anyone ever expects me to say I'm an introvert, but I am. I have my public moments, and I need those, just like you said, I need that time to myself. And I'll do that as well during the day, because you'll go through a couple hours and just be exhausted. You just need your brain to reset. So go do something alone for a short while, reset, and then get back out there and have those conversations. Just as an aside, you know, we've been doing this podcast for a while now. It's been, what, close to three and a half years since we started all this? And the first time that I ever had someone walk up to me and say, "You're with Practical AI. It's nice to meet you and all," was when I was having one of those quiet breaks.

• 14:24Chris Benson?

  I was at a conference, and I walked away, and I was going to get my little private time, quiet coffee, and just kind of reset my brain, and someone walked up. And it was a special moment, and it went from private time to having a wonderful conversation. So I found some private time later, but-

• 14:39Daniel Whitenack?

  Yeah.

• 14:40Chris Benson?

  I'll never forget that moment.

• 14:59Speaker 01

  This episode is brought to you by iMerit and their upcoming ML Data Ops Summit in partnership with TechCrunch. It's a virtual event happening December 2nd, 2021. Check out the speakers and register at imerit.net/dataops. The event is gathering more than 700 attendees from top AI and ML companies and feature major speakers, including Facebook AI, Cruise, Zooks, GE Healthcare, and more. And I'm here with Ivan Lee, the founder and CEO of DataSor, who's also speaking at the event. Ivan, I know you'll be speaking at the conference on this subject, but can you share a teaser of what's happening right now in the NLP space?

• 15:35Speaker 03

  If we look at the advances in NLP over the last few years, there have been some really exciting developments. Perhaps most notably OpenAI's GPT-3 and their ability to just really start mimicking humans in generating snippets of English language. What we've noticed is that perhaps of all the branches of AI, NLP is one of the most mature. And there were some obvious use cases when we were starting out. There's things like the ability to handle customer support, improve upon chatbots. These were very clear verticals that we wanted to go after. But as we learned more, it turns out there's applications in the legal industry, in healthcare,

• 16:15Speaker 03

  in financial. There were a number of nonprofit organizations using us to label COVID-19 research and be able to just make sense of all the abundance of research that was coming out. We were kind of astounded by the creativity and the ways in which NLP could be produced.

• 16:31Speaker 01

  All right, learn more and register to attend for this free virtual event at imerit.net/dataops. Again, you'll hear from top AI and ML speakers who've successfully deployed machine learning data operations in their organizations. Again, this event is free and is virtual. Learn more and register at imerit.net/dataops.

• 17:07Daniel Whitenack?

  So Chris, speaking of conferences, last week at the conference I went to, you know, typically, well, I don't know if typically, but many conferences will have like a conference social dinner type thing. Yes. And a lot of those have like a keynote speaker. And I won't reveal who the keynote speaker was at this one, but the topic was technology and automation and AI type stuff. And the keynote speaker did a really amazing job. Like, you know, keynote speakers who do that so often just really know how to bring a large crowd with them on a story arc and kind of tie things together.

• 17:48Daniel Whitenack?

  So he did a great job, but you know, some of the things he talked about, I think were meant to kind of provoke people in a thought process. And I think he kind of coined some of his own terms in order to kind of make people think about new things. One of those things that he talked about was what he called "humology," which was basically the mashup of humans and technology. Okay. So it was a way to say, he said, "Hey, there's humans and technology," specifically talking about AI and automation, and I'm going to just sort of combine these into this really goofy word, "humology." Yeah.

• 18:30Daniel Whitenack?

  And he had what he called a "humology scale." And it was basically like a data visualization going from like, over on the left, there's tasks that only humans do, you know, computers, technology, they don't play a part. And it kind of went all the way over to the other side where humans are not involved and there's kind of somewhere in between. And he was kind of trying to challenge people to think in their own business processes, if you were sort of over on the left-hand side too far, you're likely not leveraging AI and automation in the way that you should, regardless of what industry in, you're not sort of leveraging the technology that's available in the marketplace to its full extent.

• 19:15Daniel Whitenack?

  And the example that he gave, I thought was a really good one. And so the example he gave was taking care of weeds in a field of like agriculture field.

• 19:27Chris Benson?

  Yeah.

• 19:27Daniel Whitenack?

  In the farming context. On the left-hand side, at a certain point, you had humans doing that completely, right? Like you just pay a bunch of people, they'd have a bag or something and go into the field. They pick the weeds, put them in the bag and, you know, move along. And then at a certain point, you know, you had like weed aside, is that the right word? Weed killer?

• 19:49Chris Benson?

  We're talking about agricultural weed, by the way, just so for people, just to be very clear.

• 19:53Daniel Whitenack?

  Well, I mean, you could be growing weed.

• 19:56Chris Benson?

  Oh, well, I guess that didn't clarify it at all, did it?

• 19:58Daniel Whitenack?

  So it's not mutually exclusive, I guess.

• 20:01Chris Benson?

  There you go.

• 20:02Daniel Whitenack?

  But at a certain point, you had weed killer, right? And people would walk through the fields and apply the weed killer to weeds. That also sort of brings in an element of technology or chemistry to that, right? And then at a certain point, they said, well, that takes a long time to apply with humans. So I'm going to drive this truck through my field, right? And just dump this weed killer, you know, all in the field, right? Which is maybe effective, maybe pretty terrible for the environment.

• 20:33Chris Benson?

  Yeah.

• 20:34Daniel Whitenack?

  But it's faster than, you know, humans doing it. And now you sort of move to later or more recently where this actually implements on farm equipment that will drive through and it will use computer vision to see weeds, distinguish them from other plants and spray them specifically. So you save a lot of money on weed killer. You're better for the environment, everything like that. And then he showed a recent example where, which I hadn't seen before, but apparently there are like companies like John Deere and others that are working on autonomous machines like this, because those ones that drive through and spray select. Still have a driver in them.

• 21:17Chris Benson?

  Right.

• 21:17Daniel Whitenack?

  But they're, you know, working on ones that are totally autonomous. So it's not, how he phrased it was, it's not totally, even that is not taking the human out because they still have to come back and get charged or refueled. Right. And like turned on and put out into the field. So a human is still involved in some way. But it's much more towards that other side of the, of the scale.

• 21:40Chris Benson?

  The weed spraying via CNN is actually several years old, you know, at this point, I remember, I remember in talks, three, four years ago that I was giving, we were, I was using that as an example. The autonomy, I agree, is more recent, but it only makes sense to go do that. And I think we're going to see that across in, in your humology, you know, context. I mean, thousands and thousands of use cases, it's, we already are.

• 22:05Daniel Whitenack?

  Yeah, yeah. I kind of liked the line of thought that he gave because it brought out a few things. One was, if you're not automating anything, you probably should be.

• 22:15Chris Benson?

  Yeah.

• 22:15Daniel Whitenack?

  Second is sort of doing a brute force application of technology can be harmful. Right. There is that element in that story. Right? But that's sort of worked through over, over time, at least in that example. But yeah, it's, it's an interesting line of thought. I think it prompts a lot of, a lot of thought, which is probably why it was good for that, that keynote. It brings up though, this element of, is AI taking all of these jobs? It's another element of what it, what it brings up. Which I know that we've, we've chatted about before and in various contexts on this podcast.

• 22:52Chris Benson?

  Which of course it is because we're evolving and we're evolving in a fast. And when I say we, I mean, not just humans, but this humology concept that you're talking about, because the technology is us. And we are the technology. There is a, and you know, I don't mean in the, in the sense of being a cyborg. But these are, you know, these, this is the fire that we invented. This is the wheel we invented. This is how the modern world is working and how we're doing it. But. As we are racing up the curve of new tool creation, ever more sophisticated, we have this trailing concern about how it's impacting us. And, you know, we've talked a great deal about AI ethics and such along the way,

• 23:35Chris Benson?

  but that is certainly a trailing concern because we go do things. And we're a little bit like a child, you know, running around, learning something new and not really understanding that that's a sharp knife that you're holding there. And as they wave it around, it's something that we're trying to figure out as we go. I don't think that's going to stop anytime soon. I think we barely even have a handle on the fact that as we are constantly finding these new applications, we're going to be making mistakes all over the place. And I think you'll see that. I mean, we've seen an unending series of stories. If you think about it over the last 10 years, about data and AI applications going awry.

• 24:17Chris Benson?

  But they continue to happen and I don't see any end to that as we do that. And I think that's just part of our very, very fast evolution as we go forward.

• 24:27Daniel Whitenack?

  Yeah, so how do you and your, the conversations that you have with people and this idea of, you know, AI taking jobs and sort of the benefits and downsides of automation? Do you find that that comes up in your conversations with non-practitioner people? Like maybe the people that you do like wildlife volunteering with? Does that ever come up?

• 24:52Chris Benson?

  It does. It comes up in just about every aspect of life because this is now touching everybody. And it affects almost all the jobs out there at some point or at some level. It may only be as a, there may be some tooling. There may be data about those jobs. It may not be the primary job thing, but, but yeah, I'm seeing that a lot. There's a dissonance between humans and the tools that we're creating right now. And that we have biological brains that have always evolved, but evolved at a much slower rate than, than over the last few thousand years. And particularly the last few centuries and particularly the last few decades. And up to where we're at now, and it doesn't show signs of slowing down.

• 25:34Chris Benson?

  So we're not used to this much change this fast. And yet we are doing it. And so, yes, it's changing what it means to have a life. I have a, as you know, a nine-year-old daughter at this point. And I, as we've had these episodes, I've said six-year-old, seven-year-old, all that stuff. Nine-year-old at this point, I am struggling as a parent to try to figure out how to steer her so that long after I'm gone, she has a good productive life. And I think it is a fairly unique challenge, given the steepness of the curve of change right now compared to any time in history. She's going to have to find a way to add value to the world enough to earn a living

• 26:15Chris Benson?

  and do that cooperatively with all the technology things around her that, once upon a time, my dad might have done that, but no more.

• 26:23Daniel Whitenack?

  Yeah. It's also just interesting in the sort of aspect of knowledge discovery and all that. Like, used to be the problem, like, if I think of even my parents' generation. My mom, growing up in a very small town in Oklahoma, Miami, Oklahoma, spelled Miami, but clearly Miami, Oklahoma. You know, if you think about what information was available there, you know, there's no internet. So there was information available at the library, right? The access pattern to that information was, you know, looking up various subjects, finding the book, trying to parse through what information was there.

• 27:07Daniel Whitenack?

  So now even that pattern is really disrupted because so much is available to us. The problem is not like, I can't find information on this subject, but I find way too much information on this subject and I can't, either I can't verify what I need to verify or I can't sort of distill down the pieces of information that I need to distill down, which, I mean, that personally for me in terms of like, it's exciting that all that information is available, but it's also exciting that sort of AI techniques or machine learning techniques are starting to be applied in that realm as well in terms of helping us find relevant things and connect the dots.

• 27:51Daniel Whitenack?

  Of course, there's a danger in that as well in terms of bias and what's presented to us. But I think in its best form, it can be applied with, you know, really great benefit.

• 28:01Chris Benson?

  At risk of acting a little bit like Father Time on this, I'm going to leverage my old age at 51 currently.

• 28:09Daniel Whitenack?

  I looked up things in a library too, Chris.

• 28:13Chris Benson?

  I'm going to go there for one second, just for some context. There were no cell phones at all until I was 20 years old, around 20, give or take. And I spent a lot of time as a kid at the public library, hoping that that little tiny local library with its world book encyclopedia, you know, that was my way of getting to the world. And the reason I'm saying this isn't just to emphasize my geriatric state, but to point out that the world, that's 30 years ago. The world has changed enormously in that time and we tend to lose sight of that very easily. There has been more change in the last 30 years than for centuries before that.

• 28:55Chris Benson?

  And then if you add the last couple of millennia, it's for tens of millennia before that. It's accelerating so much that we are in a unique state. If you go forward in time after you and I are gone, they're going to look back on this as pioneering days. We are the, the people sitting out across the American prairie with a little backpack and horses. It's just in this case, it's

• 29:16Daniel Whitenack?

  It's only a single GPU in my computer.

• 29:18Chris Benson?

  With only a single GPU to do your calculations. Exactly. But that's where we're at right now as the world is changing dramatically, more dramatically than it ever has. And so I won't belabor this any longer, but it's worth once in a while remembering just how far we've come in just such a short amount of time.

• 29:51Speaker 04

  I'm Karhal Azou, host of Shipit, a show with weekly episodes about getting your best ideas into the world and seeing what happens. We talk about code, ops, infrastructure, and the people that make it happen like charity majors from Honeycomb.

• 30:05Speaker 05

  We act like great engineers make great teams, and it's exactly the opposite. In fact, it is great teams that make great engineers.

• 30:14Speaker 04

  And they farley when the founders of continuous delivery.

• 30:17Speaker 06

  Start off assuming that we're wrong rather than assuming that we're right. Test our ideas, try and falsify our ideas. Those are better ways of doing work, and it doesn't really matter what work it is that you're doing. That stuff just works better.

• 30:30Speaker 04

  We even experiment on our own open source podcasting platform so that you can see how we implement specific tools and services within changelog.com, what works and what fails.

• 30:41Speaker 07

  It's like there's a brand new hammer and we grab hold of it and everyone gathers around, we put our hand out and we, we strike it right on our thumb. And then everybody knows that hammer really hurts when you strike it on your thumb. I'm glad those guys did it. I've learned something instead.

• 30:56Speaker 04

  Yeah, I think that's a very interesting perspective, but I don't see it that way.

• 31:01Speaker 07

  Okay.

• 31:02Speaker 04

  It's an amazing analogy, but I'm not sure that applies here. Listen to an episode that seems interesting or helpful, and if you like it, subscribe today. We'd love to have you with us.

• 31:32Daniel Whitenack?

  Okay, Chris, well, we probably should mention a couple of things that have run across our, or have crossed our paths in the news recently. Current affairs. Current affairs in the AI world. One of the ones that I wanted to mention and also make people aware of more so because it is an ongoing project is this project that That it's being run as a highly distributed collaborative research effort called the Big Science Research Workshop. Now, the Big Science Research Workshop, I think, was inspired, and first of all, we'd love to have some people from there on the show at some point.

• 32:11Daniel Whitenack?

  But I did want to make people aware of it. So 600 researchers from 50 countries and more than 250 institutions. And what they're doing is they're getting together in sort of a massive science collaboration project. In the same vein as, you know, some people from my world originally in physics might be familiar with the CERN project or something like that where Absolutely. There's a bunch of physics researchers that got together because what they realized is in high-energy physics,

• 32:49Daniel Whitenack?

  the scale of the experiment that we need to do is larger than any experiment that any of our institutions can do in and of itself. And so now we need to have a massive global collaboration in order to create this new high-energy physics accelerator and learn about the fundamental, you know, things of our universe. Well, in the same vein, this kind of big science workshop is a large workshop or research project focused on large multilingual models and data sets. So as we've talked about over the past, however, long, you know, NLP has been advancing rapidly. And one of the trends are these big multilingual language models.

• 33:34Daniel Whitenack?

  And typically at this point, those are so big that you need huge infrastructure to run them. You need sort of a huge data governance and curation sort of system and method to get all of the data together, to manage it, to distribute it across your trainings and manage those trainings. And so that's really the goal of this project. And the reason why I bring it up now is that they released one of their First models recently called T zero. And a lot of people are kind of talking about that because it outperforms in many ways. Not in all ways, but in some ways it outperforms GPT three.

• 34:16Daniel Whitenack?

  But it's also 16 times smaller than GPT three, which is a is a huge model, of course. So really interesting. There's a couple of other interesting things about it that I want to mention. But any initial reactions to that kind of, I guess, the strategy that they're employing to to make this happen?

• 34:36Chris Benson?

  I think that's a natural evolution of, you know, what we've been talking about. We're in those pioneering days from what we were saying a few minutes ago. And there's a point where to get progress, you've got to scale and you've got to try new techniques. And it can't be every little pioneer on their own. So I'm delighted to hear about this and and I want to learn a little bit more about what T zero can do. I don't know enough about it. What have you learned about it so far?

• 35:02Daniel Whitenack?

  Yeah, well, I wasn't part of the group that that did this, although I I've attended a couple of related things. And one one effort to kind of get some of the data together. They're sort of hosting these different workshops to help get data together. At least in my understanding, some sort of, or maybe the most interesting thing about it is that GPT three. So an example of large language models like GPT three, for the most part, have been trained on sort of proxy tasks like masking, which means like I'm going to take some words out of various places and see if I can sort of figure out how to slot them back in into text.

• 35:43Daniel Whitenack?

  And that has been used as a proxy task in order to kind of help learn these really useful language representations. The strategy that they took with T zero had to do with prompts, which is quite interesting. So they took a whole bunch of different data sets that had to do with a whole bunch of different NLP tasks like. Paraphrasing or summarizing, question answering, named entity recognition, natural language inference. These are all different types of tasks that people do with language. And they reframed all of the labeled examples as prompts to the model. And what I mean by that is a prompt means like, here's a couple of examples that I have pulled up.

• 36:28Daniel Whitenack?

  For question answering, it could be something like, I know the answer to what team did the Panthers defeat? Is in blah, blah, blah, blah, blah. Can you tell me what it is? So it's literally just like a prompt like you'd give to another person. But it's flexible to other kinds of tasks. Like suppose the banker contacted the professors and the athlete. Can we infer that the banker contacted the professors? So that's more of a natural language inference thing, which has to do with logical agreement. You could reframe like sentiment. You could reframe summarization. Summarization would be like, you know, blah, blah, blah, blah, blah.

• 37:12Daniel Whitenack?

  How would you rephrase that in fewer words? All of these statements are just prompts paired with the corresponding output. And that I think is one thing that makes this kind of really stand out versus other models that have been trained.

• 37:27Chris Benson?

  And for our listeners, I'm going to embarrass you and say, Daniel really knows his NLP very well. He's quite the expert. And I know you won't say that about yourself, but I've known you long enough to know that's true. What do you see as the advantage of that particular strategy relative to previous ones? What is the value that they are managing through that approach?

• 37:47Daniel Whitenack?

  Yeah, so I think you have to think about how they're optimizing this model to be used. I think that they have recognized very significantly that this is the sort of zero shot thing that people want to do with modern language models. They want to give it their own unique prompt and have it immediately sort of know what to do.

• 38:10Chris Benson?

  Okay.

• 38:11Daniel Whitenack?

  And so an example of this would be recently, we needed to create a data set. In our own work that showed some contradictions in text. But we had a bunch of text and we needed to sort of transform some of that text to where it contradicted itself. And so what we did was we created prompts like this and fed it. In that case, we fed it to GPT three because you can do a similar type of interaction there where we kind of gave it a couple. In that case, it was sort of few shot. We gave it a couple of examples of the prompts that we wanted, like, you know, if this is this, then this is not this sort of thing. And it learned how to produce those sort of samples for us.

• 38:54Daniel Whitenack?

  This is the sort of thing I think this model is geared toward where people can come up with their own unique, custom, flexible prompts. And this should generalize quite well across a whole variety of tasks that people come up with. And so I think the advantage is that they're really focusing from the start in the training data around this sort of zero shot usage of the model.

• 39:20Chris Benson?

  Just to ask a very basic question, this is, this is yet another transformer approach, right?

• 39:26Daniel Whitenack?

  Yeah, so I believe that the model architecture that they base this on was similar to that of T5, which was a text to text model that's transformer based. So it's an encoder decoder sort of language model that I believe was from Google research not too long ago. Correct me if I'm wrong, listeners, and I'm getting the wrong information, but I believe it came out from Google and that's been around for some time. So that's why they called it, I think, T zero was partially, it's the first model that they came out with, so zero. And it's kind of inspired by T5, this transformer based encoder decoder.

• 40:07Chris Benson?

  Let me ask you a question while we're on this line of conversation. In recent years, we've really seen, you know, transformers came out and, you know, have just completely revolutionized the NLP space. In your view, just summarize, would you say that's really the only thing that matters right now? Cause that's what we've been talking about the last couple of years, every time we hit is, is there anything outside of that kind of transformer approach that you've seen that is still valid? Have we, have we eclipsed some of the older, older model approaches at this point? Or is there still any diversity there in terms of completely different mechanisms?

• 40:45Daniel Whitenack?

  Yeah, it's a good question. I think there's a lot of people doing sort of interesting things neural network architecture wise. I know in the natural language type of space, in particular around speech, I've seen some of the recent sort of streaming and space efficient speech models utilizing various new kinds of convolutional architectures. And so I think that's still, you know, like, like we kind of talked about with the weed spraying thing in the fields. That's what was used at that time, quite some time ago, and it's still finding new, slightly different applications. But then there's also other

• 41:29Daniel Whitenack?

  new things that people are exploring to either combine multiple kinds of data, multiple modes of data, you know, video, audio, text. There's things that people are doing with natively graph structured data in graph neural networks, which I think is pretty fascinating stuff. So I think there's a variety of things that I wouldn't, I wouldn't phrase it as transformers as all that matters, but I think transformers have been, you know, no pun intended, transformational.

• 42:01Chris Benson?

  Transformational.

• 42:02Daniel Whitenack?

  It's also, I mean, it's impacting video and computer vision and that sort of thing too, because it's now being applied in those spaces. So it's definitely been a big deal.

• 42:12Chris Benson?

  Sounds interesting. Well, that was fascinating conversation there.

• 42:17Daniel Whitenack?

  I'm glad to have prompted, prompted it. My puns are getting worse, or my jokes are getting worse as the episode's going along.

• 42:26Chris Benson?

  Well, we're almost at an end. I was gonna say, we'll get on the hook of going into some of those, those new approaches. The multimodal sounds fascinating. So we'll have to have an upcoming episode to dive into that.

• 42:37Daniel Whitenack?

  Yeah, for sure. You know, in each of these episodes, we like to provide some some learning resources for people. I think that you found a couple. It seems like both of us have been seeing quite a few things recently from IEEE Spectrum. So, great, great work. If any of those working on those articles are out there, it seems like you've been been doing good recently. But this how deep learning works inside the neural networks that power today's AI. I appreciate you forwarding this along to me. So, what caught your attention about this?

• 43:11Chris Benson?

  Just it was another take, you know, for a long time, we had seen kind of introductory articles out there. But as over the last few years, as things have progressed, we haven't seen as many of those lately. And so it was kind of nice. This was a fresh one. This was a new one. And it still just hits the basics. But there are people coming into our field every day right now that are just learning the basics. And I like the idea of throwing out one that takes another take at the basics and tries to communicate it well. It was as simple as that. It's keeping it fresh with the new material for people who are just coming into the field. And then I saw another one, which is five deep learning activation functions you need to know. I remember back when I was learning this,

• 43:52Chris Benson?

  understanding what the activation functions are, not just mathematically, but then knowing how to use them and where to use them is one of those things that sometimes catches people. It caught me a little bit when I was trying to learn it. And so the article basically kind of walks you through a quick summary of each one and talks about where you could use them. And I thought that was also as a an introductory article would be a really good thing for people to get to because it knocks over one of those hurdles quickly. And so I just wanted to throw out those for the people that are just coming into the field, welcome. And there's lots of good material out there and some of it remains brand new.

• 44:27Daniel Whitenack?

  Yeah, I would I would echo that. It's worth getting into the weeds a little bit if you're getting into this field and understanding things like activation functions. As time goes on, the tooling gets easier and easier. So you can just sort of pick stuff off from the shelf and use it. But there is a real value, I think, in getting some intuition around.

• 44:47Chris Benson?

  On these things. So great finds, Chris. I really enjoyed our conversation today. Appreciate, appreciate your thoughts on all the subjects. I did too. It was a good one. We'll talk to you soon and look forward to new guests next week and carrying on the conversation. And have a very good. I know you're taking a little time off around Thanksgiving, so enjoy your vacation. Veggie turkey day for us. Absolutely. Talk to you soon, Daniel. Happy Thanksgiving. Bye. Happy Thanksgiving.

• 45:17Speaker 07

  That's our show. Thanks for listening. For more like this, check out our master feed. It is all change log podcasts in one easy to consume place. Let your podcast app snag everything we produce and then pick and choose which ones to listen to. Subscribe today at change log.com slash master or just search for change log master in your podcast app of choice. You'll find it. Special thanks to Brake Master Cylinder for providing our music and to our longtime sponsors, Facely, Launch Darkly and Linode. That's all for this week. We'll talk to you again next time.