COVID-19 Q&A and CORD-19

So many AI developers are coming up with creative, useful COVID-19 applications during this time of crisis. Among those are Timo from Deepset-AI and Tony from Intel. They are working on a question answering system for pandemic-related questions called COVID-QA. In this episode, they describe the system, related annotation of the CORD-19 data set, and ways that you can contribute! Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean’s developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.99% uptime SLA, and 24/7/365 world-class support to back that up. Get your $100 credit at do.co/changelog https://do.co/changelog. AI Classroom https://datadan.io – An immersive, 3 day virtual training in AI with Practical AI co-host Daniel Whitenack Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Timo Möeller – GitHub https://github.com/Timoeller, LinkedIn https://www.linkedin.com/in/timo-moellerTony Reina – GitHub https://github.com/tonyreina, LinkedIn https://www.linkedin.com/in/skysurgeryChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: COVID-QA https://github.com/deepset-ai/COVID-QACORD-19 https://pages.semanticscholar.org/coronavirus-researchEpisode featuring SpaCy https://changelog.com/practicalai/68Sentence transformers https://github.com/UKPLab/sentence-transformersEpisode on BERT https://changelog.com/practicalai/22German BERT from Deepset-AI https://deepset.ai/german-bertHugging Face Transformers https://huggingface.co/transformers/Haystack from Deepset-AI https://github.com/deepset-ai/haystackDeepmind molecular structure with RL project https://deepmind.com/research/open-source/computational-predictions-of-protein-structures-associated-with-COVID-19Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:17Speaker 00

  This episode is brought to you by Digital Ocean. Digital Ocean's developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, TM accounts, worldwide availability with a 99.99 uptime SLA and 24/7 365 world-class support to back that up. Digital Ocean makes it easy to deploy, scale, store, secure, and monitor your cloud environments. Head to do.co slash change log to get started with a $100 credit. Again, do.co slash change log.

• 0:59Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM. Okay, here's Daniel and Chris.

• 1:24Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack, one of your co-hosts. I'm a data scientist with S I L International. And I've got my co-host with me, Chris Benson, who is a principal AI strategist at Lockheed Martin. How are you doing, Chris?

• 1:43Chris Benson?

  I am doing okay. I'm safe and I'm well, and so is my family. So that is a good sign for me.

• 1:50Daniel Whitenack?

  Yeah, that's great. I'm recording in a new location because we've got a bit of a full house at the moment with my brother-in-law's being back from college and living with us. And so I've transitioned my studio out into the dining room. So it's been an interesting week in that sense as well.

• 2:08Chris Benson?

  We're all making adjustments these days. The world is an unusual, unusual time.

• 2:13Daniel Whitenack?

  Yeah, yeah, we're all making adjustments. But yeah, I've actually, it seems like I've been more busy work-wise since the crisis started than even before, because S I L has been making various efforts to contribute in beneficial ways related to COVID-19, including trying to translate the phrase "wash your hands" into as many languages as we can. Part of that is machine-generated, and then part of that is just crowdsourced translations. And I think we're up to 454 on my last check.

• 2:49Chris Benson?

  I know, I saw you tweet about that a few days ago. And so if you're not following Daniel, you definitely should. And you can see the work that he did there.

• 2:58Daniel Whitenack?

  Yeah, and those conversations and that work also led to some other discussions with one of my contacts at Intel. And she pointed me to this other project called COVID QA, which some people at Intel started collaborating with this team from DeepSet AI. And I was super fascinated by this project and also interested in potentially contributing because they're looking to up the language support as well. But I had a conversation with them and they've agreed to be on the podcast today. So we've got Timo from DeepSet and then also Tony from Intel. Welcome, guys.

• 3:37Speaker 04

  Hey, welcome. Thanks so much. Thanks for the introduction. It's really been a great week and two weeks, actually, because this is how we started this COVID QA project. But should I talk a little bit about myself first to begin with?

• 3:54Daniel Whitenack?

  Yeah, please do. If you want to introduce yourself, and then we'll ask Tony to do the same.

• 3:59Speaker 04

  Okay, ja, so hey, I'm Timo and I'm co-founder of NLP Startup in Berlin and I would say a total NLP, Natural Language Processing geek, I studied data science and computational neuroscience and then co-founded the startup Deepset two years ago in Berlin, which is actually really a great place for startup, a lot of talent are coming here and also a lot of open source companies are based in Berlin, for example, Spacey or maybe you know Raza.

• 4:35Daniel Whitenack?

  Yeah, we've, we had. Spacey on. Uh, Spacey on the podcast a little bit ago and yeah, Berlin sounds like quite a place to be a developer. Um, I definitely need to make a trip there. I was gonna say, we need a practical AI road trip. Exactly.

• 4:49Speaker 04

  Yes, totally. Yeah. And yeah, it's a totally great vibrant city. Of course nowadays it's a bit more empty and calm. Um, when you go, for example, we have a huge airport that got shut down after the change in government and this field is completely empty nowadays and normally it's full of people and very lot of people doing sports or celebrating there. But yeah, um, so at Deepset, I'm responsible for innovation because we believe that there has been a lot of advancements in deep learning and also natural language processing and this has to be brought

• 5:30Speaker 04

  to the industry, but also what is really important to us is getting NLP technology to work on German language and for this we are very deeply rooted in open source technology. We train a bird model like these language models that got open sourced by Google and we trained these on a lot of German text data and also open source to this. And this is giving us a lot of traction from the community, a lot of researchers, German researchers are using this. And yeah, this is just a really great time to contribute to open source projects.

• 6:06Daniel Whitenack?

  Awesome. Yeah. Thanks for the intro. Tony, you want to let us know a little bit about your background and how you eventually ended up where you're at now?

• 6:13Speaker 05

  Yeah, absolutely. Thanks for having us. So I'm Tony Reina. I'm a medical doctor and data scientist, and I'm a chief AI architect for health and life sciences at Intel. So my primary role is actually taking artificial intelligence algorithms and trying to make them run faster on, well, on Intel products, obviously. A lot of what I've been doing has been in the medical imaging space, so CTs, MRIs, you know, things like that, but been branching out into genomics and particularly natural language processing, so the NLP stuff. Timo and I kind of first met with some of the German work that he was doing, which I like the playfulness of NLP. They named things after Sesame Street characters like Bert and Ernie and Elmo and things like

• 6:58Speaker 05

  that, which is kind of, kind of a fun group to be working with when you, when you work with researchers that really love, love to, you know, have kind of fun things to do.

• 7:06Daniel Whitenack?

  Makes the logos a lot better.

• 7:08Speaker 05

  It makes the logos a lot better. Nobody, nobody forgets Bert now, so.

• 7:13Daniel Whitenack?

  Poor Bert.

• 7:14Speaker 05

  Poor Bert. I mean, but you know, Bert's now like, you know, a world celebrity now in terms of AI researchers. But yeah, no, it's, it's really great. And, you know, I mean, Bert's only been a couple of years now, maybe even less, that, that has been in existence and has just kind of taken the field by storm. So, so yeah, this, this project with, with Timo that we, we kind of looked at since we were already connected, he just popped up on my LinkedIn page and said, hey, we're, we're doing this COVID question and answer thing. Would love to get some help. And I said, well, let's figure out how to help out. So that's what we've been doing is, is obviously, we've been really busy at Intel just working, you know, as everybody is around

• 7:57Speaker 05

  the planet, basically, trying to figure out ways we can help with COVID. And obviously, we're a tech company, so we're not healthcare. We're not gonna, you know, Be able to go out there and do magic like healthcare providers are doing. But we're going to do what we can, and this is one of the ways we think we can really make a difference.

• 8:16Chris Benson?

  Yeah, this is just such an unprecedented time. And as we just for it's moving so fast that for context, for listeners who are tuning in, we're actually recording this on Tuesday, March 31. And we don't normally say that when we record episodes, but given the topic and given how fast this is evolving, I thought that a point in time was worth having. Just to set the context and then kind of I'd like to come back over to you, Tony, for a little level setting for us. I know that right now we're at a point where there's 203 countries, areas and territories that have COVID-19 cases. As of today, the World Health Organization said 754,000 and change. Pardon me, just to round out the numbers of cases.

• 9:01Chris Benson?

  There's almost 37,000 cases around the world that resulted in death. In the US, we're at 163,500 cases and we are approaching 3,000 deaths, which we may hit today based on the current run rate, which would here in the US, which would put us on the same as 9/11 in terms of that. So it's a moment in history that none of us have ever experienced. There's nothing, I guess, other than maybe the Spanish flu of 1918 that's comparable in any way. And I know that has limited comparisons. I'm wondering if you can kind of level set beyond just the numbers that I called out that are on the websites everyone is following, where we are today,

• 9:42Chris Benson?

  what that looks like from your perspective as a medical doctor that's dealing with this. And then obviously we'll talk about how we're using data to start attending to these problems that we're facing.

• 9:53Speaker 05

  Yeah, I mean, I think just from the medical aspect, I mean, I haven't practiced in over 20 years, basically. So I mean, I wouldn't be the best person to answer about kind of the clinical things. And but this is the strangest part of this whole thing is that when people ask me, how was it going? All I know is because we're kind of locked down and stay in place and shelter in place, I can tell you how it's going at my house and everything else I get from reading and from listening to the news and things like that. And I think that's what's just kind of curious about this is I I feel like I want to be out there and doing things. And even my, my wife's a retired psychiatrist from the Navy.

• 10:34Speaker 05

  And, you know, she was actually even thinking about, you know, should I kind of lend a hand somewhere? You know, what can I do? And, you know, and I guess that's a great thing. And, but that's also kind of where we're at is everybody wants to help. And yet it's this odd situation where, you know, the best thing for most people is just to shelter in place and make sure that we don't, you know, keep spreading things and, and trying to get it under control.

• 10:57Chris Benson?

  That's a great point right there.

• 10:59Daniel Whitenack?

  Yeah, definitely. And one of the things you mentioned, which really struck a chord with me is the idea that we're all kind of sheltered in place, at least for the most part, a lot of people are. And we're getting information from various sources. There's so much information swirling around. Some of that's recent, some of that's not recent, some of that's from trusted sources, some of it's not from trusted sources. Um, we're hearing anecdotes from our friends and family. They're hearing things and things are getting passed secondhand. Could you talk a little bit, either one of you, about, you know, what you've seen in terms of the spread of accurate information and the problems related to actually information spread and, and the virus?

• 11:40Speaker 05

  Yeah, I think Timo should go on that because he's definitely the one that started this on trying to get factual information out there.

• 11:46Speaker 04

  Yeah, exactly. So of course, I mean, um, social media is quite difficult to dissemble, um, like really truthful information. And this is exactly how we started the COVID QA project. Um, it was two weekends ago, there was a hackathon organized by the German government and other authorities. Uh, it was actually a huge event, 45,000 people in one Slack workspace.

• 12:13Chris Benson?

  All virtual.

• 12:14Speaker 04

  Yeah. Yeah. And all remote and like a beehive buzzing about. And part of this hackathon, um, we decided to focus on getting factual information. And that's why we looked at official government pages and, um, already saw quickly that if you look at a single government page, there's not so much information. And that the information need is actually spread across a lot of official pages. And this is exactly the birth hour of COVID Q where we wanted to aggregate these official sources and make them available and searchable in a meaningful way. And yeah, this was during the hackathon two weekends ago and there were about twenty five developers

• 12:58Speaker 04

  just jumping on to this project. We were five core developers from Deepset, worked basically the whole weekend through. And with the support of this external people, it was really fun to develop the UI, to develop the backend, develop scrapers that scrape these sites and bring all pieces together. And afterwards, after the hackathon, there's also people now interacting, but also people from external coming and wanting to collaborate, wanting to help, wanting to extend. And this is exactly how we got in contact then with Tony, us following on LinkedIn. And I think this is the most great part of this project,

• 13:41Speaker 04

  to have really like a community that is fast, agile, that is not bound to bureaucracy, that there's no improvement processes, like long improvement processes. It's just we have a situation and we need to work on it. And this could help people actually saving their lives or the lives of their relatives.

• 14:01Speaker 05

  That was the nice thing about, you know, with Timo's group is that Deepset was already set up to kind of do NLP and do it at scale. And so it was one of these things where I knew coming into it that A, they had something already in like the first weekend that you could work with. And B, they had the engineers and they had the data scientists that could make this thing scale. So, you know, they really just needed, you know, resources to kind of come in and help them to make it scale, but they had the machinery ready to go.

• 14:29Daniel Whitenack?

  So I'm curious, before we, and I want to dive into that machinery here in just a second in terms of like the end goals of COVID QA and its functionality and kind of some of the things under the hood. But before we get into that, maybe we could just kind of set a foundation in terms of, you know, after you've kind of looked at what, what sources of data are out there, what sources of information are out there, what they're talking about and what people are asking, what, what is the sort of information that people really need to know during this time? Is it symptoms? Is it, you know, best practices for hygiene and hand washing? What are you seeing as kind of some of the main pieces of information

• 15:11Daniel Whitenack?

  that, that really need to get to as many people as possible?

• 15:15Speaker 05

  It depends. So I think it's, there's two groups that we're getting at with this. The first group is just the layperson that's, you know, out there and they're the ones that are going to hit the tool that's as it exists right now, which is, you know, one that basically will sift through a lot of World Health Organization and the CDC kind of FAQs and they're looking for, you know, what's the best way to disinfect, you know, my house or, you know, what's the best way of, you know, washing my hands or, you know, can I eat this or, you know, will this help? That's where it is right now. So that's kind of the first group of people that would be using this. And then what I thought was interesting was, was coming in to add the second group, which is going to be the researchers that

• 15:57Speaker 05

  want to look for new things. And these are the data scientists and geneticists and physicians and epidemiologists that want to come in and actually do research on COVID and on coronavirus. And so one of the things that Tim and I talked about was there was a data set that was released on Kaggle by the Allen Institute, by the White House, NIH, Georgetown, CZI, MSR. It was a whole group that put it out called CORD19. It was the Coronavirus Open Research Data Set Challenge. And it's something like, I want to say like 25,000 PubMed articles. So these are peer reviewed. You know, high-quality articles that they basically just as a search on like coronavirus

• 16:42Speaker 05

  and virus and, you know, and got all the articles, basically. And, and so the idea was, well, you know, Burt and all of these great models have, you know, things called like extraction AI where you, you get to do a question and answer system for this large body of articles. And so that the question would be, you know, when the Kaggle thing went out, it was like, here's a bunch of data, can you find interesting things to do with it? And I thought, well, the first thing you need with a mountain of data is a way to sift through it for actionable, you know, relative data that's actionable. And, you know, Tima's group had something called Haystack, which was like, you know, trying to sift through a haystack for a needle.

• 17:22Speaker 05

  And I thought, what if we take this, we annotate it using like the Stanford question and answer type of models, the squad models, and be able to actually have researchers, give a free tool that researchers can go through the core dataset and be able to type in random questions that are, you know, things that are not going to be how to wash your hands, but things that are like the beta subunit of the globulin of the such and such, you know, whatever. And it will actually give you a relevant answer and a few articles, published articles that you can actually look to and go through these 25,000, you know, articles and get the real meat of the issue.

• 18:02Speaker 04

  Yeah, also, I really like this dual use of, of this project. And to come back to the question, I think I looked at quite a lot of FAQs for the general public. There, the most important information is really informing people how Corona spreads and how to prevent the spreading. I think if this is hitting larger cities where people are crowded together, this information needs to be there in the right way and in a trustable way. I think this is really important. And then this dual use for the general public and then, as Tony mentioned, for the researchers, this will be incredibly useful to speed up the innovation process.

• 18:59Daniel Whitenack?

  Hi there. This is Daniel Whitenack, one of the co-hosts of Practical AI. And when I'm not working on Practical AI, I'm developing my own AI applications, or I'm training teams at other companies. Been doing this for over 10 years now, and I've trained more than 1,000 people. Now I'd like to invite you to my new live online training event called AI Classroom. In AI Classroom, I'm going to teach the practical skills I've learned over the years using the latest open source AI technology. You will learn both AI theory, along with practical hands-on implementations in both PyTorch and TensorFlow. After attending AI Classroom, you'll be able to understand the latest models,

• 19:41Daniel Whitenack?

  implement your own models in code, train computer vision and NLP models, create model inference servers, and experiment with state-of-the-art methods like reinforcement learning. AI Classroom is taking place this May. It'll be taking place live and completely online in a high-quality virtual classroom, so no travel is required. There will be two cohorts with convenient time zones for Eastern and Western hemispheres, so don't miss out. Tickets and more information is available at dataDan.io. That's dataDan.io. And early bird pricing lasts until April 3rd. See you online in AI Classroom.

• 20:34Chris Benson?

  So I guess coming out of that and into kind of looking at the next layer, I'm wondering, we've kind of talked about what COVID QA is, and we've kind of talked about it being based on the CORD-19 dataset. I'm wondering if, at this point, now that everyone has a kind of a sense of what you're trying to accomplish, if you could kind of dive in into specifically what this is that you're putting out there and making available to the public? And as we get a sense of that, we'll dive into kind of how it works and what's the technology underlying it?

• 21:07Speaker 04

  Yeah, let's maybe also best then separated. This tool use one for the general public, explain this. I'm going to more detail and then also the researcher use where we mainly use extractive technology. Sounds great. Exactly. So for the general public, it is basically matching user searches, user questions to the questions we crawled from the official FAQ pages. And the technology is based on open source technology, PyTorch, Hugging Face Transformers, and also our other framework haystack that can basically do question answering at scale. And we started off with the question matching in a very simple way.

• 21:50Speaker 04

  So basically, we just indexed the questions in Elasticsearch and incoming queries were then matched with this Elasticsearch index, which is basically just a rule-based matching. And we thought this is like a good baseline for people to continue working and developing because it's easily extendable to other languages. Elasticsearch doesn't really care about the language so much that has been inputted, and it is also super fast. And during the hackathon and the last days, we experimented a lot with BERT-based embeddings. And if you just, so BERT is like a language model where you stick in text and you get

• 22:32Speaker 04

  then a vector representation of, basically like a virtual vec was before for words. It now works on sentence or document level to get like a document representation of it. And these models, they really don't work so well when you just take the embedding and you compute similarities, like for example with the cosine similarity metric. They don't work so well out of the box. So you need to find ways to. Adjust these language models to suit your needs. And there we used a really nice library, Sentence Transformers. It's from a German NLP laboratory, UKP Lab. Niels Reimer is also a main contributor there.

• 23:15Speaker 04

  And this basically takes a BERT model and creates a clone out of it, like a Siamese network. So the weights are totally the same. You stick in the query that the user types in. And on the other side, you stick in the questions that you have already crawled. And then you get representations for both. And then you can compute similarity metric. And this whole network is trained end to end with exactly this user questions and the questions you have. And this works really, really great. Like the more data you feed into this network, the better it can match questions. And we've then also seen over the course of the hackathon that this is the way to go.

• 24:00Speaker 04

  And we need to extend this also to to other languages, because the questions from official FAQ pages are phrased in a very official tone. And people who want to ask questions more write in a colloquial manner or also there are spelling mistakes. And these models cover this by part quite well. And this is why we are actually trying to push in this direction more and more.

• 24:26Daniel Whitenack?

  So I'm pretty curious about that. And Chris could probably guess that I'm very curious about that because of my interest in languages, which we've talked about a lot. So you started talking about the sort of ElkStack or Elasticsearch, you know, matching with the index and then talked about BERT. So I'm curious, there's of course a lot of marginalized language communities out there that also need this sort of information and are only becoming even more marginalized because they don't have access to proper health information.

• 24:55Chris Benson?

  And I'm curious, so with that sort of flow that you talked about, you have a language model, let's say like BERT or a transformers model. So you could train this sort of model, assuming you had had data in in the language. And then there's this sentence transformers and matching piece that you talked about. In terms of transitioning that piece and the training of that piece, you mentioned you you have kind of your set of scraped questions and then the set of user questions. Do you need annotation in terms of, you know, matching known user questions to the properly matched FAQ, you know, scrape data? Is that what you need for for that certain piece?

• 25:37Speaker 04

  Exactly, this is exactly right. And we created manually, so the core team distributed a set of like 30 questions and we manually created rephrasings of these questions to basically evaluate the models. But now we also implemented a feedback mechanism into the UI and also we have a Telegram button, can maybe talk about the Telegram button as well later. And there, people can actually give positive feedback or negative feedback saying that maybe the content is irrelevant, it doesn't match the question or the content is outdated, for example, between formats that we have to adjust our scrapers in a way. And this, we hope, will scale to other languages

• 26:20Speaker 04

  and all the data that is coming from this will be open sourced in this COVID QA repository and will make this also available to other researchers that they can improve a question matching for COVID-related questions.

• 26:33Chris Benson?

  This is super cool. I love what you're doing on this. I guess one of the things I wanted to ask is maybe as people focused on these technologies and we're doing this day to day and our live and, you know, the efforts that we're engaged in in our own projects are built on these data sets that we have. And yet, as we look at this crisis and we're looking at the fact that the data set may or may not have everything you need, which you kind of alluded to there, you know, in terms of how applicable it is and getting that feedback. What kind of strategies are you thinking of in terms of being able to provide the outputs that maybe some users are needing if they're not in the CORD 19 data set inherently? Is that just a limit? Is that a hard limit of what you can tackle?

• 27:15Chris Benson?

  Or have you thought about how to extend beyond the limitations of the data set you're working with?

• 27:21Speaker 04

  So exactly like this is then the the second stage where we have more an extractive QA that takes some unstructured text database like the Court 19 dataset, for example, and then extracts question. We think that this will be related to researchers, but we could also envision that more text that the general public would be interested could be searchable with this system. The only problem there is that this extractive QA mechanisms are incredibly hard to scale to a huge amount of users. Yeah. So we would possibly do this for the general public in more like an offline way where we collect questions. And if a lot

• 28:04Speaker 04

  of questions come up that cannot be answered, we might need to use these extractive QA models to answer them from different data sources.

• 28:15Daniel Whitenack?

  Yeah. So just to kind of follow up on that, I think that it's worth noting here that I think it's really cool how you've approached this because there is existing sort of question and answer data out there that's from a trusted source. So let's say FAQ pages from the World Health Organization or something like that. So in the first case where you're talking about doing this matching with the the transformer models, you're actually matching a user query to a trusted source answer for that question because it was posted on an FAQ site. But then in the second piece, what you're just talking about in terms of extractive QA, really now what we're talking about is saying, okay, well, and correct me if I'm wrong, but I think I think the goal here would be to say, well, the user isn't

• 28:58Daniel Whitenack?

  asking exactly what's on the FAQ site, or maybe the user isn't sort of general public user, but they're a research user. And like you said, Tony, they want to know a very specific question. It's not on like a FAQ site, but it is in some research article or it is on some trusted source page. So this is like totally unstructured data. It's just like an article. And you want to ask some random question about that article. And that's where, you know, this extractive QA comes in. Tony, could you maybe comment a little bit about that and how this sort of extractive QA model is maybe different

• 29:40Daniel Whitenack?

  from the sort of embedding matching that we're talking about in the other case?

• 29:44Speaker 05

  Yeah, sure. So, I mean, the way these models typically work, they talk about a language model. I usually think of it, it's learning the statistics of a language. So it's effectively like learning I before E except after C. Or learning that, the way I kind of go over it is if the Dewey Decimal System is this random alphanumeric number system, but you give it to any librarian and they're able to take very different books and be able to basically place them correctly in a library in an ordered kind of structure. And so a lot of these models are doing the same thing. They're basically just looking at patterns of word co-occurrences and the statistics of how words occur. And what we're trying

• 30:28Speaker 05

  to do here is for the models that have already been trained, they're usually trained on things like Wikipedia or English language, German language text, which is great, but for these sorts of things, you really want to get into the domain-specific terms. So the medical terms, the genomics terms, the more difficult and more infrequent terms that won't be showing up in Wikipedia, and trying to learn the statistics of that data set. So they have existing things like Cybert and BioBert, which are BERT models that are built using things like the BioASQ data set. And so, Timo and I kind of had talked and

• 31:11Speaker 05

  we said, what if we took the CORD-19 data set, which is supposedly, we've looked at it now, there are mostly coronavirus articles, but there are lots of other articles as well in there. So the first rule of data science, the data is always going to be dirty coming in. So what we did is, we said, I'm an MD and Intel has a lot of contacts. And so I contacted people from the American Medical Association and people that I knew, and just basically put out a call and said, hey, could we get some domain experts, physicians, nurses, PhDs in biosciences, people that are probably, in some cases, sitting at home. I heard on the radio that third and fourth year medical students are being told to kind of stay home.

• 31:56Speaker 05

  And I was a third year medical student, I mean... I know how difficult it is. You want to be there. You want to be doing something. You're incredibly intelligent and you have all of these skills that you've spent the last two years doing. So I just put out a call and we set up a Slack channel and Timo's group had, you know, Deepset had this annotation server. So we put up this Chord dataset and it's essentially the Slack channel allows, I think we've got, we've got like 24 in the Slack channel right now. We just, we just started yesterday on the annotation and we've, right now, we've got over 100 question answers off the dataset, just in the, in the first day. And so these are things like, you know, I'll read you some of the ones that I'm looking

• 32:39Speaker 05

  at now from, from the website, you know, how many amino acids are in the SARS-CoV-1 protein? And the answer is 76 amino acids. And so this is something where, you know, Wikipedia's not going to be able to get you there. These are directly from the articles. It has a link back to the article that you're, you're talking about. Things like what does the SARS-CoV-1 protein activate? NLRP3 inflammasome. Again, very, very detailed kind of question and answer things that are either specific to viruses or specific to epidemiology or specific to SARS itself or, or COVID or MERS or any of these kind of similar pandemics that we, that we've seen. And the nice thing for the domain experts is they just log into a website. All

• 33:24Speaker 05

  they need is a web browser, an internet connection, and as long as they can highlight some text, they're good to go. And so we just throw them at it, give them, you know, some walkthrough videos and let them go away and annotate.

• 33:35Chris Benson?

  I'm kind of curious, as you're talking about the specifics of amino acids and such, as the, I know that China had done a complete genome of this virus fairly early on and, and published it. Have you been using that? Has that been helpful? And has that informed any of the work that you guys have been doing in the project here?

• 33:55Speaker 05

  Not for us. So what we've got so far are just the published articles that were on PubMed that were pushed into the core dataset, but it's certainly something that we could add into into things. So kind of the the mechanics of how these question and answer systems work is that it's kind of the annotator kind of goes backwards from the model. The annotator reads through the article. So this is a published, you know, peer reviewed article and the annotator comes across a certain fact and they say, that's interesting. That's something, you know, that's specific, that might be interesting. So they highlight it and they click question and they make up a question based off of the highlighted text in the article. So if they had the genomic sequence or something like that, they could certainly, you know,

• 34:38Speaker 05

  if that were in the in the text article, that's something that they could definitely highlight and make up a question to. And then the great thing about this is, you know, on Timo's side, when he, you know, creates an extractive AI model for it, it can actually extrapolate and then say, okay, I I understand the context and the statistics. And so if you threw me a new question that wasn't something that the annotators had ever come up with, it should be able to do a pretty decent job at kind of figuring out what it's looking for in that article. So if I put up a brand new article, if I put up a genomics article to this website and ask some questions, it should know where to look for in the text.

• 35:19Speaker 05

  And that's what's coming back is the model is not just making up words. It's identifying, it's highlighting the text and saying, here's the highlighted answer in the text. Does this seem right to you?

• 35:36Speaker 01

  The change log is deep discussions in and around the world of software and it's been going for over a decade. We interview hackers like Chris Anderson from 3D Robotics.

• 35:46Speaker 06

  At the time drones were like predators and global hawks and military industrial and they were classified and super, you know, $10 billion things. And we had just built a drone with Lego pieces around the dining room table programmed by a nine year old. And it's like, okay, that should not be possible. You know, it's when when a nine year old can do something that is classified, that literally export controlled as ammunition with Lego with toy pieces, it was something important in this world has changed.

• 36:15Speaker 01

  Leaders like Devin Zogel from GitHub.

• 36:18Speaker 07

  In the like 10 to 15 year range or 20 year range, what I would really like is for if you have like three 12 year olds hanging out and one of them is like, I want to be a firefighter. Another one's like, I want to be a lawyer. I want one of them to say that I want to be an open source developer.

• 36:33Speaker 01

  And innovators like Amel Hussein.

• 36:35Speaker 08

  I've yet to kind of see applications at scale that don't use multiple languages, that don't you have just arcane stories behind why this weirdo thing exists. You know, like, all right, when you open this file, you're going to have to turn around three times and tap your nose once.

• 36:52Speaker 08

  Like it's just the most hilarious stories, you know, but applications are living, breathing. They have craft that's normal. So I want to normalize weirdness because that's just how applications evolve over time.

• 37:08Speaker 01

  Welcome to the change log. Please listen to an episode from our catalog that interests you and subscribe today. We'd love to have you with us.

• 37:27Daniel Whitenack?

  So we've kind of talked about the QA annotation that Tony has kind of helped spin up and really utilizing that expert input from doctors, from medical students, from medical professionals on the CORD 19 dataset. I was curious to kind of push that back to you, Timo, and see what your thoughts are in terms of, let's say that Tony was able to get all this annotation in place and it sounds like there's a great start on that. How do you see that being integrated into the COVID QA system itself? And maybe how do you see, you know, the two sides of the COVID QA system developing?

• 38:06Speaker 04

  Yeah, exactly. Good question. So I would say it starts with a scale of data that is needed. So for this question matching, we can either use already pre-existing external data sets. And then this matched questions, maybe 1000 or 2000 per language. This is enough to get a really good matching system already going. But for this extractive question answering, there's much, much more help needed. The scales are there, like this common data sets. Squatch from Stanford is in 150,000 question answer pairs are there. And that's why we think it's really great to have external help.

• 38:45Speaker 04

  Because this will be the largest lever, getting this data in a format that we can then use in the frameworks. This framework, we actually use a haystack, which is basically enabling question answering on a larger scale. So normally, if you just use a language model like BERT, you basically take paragraphs, a small paragraph, like maybe 2000, 3000 characters, and you ask a question and you compute this. But for a large document base, this would be very infeasible. And then you need a two-stage process. In the first stage, you pre-select documents that could be relevant with a very cheap and very fast solution.

• 39:28Speaker 04

  And then you apply more powerful models like BERT. And this is definitely not a real rein, it's definitely not a new invention. For example, there's a framework out from Facebook, it's called DrQA. This is exactly doing this retriever and then reader architecture. But haystack is doing it in a bit more modular way and modern way with a BERT-based extractive question answering system. And we think there's a huge gain in performance. And so we can take these labels that Tony and the collaborators produce and stick them into frameworks to train end-to-end systems that answer questions on this large Chord 19 dataset.

• 40:13Daniel Whitenack?

  So I'm curious, you mentioned kind of the scale of the data and you know, as we've, SIL has been working to get translations in place over the past days, definitely you know, translations in the thousands seem, you know, within reach. Annotations in the hundreds of thousands is definitely a tough thing, especially when you're relying on experts. But I was wondering if you could speak to, you know, I know some of these sort of domain adapted models, so CyBird or other ones, do I have it right that those are transfer learned from another model? So like if you have a model trained on the squad dataset for question and answer, which is totally general domain, is it possible to then transfer

• 40:56Daniel Whitenack?

  learn a domain adapted question and answer model with the data that Tony's working on?

• 41:01Speaker 04

  So it's a little bit difference. You have to separate a base language model that can just transform text to vectors and then you have to take this language model and adjust it to suit your task intent, for example, document classification or extraction of named entities like persons and cities and also question answering. You have to attach a prediction, add so another small neural network on top of this language model and then train this whole joint network on this target task. And models like CyBird or BioBird, they are just pre-trained on a large biomedical corpus. What we are also doing right now

• 41:46Speaker 04

  is we took a bird on English data and adjusted it then because this process of adjusting this language model to a domain is not that computationally expensive, for example, training the whole network from scratch. So we took this network, adjusted it to the Core 19 dataset and if we take this adjusted court bird, let's call it like that, and stick in the data, the labels, the question answering labels, it will hopefully perform better than just a plain bird trained on English.

• 42:19Daniel Whitenack?

  So you know, I guess, how would it be useful, would it be useful to get more annotators involved in this? And if so, what types of skills do you need with annotators to make it useful for them to apply annotations to the dataset? Who can do that and how many more people would be helpful?

• 42:37Speaker 05

  As many more as possible. I had a feeling. What we're primarily looking for are people that are kind of masters level or above in the biomedical sciences. So, I mean, you don't have to have a PhD to do this, but I would like to see someone who is comfortable in reading an academic paper and being able to explain it to someone else, or be able to point out the salient points. Some of the other things that are useful, even if you're not one of those expert annotators, or you're not sure if you're an expert annotator, are just proofreaders. So we've got on the Slack channel a sub-channel called Second Opinion, just like the medical jargon. And so Second Opinion is where we have somebody that just is looking through the current answers

• 43:21Speaker 05

  and the current question and answers and going, hmm, I wonder if that seems quite right because that doesn't seem to make sense to me. And so they'll put it up and say, hey, question and answer 234 is kind of weird. Can I talk to the person who annotated that? Or can I talk to somebody else who might be able to give us a yea or nay whether that's a good annotation or not? So things like that are always useful. And I'm getting good response so far, but I'd love to always get more, as we talked in the beginning, just people that are kind of at home trying to figure out things to do. Again, we've got geneticists who might not be doing anything right now. We've got people with biochemistry degrees that maybe they're not doing anything right

• 44:05Speaker 05

  now, or maybe they're grad students or would love to talk to them and try to onboard them for this. If you have an internet connection and know how to use a web browser, you're set. I have a daughter who's a third-year med student, so I'm definitely... Oh, perfect. She is, cover, contact me, absolutely. We'll get her going today. I'm definitely going to bring it to her attention. We made it 25 at least on this. Excellent. We made it to 25.

• 44:29Daniel Whitenack?

  And what's the best way for those sorts of people to contact the effort and get on board it? Is it the best way through the GitHub repo, or is there another way to do that?

• 44:42Speaker 05

  Yeah, so for the programmer, it's the GitHub repo. And then for the domain expert, it's probably going to be the Slack channel. And we can put the Slack channel up on the site. And we're kind of keeping those two communities kind of separated. So one doesn't get freaked out by the other. So we'll keep the coders on one side and the biologics on the other side.

• 45:02Chris Benson?

  And we can also add that into the show notes, which I think is what Daniel was about to say there. That way it makes it easy for them to slide through to those.

• 45:08Speaker 05

  Yeah, to click on it. Yeah.

• 45:10Daniel Whitenack?

  Yeah, we'll definitely get the Slack team added to the show notes. So if you're listening and you want to be involved in the annotation, take a look at those show notes and make sure you reach out. I was curious on the kind of circling back. So that's a great way to contribute from the research user side of things. In terms of scaling up the general public use case of COVID QA, it sounds like there's definitely still some needs there around language support. And maybe that has to do with, so maybe Timo, you could mention what would be best to add there in terms, maybe it's scraping more information, more FAQs or adding them. And then also on the development side, where are your biggest needs right now in terms of making COVID QA

• 45:55Daniel Whitenack?

  really useful for the general public? What are some needs that you have, whether that's front end development, maybe not even AI related? Or maybe it is AI related?

• 46:05Speaker 04

  I'm really great that we are separating labeling process and developing process because it will get super complicated. And I also wanted here to thank Tony that he's taking also a lot of initiative for supervising and pushing the label process. Because I think I have never heard of an open source labeling process. And I think this will be a mess at some point. And it will be very complicated to interact and also to supervise the quality of the labels. But I think this is the great part of this challenge. We just have to try and have to make it work. So this is really great on the labeling part. On the development part for computer scientists of any sorts, there's a lot of help needed.

• 46:49Speaker 04

  So this is just a hackathon project. It's not like a full-fledged professional industry solution, so we need a lot of help. I'm in contact actually with a data scientist also from Intel through a contact from Tony. And she's working on an intelligence scraper. So right now we have very manual scraping processes for each page that adjusts to the HTML structure. And we would need more intelligence scraper that you can just point to an FAQ page. And it automatically extracts questions and the answers. Of course, there will be errors, but I mean, this is a little bit unavoidable. Maybe we can do a review process there afterwards.

• 47:31Speaker 04

  So this intelligence scraping will be extremely helpful. And then you also mentioned then that bringing this question matching to other languages. This is something that is personally very important to me because I think this will create the biggest societal impact. And there's a lot to do because for now we have the question matching algorithm with sentence transformers. And BERT just implemented for English, but making this work for other languages with multilingual language models, for example, with this cross-lingual language model, open source by Facebook, for example. This would improve the experience a lot. This on the modeling side.

• 48:17Speaker 04

  And then also a huge help we need on getting this actually to people. And after the hackathon, we got contacted by a person, I don't even know his or her personal name. It's the Apache sixty-four. And this person just programmed a Telegram integration. So this service has an API where it can match questions and you can call this API. And he integrated this into Telegram. And this bot is just, yeah, it's working. Also, we integrated the feedback mechanism to feedback the user information back into our system. So this help is really appreciated.

• 48:58Speaker 04

  But what I think could be important there would be maybe a WhatsApp integration. And maybe even if we extend this really to to low resource languages where people might not have access to mobile phones with internet, maybe have like a text message interaction. But this would be a little bit further away, I would say.

• 49:20Chris Benson?

  So I guess as we get toward the end here, I want to ask a question and I'd like each of you to give me your perspective, since you're coming from two different places on it. As we look, we are in this global crisis, which is unique and has stressed all of us and forced us to think creatively in ways that we have just never done before. It's sort of like living in a science fiction novel to some degree. And so as you guys are looking at the role of artificial intelligence at the world, within the world, and we're looking at, suddenly we have this crisis upon us. How do you see artificial intelligence technologies and data technologies impacting our way through this crisis at large?

• 50:03Chris Benson?

  Not just the project that you're in, but its kind of role in the larger world? How has your perspective possibly changed over the last few weeks with regard to that? And what opportunities do you see as the most exciting, you know, in terms of the path forward now that you are involved in this and seeing the results that you are? Timo, you want to go first?

• 50:21Speaker 04

  Yes, totally. So I would say this will like enlarge the way corporates contribute to like a solution that is helpful for everybody. For example, DeepMind has announced quite early a solution where they basically analyze the molecular structure with reinforcement learning. And if basically through interdisciplinary collaboration, that is now made much, much more possible, less bureaucratic and very fast and agile, I think a lot of great solutions can emerge. And I also think that a lot of corporations give their employees actually some dedicated time to work on these solutions.

• 51:06Speaker 04

  So like a collective effort of everybody around the world to work on something that is not directly related to making profits, but to solving this crisis. And I think this is something unique, as you said, it's like a unique situation. And yeah, it will hopefully can make people collaborate a little bit closer on things that is relevant for society.

• 51:30Speaker 05

  Yeah, so I love what Timo said about, you know, trying to do things that are relevant to society. And I'm not an official intel speaker, so but I can, I can tell you that, you know, we're very creative people and we're allowed to do a lot of, you know, things kind of what interests us in addition to our usual, you know, what pays the bills. This has actually been kind of interesting that now the things that are interesting, the whole company and in fact, the whole, you know, world, basically now, all the extra stuff is now going toward what can I actually do. In terms of the AI stuff, I mean, I kind of go back to, you know, we're sheltering in place and we're trying to get through, you know, kind of the scale without being connected. And somehow AI is kind of helping us to get through the mountain of data that's coming in and trying

• 52:15Speaker 05

  to maybe focus us a little bit better. I mean, it's designed to be a tool, you know, it's not designed to replace anything. It's just, it's designed to be a really, really nice way of sharpening the edge to figure out exactly what we want to do and what's possible. So that's where I see, see AI coming in.

• 52:33Daniel Whitenack?

  Awesome. Well, we appreciate you both taking time to join us today. I know that especially during this time, there's a lot to work on. So thank you for taking time. And definitely if you're listening out there and you are wanting to contribute in a positive way, using your development skills, using your AI skills, using your health knowledge and your medical expertise, please check out this project. The links are in the show notes and reach out as well. If you're having trouble figuring out how to get connected, there's our Slack team as well, which you can find at changelog.com/community. And we're happy to get you connected to Timo and or Tony and their team. So just make

• 53:16Daniel Whitenack?

  sure you get connected and contribute. And thank you both, Timo and Tony for joining us. Really appreciate it.

• 53:21Speaker 05

  Thanks, Dan. Thanks, Chris.

• 53:23Daniel Whitenack?

  Yeah, be safe, stay healthy. Thanks for inviting us.

• 53:29Speaker 01

  Thank you for listening to this episode of Practical AI. More like this at changelog.com/practicalai. There you'll find our latest, as well as lists of our most popular episodes and the ones we recommend. If this show has helped you on your AI journey, please leave us a five-star review on Apple Podcasts, heart us on Spotify, star us on Overcast, and tell a friend what they're missing out on. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo, and our music is brought to you by the beat freak, Breakmaster Cylinder. We have awesome sponsors, please support them, they support us. Thanks again to Fastly, Linode, and Rollbar. That's all for now, we'll talk to you next time.