Exploring the COVID-19 Open Research Dataset

In the midst of the COVID-19 pandemic, Daniel and Chris have a timely conversation with Lucy Lu Wang of the Allen Institute for Artificial Intelligence about COVID-19 Open Research Dataset (CORD-19). She relates how CORD-19 was created and organized, and how researchers around the world are currently using the data to answer important COVID-19 questions that will help the world through this ongoing crisis. Sponsors: Linode https://linode.com/changelog – Our cloud of choice and the home of Changelog.com. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2019 OR changelog2020. To learn more and get started head to linode.com/changelog https://linode.com/changelog. AI Classroom https://datadan.io – An immersive, 3 day virtual training in AI with Practical AI co-host Daniel Whitenack. Get 10% off using the code PRACTICALAI10. To learn more and purchase tickets go to datadan.io https://datadan.io/. Featuring: Lucy Lu Wang – Website https://llwang.net/, GitHub https://github.com/lucylw, LinkedIn https://www.linkedin.com/in/lucylw, X https://x.com/lucyluwangChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Lucy Lu Wang - Google Scholar https://scholar.google.com/citations?user=REBtJOYAAAAJKaggle: COVID-19 Open Research Dataset Challenge (CORD-19) https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challengeCORD-19 Explorer https://cord-19.apps.allenai.orgSemantic Scholar | CORD-19 https://pages.semanticscholar.org/coronavirus-researchAllen Institute for AI https://allenai.orgUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The scientific engine has really spun up to handle this current situation. As far as I know, there's been more than 4,000 papers released since January on COVID-19.

• 0:10Daniel Whitenack?

  Wow.

• 0:11Speaker 00

  And that the number of papers continues to grow, but more importantly, the number of papers released every day continues to grow. So we're up to maybe more than like several hundreds of new papers a day.

• 0:25Speaker 02

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com and we're hosted on Linode cloud servers. Head to linode.com slash change log.

• 0:43Speaker 02

  Do not underestimate the power of the independent open cloud for developers. Yes, I'm talking about Linode. Linode is our cloud of choice and it's the home of change log.com. What we love most about Linode is their independence and their commitment to open cloud. Open cloud means being unencumbered by outside investment and maximizing value for the community, not shareholders. And that's exactly what Linode represents. No vendor lock-in, open at every layer. If you want to learn more, head to linode.com slash open. Again, linode.com slash open.

• 1:26Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM. Okay, take it away, guys.

• 1:52Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with S I L International and I'm joined as always by my co-host, Chris Benson, who is a Principal AI Strategist at Lockheed Martin. How are things down in Atlanta, Chris?

• 2:10Chris Benson?

  Doing very well down in Atlanta. Got a bit of a cold, so may cough my way through the episode, but other than that, doing great. Spring has sprung. It's beautiful.

• 2:18Daniel Whitenack?

  Hopefully just a cold.

• 2:19Chris Benson?

  Hopefully, yeah, we're just crossing our fingers. Took my daughter a couple of weeks, I'm hoping it was her cold, took her to a pediatrician a couple of weeks ago. We actually had to go in because it kept going and it was a frightening thing to say, "Well, it could be strep, could be this, or it could be COVID-19. We can't exclude that." And as a parent, that was like, whoa. So I've been, I've gotten through that. Just a cold. We're good.

• 2:41Daniel Whitenack?

  Good, good.

• 2:42Chris Benson?

  Rolling forward.

• 2:43Daniel Whitenack?

  Well, we're surviving here in lockdown in Indiana. It's actually pretty nice outside. It's mushroom season here, so there's these like wild mushrooms that come out in Indiana just around this time. They're called morel mushrooms and we go every year hunting. So my family has some property that's all forested. No one else is there, so we've found some good times just going out there and walking through the forest and getting outside. So that's, that's been nice.

• 3:12Chris Benson?

  Yeah, that sounds nice, whether you find any mushrooms or not.

• 3:14Daniel Whitenack?

  Exactly. Well, I guess a related topic because it's really the only topic these days affecting us in all of our lives in a big way. We've had another episode a couple of weeks ago about the COVID QA system, which is a question answer system related to COVID-19. And they were also using this data set called CORD-19. And today we've got Lucy Liu Wong from the Allen Institute for AI. She's a research scientist there and we're going to be talking all about the CORD-19 data set, the ins and outs and the story behind it. So welcome, Lucy.

• 3:52Speaker 00

  Hi. Thank you, Daniel and Chris, for having me on the show.

• 3:56Daniel Whitenack?

  Yeah, it's great to have you here. Appreciate you joining us. This is, of course, a big topic. Everyone on Twitter and all around is talking about this data set and how it's being used. So we're really excited to talk about it a little bit more here. But before we do that, I'd love to hear a little bit about your background, how you got into AI-related things and ended up at the Allen Institute.

• 4:21Speaker 00

  Sure. Yeah. So I guess my background's maybe a little less traditional. I started out more in biomedical engineering and physics and worked in a host of biomedical startup companies, creating medical devices. And over time, I was doing more simulations and incorporating more data science and machine learning techniques into my work and found that that was what was very motivating for me. So I decided to pursue a PhD in biomedical informatics, where I focus primarily on biomedical applications of natural language processing techniques and creating models to try to connect these automated methods

• 5:03Speaker 00

  with the type of improvements in clinical care and biomedical text mining that we so desperately need these days.

• 5:11Daniel Whitenack?

  Yeah. So when you're talking about NLP for biomedical applications, are we talking mostly here about medical records and doctor's notes or whatever that is and trying to extract relevant information from those and patterns and mine those for useful things? Is that the main sort of drive there?

• 5:31Speaker 00

  That's definitely one aspect of things. I am also very interested in looking into the scientific literature and trying to extract entities and relationships and useful information out of that body of work. And I think that's what really I'm working on at the Allen Institute for AI, or AI2. I'm part of a team called Semantic Scholar, which I think a couple of weeks back, you had an episode about Semantic Scholar. And it's a literature search engine project. And for Semantic Scholar, we've indexed, you know, 180 million papers. There's a really rich corpus of texts to work with. And as part of the research team there, I've created a number of tools and work on a number

• 6:14Speaker 00

  of projects to kind of understand more about the content of that text. And that's kind of what brought us to the Chord 19 dataset is we have this kind of underlying infrastructure for processing scientific text. And we were asked to contribute some of that expertise to creating the dataset.

• 6:32Daniel Whitenack?

  Awesome, yeah. And I'm curious, so I'll definitely reference the Semantic Scholar episode in our show notes so people can listen to that because I think that provides a really good baseline for like the sort of data that you came into this recent work having, which is a really great way to find and discover scientific information and related data across scientific literature, which is amazing. I was wondering if you could comment before we jump into Chord 19 specifically. I know we're kind of in a really interesting time where a lot of people are publishing a lot of things about COVID-19 very rapidly. What does that situation currently look like?

• 7:14Daniel Whitenack?

  Is it like, you know, are we talking thousands of papers over how much time? How rapidly are they coming out?

• 7:23Speaker 00

  Yeah, I think the scientific engine has really spun up to handle this current situation. As far as I know, there's been more than 4,000 papers released since January on COVID-19.

• 7:36Daniel Whitenack?

  Wow.

• 7:36Speaker 00

  And that the number of papers continues to grow, but more importantly, the number of papers released every day continues to grow. So we're up to maybe more than like several hundreds of new papers a day.

• 7:49Daniel Whitenack?

  Wow.

• 7:49Speaker 00

  And it's kind of intimidating to look at this source of information and kind of see what people are discovering.

• 7:57Chris Benson?

  So the fact that you have so many new coming in every day, are you refreshing the dataset? Is the dataset static at some point in time? Or is it something that you're constantly updating and refreshing?

• 8:07Speaker 00

  Yeah, so I guess maybe folks already know what the CORD 19 dataset is, but it's a kind of a collection of papers about COVID-19 research, including historic coronavirus research. So we have a collection of historic research, and then we also have all of the new research that is being released daily. We update the dataset currently at a weekly cadence, but we are rapidly moving to a daily cadence since there's just so many new papers released every day.

• 8:35Daniel Whitenack?

  Yeah, so since we kind of went that direction, I was wondering if you could maybe tell a little bit of the story of how this dataset came about. I mean, obviously, you have data about scientific literature within Semantic Scholar, and you're already doing certain things as relating to, you know, tracking entities or topics covered in those. How did the idea for CORD 19 come about? And I know that there's others involved in this, too. So there's Allen AI, but there's also, I think, Microsoft and the Chan Zuckerberg Foundation and others. So how did this come about?

• 9:08Speaker 00

  Yeah, so the entire project is kind of a coordinated effort by the White House Office of Science and Technology Policy. And I think sometime in early March, a group at Georgetown, the Center for Security and Emerging Technologies, Georgetown CSET, they reached out to us at Allen AI to help coordinate the release of this dataset along with a couple of different organizations. You mentioned MSR, Microsoft Research, Chan Zuckerberg. CAGL was also involved, and the National Library of Medicine, which is part of the NIH. So all of these groups were going to come together to essentially create this dataset to help, I guess, create text mining and information retrieval tools that could assist medical experts

• 9:51Speaker 00

  in understanding more of what was going on with the epidemic. And for Allen AI, the way that we got involved... Is we had recently sort of created a new pipeline to revamp our research corpus, our open research corpus. So we had a pipeline for essentially taking these paper documents, which are traditionally in kind of a PDF format, not very easy for text mining, not very accessible, and converting them into kind of like a structured full text format where you could run these natural language processing models on them more easily. So that's sort of our major contribution to the data set is the pipeline for both harmonizing the paper metadata that we've collected over the years and also producing

• 10:36Speaker 00

  these structured full text parses so that we can run our models over that text.

• 10:42Daniel Whitenack?

  And I know one of the big things that we talked about when we talked about Symantec Scholar was the ability to kind of find relevant data that might be buried in the wealth of scientific literature that we have about a certain subject that is of interest. So when you came to CORD 19, there's the extraction of the metadata, the actual content of the paper, but then how do you even go about saying like, these are all the papers related to coronavirus? I mean, I know, and I'm a little bit ignorant on this subject, so you'll have to forgive me. I know that coronavirus is kind of a family of things.

• 11:22Daniel Whitenack?

  It's not just this COVID-19, which is associated with coronavirus, but there's coronavirus associated with the common cold and all these things. So how do you go about saying like, this is what we're scoping down our data set to and finding that and along with that, deciding what you're going to exclude, I guess, as well?

• 11:43Speaker 00

  It's a great question, and I think it's a question with very open answers. So what we started with were a couple of trusted sources that we knew needed to be included in this data set. And those sources were a collection of papers curated by the World Health Organization on COVID-19 specifically. And we also performed searches over PubMed Central, which is a biomedical paper repository run by the NLM, the National Library of Medicine. As well as these pre-print servers Bio Archive and Med Archive, which were publishing the latest research on COVID-19. And we went out and collected papers from these sources

• 12:28Speaker 00

  using a set of essentially like keyword searches to make sure that they were relevant to both COVID-19 or the family of coronaviruses in general. Because I think historical coronaviruses like SARS and MERS are also extremely relevant in the current case.

• 12:46Chris Benson?

  I'm kind of curious, as you reached out and made the dataset available, and you look across some of your partner websites, like Kaggle has a call to action and stuff, and you're trying to get AI practitioners and data scientists to focus on kind of important questions that need answering for this purpose. How do you provide guidance in that way for people who are going to engage on the dataset? Is it something where people just grab it and do whatever they want? Is there any kind of organization across teams? You know, there's a lot of human factors involved in this. And how was that conceived?

• 13:17Speaker 00

  Yeah, so there were a lot of challenges. For Kaggle, when we opened the challenge initially, the COVID-19 challenge, there was a set of kind of 10 slightly open-ended clinical questions which were given to the community. And the engagement at Kaggle, the response we've received, has been like absolutely incredible. There's been like millions of views on the landing pages. The dataset has been downloaded 70,000 times or more. There's been lots of teams that have cropped up and self-organized to work on this dataset. I think there's a group called Corena Y. That's like several hundred data scientists and medical experts who've bonded together

• 14:00Speaker 00

  to work on the COVID-19 dataset and other coronavirus datasets. And we really want to just offer support to these community members. So there's, you know, a couple of sources of information that we've created to help facilitate these things. So on Kaggle, the forums have been super active. There have been a lot of people answering questions for each other, including from the organizations that have created these datasets. We've also established like a discourse to answer questions specifically about COVID-19, the dataset.

• 14:32Speaker 00

  So that's a great place to get answers. And finally, I think for these Kaggle challenges and for these shared tasks, one of the things that we're really trying to do by hosting these shared tasks is to connect ML experts with the medical community and kind of experts who can judge the answers that are being retrieved and extracted by these machine learning experts and see whether they have practical application in the clinic. So that's been a challenge.

• 15:05Daniel Whitenack?

  What's up? This is Daniel Whitenack, one of your Practical AI co-hosts, and I hope you're enjoying this episode and staying healthy during these crazy times. I'm working on some pretty cool AI stuff here from my home office, but I've also found that I'm having to get a bit creative and be intentional when it comes to honing my AI skills and virtually connecting with the AI community. If you're in a similar situation or you've been inspired by the Practical AI we talk about on this show, I want to invite you to a live online AI training event I'm hosting this May called AI Classroom. In AI Classroom, I'm going to teach you the practical skills I've learned over the years using the latest open source AI technology.

• 15:48Daniel Whitenack?

  You'll learn AI theory along with practical, hands-on implementations in both PyTorch and TensorFlow. And after the training, you'll be able to understand the latest AI models, implement your own models in code, train computer vision and NLP models, create model inference servers, and experiment with state-of-the-art methods like reinforcement learning. AI Classroom is taking place this May. It'll be taking place live and completely online in a high-quality virtual classroom, so no travel is required. There'll also be two cohorts with convenient time zones for Eastern and Western hemispheres. Don't miss out. Tickets and more information are available at datadan.io. That's datadan.io.

• 16:32Daniel Whitenack?

  And Practical AI listeners can use the code PRACTICALAI10 for 10% off. See you online in AI Classroom.

• 16:51Daniel Whitenack?

  So Lucy, you just brought up something that I think is really interesting, which is the sort of interaction between the AI community and the medical community. And I was actually wondering while you were talking about like, okay, this Core 19 dataset exists, but I know I have some AI expertise, but I don't necessarily have a lot of medical expertise outside of knowing like that I should wash my hands and these other things that kind of the top five that have been going around. I guess I was wondering, as you've got more experience with this kind of intersection between the AI community and the medical community, what has that interaction been like in the past? Has there been much overlap between the AI

• 17:34Daniel Whitenack?

  community and medical practitioners? And then, secondly, as we enter into this new Core 19 challenge, has that changed in any sort of way or been rapidly, you know, advancing in any sort of way?

• 17:48Speaker 00

  I've sort of worked on the intersection of these communities for a number of years, and I think there's a lot of great collaborations going on. I think a lot of folks in the computing community are incredibly motivated by these like very practical questions that need to be addressed, ways to improve patient care, ways to help with drug development or vaccine development and questions of these nature, this nature. As for kind of COVID-19 specific initiatives, so I can kind of give you two anecdotes for ways that we had annotators, or I guess medical experts interact with computing experts. So for the Kaggle challenge, seems that what is

• 18:33Speaker 00

  happening is a lot of people are developing different systems, different information retrieval, different information extraction systems, and those systems need to be reviewed by experts for usefulness. So in Kaggle, there's essentially kind of like an army of medical students and other people who are willing to provide their medical expertise and volunteer their medical expertise. Who are actually going through and manually reviewing a lot of the extractions that are coming out of these CAGL challenges and creating these kind of living systematic review pages with the answers to some of these questions. So if you go to the CAGL page, you can see these reviews kind of being created in real

• 19:17Speaker 00

  time and updated in real time as new literature is released. And another thing that I've been involved in lately is we're kind of running a TREC challenge on this dataset. So TREC is the Text Retrieval Conference, and it's been a project at NIST, which is the National Institute of Standards and Technology for the last 20 years. And these folks are really good at information retrieval and judging information retrieval systems. And the way that these systems are judged is by having expert medical annotators review all the results and provide gold rankings of what is most relevant to query and what is the kind of least relevant.

• 20:01Speaker 00

  So there's a lot of this, like how incorporating experts in the loop, incorporating humans in the loop to kind of bolster our machine learning systems. And that is not something that we're going to be moving away from anytime soon.

• 20:15Chris Benson?

  As you're talking here, I'm looking at the various questions there that are listed on CAGL, the tasks to go answer, and kind of extending this thing about this collaboration between the AI community and the medical community. The questions themselves, where did they originate from? How were they decided as the important questions that we could all give a shot at going and answering with the dataset? How was that?

• 20:37Speaker 00

  Yeah, I may be wrong, but I believe this set of questions originated from the White House Office of Science and Technology Policy in collaboration with CAGL. And you have to understand, so this challenge and the dataset, during the early days, we literally had just a few days to turn around this dataset, put it out there, and kind of publish this challenge. We wanted people to start looking at this as quickly as possible. So there were, you know, a lot of the questions that you see on CAGL right now are very open-ended, they can be interpreted in different ways, and as time has gone on, as we've learned in this last month. There's actually, some of those questions are more useful in clinic. Uh, some of those questions are less useful.

• 21:20Speaker 00

  Like clinicians already know the answers to some of these questions. So now, as we move into the second month of this challenge, there will be new, a new batch of questions released, um, to kind of motivate new work and the kind of questions that have not yet been answered by the community.

• 21:37Daniel Whitenack?

  Yeah, so I, I'm curious with that. It seems like you could have various bottlenecks in, in this situation. And one of those, I think you highlighted is this sort of useful interaction between medical practitioners and the AI people that are trying to do something with the, the data set. So I was wondering, do you have a sort of healthy community of, of medical practitioners that are, are very deeply involved in kind of looking at what's coming through Kaggle or these other teams that are kind of self-organizing? Cause I know that's one, having worked at a nonprofit for a bit. One of the things I've seen is like, oh, people really get behind like the social good like challenge

• 22:18Daniel Whitenack?

  and they work on it like on the hackathon on the weekend and then the project kind of dies. So, um, how, how are you kind of, what are the ways for people? If I want to step into the, some cord 19 related work, what are the ways that I can kind of step into that? But also get connected with the right sort of medical people to make sure that what I'm doing is useful and not just like an kind of interesting weekend thing?

• 22:45Speaker 00

  Yeah, of course. Uh, I think like now that COVID-19 has sort of taken over all of our lives, um, a lot of people are feeling very motivated to do something in this, uh, direction, um, contribute their skills. And I think I, I mentioned some groups earlier, groups like Corona Y, which is self-organizing to analyze this type of data. And that's a group made up of data scientists, machine learning experts, medical practitioners and so on. And so it's like a good place to get feedback on one's work. And CAGL forums, similarly, I think there are a couple of threads out there essentially continuing this discussion of how to connect to medical experts, how to verify results and

• 23:29Speaker 00

  so on. A lot of people have, you know, taken it upon themselves to build systems. There's tons of systems out there for searching cord 19, for extracting information out of cord 19. And, you know, to be perfectly honest, not all of those systems are going to be like highly usable or used by any kind of clinical audience. But we, as a community, need to essentially figure out which of those systems are most promising, figure out where to expend additional energy, like more development time and so on. And that's where our annotators come in.

• 24:03Chris Benson?

  Going back to what you said a moment ago, you have put this together so quickly and gotten it out there and the whole world has, has kind of dived into it, the whole world of data science at least. And, you know, that's very different from how most communities, you know, form. And I'm kind of wondering, with hindsight, totally recognizing that you had no choice, you had to get it out there and you did a fantastic job with what the kind of pressure that you were under. If you could go back, knowing what you know today, what are some of the things around community that you would like to have done and that maybe going forward as you're looking at like the tasks and how to move us going into the second month, how to move us in the next set of directions that you need people to take? What are some of the ideas that you're planning to implement there to kind of evolve this process?

• 24:48Speaker 00

  Let me try to unpack that.

• 24:49Chris Benson?

  No worries. Any way you want, it's fine.

• 24:52Speaker 00

  Sure. I think we've learned a ton over the last month. Speaking with some of our collaborators at CAGL, like Anthony Goldblum was mentioning, how they've really fallen, or CAGL has fallen into this place with this challenge where they're kind of in new territory. Like, the type of challenge that the cord 19 challenge is, is very unlike most of the challenges hosted by CAGL. And there's a very open-ended nature of it. We're like trying to discover answers, but there's no sense of like what a gold answer is. And they've reacted to that in a very like, kind of wonderful way, by essentially harnessing these medical students as a resource to make judgments on people's extractions and

• 25:37Speaker 00

  kind of putting an effort there where it seemed like the results were most useful or were going to be the most useful. So I guess that's one thing, which is trying to figure out as early as possible. Where the most useful results are and putting in additional effort there and maybe even like abandoning things that are not worth pursuing. And then I don't even remember what the second half of that question was.

• 26:01Chris Benson?

  No, it was things that you might be thinking going forward, but no worries, we can come back to that.

• 26:06Speaker 00

  That's right. I mean, I have lots of thoughts on that as well. So certainly, we're going to be supporting cord 19 for as long as it makes sense to do so, certainly until the epidemic seems to wind down a bit. There have been lots of requests for additional features and additional content. So that is one of our priorities. Additional content comes in two forms. One is simply providing more faithful parses of the papers. So right now, first things might be including things like inbound and outbound citations, tables and figures, other places where the answers might be. And these are, these have been requested by a lot of folks.

• 26:51Speaker 00

  Another is content in the sense of more papers. So one thing that we've been very grateful for is that a lot of publishers have made their COVID-19 articles open access. And by making them open access, they've allowed us to release a dataset like cord. But the fact is, if you look kind of at the dataset and at the papers that are being cited by the papers in the dataset, there's actually a lot of content that are outside of this direct core set of articles on COVID-19 and coronaviruses that are also very relevant to the content of the dataset. So it would be great if we could work with publishers or they could work with us essentially

• 27:35Speaker 00

  to provide additional content that could be useful for kind of discovering information about COVID-19 and its treatments.

• 27:43Daniel Whitenack?

  Yeah, so it's like you've amassed this kind of central hub, but from that hub of papers, obviously those papers cite other papers and those papers cite other papers and there's other related work and you can kind of go down a rabbit hole. I know we talked on the Semantic Scholar episode about the sort of graph of relations and that sort of thing as papers cite each other. I mean, there is a wealth of papers already in the dataset. What is the kind of current size of the dataset? And you know, you mentioned a few different sources. Could you just give us a sense of kind of the descriptive statistics of the dataset at this point, I guess?

• 28:21Speaker 00

  Sure, probably should have started with that.

• 28:24Daniel Whitenack?

  Yeah.

• 28:24Speaker 00

  So the dataset currently consists of more than 50,000 papers and approximately 40,000 of those papers have full text content available. And these papers, as I mentioned, come from a diversity of sources. There's a list of WHO COVID-19 papers, several hundred of those that have been curated by the WHO. And there's preprints from Bio Archive and Med Archive, that's numbers in the thousands. And actually, the vast majority of papers come to us via PubMed Central. And this number will actually continue to grow because

• 29:05Speaker 00

  many publishers are now depositing all COVID-19 content into PubMed Central.

• 29:11Chris Benson?

  So Lucy, I'm kind of curious. So recognizing that your responsibility has been putting this together and getting it out into the world, I imagine that you've probably talked with various teams or at least observed some of the efforts and work that's going on. And so I'm kind of curious, what are some of the more interesting or innovative or, you know, pick your adjective of the efforts that you've heard of and kind of gone, wow, that's a pretty cool way of approaching this? Any stories to tell us on that?

• 29:38Speaker 00

  I think folks have done a really great job diving deeply into this dataset. There's so many different kind of search engines that have cropped up over this dataset. If you go to the dataset landing page on Semantic Scholar, there's Like a list of maybe several dozen that we've heard of and are enumerating there. I'm sure there's many more that we aren't aware of. People are pursuing lots of different technologies for these kind of search engines. Some are kind of using the latest kind of state of the art transformer models, neural models for ranking. I think COVID X from Waterloo and NYU is using kind of the latest side T5 model. Very cool stuff.

• 30:22Speaker 00

  And some of the search engines are actually using very traditional methods, using Lucene or Elasticsearch and focusing more on how to search and filter using entities or other paper features. And we've kind of a funny thing that we heard from Kaggle is that for many of the questions on the CORD 19 challenge, something simpler methods, the more traditional methods have actually worked better for kind of extracting answers. And this came as a surprise to me.

• 30:55Chris Benson?

  So I have a follow up on that. And that is like for working better, you know, kind of putting that in quotes. What does that mean? Who out there is looking at the various results that are coming back from teams and making those evaluations and kind of, and obviously you've already said that you'll be steering in the new tasks on Kaggle, learning what we know. Who's making those evaluations and the decisions associated with that to keep everything focused?

• 31:22Speaker 00

  So I think this is a primarily work that has been taken on by organizers at Kaggle and kind of medical students that they have had come in and evaluate some of these answers. So really like they put in a ton of effort in curating these results. And I think as for like the metrics that we use to judge these results, I think currently like they're mostly kind of information retrieval based metrics of success.

• 31:56Daniel Whitenack?

  Yeah, to give people an idea, I'm on the Kaggle website and just looking at some of the tasks to maybe for those listeners out there that don't have a good idea of the kind of scope of these, there's information or some of the tasks that are listed are what do we know about COVID-19 risk factors, what do we know about vaccines and therapeutics, what has been published about medical care? And if I kind of dive into each of these, there's a number of submissions, but things like, you know, cord 19 analysis with sentence embeddings, COVID-19 literature clustering, full text search of research papers. And

• 32:37Daniel Whitenack?

  so you can kind of already see, and here's another one, BERT squad for semantic corpus search. So you get a sense, I think, of what you were talking about, Lucy, that some people are kind of going after these sort of transformer-based, maybe extractive QA sort of things. Others are maybe using full text search capabilities that have been out there for a while, like elastic search sort of capabilities, like we talked about on the previous episode. Also, if I'm looking at the data, just to kind of have something in people's mind, I'm seeing, of course, you have the categories of source, like the bio archive and that sort of thing. But if I'm looking at the individual papers, I can see the abstract of the paper and the

• 33:21Daniel Whitenack?

  body text. And if I'm looking at, you know, the body text of this one, I'm reading things like to assess the effects of truncation of the poly C tract on replication, blah, blah, blah, which some of that doesn't mean a lot to me, but this is the sort of data that's in there. I was wondering, as a complete noob to, you know, a lot of this medical terminology, what are maybe some good ways, I know you've got this kind of co-vis from the Allen Institute, which is helping kind of explore some of the, looks like the genes and cells and diseases and chemicals that are connected throughout the data set. What are some good ways of

• 34:02Daniel Whitenack?

  kind of onboarding into the cord 19 work for people that might be sort of new to medical terminology? Are there ways to kind of? Pick up some of that or explore some of it and form some of those connections in a reasonable sort of way?

• 34:20Chris Benson?

  The domain knowledge?

• 34:21Speaker 00

  Yeah, I think domain knowledge is one of the greatest barriers for working on this data set. And thanks for mentioning the CoViz project. That's a tool that was released by Alan AI for exploring this data set in a slightly more meaningful way. And for that tool, we essentially ran models to perform extractions of these entities from the text, entities of different classes like drugs, genes, diseases, phenotypes, things of that nature, and created a visualization to allow you to browse the relationships that are most prevalent between pairs of these entities.

• 35:06Speaker 00

  So that's a great way to kind of explore what's in the data set. There's also just like exploring the articles. We have a kind of a CORD 19 Explorer to help you do that. But in general, I think unless you're willing to spend sort of a couple of years of your life in medical school, it is very hard to understand what some of these terms mean. Certainly knowing what class or category of entity is being mentioned is important. So knowing something is a protein, knowing something is a receptor, knowing something belongs to a particular biological pathway. These are kind of key for gaining an initial understanding of what is being said in a text snippet. But that's also why we need medical experts to assess the actual utility of some of these extractions.

• 35:49Chris Benson?

  So I'm wondering, I happen to have a daughter who is a third year medical student. And I've told her very recently, because we had the other episode about this and stuff, but she hadn't been aware of it. Is there any need to connect with medical schools? Has anybody kind of taken that on to try to gather those together and stuff? Because it's been, obviously, there's been an enormous effort in a very short amount of time, totally recognizing the constraints of the reality that we're in today.

• 36:16Speaker 00

  Yeah.

• 36:17Chris Benson?

  Is that something that y'all are kind of thinking about in terms of going forward? Maybe for stage two or whatever you want to call it?

• 36:22Speaker 00

  Yeah, absolutely. I'm sure your daughter knows that, I mean, I don't know if she's still, if medical school is continuing as usual, but I think during the third and fourth years, you're mostly in clinic.

• 36:33Chris Benson?

  Yeah.

• 36:33Speaker 00

  So I know of a lot of medical schools where there are these kind of more senior medical students who really want to Contribute how they can, but really aren't able to be in clinic at this moment.

• 36:47Chris Benson?

  Yeah, they've been kicked out of the ER for, you know, and it's they're working with the health department locally. And I think that kind of alternative work is really common right now for advanced medical students.

• 36:57Speaker 00

  Yeah, exactly. So for the track task that I was mentioning that we're hosting, we actually are enlisting medical students from a number of institutions, from the Oregon Health and Science University and University of Texas and University of Washington to kind of help with provide annotations on some of these extractions. So I think depending on where your daughter is or where some of these medical students are, there's probably going to be other initiatives like this one that really need their help. So I would definitely encourage anyone to look out for that.

• 37:33Chris Benson?

  I guess I have one more follow up to that that you just mentioned. Do you think recognizing that and recognizing that we're going to be past this moment at some point, this kind of very unique moment in our history, but just as, you know, the kind of the widespread introduction of like open source software really changed, you know, industry itself from being highly proprietary to being, you know, open source became not only a part of business models, but even underlying part of a lot of commercial software that's out there. And it fundamentally changed how that worked. Do you think this is a moment just because COVID-19 passes us, you know, and we get past this, that that maybe there are other challenges, whether they be things that we've been dealing with a long time,

• 38:18Chris Benson?

  like cancers or or new things that may come, that this may fundamentally change how we attack, you know, really hard medical challenges with AI and that integration with the communities that that has happened out of necessity?

• 38:31Speaker 00

  I'm certainly hoping that to be the case. So we've definitely seen what people can do when they come together for a month or two. And it's it's incredible. Like there's so many people being engaged and building like interesting tools and useful tools. I think there's a couple of maybe things that I'd love for us to be able to extend into the future. One thing is definitely publishers coming together to release more open access content on really kind of.

• 39:03Speaker 00

  Important topics such as COVID-19 and then the community coming together, especially crossing boundaries, crossing boundaries between computing, the computing community, the medical community and policymakers to really build something useful.

• 39:20Daniel Whitenack?

  So I'm curious, a little bit of a follow up, I guess, to that, that question as well. I know there've been things, as I've worked on related work with SIL where I was thinking, oh, if I would have done this prior to this crisis, like I would be able to do something better than what I'm able to do now. In hindsight, it's easy to, to see those opportunities. I'm curious, on your side with your own research and work. I mean, you were kind of, I'm assuming you were working on various things related to Semantic Scholar prior to this crisis and now your, your head's down working on, on Cord 19 and, and getting this in shape. What are you interested in exploring in the future?

• 40:05Daniel Whitenack?

  Not necessarily Cord 19 related, but how has this whole process shifted, like what you want to work on in your own research in the future?

• 40:15Speaker 00

  Yeah, so I think this brings up a slightly earlier point, which is we really became involved in the creation of this data set because we had at Semantic Scholar built a bunch of infrastructure for scientific papers. And a collaborator of mine, Kyle Lowe, and I had also been working for the past nearly year on a way of essentially creating a full text extraction pipeline for some of these papers that we are using in Cord 19 today. So a lot of this was infrastructural work. It's not particularly glamorous, but it is really important and it really became more important in light of what happened in the last few months. I guess one thing is infrastructural improvements

• 40:59Speaker 00

  can be really important, even if it's not particularly sexy.

• 41:04Speaker 00

  And then kind of going forwards, there's certainly things I care about besides kind of creating data sets of papers. And my research focuses on making scientific literature, making this content more available to biomedical researchers and more understandable. And as you mentioned before, there's so many entities, so many kind of like very domain specific words and relationships that exist in the biomedical literature. And even for someone who is a domain expert, some of those terms can be very hard to parse through and understand. So a lot of my ongoing projects are trying to create systems that understand particular types of relationships. For example, those that understand drug-drug interactions

• 41:49Speaker 00

  or can mine them out of the literature, those that can understand medical images better. These are the types of projects that I am hoping to continue to work on in the future.

• 41:59Daniel Whitenack?

  Awesome. We're for sure going to have the links to the main dataset website in the show notes, along with the Kaggle challenge and the various other projects and groups that you've talked about. Really appreciate you coming on the show and describing a bit more about the dataset, how it came about, and your own work with it. Really encouraged by the work that the Semantic Scholar team and collaborators are doing. And thank you for your hard work on this and taking time to talk to us.

• 42:29Speaker 00

  Yeah, thank you so much for having me. And we really hope that other folks are encouraged to kind of contribute and become involved in this project.

• 42:36Daniel Whitenack?

  Yes, please do.

• 42:41Speaker 04

  Thank you for listening to this episode of Practical AI. More like this at changelog.com-practicalai. There you'll find our latest, as well as lists of our most popular episodes and the ones we recommend. If this show has helped you on your AI journey, please leave us a five-star review on Apple Podcasts, heart us on Spotify, star us on Overcast, and tell a friend what they're missing out on. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo. And our music is brought to you by The Beat Freak, Breakmaster Cylinder. We have awesome sponsors. Please support them. They support Thanks again to Fastly, Linode, and Rollbar. If you and your organization could benefit from speaking directly to all the AI practitioners out there, you should sponsor the show.

• 43:25Speaker 04

  Podcast advertising is one of the most effective ways to spread your message in an authentic way. Plus, you get the added bonus of supporting something you love. That's all for now. We'll talk to you next time.