What's up, DocQuery?

Chris sits down with Ankur Goyal to talk about DocQuery https://www.impira.com/product/docquery, Impira’s new open source ML model. DocQuery lets you ask questions about semi-structured data (like invoices) and unstructured documents (like contracts) using Large Language Models (LLMs). Ankur illustrates many of the ways DocQuery can help people tame documents, and references Chris’s real life tasks as a non-profit director to demonstrate that DocQuery is indeed practical AI. Featuring: Ankur Goyal – LinkedIn https://www.linkedin.com/in/ankrgyl, X https://x.com/ankrgylChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: DocQuery https://www.impira.com/product/docqueryDocQuery Announcement https://twitter.com/ankrgyl/status/1565437042032402433DocQuery Blog Announcement https://www.impira.com/blog/hey-machine-whats-my-invoice-totalDocQuery | GitHub https://github.com/impira/docqueryImpira https://www.impira.comUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  In Empira, we really wanted to create an experience where users could easily see whether the predictions were right or wrong. And then if the predictions are wrong, or if they feel compelled to give us feedback that they're right, they could correct or confirm things. And every time they do that, we drive the feedback into the model and incrementally train it. And so because of that design, we basically structured the machine learning approach to be one that is very, very lightweight and something that can train and evaluate really, really quickly.

• 0:39Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com. And to our friends at fly.io. We deploy our app servers close to our users and you can too. Learn more at fly.io.

• 1:15Chris Benson?

  Welcome to another episode of Practical AI. This is the podcast that likes to bring practical issues in artificial intelligence and learn as we go. I am your co-host Chris Benson. Daniel White neck is unfortunately traveling right now, so he's going to miss what I'm sure is going to be a pretty cool conversation. And without further ado, I would like to introduce our guest today, Anker Goyle, who is the founder and CEO of Empira. Welcome to the show.

• 1:45Speaker 00

  Thank you so much for having me. I'm really excited.

• 1:47Chris Benson?

  Yeah, absolutely. So we got a bunch of cool things to dive into today. I guess if you could just as a start, kind of before we actually dive into the topic, kind of tell us how you got here. Who are you? What's your story? And how did you arrive so that you could tell the world about what you're, what you're going to be talking about today, which is your company and DocQuery in particular?

• 2:10Speaker 00

  Awesome. Yeah, so I actually don't have a mathematical background in, in machine learning or AI. I've been working on relational databases for a really long time. I actually started doing research on them in school and worked at a company called SingleStore. I joined as the second employee and was the VP of Engineering there for some time. And what got me into the space is actually talking to our customers who were able to make use of data that is structured, but really struggled when the data that they wanted to work with didn't fit inside of the relational database that we built. And so I thought, you know, there has to be a better way. And looking around me, it was clear that the progress, and this was back in 2017, but and a lot has changed since then. But but even

• 2:53Speaker 00

  back then, it was really clear that the progress on the machine learning side would make it possible for people to work with any kind of data, no matter how structured or messy or complicated it is. And that's what we're all about at Empira. We're one part database and one part machine learning technology that basically makes it really easy to to work with unstructured data.

• 3:13Chris Benson?

  Very cool. You know, as you came into the industry and and, you know, getting ready to set up your company and looking at that with unstructured data, like, could you tell us a little bit about what you were walking into and why you chose the particular path in the industry that you that you did? You know, what was it that attracted you down the path that you did as an entrepreneur?

• 3:36Speaker 00

  Yeah, it was first thing I'll say is definitely a windy road. And we didn't know exactly what we were getting ourselves into when we started. So actually, when we started, we thought that the really big problems in helping companies work with unstructured data would be in helping them work with image and video content. And I think as it's becoming really clear now, with images and videos, the bottleneck is actually creation. It's not understanding. And so we learned that, you know, just purely on the market side of things a few years ago. And as kind of a funny coincidence, because one of the models that we ran on data that people uploaded was OCR, which is optical character recognition.

• 4:19Speaker 00

  Some of our customers started asking, you know, you can do this stuff with images and videos, but can you also analyze the data that is in my invoices and, you know, my forms and other documents? And we realized that there was actually a really exciting opportunity for us to help companies work with this kind of unstructured data. And so kind of a happy accident we discovered together with our customers.

• 4:41Chris Benson?

  You know, it's interesting that you mentioned that particular example because I know when I think of things like invoices and I run a separate from this, I run a nonprofit. So I kind of have that that business hat I have to wear separately. You know, I'm thinking of things like PDFs and things, you know, that are not typically what we're thinking of when we're training models, you know, it's not the form that we're usually, we're not going and pulling a bunch of data out of a database to train on or sources off the internet or whatever. So that's a little bit of a different take from your typical avenue into machine learning off the bat. How you like, like what is you started recognizing that was a challenge?

• 5:24Chris Benson?

  Did that worry you at all in terms of recognizing that you had, you were going to kind of take a different approach?

• 5:30Speaker 00

  It probably should have, but as usual with myself and co-founders and kind of how we think it didn't. And actually, Richard, who's our CTO, came up with a really powerful approach to solving this problem that uses primarily computer vision, actually, to reason about PDF files. And so for a long time, and we're foreshadowing a little bit with DocQuery, which kind of brings these worlds together. But for a long time, actually, a lot of the work that we did used computer vision. And so we thought if a PDF is like a hybrid of text and visual stuff, we leaned on the side of the visual stuff. And that has a number of advantages and disadvantages, which we've learned over time as well.

• 6:14Chris Benson?

  So you've mentioned PDFs or do you focus strictly on PDFs? Are there other file formats that you end up working with as well?

• 6:21Speaker 00

  What we do is we take almost any file you could throw at the system that you could, you know, like self identifies as a document, anything from PDF files to emails, HTML files, scanned images, pictures from your phone, just about anything. And we do a bunch of preprocessing upfront that basically normalizes anything you upload into a fairly consistent data structure. So. From whatever you put into the system, we normalize it into a bunch of pixels, a bunch of text and a bunch of bounding boxes that tell you where the pieces of text are, as well as a few other things.

• 7:01Chris Benson?

  Yeah. Gotcha. So before we dive fully into how you're approaching it at this point, what was in place, both from the early machine learning days as we're going back a few years, talking about that, but also you mentioned OCR. And what were the approaches people were taking and what was the mental model around that that you were looking at and saying, that's not good enough, based on what you were starting to think? What was the world looking like at that point?

• 7:31Speaker 00

  You know, what's really interesting about this is OCR is not a new thing. Neither is reading data from invoices or other kinds of documents. But for some reason, most businesses don't take advantage of it. And I think that's because the solutions out there are just not easy enough to use. And so we've always thought about this from the standpoint of what does it take to make something that's actually so easy to use that it provides value for someone? So the solutions that existed prior, they fell into a few different buckets. One is something called an OCR template, where basically you take OCR text and then you draw a box of X, Y coordinates around exactly where the text needs to be.

• 8:15Speaker 00

  And if you're working maybe at the DMV or something and taking identical documents and scanning them with an identical scanner every time, that approach can actually work really well. In reality, I'm sure with the invoices that you're working with in your business, it's never that simple, right? And so that's an example where the user experience and cost barrier in practice can be just prohibitively high. Another technique that was really emerging as more popular when we started is this really big pre-trained model approach. So AWUS has a product called Texttract, for example, which is actually, it's a great product. And what it allows you to do is upload any document into it and it will give you back some kind of data structure about what's in the document.

• 9:01Speaker 00

  And the nice thing about this approach is you don't need to do any of that template definition or anything like that. But the challenging thing about it is that if the results aren't what you expected, then you don't really have any recourse to solve for it. So we actually, a number of our early customers were using Texttract and building machine learning models on top of Texttract to normalize the data to be consistent. And they realized, you know, this is just not, what are we doing here? Right?

• 9:30Chris Benson?

  So it was essentially a Band-Aid that they were kind of creating on top of the product they were using or the service they were using.

• 9:37Speaker 00

  A very fancy Band-Aid. Yeah.

• 9:39Chris Benson?

  So, you know, with, and I know that we have seen kind of evolutions over quite a long time in OCR, you know, in terms of that. And you, you mentioned something though, that, that made me curious. You were talking about like, if you are using one of those early models that were pre-trained and then you didn't get what you wanted out of it. Can you talk a little bit about like, what kinds of problems might arise in terms of like, why weren't they getting it out of those models? To kind of define a little bit about the space that you're fixing, you know, going forward?

• 10:10Speaker 00

  Absolutely. Yeah. So there are two or three classes of problems. I think there are three. So the first kind of problem is, let's say you take a relatively low quality image, like a scan that's, that maybe is actually hard for a human to even decipher. Or maybe it has really bad handwriting or something like that. And you upload it into one of these products. If it can't read the handwriting or it can't read past the quality, there's really nothing you can do about it. And so that's one class of problem. Another class of problem is if you just consider a single document and you upload it into a service like this, it may not actually pick up all of the fields that are in the document.

• 10:54Speaker 00

  So one of the problems that we see, it's almost like, you know, bald spots or something in the document. It'll just miss things. And if it misses something, there's no way of telling it like, "Hey, please don't miss this field next time." You know, there's, there's no input like that that you can provide.

• 11:11Chris Benson?

  Because it's, it's all pre-trained and you, you've got what you got to work with at that point, right?

• 11:14Speaker 00

  Exactly, exactly. And the third thing is that if you imagine, you know, working with many, many documents, they all might have different bald spots. And so you might have two documents which for a user have the same schema, meaning they have the same fields that you want to extract, but you upload them into a schemaless service and you get back two different schemas. And that's actually where some of our early users were implementing their own machine learning models to try to translate from the schema that the pre-trained model produces to the schema that they actually want to work with.

• 11:47Chris Benson?

  That is not a problem I had really considered. That's an interesting side effect that you get on that. So you end up training in those early models, you end up, you know, having the trained model, you are running the document through the model. It comes up with both the white space issues and it also leaves you with the problem of an inferred schema that was not intended. And then I assume that at the end of that, you're trying to kind of get it all corrected back to what it needs to be. So that's a lot of manual effort there. You may have some tooling to help you along, but there's a, there's kind of a manual cleanup process that you're having to go through.

• 12:29Speaker 00

  Yeah.

• 12:29Chris Benson?

  So definitely interesting with, with, one of the things I wanted to, to ask about as far as that goes is you talked about OCR, but we're also talking about language models here. And you said that you were starting with the visual model. So we're not yet talking about any kind of NLP, natural language processing or anything like that, I'm assuming. We're talking about some sort of early visual model that's pre-trained.

• 12:54Speaker 00

  That's right. Although in Empira, the model that we had early on was actually not pre-trained. Because of how it works, it would actually learn just on the user's documents that they uploaded.

• 13:05Chris Benson?

  Interesting.

• 13:06Speaker 00

  Yeah.

• 13:44Chris Benson?

  So having kind of laid the landscape there of what you were kind of walking into in terms of of problems to solve and ways of making a better experience for people that needed this, could you kind of describe how you started thinking about that process in terms of like specifically where could you see things that needed improvement, so that we kind of get a sense of how we would ultimately get to what I'm going to get to in a moment, which is kind of where DocQuery has landed. But kind of tell us a little bit about how, what that pathway from I've identified the landscape to here's a much better way of doing it.

• 14:25Speaker 00

  Yeah, so we kind of set ourselves up with a few constraints early on. One of them was that we wanted to make the product completely self-service. And our definition of that was that a user can sign up on our website without talking to anyone onboard onto the product and then evaluate whether it works on their documents or not. The second thing is that we wanted to support documents of any schema. So if we hadn't seen that particular document type before, that's totally fine. We'd be able to learn about it on the fly. And third thing was that we wanted the product to be incredibly easy for a non-technical user to use and work with. And so what we did after performing a lot of user research is realized that most of

• 15:09Speaker 00

  our users are either beginner or advanced Excel users, meaning we could safely assume that our users were able to work with Excel at a basic level, like entering data and some basic formulas and stuff. And then we could also assume that some of our more advanced users are really, really powerful Excel users. And so in Empira, even from the very start, you've been able to kind of create these really complex expressions and formulas and stuff. And we realized the reason for all of this is that And if you sort of tie it back to what I was saying about pre-trained models not evolving when you notice something is wrong, we really wanted to create an experience where users could

• 15:52Speaker 00

  easily see whether the predictions were right or wrong. And then if the predictions are wrong, or if they feel compelled to give us feedback that they're right, they could correct or confirm things. And every time they do that, we drive the feedback into the model and incrementally train it. And so because of that design, we basically structured the machine learning approach to be one that is very, very lightweight and something that can train and evaluate really, really quickly. And so that's kind of the overall approach for how we tackled it.

• 16:24Chris Benson?

  So I have what seems to me like it may be an odd question, but as you were kind of talking your way through that, it's what came to mind. What are the things that you need to really be able to do with the document? You know, with doc query being called doc query, for instance, like, what does it mean to query a document? I mean, because that could be interpreted in so many ways. It started something as simple as people doing Ctrl F to do a find on a document.

• 16:50Speaker 00

  Oh, my God. I love this question.

• 16:52Chris Benson?

  Yeah. What are the things that that matter? Because it occurred to me, I don't know what those are.

• 16:56Speaker 00

  Yeah. So I'd say like, from a user's standpoint, there are a few different things that they're really interested in. And then we can talk a little bit about the sort of Empyrus technology and what part of that we hit and what part of it we missed until we introduced stock query. But users care about, one is integration. So there are really common workflow for a lot of different types of documents. And I'm sure you'll relate to this from your nonprofit business as well. But you receive documents through email. You have to interpret them to some extent. And that could mean, you know, reading the whole document, or just eyeballing something and figuring out where it should go next. And then you need to take that information and shove it somewhere.

• 17:38Speaker 00

  And what that looks like in a workflow like accounts payable, for example, is receiving an invoice through email, opening the invoice on your screen, and then manually keying in the information into your ERP system. And there's usually some judgment or interpretation that goes in as well. So it's not, these are, things are never totally literal. You might be making sure that, you know, the purchase order number that's on the invoice is actually one that's in your database. You might check that, you know, shipping plus subtotal plus tax equals the total and sending an email back to the vendor if it doesn't, or doing some other stuff as well. And so that's kind of like the basic workflow. The other thing that people really want to do is ask questions.

• 18:21Speaker 00

  So, you know, not just sort of run the formula of like, does this plus this plus this equal that, but say like, are these two numbers equal? Or of these 100 invoices, which ones are due next week? Or what was the most expensive line item on this invoice? And that, it kind of overlaps with search, although what we see is that people, they're looking for answers to, to questions that are fairly analytical in, in nature. And a lot of this is done, you know, very, very manually today.

• 18:53Chris Benson?

  It is. So it's kind of funny, and it's funny that you, that you referenced me doing the nonprofit thing because these are agonies. They're little things that I know for a fact, because just to bring in my own experience into the conversation, my wife and I are doing these, these administrative tack, you know, things that we have to do. We have a group of volunteers and all, but most of the admin falls to us. And they're tasks that neither one of us is, is particularly trained in, nor particularly are they things we love to do. And so as you were describing that, I was like, oh yeah, that was a pain. Oh yeah, that's painful. Yeah, that's painful as well. So it's interesting that you've identified all of these pain points. And I, I realize you're not specifically talking about nonprofits or small organizations,

• 19:38Chris Benson?

  but indeed they are things that definitely impact us as users.

• 19:42Speaker 00

  We do actually have quite a few nonprofit users and customers of Empira. So it's, um, we've heard this feedback very directly from them as well. Yeah.

• 19:51Chris Benson?

  So as you've kind of recognized all this, can you talk a little bit about what Empira has done and how DocQuery fits into that? And like, within the scope of, you've kind of laid out the problem and you've laid out kind of approach to solution, could you talk a little bit about how that is kind of realized in Empira in broad and specifically in DocQuery?

• 20:12Speaker 00

  Absolutely, yeah. So if you think about what I mentioned with Empira, there are a few things that really stand out. One is that users can work with any field that they want. They can create any schema that they want. And the second is that we really care about ease of use and simplicity. And so if you rewind back a few months, we were in kind of a state where you could create whatever field that you wanted, but you had to provide at least one label on the document, like you had to highlight and click something to teach the model. And even though you didn't have to do it for every single format that you uploaded, you had to do it for most of the formats that you uploaded, at least one label. So if you imagine with invoices, if you had like 100 different vendors, you might

• 20:57Speaker 00

  need to provide like 50 or 60 labels to teach the model about the breadth of vendors that you had. And so what we started thinking is, okay, how do we solve the problem of making it so you don't need to provide any labels in this case? And not only would that provide, you know, a much better user experience, but it also would mean that we'd be able to address the long tail of variety a lot better. And that means that if you upload something that we haven't seen before, or doesn't look like something that you've trained your model on, it still has a fighting chance at extracting the data correctly. And so we started open-endedly exploring, like pull our head kind of out of the sand

• 21:41Speaker 00

  of all of our Empira context, and open-endedly started exploring what else was out there. And actually, the first thing I did, I remember doing it on The car ride to the airport from New York back to San Francisco was copy paste manually the text out of a bunch of invoices. To your point earlier, like PDFs have all the structure, but I was just copying it out and basically ignoring all of the structure. And on Hugging Face's website, trying out a few different models that are pure text question answering models and pasting the text into the website and asking questions like, what is the invoice number and what is the total? And I was just blown away by

• 22:22Speaker 00

  how accurate it was. It wasn't not even like 60% accurate, right? But but still like with no context about this problem, nothing to do with invoices, no training data about invoices, no PDF structure or anything like that. It was like that accurate. And so that kind of blew my mind. I mean, if it was that accurate with something that was so distant from what we were doing, it meant a few things. One is, we could probably do better if we put in a little bit of effort. Two, you know, we had this epiphany that the framework of question answering allows a sort of infinite canvas of any fields or any questions that that you want, which is very in line with our product's philosophy. And then three,

• 23:05Speaker 00

  because something that has never looked at any documents like the ones I was pasting into the text box, because it was working so well with that, that probably meant that it would solve that generalization problem that that I mentioned earlier. And so that sort of experience, I still remember the car ride and I still remember, you know, working on my hotspot and stuff and furiously kind of playing with it. That sort of kicked off this whole idea.

• 23:31Chris Benson?

  That's very interesting. And I know this is fairly recent. You've actually hit a whole bunch of things that I want to I'm going to touch on with with a couple of follow up questions. First of all, this was a recent announcement. It was only on September 1st that you announced dot query. And you've in another thing that you mentioned just now was Hugging Face and stuff. So I'm curious about several things that, you know, I'll kind of throw throw several out to you. How has that model evolved that you've had as you've done this, you had started? In the visual, you talk about large language models in your Twitter. There's obviously an evolution of, you know, deep learning technologies that you're applying here. And as you did that, how did hugging face fit into that?

• 24:14Chris Benson?

  We have a habit of talking about hugging face quite a lot on the show. We're big fans. So how did all that come together?

• 24:22Speaker 00

  Yeah.

• 24:22Chris Benson?

  The evolution, hugging face, everything fit in with that?

• 24:26Speaker 00

  So we're also big fans of them and we've actually had the distinct pleasure of collaborating with them on this problem. So essentially what happened is they have this cool thing called a pipeline. And a pipeline for people like me who are not machine learning experts and barely understand what logos are, like any of this kind of stuff, abstracts away all of that complex machinery and makes it really easy to work with models. And so the pipeline that I was experimenting with is called the question answering pipeline. And it's all over their website and any model that kind of fits the question answering framework works with it.

• 25:06Speaker 00

  So after we saw this, Richard and I chatted and we were aware of some work out of Microsoft for a project called Layout LM, which is a language model that in addition to taking text as input, it also takes bounding boxes for each word of text. And so that kind of introduces the geometric information into the model that is actually super relevant to our problem. And just to give you an example, you might have the text invoice number and then the the actual invoice number might be to the right of of it. And if you turn that into plain text, then even, you know, a plain text model could pick up on that relationship. On the other hand, you might have, you know, the word invoice number and then the text beneath it. And you might have some other text to the right of the word invoice number.

• 25:51Speaker 00

  And without the bounding box information, it's actually really hard for a model to be able to pick up on that kind of relationship. And so Layout LM seemed like a really promising approach to solving that. But for some reason, when we dug around hugging face and, you know, scoured GitHub and, you know, Google at large to see if there was a question answering pipeline that worked with Layout LM, we just couldn't find anything. And it seemed to us like, wow, you know, if we're, if we had this awesome experience working with text-based question answering, and we know we're not the only people trying to work with documents, but there's nothing quite that easy out there, maybe we should

• 26:35Speaker 00

  kind of take the lead on this and make it just that easy to do document-based question answering as well. And so we reached out to the team at Hugging Face, you know, actually just by filing a GitHub issue, and they were incredibly receptive to the idea. And, you know, over a month of collaboration and working with them, we actually contributed the document question answering pipeline that's now in Hugging Face and a model that's pre-trained and MIT licensed and everything that you can play with and work with and even put into production that works with it and actually makes it that easy.

• 27:23Chris Benson?

  So that's really cool. What motivated you to, to make this an open source distribution? You know, like what, from the business as you have put together, you've identified the problem, you have a new approach that you want to take, you're taking advantage of really, really leading-edge technologies from Hugging Face in terms of their pipeline. What made you decide as an entrepreneur to release DocQuery as open source? What was the, what was the business motivation there?

• 27:52Speaker 00

  Yeah, so I think there may be three, three reasons for it. The first is not the business motivation, but just the personal motivation. When things are open source and they're easy to work with, it removes all barriers to innovation. And I think selfishly, as someone who cares a lot about innovation, but also as a member of the tech community at large, I think being able to contribute to people innovating and making it easier for them to Innovate and play with ideas. It's just very important to me. From a business standpoint, the second thing is you could think about it in terms of distribution. So in exchange for providing something that's generally useful to a large community of people,

• 28:33Speaker 00

  we have the opportunity to get some mind share and for them to familiarize themselves with us as a company, to experience technology that we create and, you know, form an opinion about how credible we are as product builders and so on, in a way that that doesn't require them to give us, you know, email or talk to a salesperson or anything like that. And so, you know, just purely from the standpoint of distribution, it's actually really valuable to us as a company to have the mind share and attention associated with it. And then the last thing is being confident about what our sort of proprietary strategy can be in the context of having open source stock query.

• 29:19Speaker 00

  And there are a couple of things that make me really confident that we can still be, you know, a really successful proprietary product. The most important one is that when you, as a customer, use our product, you have this really real-time data flywheel, which allows you to correct things, review things, integrate things, and the models will keep improving just for you to be able to do that. And time and time again, we've seen how important that is for people to put models into production in commercial settings. And we know that the ease of use UI, security integrations, workflow involved is something that is actually really hard to build and engineer yourself.

• 30:02Speaker 00

  And so we know that that's extremely valuable and we feel confident in that. And so if something, you know, for things outside of that, that kind of opens up the possibility of open sourcing them and still being able to derive a lot of value from this kind of core proprietary product.

• 30:19Chris Benson?

  You said something that struck me right there about kind of having that level of confidence and the fact that you already knew it was hard to kind of build those things out. That is something that stops a lot of would be entrepreneurs right in their tracks. You know, is that you've kind of dived into the deep end of the pool. You've said a couple of times in our conversation that, that, you know, you were not coming into this as a, as a world-class deep learning expert yourself, you know? You've built a team obviously around, but, but you were coming in as someone with, with an idea. What gave you the confidence or the bravery to kind of, to dive into the deep end of the

• 31:00Chris Benson?

  pool and do something that we normally associate with, with people who might have a different background, you know, have all that heavy math and years of deep learning modeling and stuff like that? How did you get past that? Because there are probably a thousand people listening right now that, you know, they want to be entrepreneurs, they've tried it, maybe they've tried and failed. How did you get past those hurdles?

• 31:24Speaker 00

  Yeah, so I'll give you the, the real answer and the inspiring answer.

• 31:29Chris Benson?

  Okay.

• 31:29Speaker 00

  The real answer is just stupid naivete. Like, I didn't even think about that. And I've learned and been humbled so many times by so many smart people over the past 10 years. And I'm still pretty stupid and still pretty naive, and I hope I am that way for some time. But that's the real answer. Now, the, the, I think more hopefully inspiring version of that is that as someone who is not deeply familiar with the math and deeply entrenched in the existing workflow for how things operate, it gives you a really unique perspective on what it would take to make

• 32:09Speaker 00

  something easy to use and simple enough that non-experts can take advantage of it. And I think a lot of what you're doing as an entrepreneur is bringing together two perspectives. The one perspective are the people who you can feel need something. And the other perspective is the perspective of the people who feel they can build that thing. And as someone who's not a machine learning person, it's very easy for me to go onto Hugging Face's website and Play with the question answering model and then try to read the documentation about the layout LM model, which had no examples and no nothing that easy to use and and see

• 32:52Speaker 00

  the difference, simply because I I was, I just didn't understand enough about the, you know, model complexity and and and so on to actually understand. And so I was able to see that difference. And I I think actually knowing more than I had at the at the time would have prevented me from from doing so. Now that I've actually learned a decent amount about this stuff, I don't have that same experience when I'm like reading through papers about models or documentation and I almost miss it.

• 33:21Chris Benson?

  I'm curious, as as you've done that, and by the way, you you're like, I I really think that what you said was quite wise in terms of kind of having that always willing to learn, you know, knowing that you're never there. I think so. You've had several really great insights, I think, in your process. And, you know, one of which was the benefit of doing it as open source, which which scares a lot of people off, obviously, in terms of as a business model. But, you know, one of the things that we know is that when you have a great product, you're solving a problem well, and you put it out there like that, it makes it very accessible, as you mentioned earlier. So, adoption tends to be much higher when you do that, because it's, you know, people can can dive in at whatever level

• 34:04Chris Benson?

  the, you know, they're comfortable with and give it a shot and figure out, you know, how to engage you going forward. As you do that, like, what what do you think is are the next steps for DocQuery and Pira at large there? And then, and then I'll ask you the kind of a broader question after that. But I'm kind of curious, very specific to DocQuery, where do you think it's going to go over the next year or so, from an adoption standpoint? And in terms of like, what's your short term vision for that?

• 34:31Speaker 00

  Yeah, so in the very near term, thanks to like, just a fantastic Flood of feedback of users both through GitHub and Discord, among other channels. We have a pretty good sense of the types of questions that people want to ask about a document that they can't currently ask with DocQuery. And the really beautiful thing about the question answering framework is that it actually encourages that creativity. People can really easily type whatever question they want and either get an answer or not get an answer. And so the two kinds of questions that people keep trying to ask that we're not able to answer about a single document with DocQuery are what kind of document something is.

• 35:16Speaker 00

  So like, for example, is this an invoice? Or is this a purchase order? Or is this an invoice from this vendor? And the other kind of thing they're trying to ask are questions about tables. And we actually support, and an example of a question about tables would be like, give me all the line items on this invoice. Or what are all of the descriptions? Or what is the first or second or third description? Or what is the highest total value? Or something like that. And these are things that we're, we actually are fortunate to have a good amount of data for, and in the very near term are basically expanding the question answering model to be able to support.

• 35:57Speaker 00

  We have looked at other model frameworks, for example, things like document classification as a framework, or like visual table detection and stuff. And we have a lot of experience trying these things out within the Empira product, but we feel pretty confident that we can basically expand the question answering framework to support them. And we just love the fact that it's an infinite canvas. The next step from there, which I'm extremely excited about, is allowing people to ask natural language questions over multiple documents or a pile of documents, if you will. And that could be things as simple as like, what are all the invoices? Or find me all the invoices in my Google Drive folder?

• 36:40Speaker 00

  Or things that are more complicated, like what are the invoices that are due next month? Or which invoices am I past due on? Or which invoice from this vendor is the one that's most relevant to this contract? Or something like that.

• 36:55Chris Benson?

  Is that farther out, though? Or is that something that would be, is that something you think is, like, are you, are you close to that? Or do you think it's going to take a little while to get to that point?

• 37:05Speaker 00

  Well, the model's training right now. There's a few moving parts that we're trying to figure out.

• 37:10Chris Benson?

  That was a great answer right there.

• 37:14Speaker 00

  I mean, like, literally training right now. I kicked off the most recent run right before the podcast. I'll give you the teaser for how that works, which is that, actually, we've studied this problem a lot through Impera's product. Because, long story short, people actually do this kind of stuff with Impera. You can extract fields and then you can write queries over the fields as well. And we actually have a pretty powerful query language that makes all of this possible. And what we've realized is that you can take natural language and basically compile it into a query, which consists of both relational algebra and other models or questions to ask of documents. And so we're cooking this framework and making it work.

• 37:59Speaker 00

  And we've seen some really exciting initial results, but I don't think it's going to be too long before that's possible. And then, as we think about it further, one of the things that we did, and I'd encourage anyone who's interested in the space to throw any idea you have at it, is we opened up a discussion on GitHub about, like, what are things that you'd like to be able to type that have to do with documents? And what's interesting is, a lot of the questions or things that people want to type are also actions. So things like organize all of these documents into folders by their document type, or forward along, you know, all of these things to this email address. And so I'm not exactly sure how we're going to tackle this in the open source part of the equation

• 38:44Speaker 00

  versus our product versus integrations with other products. Because even our product doesn't do all of these things. But I think purely from the machine learning standpoint, we're starting to think about what the right framework looks like, both on the machine learning side and on the application side, to make it possible to type things like that. And then, the last thing I'll say is that, as we push further into DocQuery, it's become increasingly clear to us that even though this question answering approach is incredibly relevant to working with documents, and it happens to work really well.

• 39:21Speaker 00

  This framework of having one or more things of data and asking questions about it is an incredibly powerful paradigm for people to work with data. And so our vision is increasingly becoming, making it really easy for anyone to ask anything of any data. And how we sequence those parts together, we're still learning. I suspect one of the really great benefits of open sourcing DocQuery is going to be engaging people in the community who have different flavors of this use case to apply it in different domains. We probably won't build models that analyze video, but you could use like 75% of DocQuery

• 40:06Speaker 00

  to manage getting the question, semantically representing it, turning it into relational algebra, yada, yada, yada. And someone really smart in the community could plug in the video aspect of it. And so that's kind of where I see the future of this. I think open source in particular is going to be a really powerful vector for us to engage a much larger audience than our limited engineering bandwidth has the capacity to support over the long term.

• 40:35Chris Benson?

  Well, Anker, that is very inspiring. It's kind of funny because on a day-to-day basis, many of us would think of just kind of document management as a fairly mundane thing, but it's such a huge impact on people's lives in a billion small ways in terms of making that better. It's definitely something that brings a lot of value to a lot of people around the globe. So, thank you so much for coming on the show. It was a fascinating conversation. Thank you for what you're doing. Thank you for taking the approach that you've taken. And I'm looking forward to finishing up as this little nonprofit manager. I'm excited to use that to make my life just a little bit better going forward. Thanks a lot.

• 41:20Speaker 00

  Awesome. And send us any feedback you have. We'd love it.

• 41:23Chris Benson?

  Absolutely.

• 41:32Speaker 01

  All right, that is our show for this week. If you dig it, don't forget to subscribe. Head to practicalai.fm for all the ways. And if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to Fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.