Arvind Jain on Building Glean and the Future of Enterprise AI

In this episode of Gradient Dissent, Lukas Biewald sits down with Arvind Jain, CEO and founder of Glean. They discuss Glean's evolution from solving enterprise search to building agentic AI tools that understand internal knowledge and workflows. Arvind shares how his early use of transformer models in 2019 laid the foundation for Glean’s success, well before the term "generative AI" was mainstream. They explore the technical and organizational challenges behind enterprise LLMs—including security, hallucination suppression—and when it makes sense to fine-tune models. Arvind also reflects on his previous startup Rubrik and explains how Glean’s AI platform aims to reshape how teams operate, from personalized agents to ever-fresh internal documentation. Follow Arvind Jain: https://x.com/jainarvind Follow Weights & Biases: https://x.com/weights_biases Timestamps: [00:01:00] What Glean is and how it works [00:02:39] Starting Glean before the LLM boom [00:04:10] Using transformers early in enterprise search [00:06:48] Semantic search vs. generative answers [00:08:13] When to fine-tune vs. use out-of-box models [00:12:38] The value of small, purpose-trained models [00:13:04] Enterprise security and embedding risks [00:16:31] Lessons from Rubrik and starting Glean [00:19:31] The contrarian bet on enterprise search [00:22:57] Culture and lessons learned from Google [00:25:13] Everyone will have their own AI-powered "team" [00:28:43] Using AI to keep documentation evergreen [00:31:22] AI-generated churn and risk analysis [00:33:55] Measuring model improvement with golden sets [00:36:05] Suppressing hallucinations with citations [00:39:22] Agents that can ping humans for help [00:40:41] AI as a force multiplier, not a replacement [00:42:26] The enduring value of hard work

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02Lukas Biewald?

  You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald. This is a conversation with Arvind Jain, who I've known for a while. Arvind is the CEO of Glean, which is one of the most successful enterprise companies using LLMs, maybe right now the most successful enterprise company that foundationally uses LLMs. And I really want you to not be fooled by Arvind's humble style. He's one of the most successful founders in Silicon Valley. Before, Glean, he founded Rubrik, which IPO'd for, I believe, a value of over $20 billion. So he's been phenomenally successful twice.

• 0:46Lukas Biewald?

  Very low key, but clearly very driven and has some real insights. I hope you enjoy this conversation.

• 0:54Lukas Biewald?

  Arvind, thanks for taking the time to talk. It's been a while. I think you have an incredibly successful company, so I feel maybe embarrassed to ask you this, but I think it's not a consumer brand, so people might not have heard of it. Can you explain what Glean does, please?

• 1:09Speaker 01

  Yeah, absolutely. Well, first, Lucas, thank you so much for inviting me to this. Glean is an enterprise company, enterprise AI company. What we do, I mean, think of us as chat GPT, but inside your company. Glean connects you with all of your internal systems, your data, your knowledge inside your company, and then gives you a chat GPT-like experience where people come, ask questions, and Glean will answer those questions for them. And it's going to use all of the world's knowledge, but also additionally, it's going to use all of your internal company's data and knowledge in context to help answer those questions for you. So that's like, simply put, like, you know, that's what we do.

• 1:50Speaker 01

  Of course, like, you know, as AI continues to advance, the same platform that we use to build our Glean assistant, you can also use that as an agent building platform to build all kinds of AI agents on our platform.

• 2:09Lukas Biewald?

  You know, it's kind of interesting about your company is it's been, you know, one of the real successes, I think, or at least from my perspective, one of the real successes in terms of real L M applications for enterprise that that works well. But you know, if you look at the timing of when you started it, it was before it was obvious that, um, you know, L M's were going to work so well, at least, at least to most people.

• 2:31Speaker 01

  Yeah.

• 2:32Lukas Biewald?

  How did, how did your vision for the company start and how did it kind of evolve through this period of of L M's taking off?

• 2:40Speaker 01

  Yeah, so Glean started in early 2019. You know, in those days, nobody was talking about large language models and the term generative AI did not exist to my knowledge at the time.

• 2:53Lukas Biewald?

  Totally.

• 2:55Speaker 01

  And we were not thinking about that either.

• 2:59Speaker 01

  And we had a problem at hand that we were, that we were interested in solving and which was search.

• 3:06Speaker 01

  We, you know, we felt like, you know, this is my experience in every single job that I had before Glean. It's so hard to find things inside the company. When you do your work, you need some information, you're looking for a document. It's extremely hard to actually find information. And, and, and it's because, you know, like, you know, as enterprises, you know, we have so much information, so much data, and it's spread across many, many different systems. I guess, you know, in, you know, in enterprises, you know, tend to have like, you know, hundreds of thousands of different systems.

• 3:40Speaker 01

  And stuff is all over the place. And so when, when I'm looking for something, like I have a quick question, I need an answer, or I'm looking for a dog that I know exists, but I still cannot get to it. That was sort of like that, that caused a lot of frustration. And, and in fact, like, you know, there've been a lot of research studies that have been done. People would spend one third of their working time just trying to find things. And so that was, that was the first problem that we wanted to solve. We wanted to build a Google for people in their work lives. One place where you go, ask your questions and we want to bring the right information back to you. And sometimes, you know, it may not even be the information. It may be, he's just looking to connect with the right people who can help you on a given topic. So like we become that one place, we'll come and find data, find documents, find people.

• 4:25Speaker 01

  All in one place, you don't have to think anymore where to go and look for things. So that was, that was the first part that we wanted to build. The interesting thing is that in early 2019, transformers were already a thing. Like it was not like the rest of the, the world didn't really care as much. But inside search teams at Google, we were seeing a really good impact of the transformer technology on some of the core components of a, of a search engine. And, and we saw this real promise, you know, in that day, you know, in early 2019, you know, where we could conceptually understand information. Like, you know, you didn't have to actually go and build a keyword based search system anymore, which is sort of like, you know, like quite dumb.

• 5:07Speaker 01

  Like if you think about like how, you know, we match information to people's questions. Like it's happening at a very surface level with keywords. But Google had actually put BERT based models in open domain in late 2018. And we were able to use them and then actually go and pre-train these, what we call now small language models on your enterprise corpus and start to sort of deeply and fundamentally understand what your business is about and what, you know, any given document is about. And, and then do this sort of conceptual semantic matching of information to people's questions. So, so, so interestingly transformers actually played a foundational role in Glean's evolution.

• 5:49Speaker 01

  Like, you know, we probably the first company to use transformers in enterprise, you know, in, in that day. Uh, cause it was like really, really applicable to building a good search experience in the day. So that's, so that's, that's how we got started. And then of course, over the, as, as these models, like in, in, in 2019, the transformers had no capability to generate information. They had. No, like none of the reasoning capabilities that models have today. But they had this, you know, embedding based conceptual sort of mashing, you know, capability, you know, which which we were able to use. But as these new capabilities came that allowed us to evolve our product. And what used to be a Google for you in your work life.

• 6:30Speaker 01

  Became more like chat G P T for work, like, you know, it can go one step further and now we now we don't have to just surface information to you. We could actually read that information using A I and actually give you precise answers to your questions. So sort of that the evolution has been like that over the la- over the last six plus years.

• 6:48Lukas Biewald?

  So you're you're politely telling me, I think that you actually did see the power of transformers, at least in in twenty nineteen when you started the company, which is which is super cool.

• 6:57Speaker 01

  Well, I mean, I think, yes, yes, I mean, because because we saw it, actually, like, it was not, it was not a vision, it was, you know, we actually. It's not, we saw like how, you know, it could really deeply, I can, we could take two documents. Documents that had no real word overlap, like, you know, like, you look at the actual choice of words using those documents very different. But still, you know, a a bird based model, you know, trained on your enterprise data. Could actually tell you that, like, hey, these two, these two documents are very close to each other semantically. And so that, so we, that was, that was, you know, in in those, remember in twenty nineteen, when we, you know, today, we expect a lot from A I. But A I was not there in twenty nineteen. And that itself was big, that was still a big step function in terms of like our ability to, like, you know, really.

• 7:43Speaker 01

  Conceptually match people's questions with with knowledge. We saw that power. And and then we decided to sort of bring it to enterprises.

• 7:52Lukas Biewald?

  And so I mean, you're in this world where like, you know, these labs are investing massive resources, constantly coming out with with new models, like. Do you still, you know, fine tune a custom model for every customer or is that important to you? Or like, what, how do you think about like what parts to kind of, you know, buy or use or parts to build a house?

• 8:12Speaker 01

  Yeah. Well, I mean, the company like our like the thing that I actually keep carrying our technology team is that do not reinvent things that have been invented already. I mean, this is we will not like that's not that's not doing service to us or to our customers. And so our approach always is to actually maximally use innovation that's happening outside because it's available to us. I mean, the good thing is that all of the innovation that's happening in in L L M space, all of these models, companies like Google and OpenAI and Anthropic, they're all making those models available to us to use. So we actually like to use those models and we do we use them quite a bit in our stack.

• 8:56Speaker 01

  So the way like the way our technology works is that we we we actually pre-train and you know, we pre-train and build some models like, you know, for for semantic matching. That that we have to do ourselves because these models are all about your enterprise and your data and sort of building an understanding of what your what your business is about. And so there we actually will go and connect with your enterprise data and data and knowledge. We'll take all the documents, we actually build, train the model on it so that it can start to understand how your business talks and speaks. How how like what you know, what code verse you know, you use versus your lingo. So for that, it's actually important to actually go and train a model on your enterprise corpus.

• 9:39Speaker 01

  But these are very small models, like, you know, we don't actually train large models. And and these models allow us to actually build custom embeddings, you know, for your enterprise content. And so that when people come and ask for things, we can match, you know, their requests with your, you know, documents in your corpus in a in a much better way, using semantic matching. So so those models we build ourselves. But then when it comes times to.

• 10:06Speaker 01

  To actually do something, you know, with the information. So imagine the flow of like when you come and ask a question in Glean. So the first step of it is, well, you know, we see your question. We're trying to understand what it is about. And we're going to use our semantic embedding models, the one that we train and, you know, ourselves, to assemble the right pieces of information to get the best knowledge within your company that we think can answer those questions. And now we're going to actually take, we're going to take this, you know, information and, and ask AI to actually synthesize an answer from it. And then for that, for that, we don't actually use our own models, you know, use out of the box models like GPT or.

• 10:49Speaker 01

  And similarly for reasoning, like when you actually take a complex question and try to break it down into, into like a multi-step agent to go workflow again, like, you know, we use the reasoning capabilities of the largest information models. So that's one of the architectures. So yeah, this is more the don't, don't train, don't fine tune when you don't need to. But, but there are certain things, you know, in the search stack where it makes sense to do that. Maybe I'll give you one or two more examples. Please, yeah. Well, what one, so in, in search, of course, you know, part of it is like, you know, you, you get a question from a user and you're trying to semantically match it with the right information. So that's what I described right now. But then there are like, you know, smaller parts of the stack.

• 11:33Speaker 01

  For example, where to expand, like, you know, to expand, like, you know, in retrieval systems, you have to spell check users' questions. You'd actually find synonyms, you'd refine acronyms and, and for those models, what we, what we realize is that you can take, you know, these really, you know, sort of small, open, open domain language models and you can fine tune them for those specific tasks. So for example, for spell checking, we, we have a fine tuned version of a language model that will do it for us. So, so for like, you know, for some point tasks, like small, very, very specific things, you know, the, these fine tuned models actually help us do those things faster and cheaper. They don't, they don't add more.

• 12:16Speaker 01

  More capabilities like you could actually take G P T like a large G P T model to do like to actually do spy check for you like it'll do a fantastic job. It's just that it's a work could actually use use that model for that kind of a task. So so there's a little bit of that finding that's happening in the search stack for us, but but otherwise we are for reasoning for generation, we use out of box models.

• 12:38Lukas Biewald?

  I think another one of your kind of advantages or parts of your technology that that you talk about a lot is is like the the security model that you have. Right? So it's like, I mean, in enterprise search, there's such complicated rules around like who can access what data and as these models get more complicated like, you know, like a building in embedding model, you could imagine it somehow like leaks information across embeddings. Like, can you talk a little bit about how how that works, how you think about it?

• 13:04Speaker 01

  Yeah. Yeah. So the way Glean works inside your enterprise is that as we connect with all of these internal systems like Google Drive or Slack or Salesforce. We are reading, we are understanding content that lives in those systems, but we're also understanding the governance of that data, like who are the people who can access that information? For any given document, for example, we'll look at the list of people who are authorized to see it. And this is this is fundamental because if you are going to, you know, deliver A I in the enterprise in any way, you have to do it in a safe and secure way. You can't you can't have like A I based systems start to leak information internally to your employees who didn't have rights to see those information.

• 13:47Speaker 01

  So so so the way our system works is that we connect with these individual applications, we understand their permission model. And then we actually, you know, bake that into our into our into our core index indexing technology. Is the permissions are baked in right into our, you know, our index indexing system. So now when you come in, you know, ask questions in Glean. We will, uh, we will retrieve information for you, uh, which can answer that question. Uh, but we'll actually only retrieve information that you have rights to use. Uh, we know who you are, you're signed in and we'll use our enterprise identity to establish that these are the documents that you are, have access to. And these are relevant to this topic. And then we make actually models work on it.

• 14:31Speaker 01

  So that's, that's the, so that's the fundamental, like, you know, in a RAG style system, you can actually, you know, make AI safely work on data that only you have, you know, that you have permissions for. But then you also mentioned this interesting question around embedding, like if you're actually building custom embeddings, what content do you actually train it on? Because if you train it on all of your enterprise corpus, it will indeed like start to leak information. And you won't realize, you know, in what form or shape it actually does it because ultimately you can't understand behind the scenes, you know, how AI works. So, so what we do is, you know, when, when we're actually training models, custom models, we typically would train them on, um, content that we think is safe to train on.

• 15:14Speaker 01

  So there's a lot of, you know, uh, curious psych going behind the scenes to, to understand like what, what information is actually safe to train on. And, and so, so yeah, so if you, if you sell, if it's, if you use it only on a subset that is safe, then your embedding models are actually not leaking information.

• 15:33Lukas Biewald?

  I see.

• 15:36Lukas Biewald?

  I guess I'm speaking of security, I think a really interesting fact about you, at least from my perspective, is you're also the founder of Rubrik, right? Which is like, maybe, maybe not everyone has heard of it, but an incredibly successful company that, you know, recently IPO'd. So I feel like you're like, you know, kind of twice like unicorn verging on like Decacorn, maybe, uh, founder. And it's so amazing, like you're so like kind of humble and understated. I just wonder like if you have any insights into kind of, you know, what you're doing differently than other people? Like what, what do you kind of attribute these, um, these two successes?

• 16:14Lukas Biewald?

  Cause I think like, you know, I would think like, you know, 10 billion plus exit is gotta be like one in a thousand, one in, you know, 10,000, where you start. And so the fact that these two are correlated, I think is a powerful indicator that there's more than luck involved, at least in, in your case.

• 16:31Speaker 01

  Well, I mean, I think it is correlation and I think no more than that. But I think what I would say like, you know, from our experiences. We started Rubrik, that was early 2014. And we picked a problem which we knew every business faced. It was really hard to protect your data, especially like, you know, as, you know, the attacks are on the rise. You know, with ransomware and other things. And, and we felt that, you know, in, at that time, like 2013, when we were contemplating, you know, building, you know, starting Rubrik. We felt that all the energy and attention, all the best tech people were all focused on consumer technology. And enterprise was sort of lagging, you know, at the time.

• 17:17Speaker 01

  We didn't have a lot of great companies, you know, like the SaaS revolution was still fairly early at the time. And we felt businesses were being ignored. And the, just building, just helping them with this very fundamental problem that like, you know, like here will help you, help keep your data safe.

• 17:37Speaker 01

  Was something where there were no products that were being built, that were less than like, you know, two decades old. So, so we, we, we saw an important, large problem, you know, with large market. And, and then after that, it's about just go and, good job, you know, building a product, good job, building a business. And, and, and I think we were fortunate that, like, you know, we were able to sort of go build a good team, you know, that then of course, you know, generated all the success. I would say that, you know, with Glean also, similarly, we picked a problem which is not a niche problem. Like, you can go and talk to every single person, anybody that you know, like, you know, bet, like, you know, ask them that, hey,

• 18:21Speaker 01

  it's easy for you to find things inside, you know, inside your company, and they'll all say no. And it was a pretty obvious problem, again, nobody was working on it at the time. And we felt that we had to actually be the pioneers and, and, and solve this. I got a lot of advice, by the way, at the time, not to actually get into this business.

• 18:41Lukas Biewald?

  Well, yeah, because, okay, sorry, I knew you were going to give me something like really humble like this. But I think that what these two have in common in my mind, your two companies, is they're kind of like. Like ideas that like a lot of people have that like don't work. Like, I feel like I've I've sort of heard those pitches for a year or two. You know, companies quite a lot in my life, but then you were like wildly successful doing both of these these different things where I think it must be really like an execution oriented thing. Like, it's not like you kind of came in, it seems like with some, you know, kind of wildly new product that nobody was expecting. It's like, it seems like you did like a phenomenally good job in a

• 19:20Lukas Biewald?

  in a kind of market that didn't seem so great, or it sort of seemed like broadly crowded and specifically hard to execute. And I'm not sure, but I just, I want to draw some insight out of you.

• 19:30Speaker 01

  Yeah, well, I mean, I think like, yeah, that's, that's like, maybe I'll add one thing. Look, like, think about search. A lot of people actually told me not to work on search and enterprise search. Because there were only failures.

• 19:45Lukas Biewald?

  Yeah, exactly.

• 19:46Speaker 01

  In the two or three decades, two or three decades of enterprise searches, including like products from Google. Like, you know, every time, you know, Google tried to actually build an enterprise search product, it was always, you know, always failed. So, so when, when you see like domain, like it's sort of like, you know, almost people started to feel like, you know, this, this problem is like maybe too hard to solve, or maybe it's not even worth solving. Because, you know, if nobody builds a big business, maybe, maybe there's no need to actually build a product like that. So that, so that was a mindset and, and that's why there was no innovation in search, you know, in, you know, inside the enterprise. For a long time, there were no startups, like, you know, it started in, we started in 2019. There, there was no search company that you would know of that got started like, you know, like, you know, 10 years, you know, in that 10 year period.

• 20:31Speaker 01

  Like, this was a dead, dead, dead, you know, probably the, like, one of the worst areas, you know, where no investors want to invest. But this, this is interesting, like, you know, you, you see a problem that everyone has, like, you know, everybody faces this pain point. Yet there are no good products. And then there are also technology trends, like, we saw some new things that happened at that time. One of them was SaaS, the whole SaaS transformation, which made the problem so much more worse. But also, it actually made the problem tractable in the sense that, you know, SaaS systems actually allow you to actually go and read data inside those systems. Before, we couldn't even do that, it was just so hard to get hold of data in the enterprise. We also saw

• 21:15Speaker 01

  Transformers, like, you know, as a really core new capability that would help you deeply understand content and actually build a better search experience. So, so it was like, you know, we did observe those, we observed those trends, uh, which made us, you know, made us actually confident as to take that contrarian bet and say that like, look, you know, no, we think, you know, we can really now go and build a good product and, and build a huge business. So, so, so I, I think part of it is that, like, you know, I don't know, like, you know, I'm, I'm an engineer, by the way, like, you know, by, you know, in my buildup and how I'm, you know, who I am as a person and doubts are part of my existence, like, you know, like, that's how I think like most engineers are. So, so I was surprised at myself, like, you know, when

• 21:57Speaker 01

  everybody else told me not to do this, I don't know what actually, uh, was it in, which actually made me have that conviction. So, surprised myself at that. But, you know, like, and that, that, maybe that, that's, that's, that's one thing to think about is like, large companies, if you're going to build them, like, you know, of course, you have to tackle, like, you know, problems, you know, which are very universal in nature, like, you know, broad, broad, like, you know, which have broad impact.

• 22:23Lukas Biewald?

  Do you think that you, like, operate differently? Like, I mean, another thing that's interesting, right, is your product is to kind of help other companies operate better, you know, more efficiently. And I assume you're, you know, kind of dog fooding your own, your own products. Like, do you think that you, like, if I, if I was like a new hire and I was asking you, like, should I join or something like that, would you tell me, hey, um, you know, get used to, like, you know, kind of a special kind of culture that we have here at Queen?

• 22:51Speaker 01

  Um, that's a, that's, that's, that's a good question. The, um, I think, I don't know if we have a particularly unique culture, like, a lot of our learnings and my learnings have come from Google, you know, which is where, um, I spent a lot of my, um, you know, work, life, you know, at. And, and Google was a really special company, like, it was very different from, and, and I'd actually worked at three companies before Google, um, um, at that time. And

• 23:24Speaker 01

  It was so unique, you know, in the, in the sense that it actually put technology and innovation in front of everything else. As an engineer at Google, you were the king. Nobody gets to tell you anything like you go and, you know, build, build like, you know, whatever technology you want to build, like, you know, it doesn't matter. Like, you know, whether you get to decide whether it's important that you want to build that system. And you had all the resources to actually then go and build those things. So, um, and so, so there was that Google model, which is that you hire really, really smart people. You know, really good engineers who are, who are very motivated to do something special. And then just let them be and like, you know, they'll figure out the right things to figure out.

• 24:08Speaker 01

  They'll figure out how to organize, you know, with other, with other team members and, and do a great job. And frankly, like, you know, for a man, for a manager, that also is a pretty, pretty good situation to be, you know, you don't have to do any work.

• 24:22Speaker 01

  And, and so like, you know, I, I sort of like, you know, I love that model and we always followed that model like now at team. You know, where we, I have a really, you know, amazing team. And, and like, you know, like my role is to generally be not in the way, like, you know, I can't help myself. I do come in the way sometimes, but, but, but often like it's mostly like, you know, it's about that, like, you know, building innovation first culture. And, uh, and success will come.

• 24:51Lukas Biewald?

  Do you think that having like a much better, um, kind of search and even now you're building these like agent, agentic systems. It sounds like that could be even custom to, to what a company is trying to do. Do you, do you think that is causing or will cause companies to function dramatically differently in the future?

• 25:13Speaker 01

  Well, I mean, the way we work is fundamentally going to change both, both at a individual level and also at a overall business level. In fact, you know, the, um, the vision for the future, like I believe this is, you know, this is how, you know, our work lives are going to look like in the future. So for any person, um,

• 25:37Speaker 01

  Regardless of like, you know, how senior they are, um, you will have an amazing team of assistants, coworkers and coaches that are going to basically help you not only do ninety percent of the work that you need to do, um, but also help you get better at things that you do. Uh, and like, you know, if you think about today, you know, there are certain people in the company that have that luxury. Like, you know, if you're CEO of a company, you've got to have your coach, you've got to have your assistants, you have a chief of staff, you have an executive team. And you have all this, like, you know, really amazing sort of people around you that that you're surrounded with, which actually helps you become, you know, that true multiplier, you know, for the company.

• 26:21Speaker 01

  And and and AI is going to change that. Like, today, only the CEO or the senior executive can actually have that luxury and that help. Uh, and AI sort of democratizes that and brings that help to every single person. So everybody, everybody's gonna have this dream team around them. And you get to, in fact, you know, go and build that dream team, uh, with, you know, AI, you know, powered like, you know, agents. You can have some, you know, you can have an agent that works like your assistant, another one that acts like your coach, yet another one that actually acts like a coworker and shares a workload with you. Um, and and that's, that's the world that we're going to be in. Um, so the, so you're no longer alone, you have, you have this team, like, you know, continuously like around you.

• 27:04Speaker 01

  Um, and similarly for like the companies, um, most of our business processes that we have today, you could envision that parts of those processes, like, uh, that can actually be easily be handled, like, you know, through some kind of an AI agent. Um, so there's a big transformation ahead of us. Um, it's gonna reshape, uh, how individuals work. It's gonna reshape how a company's, you know, how an organization looks. Um, there may be, uh, people's roles are gonna change. I think about software engineers, you know, you hear that in the future, we're gonna become more, more code reviewers as opposed to like, you know, people who write code.

• 27:47Speaker 01

  So, so, so there's, you know, that, that kind of like, you know, change in like where we spend our time is gonna happen to all of us.

• 27:54Lukas Biewald?

  It's funny, I think about like what's in our A ways and Biases, like in our Notion and Google Drive and all these things. And we have a lot of, for example, like

• 28:08Lukas Biewald?

  OKRs and kind of like our quarterly plans. I keep being tempted to paste them into Gemini and the context window is like big enough. And put in my like current OKRs, I got through the quarter and just be like, which ones do you think we're gonna miss? Or what do you think about this strategy? Do you think it's a good idea? Like if you're kind of look back at what we've done. Do you ever do stuff like that, like not like tactical coding stuff, but sort of like start to ask about broader strategic questions?

• 28:41Speaker 01

  Yeah, actually, the

• 28:46Speaker 01

  so like, you know, today with Glean,

• 28:50Speaker 01

  these are the kind of things, you know, that it does a really good job at. Because, you know, Glean is actually connected to all those systems. So you don't actually even have to cut and paste, you know, that OKRs from that Notion or Google Drive doc. Because, you know, it already is in that, you know, that overall enterprise wide search index that Glean has. So some of the things that, you know, that are my favorite activities, you know, with AI now are, number one, the when I want to learn about technology or what like a given team is working on inside our company, or how do we actually design a particular system? So now, instead of like, you know, like setting meetings and asking them to give me an update and walk me through like some kind of a presentation,

• 29:35Speaker 01

  I actually ask Glean to actually, you know, generate a latest, you know, up to date tutorial on that topic for me. And that's all I say. Like, and I have this agent, you know, where I've actually, you know, where we've told it to actually say that, like, you know, when I'm looking for a tutorial on a given topic, like make sure to actually go and look at all the design docs that have been written. But then, you know, design docs often get obsolete. And so like, you know, read the design docs, see if it was updated, but then after that. That actually start to look at more like, you know, Jiras that got resolved, you know, code that was committed into that, into that component. Um. And sort of like, you know, get the latest view of like, you know, how things work. So like, don't give me the stale dog, you know, give me the latest on this, on this system and how it works.

• 30:20Speaker 01

  This is actually pretty cool, like, you know, because now like there's this concept of evergreen documentation. Like I, I, I don't have to deal with, you know, stale stuff anymore because you know, AI is actually doing that. Human like thing where it's reading stale information and new information and it's actually deciding what's the right and upgrade information because it's a fact. Um. Similarly, um, like an example, similar to what, what you mentioned. So we, so we're a new business and. Um. New businesses, you know, like, you know, if you build a good product, you don't have a churn problem. And as we, so we actually didn't have any dashboards in our, in our data. I, you know, like in, in our, in our sort of, you know, data dashboards around, um, around chair.

• 31:04Speaker 01

  I was just simply a thing that was not, nobody actually built any analysis on it because it's not relevant.

• 31:10Lukas Biewald?

  Um. I love how humble you are. That's, that's, that's amazing. I mean, you're, you're old enough that you must have a very sticky product. If, if six years in or something, you're not worried about.

• 31:19Speaker 01

  Yeah, the like at some, but something also, but well, but, but, but like, guess what? Like, you know, we had some churn that happened last quarter. And then like, you know, suddenly got worried like, okay, what's happening? Like, you know, like what's going to happen to all the other customers? And there, there is the customer health and like I'm exaggerating a little bit. I mean, like, you know, some teams have it, you know, you know, like, but we didn't have like really good versions of it. So the, we wanted to do an analysis and so, um, one of our team, one of our team members in finance, they're not, you know, she's not an engineer. Hasn't built anything like, you know, from a software system perspective, but she went and actually asked this question and, and told like, you know, in green.

• 31:58Speaker 01

  And said that like, go and look at everything with customer of ours. So you can actually ask his for like, you know, when you can look and see his first to get all the list of all the customers. Um. And for each one of them, go and look at like, you know, our shared Slack channels with them, go look at what the conversations. Like, you know, sentiment there, go look into our product dashboards and look at the usage. And then, um, and then like, you know, take, take all of that data and based on that, like, you know, now give me a risk profile, like, you know, whether it's, you know, like a account that is healthy, like green, you know, or red or, or yellow. And, and, and, and actually sure enough, like, you know, AI is actually pretty good at this and like, you know, you know, generated a report, which basically tells us like, you know, all the top risk accounts and, and the ones that are not.

• 32:43Speaker 01

  And actually frankly, I would say like it was actually better than like any dashboard that we would have built because it was able to actually make use of, you know, subjective textual information, like, you know, which, you know, obviously dashboards can't, like dashboards can only look at, you know, usage, like, you know, usage trends and some like, you know, structured information. So, so, so these are like, you know, certainly like, you know, new capabilities, like, you know, where you can tap into this, like 95% of your enterprise data is unstructured. Uh, and this, you know, is full of insights, like only if somebody had time to actually, you know, extract them and like, and that's where the AI sort of comes in super hand.

• 33:19Lukas Biewald?

  Totally. Well, that's such a compelling, um, story. I mean, I, I wa- I want to do that as soon as we get clean installed here.

• 33:32Lukas Biewald?

  Wow. How do you, um, you know, one thing that I think about with you guys is, you know, your, your customers probably don't let you, you know, pull metrics on their data.

• 33:44Speaker 01

  Yeah.

• 33:44Lukas Biewald?

  Or maybe they do, but how, how do you know if you are like shipping a new model that's, that's better? Or if you're embedding is better, like, how do you actually measure your, your progress?

• 33:55Speaker 01

  Yeah. So, so first, like, you know, we have like customer zero, which is clean, uh, our own like instance. And as we build, um, as we sort of, you know, bring new models and like, you know, we have a valuation framework, um, where, uh, there is, like, what we do is, um,

• 34:18Speaker 01

  For for in any, you know, at any customer deployment, actually,

• 34:25Speaker 01

  there is plenty of information available where there is like you can have question answer pairs. So people have asked questions and you know, somebody's answered them. You don't even have to, you don't even have to actually like, you know, create these golden data sets. Sometimes you can just obtain them from actual like real conversations that people have. Like for example, like in Slack, somebody's asking questions or people are spawning with answers. And there's a lot of reaction and thumbs up to that, that information. And that, that's like a really good example of a golden sort of set of like, you know, what a question is and how I should be answering it. And so, so we build these golden sets, you know, like for each one of our customers.

• 35:07Speaker 01

  And then it'll bring a new model and we will actually run, run like, you know, all those questions through the new model. Or the new, like it's not, doesn't have to be a new model, like and there's also any change that we make in our own systems, right? Like it's, it's like our system, you know, there's a lot of, you know, things happen in there, including changes in the search and retrieval system. And then also the more, the models that we use. So, so you sort of, you measure, um, you know, like how the answers are now looking, you know, with, um, you know, on the golden set, you know, with this new change that you're rolling out. And, and then you also use AI to sort of, uh, cause you know, like these are subjective answers.

• 35:48Speaker 01

  Then you have to actually also use L L M's as judge to sort of see like, you know, how close you are to what the golden response is. So that's the fundamental way for us to measure, like, you know, our, like how well our end to end system is working.

• 36:05Lukas Biewald?

  Do you have, um, issues with hallucinations? Like I feel like in almost every domain that kind of drives us

• 36:11Speaker 01

  Yeah.

• 36:12Lukas Biewald?

  crazy. Like do you, do you take like special measures about that? I would think that from an enterprise perspective, they might feel more sensitive to hallucinations than even a consumer.

• 36:21Speaker 01

  Yeah, I mean hundred percent. I mean, I think AI has to, I mean, and hallucinations are there. Um, the, we, we have some safeguard against it. Um, but, but that, that's not to say that Glee is going to actually always like answer questions exactly the right way. Like, you know, I think the, it's not, you know, it's not, it's not there because there's so many things that can go wrong in terms of like when somebody comes and asks you a question, like, can you answer it properly? Sometimes the information is not even there to answer. Sometimes the information is stale or out of date. And that you're using to answer, you know, questions for people. So the, the failure scenario are so many. Um, the, and one of them is actually the hallucination of the model itself.

• 37:03Speaker 01

  Um, but for that, for that, for the hallucination part of the model, like what we do is remember that in our system, when you come and ask questions, we are the ones who first assemble the raw materials. Like, you know, the knowledge that we're going to use to answer that question. And we give all of that knowledge at the prompt time to the model. Along with the question and the model comes back with an answer. What we do after that is that line by line, you know, we'll take every, um, you know, the response of the model. And we'll try to see like, you know, if we can actually find, you know, that same piece of information and, you know, in the input that we gave to the model. And then we'll sort of like, you know, in fact, you know, we'll use that for our core citations.

• 37:45Speaker 01

  Uh, framework and we'll cite on line by line basis, where is that human generated information that was used to produce, like that AI used to produce this, you know, this particular line. And when we don't see it, we either suppress it or we actually would don't, so don't show the citations. But so that way you can actually like tame the model a little bit. Like, you know, if you're not, if you see, you know, information that's being produced, which has, which is not present in the input that we gave. Um, then we can sort of give up and, you know, tell, tell the user that look, you know, like, you asked a question, here's some relevant piece of information, but we actually were not able to figure out an answer. So, so, so that's how we suppress hallucinations a little bit through this sort of citation and reference checking.

• 38:28Speaker 01

  Um, but the, but the bigger problem I would say in enterprise is not that. The bigger problem in enterprise when people ask questions is can you even, um,

• 38:38Speaker 01

  You know, get that right information from all these different systems that you have. And can you get the information that is actually up to date and actually correct, you know, written by a subject matter expert? Most of the losses actually happen on that side.

• 38:53Lukas Biewald?

  I see. Do you do you imagine a world where um these agents start just asking? People if they can't figure something out, like just like kind of routing to, I mean, I think I would kind of like that if I could just deploy a bunch of. Agents to collect information for my employees.

• 39:10Speaker 01

  Yeah, well, that's an interesting use case. So so in in Glean, so we so one thing that we do is. We are we are like.

• 39:22Speaker 01

  Like, we don't think of our product as hey, take all the information in the company and standard and then answer questions for people. It's very much a people oriented product. So like when we go and uh. Like deploy Glean inside a company, we actually build a deep understanding of the enterprise itself, like what the business is about. Also, who are the different people? What do they work on? And and in fact, it's a common scenario, like, you know, when somebody comes and asks questions and we can't answer. Sometimes our answer will be that like, look, you know, we don't like have the information, but these are the people who are working on this topic. And if we want to connect with them. But then like agents, like, you know, sort of you you flip the the question like, you know, whereas like instead of humans asking agents to do work, like you're saying, agents are actually pinging.

• 40:07Speaker 01

  You know, people to retrieve some information from them. Like, you're like, that also seems quite legitimate because the. That's probably the that's probably the future world, like, you know, where there is, there are these deeply intelligent. Agents, you know, which are sort of like a they work alongside humans. So, yeah, like, you know, humans can ask agents questions and agents can also ask humans questions.

• 40:31Lukas Biewald?

  Do you have like other thoughts on where um like organizations go as they interact with AI more deeply?

• 40:41Speaker 01

  Well, I think I would just add one thing. There's a lot of

• 40:48Speaker 01

  this ROI conversation that tends to happen with AI. That the, like, you know, I'm gonna have AI come and like, you know, do like a lot of the work that we do with humans today. And we'll be able to replace them. Maybe I'll have a small team. Maybe, you know, we'll have a single person company, you know, that is a billion dollars in business.

• 41:13Speaker 01

  And, and there are all these sort of

• 41:20Speaker 01

  sort of viewpoints around like how AI replaces humans. And, and I feel like, you know, that's, that's missing the point in a big way. Because like ultimately, like a business and its value comes from its people. I mean, that's, you know, that's, that's, that's where the intellected, that's where like, you know, the capacity to do work is. And it's, it's so much better to think of AI as an enabler to all of those people. So like, you know, as a business, like, you know, I, for example, I don't actually shrink my team size.

• 41:52Speaker 01

  I sure to grow it. But I want to be able to do 10 times more work so that we can, like, you know, we can succeed in a much bigger way. And so that's, that's, that's, so I didn't like, you know, when you think about enterprises and how they change with AI, you know, people are going to have like one of the two mindsets. Like either they're sort of, they're feeling the pressure, they want to reduce the team size significantly, or you can actually go and keep investing in people and just like, you know, improve your top line significantly.

• 42:20Lukas Biewald?

  Do you think that the skills that make people successful will change in this world?

• 42:26Speaker 01

  I think the fundamental skills remain the same. Like, I think what makes one successful, in my opinion, is, you know, hard work, dedication and desire to succeed. And I think that doesn't fundamentally change. But of course, like all of us have to adapt. Like, you know, one of the things, like, you know, for, I think about software engineers, like, you know, the, maybe it's no longer necessary for you to sort of remember, like, you know, what all The different APIs and frameworks are available because you know like AI can always sort of bring the right information back to you. So you can spend more time thinking, um, more time designing, less time writing code maybe. So like, you know, there'll be, but I, I feel like, you know, those are sort of like incremental, these changes will happen on an incremental basis.

• 43:08Speaker 01

  And we won't realize, you know, by the time we have transformed ourselves, like it's sort of, that's my, that's my feeling. It will feel like a step function change, like, you know, like every day there's one new task, you know, that we will start using AI for. And, and, and before you know it, like, you know, like you're fundamentally different from what you were two years back.

• 43:26Lukas Biewald?

  All right. Well, that seems like a nice place to end. Thanks so much for your time, Arav.

• 43:30Speaker 01

  Yeah, thank you, Luva. It's gonna be fun to have this conversation.

• 43:32Lukas Biewald?

  Yeah, appreciate it.

• 43:34Speaker 01

  Thanks.

• 43:34Lukas Biewald?

  Yeah. It's okay. Thanks so much for listening to this episode of Grading Descent. Please stay tuned for.