#254 Prashanth: Why Developers Still Trust Stack Overflow in the Age of AI

This episode is sponsored by Oracle. OCI is the next-generation cloud designed for every workload – where you can run any application, including any AI projects, faster and more securely for less. On average, OCI costs 50% less for compute, 70% less for storage, and 80% less for networking. Join Modal, Skydance Animation, and today's innovative AI tech companies who upgraded to OCI…and saved. Offer only for new US customers with a minimum financial commitment. See if you qualify for half off at http://oracle.com/eyeonai In this episode of Eye on AI, host Craig Smith speaks with Prashanth Chandrasekar, CEO of Stack Overflow, about how one of the internet's most trusted platforms for developers is adapting to the era of generative AI. With over 60 million human-curated Q&A pairs, Stack Overflow is now at the center of AI development — not as a competitor to large language models like ChatGPT, but as a foundational knowledge base that powers them. Prashanth breaks down how Stack Overflow is partnering with OpenAI, Google, and other LLM providers to license its data and improve AI accuracy, while also protecting the integrity of its community. He explains the rise of OverflowAI, how Stack Overflow for Teams is fueling enterprise-grade co-pilots, and why developers still rely on expert human input when AI hits its "complexity cliff." The conversation covers everything from hallucination problems and trust issues in AI-generated code to the monetization of developer data and the evolving interface of the web. If you want to understand the future of developer tools, AI coding assistants, and how human knowledge will coexist with autonomous agents, this episode is a must-listen. Subscribe for more deep dives into how AI is reshaping the world of software, enterprise, and innovation. Stay Updated: Craig Smith on X:https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Intro (02:31) Prashanth's Journey from Developer to CEO (05:18) Why Stack Overflow is Different from GitHub (08:51) The Power of Community and Human-Curated Knowledge (12:53) Stack Overflow's Data Strategy for AI Training (17:26) Why Stack Overflow Isn't Competing with OpenAI (20:36) How Stack Overflow Powers Enterprise AI Agents (26:13) OverflowAI, Gemini, and the Future of Dev Workflows (30:09) Inside Stack Overflow for Teams (33:29) Safeguarding Quality: The Fight Against AI Slop (38:32) Licensing, Attribution, and Protecting the Knowledge Base (43:19) Business Strategy in the Age of Generative AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  At the moment, we are not really looking to do something equivalent to what the L M providers are doing. I think what we were trying to do is to really assess the quality of our knowledge base and what does that mean? We will incorporate A I for our users ultimately in the context of making their lives a lot easier and productive on our side. The kind of the system is truly automated, completely automate all your work, especially in more complex tasks. I can totally see the foundational work, the simple questions, the simple programs, absolutely right. It makes a lot of sense for those to be solvable and through this new mode of the user interface. But ultimately, there is still a pretty large gap, especially for more advanced programming needs, where you're absolutely going to need to know how to debug it.

• 0:43Craig S. Smith?

  There's a growing expense eating into your company's profits. It's your cloud computing bill. You may have gotten the deal to start, but now the spend is sky high and increasing every year. What if you could cut your cloud bill in half and improve performance at the same time? Well, if you act by May thirty first, Oracle Cloud Infrastructure can help you do just that. O C I is the next generation cloud designed for every workload, where you can run any application, including any A I projects, faster and more securely for less.

• 1:21Craig S. Smith?

  In fact, Oracle has a special promotion where you can cut your cloud bill in half when you switch to O C I. The savings are real. On average, O C I costs fifty percent less for compute, seventy percent less for storage, and eighty percent less for networking. Join Skydance Animation and today's innovative A I tech companies who upgraded to O C I and save. Offer only for new U S customers with a minimum financial commitment. See if you qualify for half off at oracle dot com slash I on A I. That's oracle dot com slash I on A I.

• 2:05Craig S. Smith?

  E Y E O M A I all run together. Oracle dot com slash I on A I.

• 2:13Craig S. Smith?

  Why don't we start, Prashant, by having you introduce yourself, give a little bit of your background, how you came to Stack Overflow and some background on what Stack Overflow is and how it got started? And then we'll talk about the future.

• 2:30Speaker 00

  Sounds great. Yeah, well, again, thank you for having me. It's a pleasure to speak with you. As my background, fairly straightforward, I've been in technology for a couple of decades. Started out as a software developer at companies like National Semiconductor and Cabletron back in college, in fact. And then quickly realized that I was quite interested in understanding how tech companies worked and how they operated. So through a sort of a multi-dimensional career path in consulting, in banking, all focused on technology, companies or technology functions or using technology in some fashion. Then sort of landed in my first operating role at a company called Rackspace, which is in the cloud services space.

• 3:14Speaker 00

  Spent about seven years at that company, Q and A Texas, that's the reason I moved from New York City back in 2012. And that was just a great experience, learning how to really scale fast-growing businesses, building great teams, and especially entrepreneurial or intrapreneurial endeavors. And that was an exciting thing to focus on. And then in 2019, after about seven years at the company, I was approached about this role at Stack. And the whole mission was, you know, the company is obviously well-known as a, and I'm happy to explain what it is, but well-known as a public platform that serves, you know, millions of people around the world. How do we transform it into an enterprise-focused company was the mandate.

• 3:57Speaker 00

  So given some of what I had done at my prior company, Rackspace, even though it was in a different industry, some of the fundamentals translated. So that's the reason I joined. And, you know, very, very excited about what we've accomplished over the past six years at this company. And I've been here, you know, through a multiple, a couple of different sort of interesting transitions along the way. And you're an unit to one, so which I'm happy to talk about.

• 4:20Craig S. Smith?

  Yeah. And my understanding of Stack Overflow is it's kind of like Quora for coders. And I mean, and correct me if I'm wrong, but you know, you're, you're, you're working on something and you are looking for a solution or a...

• 4:38Craig S. Smith?

  Some some piece of code that that you can't put your fingers on, you go to Stack Overflow. And, uh, you know, what do I do? And somebody sends you a solution. It's sort of a collaborative platform in that sense, uh, and and relatively live so that people aren't waiting, uh, overnight for an answer. Um, how did how did it start? And I've always wondered why that function doesn't exist, or if it does exist, why Stack Overflow is dominated, uh, why that function doesn't exist on, say, GitHub or or another platform?

• 5:18Speaker 00

  I think it's absolutely, um, you know, it's been a, it's quite a journey since the company was founded back in two thousand and eight. I have to say that I think the comparison with other sites like the ones that you mentioned, I think they oftentimes do get included in the kind of broader family of sites, but we're quite distinct, uh, in that when we were formed, we were formed because the founders wanted to create a really authoritative knowledge base for technology information. Uh, so when developers and technologists were writing code, as an example, they got truly, uh, they could really get unstuck, uh, based on trustworthy guidance from the world's experts all around the world. Uh, so this is not a forum site, never been a forum site where people just have, you know, subjective conversations. Uh,

• 6:00Speaker 00

  we may open up new avenues, as I've described, to ma- to do more of that in the future, but the first, let's call it sixteen years, seventeen years of the company have been all around creating like this really accurate knowledge base of information, uh, which is sort of the canonical answer to a question in technology. So, you know, that is, uh, I would say what's distinct about it. Um, and, you know, over the past, again, as like I said, sixteen, seventeen years, uh, we have accumulated our community, uh, thanks to their goodwill, have accumulated something like sixteen million questions and answers on, again, every possible technology topic. Uh, and all that is very, very useful. Uh, not only, it has been useful in the past in the context of people just going to the site through, let's say, a Google search and

• 6:42Speaker 00

  finding an answer to their question when they're stuck, uh, but also now in the context of leverage. This data for L L Ms and agents and agent AI inside companies, as I'll explain. So it's just been a really, really phenomenal, you know, exercise, a community exercise to bring all this human generated expertise together and knowledge together. To to move humanity forward, which is, I think, which is amazing. The the other question that you asked was around, why hasn't this been, you know, why, what's special about this? Why hasn't this been replicated by other platforms, etcetera? I think it comes down to sort of the problem that you're trying to solve ultimately. I think every, every product has a kind of a core problem that they're trying to solve.

• 7:24Speaker 00

  I think something like GitHub, and I can't speak for them completely, but I think, but I can, from the outside in. Obviously, they came from the code sharing and code collaboration space. We've never been in the code collaboration space necessarily, but we have been in the technical knowledge sharing space. And so it's really, you know, it's a great combination between something like the real code that's stored on a place like GitHub or any of the other repositories. That was just phenomenal for collaborating on code as a team. But also then leveraging all the context that is on Stack Overflow, because it has all the the background information on why code needs to be written a certain way, what are the right packages to be using, etcetera, etcetera.

• 8:06Speaker 00

  So I think that, so it's a very different problem that it's solving relative to sharing code. Now, both platforms over time can do all sorts of things to expand and and so on. And I'm sure both companies will will approach it that way. But then, but the original nexus of it was that it was in the two directions I described.

• 8:24Craig S. Smith?

  Yeah, how live is the platform? Again, I'm not a user, so I always imagine that it was, you know, you ran into a problem. You could post a query and somebody around the world who's online and would see it. Or is it really a, you know, a knowledge base of?

• 8:48Craig S. Smith?

  A massive FAQ.

• 8:51Speaker 00

  Yeah, it's an astute question. It is a very live platform. So we have thousands of questions being asked literally on a weekly basis, which is great because we have millions of people all around the world that are not only, you know, being able to benefit from the collective goodwill of, you know, 16 years of contributions, but also they're able to answer questions that are open because they're experts on a certain topic and they get, you know, recognized for being experts. And that's, you know, amazing for not only their primary motivation, which is to help their fellow developer and technologist out, but as a secondary motivation, there will be obviously recognition and that translates to all sorts of things like, you know, you get great jobs as a result of that. You know, we have even a partnership with Indeed on the on our website where you're able to apply

• 9:34Speaker 00

  for jobs and so on. But the nature of the platform is again, a very, very strict Q and A

• 9:44Speaker 00

  form factor where, you know, it's not a discussion. Again, it's a very specific question and a very specific answer. Now, very recently, what we have announced is that that we're going to keep absolutely because it's very important to have that accurate data set in the canonical answers. But we are now experimenting with even faster engagement modes in other parts of the site. So distinct from Q and A, we have opened up, for example, an ability for users to chat real time with each other, right? And so we have opened up that highway, if you will, which has been exciting to watch as we have, as we've exposed that ironically, that that feature actually has existed for 16 years, but has been buried deep in the site for moderators, as an example, to be

• 10:25Speaker 00

  able to sort things out when things happen on the site. So but we realized that some of the newer generation folks love to have instant access or directional guidance versus, you know, perhaps the canonical answer to get formed, you know, in pristine fashion. So, so we will experiment with other content types, you know, even maybe discussions, which is another feature that we have, which is a little bit like, not as strict as Q and A, let's say, but they're all they've got their own sort of lanes, distinct lanes. So we don't pollute the site and the idea is to have high quality information, but but but very community oriented type of interactions.

• 10:59Craig S. Smith?

  Yeah. And and who decides what's canonical? I mean,

• 11:05Craig S. Smith?

  Who's the Pope, so to speak?

• 11:06Speaker 00

  Yeah, indeed. So, uh, it very much the community. I think, you know, this is what's so fascinating about this site is that, uh, this is a very much a self-reinforcing, uh, the experts in the world have all, uh, have all sort of participated, are participating and contributing to the site. And, uh, the community votes up or votes down answers based on, you know, those actually being effective or not effective, uh, in, in getting people unstuck. And so it, it, funnily enough, it works really, really well. Because the intent behind the site and the rule sets that is, that have been established, uh, governs the site in a way that rewards quality. It, it, it absolutely does the opposite for, for the, you know,

• 11:50Speaker 00

  maybe even a little bit more, uh, harshly than people would like. Uh, but it is definitely sort of a school of hard knocks when it comes to, uh, high-quality canonical answers to questions.

• 12:01Craig S. Smith?

  Uh, and the, um, hang on just a sec here.

• 12:07Craig S. Smith?

  Uh, the, um, I can see this, this is, uh, I mean, two things. It's, uh, the, the live chat would be wonderful because things are changing so ma- so fast. Uh, uh, I imagine it's hard to keep up with, with the, uh, the question and answer side. Uh, but also, um, and, uh, as, as a knowledge base for training, uh, these, uh, coding agents and that sort of thing. Are you, uh, doing that? Do you, are you building your own agent? Or are you licensing, uh, the content to, uh, people that are building agents?

• 12:53Speaker 00

  Yeah, a little bit of all of it, actually. So we, uh, back in 2023, what we started noticing is that, uh, when all the LLM, uh, providers started announcing their developments, you know, we, we noticed obviously our data had been used, uh, by, by most of them. So we, uh, if not all of them. Because you know, we are one of the few high quality, again, speaking about this canonical knowledge base, we're one of the few cases that has this high quality information set, which is, you know, quite very, very useful. And what we did was we actually first took an open source model and fine tuned it with our own data. And we noticed something like a thirty percentage point improvement in accuracy. As we know, LLMs continue to hallucinate,

• 13:36Speaker 00

  they hallucinate less these days, you know, probably because we have done all these licensing arrangements. But, but generally speaking, they do hallucinate. So, but the quality goes up dramatically when you use our data. So when we saw that, and we of course put up things like anti-scrapers, and you know, we prevented people from, you know, grabbing the latest information or even the historical information, we, we started engaging with all the LLM providers, the cloud providers, et cetera. And over the past eighteen months or so have constructed a significant number of partnerships with everybody that you can imagine in the ecosystem. So, everybody from the Open AIs and the Googles, to all the cloud providers and more, you know, we have, we have struck partnership deals. And these

• 14:21Speaker 00

  are not only licensing deals, let's be very clear, but they're also bidirectional in nature. So yes, we want to get paid for the data so we can invest in, back into our community platform, so that that, you know, the world can keep going around and around that way. But also because we're solving real problems for these LLM companies. Because number one, you have, you know, what we call LLM brain, brain. But if you don't actually have humans creating new knowledge with, you know, from their minds and, you know, documenting it, then, you know, how are these LLMs going to obviously train? And that's important. Even in a world of synthetic data, you're going to need human generated, you know, knowledge. That's number one. Number two is, you know, when oftentimes, I'm sure Craig, you've tried it, you mentioned Rabbit as an example, but there is a complexity cliff

• 15:06Speaker 00

  with these AI chatbots or assistants or, you know, LLM models. At some point, they're going to tap out on complexity and it's just going to say, here's the best I can do. And, you know, it's on you, then you're on your own. And so when that happens, what we are trying to help, you know, Gen AI and LLM companies with is, hey, why don't you actually engage our community when, in that moment for the user? Why can't the user actually ask a question straight on Stack Overflow, straight from there? And then, by the way, we have a, a legal requirement for them to attribute our content every time. Our, you know, our content is used in a, in a block, black box answer as an example, which I'll get to here in point number three. But to be able to answer that question again generates new content on our website.

• 15:51Speaker 00

  And that knowledge can then get curated and added and, you know, pruned, et cetera. And obviously community curated. Uh, and, uh, that allows them to leverage all that data for future model training needs and other agent techniques over time. So that's the second issue, this AI, uh, you know, answers are not really knowledge all the time. And so we want to help them. And the third issue is around trust, Craig, because around the hallucinations, only about forty percent of our community trusts what's coming out of AI models. So we have, uh, we have now, uh, made sure that all these partnerships require them to attribute our, our data and links to our original answers every time an answer is used in the context of an LLM answer.

• 16:33Speaker 00

  So those are the three big problems, um, along with these, with these licensing agreements.

• 16:37Craig S. Smith?

  Yeah. Uh, on the, on the, um, link, uh, well, first of all, on the model that you guys fine tuned with Stack Overflow.

• 16:51Craig S. Smith?

  Are you pursuing that? Is that a, a product that, that's going to, um, um, become better and, and more comprehensive? I mean, are you, are you, uh, competing with, um, OpenAI and, and some of the others on code generation? I mean, you know, Sam Altman's been out or, uh, Dario has been out talking about how, you know, the, the systems are going to write complete code very soon.

• 17:24Speaker 00

  Yeah. Yeah, no, we, we don't, uh, at the moment, we are not really looking to do something equivalent to what, you know, the LLM providers are doing. I think what we were trying to do is to really assess the quality of our knowledge base and what does that mean? Uh, we will incorporate AI for our users ultimately in the context of making their lives a lot easier and productive on our site. Right? So, uh, so as an example, you know, we've integrated Google Gemini into our public platform and in our enterprise, uh, product called Stack Overflow for Teams, which is the private version of Stack Overflow that companies use. And we have, you know, thousands of enterprises using that product. Where Gemini, Google Gemini, powers our public platform and provides assistance assistance when users are trying to ask questions on the site.

• 18:08Speaker 00

  And Gemini gives them, you know, very friendly recommendations in the privacy of their Gen AI experience versus sort of having to sort of do all that in public in the context of your, you know, other subject matter experts around the world. Which is, by the way, addressing a historical concern that people have had about our site but it's a little bit, you know, it can be quite, quite an experience for a newcomer especially. So this allows, you know, a newcomer to feel a lot more welcome on the site. And then on the enterprise side, we have integrated our content into all parts of the workflow. So into the I V E in Visual Studio Code, into Slack, into Microsoft Teams. So you can ask natural language questions in any of these places, along with the force on the platform, and to be able to get Gen AI summarized answers and real assistance.

• 18:52Speaker 00

  And we will take both these concepts, you know, we'll keep improving these on both sites, on the public as well as the private. And the private one runs, by the way, on Microsoft Azure. So we work with everybody and, you know, we think of ourselves as sort of Switzerland. And then over time, there may be other opportunities to help our users with more assistance on the site. We even launched about in 2023, AI search and retrieval functionality on our public website and gave the ability of what we call overflow AI, where people are able to ask a question in natural language and get answers straight from the corpus of 16 million questions, but a Gen AI summarized answer. So we will continue to experiment with things like that as the industry moves very rapidly.

• 19:34Speaker 00

  And, you know, again, we're watching everything, including open source, which is quite fascinating to observe. So we will, you will see more innovations from us over time.

• 19:43Craig S. Smith?

  Yeah. One of the things you mentioned, the, the integration with some, some of these code generation platforms or, or, you know, putting it on Visual Studio, that's for programmers as they're working, for example, in Visual Studio. And they have a question and they, they can, without having to go to another window or something, they can immediately tap Stack Overflow's knowledge base. But increasingly, as these agents are writing code.

• 20:28Craig S. Smith?

  How is, will the agent be able to access that knowledge?

• 20:33Speaker 00

  Yes, yes.

• 20:34Craig S. Smith?

  Yeah.

• 20:35Speaker 00

  Yeah, great, great question. So what is fascinating is that we almost see in the context of our data, there are two big opportunities for us. One is all the L L M pre-training examples that I just gave you with all the big L L M companies, the cloud hyperscalers and everybody that we've got a relationship with there. But more recently, what we have with the explosion of course, agent tech A I as a topic in the industry and very much the pursuit of the ecosystem to create these agents, especially in companies so that they can assist every employee to be able to do their work a lot more effectively and productively, free up a lot of their capacity to other things. We see an absolute opportunity for our content, both our public platform content or knowledge

• 21:20Speaker 00

  on Stack Overflow, plus the private Stack Overflow knowledge base with the enterprise product, which is again within thousands of companies, where they have a proprietary Stack Overflow within their company to share information. We are noticing Craig, multiple examples where companies like Uber and a whole bunch of other super, you know, kind of, I would say, forward-thinking companies, building A I assistants or co-pilots, or you know, you can call them agents as we move into this next world of, you know, creating really sort of performing action. And they're able to tap into the knowledge store of Stack Overflow for teams and now the public Stack Overflow knowledge

• 22:00Speaker 00

  to be able to empower those agents to be really, really effective and accurate in the moment where the developer is trying to, you know, let's say, delegate tasks to that agent to perform things. So as we know, whether it's L L M's or A I agents, they're all very dependent on high-quality knowledge to be able to either provide information, you know, search results or perform actions on your behalf. And so that is why I think we believe this is a tremendous opportunity for us, both for our enterprise business as well as our public platform.

• 22:28Craig S. Smith?

  Yeah, yeah. Just as an example, again, I'm not a programmer, but, but I'm eager for these multi-agent systems to, to work. And I've got access to a couple of them. And you know, as if I try and write a simple.

• 22:52Craig S. Smith?

  Piece of code, you know, for myself. Inevitably, uh, when it runs, it run, I get a bunch of errors. And, uh, you know, the, I'm sure what many other people are doing as well. Then you copy the error back into the system or, you know, into a chatbot or whatever it is. And it gives you a solution. You copy that back. And you, you know, you end up, uh, going down a rabbit hole. It's like a house of mirrors. You don't, you pretty soon you're lost.

• 23:30Craig S. Smith?

  Uh, could the integrations with, uh, multi-agent systems address that? Where, where the system? I don't, I don't know. I assume, um, they're accessing a knowledge base in the weights of an L L M to solve. You know, to answer a question or, I mean, um, to solve an error. Uh, but, uh, I don't know whether it's just the inherent probabilistic hallucinations of, of the models. Uh, anyway, it just doesn't seem to be very effective.

• 24:12Craig S. Smith?

  But have you, are you working on that where an agent? Uh, if it runs into a problem, then could refer to sack overflow and, and get a solution that'll actually work?

• 24:27Speaker 00

  Hundred percent. I mean, this is exactly what I meant when I said, uh, the A I, the G N I tools that people are beginning to leverage and testing and, you know, piloting and so on, do hit a complexity cliff.

• 24:40Craig S. Smith?

  Right.

• 24:40Speaker 00

  Like your, like your example, I have, you know, I've also experimented with plenty of, you know, coding, uh, you know, G N I tools. And when you, you do actually hit a very much a sort of a cliff where it's able to, it's really not able to keep all the context that you have in mind and it's able, it's, it's, you know, it has to do with, uh, you know, a data scientist can explain all the reasons why our machine learning scientist, a machine learning expert. But the, the point being that. There's still a dependence on high-quality knowledge and it's not where the kind of the system is truly automated, we can completely automate all your work, especially in more complex tasks. I can totally see the foundational work, the simple questions, the simple programs,

• 25:22Speaker 00

  absolutely, right? It makes a lot of sense for those to be solvable and through this new mode of this user interface. But ultimately, there is still a pretty large gap, especially for more advanced programming needs, where you know, you're absolutely going to need to know how to debug it. So oftentimes, we do see people come back to Stack Overflow after they hit the complexity cliff, which is why I was explaining previously, why don't we get users to be able to ask questions straight from that point of, you know, kind of issue that they have, even in the GNI tools, straight on Stack Overflow, so they complete the task. And, you know, we we help the GNI user. So, you know, absolutely, you know, and over time, things will get better. There's no question, because you know, that's just, you know, progress. And, you know, we will just want to make sure that

• 26:06Speaker 00

  we're adding value to the developer. What we are trying to do is focus on the developer, add value to them. And two additional concepts that are important here is, one is, you know, going wherever the user is, right? And so our fundamental thesis is to make sure that we serve our developers wherever they are. And we call this concept Knowledge as a Service, because we just want to provide Stack Overflow knowledge, a public or a private Stack Overflow knowledge, wherever the developer is in their workflow. But that's in a coding, GNI tool, that's in an IDE, whether that's in Slack or Microsoft Teams or anywhere else in your in your company or what you're using. And then secondly, you know, we want to make

• 26:47Speaker 00

  sure that we are really sort of serving these users, you know, kind of in an authentic fashion, where they are getting access to true experts, you know, human experts when they need them, that they can trust this information, and that the world's best JavaScript programmer, either on the public platform or your best JavaScript programmer within your company, is able to validate the answers that are coming out of these AI, AI, you know, answers from these LLMs and GNI tools. So.

• 27:19Craig S. Smith?

  Yeah, just as you're talking, the Gemini integration where that's

• 27:28Craig S. Smith?

  if you're if you're working with a code generation system,

• 27:34Craig S. Smith?

  you can go to Stack Overflow and and can you copy code into Gemini and ask it to query Stack Overflow about what to do in this situation? Or do you have to explain it in natural language with what you're trying to do?

• 27:54Speaker 00

  You can do either. I think that if you look at the Gemini experience, so now if you go to Google Gemini Code Assist, which is their code editor, you can you can type in, how do I do X, Y, Z on Kubernetes? And it's going to provide a GNI summarized answer, and it's going to have links back to Stack Overflow underneath it. And you can click those links, come to the, you know, the canonical answer that has been generated by an expert, you know, God knows when. And that is then going to give you confidence that, you know, that this answer is in fact accurate. So absolutely, you could paste some things into Gemini and, you know, that Gemini, all the GNI experiences are similar. I'm not sure about, you can probably state very specifically, hey, go look at Stack Overflow and find out.

• 28:35Speaker 00

  But I don't think you even need to do that because it's just, it's going to generate the links automatically because that's what the data is trained on. The other thing I was going to mention is that in the same spirit of going where the developer is, many years ago, Craig, you know, we always thought about Google search as, or you know, any search engine as the user interface for Stack Overflow. Ninety-nine percent of folks went to a Google search when they had an issue. They would type in a question and it would land on Stack Overflow because the answer was always accurate. Now, the user interface has changed where increasingly people seem to leverage not only a search engine, but also these GNI tools as their primary mode of engagement onto the web.

• 29:17Speaker 00

  And so really what's happened is that people changed their mode of, you know, how they ask questions and in looking for answers. And just more broadly, I think search itself, I think, is sort of a somewhat of a different, it is changing, obviously, as we know. And that is what we are trying to adapt to. We're trying to adapt to the fact that previously we had, we were linked to the user interface being Google. Now the user interface is all these GenAI tools. Great. We will also then make sure our data and our knowledge is provided in all these GenAI tools. So we wanted to move with the times.

• 29:50Craig S. Smith?

  Yeah. You on, on Stack Overflow for Teams, you mentioned a private Stack Overflow. I mean, how does that work? Is that, is, is, yeah, how does that work?

• 30:09Speaker 00

  Yeah, yeah, absolutely. So back in 2018, this is about six months or about a year before I joined the company. There were some early signs that large companies really loved Stack Overflow, because you can obviously get this community curated knowledge, uploads and downloads and gamification and all these things. A really effective way to keep knowledge up to date and accurate and with a communal sense. And a lot of folks wanted to leverage this internally within their companies, because they wanted all that data and knowledge to be proprietary within their own company versus it getting, showing up on the public web. And so they, so the company launched Stack Overflow for Teams, a private enterprise version of Stack Overflow for companies

• 30:52Speaker 00

  to use for knowledge sharing within their companies. And that's effectively what it was. And, you know, companies like Microsoft and, you know, Apple and, you know, all these sort of big companies that were, you know, leveraging the platform. And the goal was effectively, how do we make this sort of the default knowledge base for all technology technologists within companies? And so over the next several years, we very much went on that drive. And so, you know, we now are used in pretty much every financial institution that you can think of, all the banks, all the retail companies, all online retailers, the, the big tech companies certainly, all use Stack Overflow for Teams as a private knowledge repository for their technologists' information.

• 31:34Speaker 00

  And now that is being surfaced, as I mentioned previously, in the case of Uber as one example, where they're leveraging for their AI co-pilots or AI assistants and those co-pilots are now leveraging the data that's been created, thousands of questions inside Uber are now being leveraged in an agentic sense inside the company through this.

• 31:52Craig S. Smith?

  Yeah, and the this private

• 31:59Craig S. Smith?

  knowledge that's that's held within Stack Overflow for teams is you also have access to the to the public Stack Overflow.

• 32:10Speaker 00

  That's correct. That is correct. Absolutely. As part of your search, when people use the product, absolutely, they how do I do XYZ on, I'll use the Kubernetes example again. So you're going to get, you know, from the 60 million questions and answers, the answers on that topic, which are more public. And then you're going to get, in addition to that, anything that's specific to your company around Kubernetes for the new companies. You're going to get both results that they're very clearly delineated as this is a private versus a public answer and so on. But really, really a powerful mechanism for for users.

• 32:44Craig S. Smith?

  Yeah, and another question I had, how is, I mean, the the answers that are generated or that are that are given on on Stack Overflow by experts, are are you seeing anyone using an LLM to write those those answers? And is that a danger that that?

• 33:17Craig S. Smith?

  You know, is the more gen AI content that gets into the knowledge base?

• 33:25Craig S. Smith?

  You know, it'll cause some sort of problem. I'm not sure what.

• 33:29Speaker 00

  Absolutely. We we very explicitly prevent that from happening. And you know, back in twenty twenty, uh, late twenty twenty two, our community very correctly basically said, look, we can't just generate. We saw a massive spike actually in content uh on the site when people would just started using GNI tools and you know, tools like chat GPT. And a lot of it was incorrect because of the, you know, the hallucination issue and it sounds really good, but actually isn't incorrect. And given like the the goal of the Q and A part, which is to create this canonical, you know, human curated knowledge base, it was literally, you know, the opposite of that. So that so be very quickly, the community and the company, we both together banned all that content saying, look, you can't just use,

• 34:12Speaker 00

  you know, chat GPT or any GNI tools to generate answers. Uh, but let's let's go speak, you know, let's go understand how we can use this phenomenal technology to solve real problems. So we've spent the next year or so uh talking to our community, figuring out that they don't trust what's coming out forty percent or only have then trusted, but seventy percent of them wanted to use GNI tools in some fashion, which is interesting. Um, and so we incorporated GNI on the public platform, as I described, to to provide assistance on questions in the, you know, enterprise product. Uh, and and and now we are also experimenting with answers using AI, but we're trying to make sure that we don't do it in the sense of Q and A, because Q and A is a very pristine sort of knowledge base. We do not want to pollute it with AI slop, so to speak.

• 34:57Speaker 00

  If you know, the internet is now polluted with a lot of just, you know, AI generated content, and we do not want to be one of those sites where that happens. There may be very specific use cases in the other lanes, as I was describing to you in chat or discussions, or maybe even other challenges and other things we do on the site uh within multiple content types, but outside of Q and A, because we want to again protect the quality of that Q and A base, uh, you know, very, very specifically.

• 35:22Craig S. Smith?

  Yeah, um, the the other thing is you've in this this Q and A uh uh knowledge base, you've got, I think you said sixteen million.

• 35:34Speaker 00

  Sixty, zero, six zero.

• 35:36Craig S. Smith?

  Sixty million uh question answer pairs. Uh, I've talked to uh the people about breaking data silos and.

• 35:47Craig S. Smith?

  You know, rationalizing and ontology and all that stuff.

• 35:54Craig S. Smith?

  And, and when you start breaking silos, you find out that you've got contradictions. Do you have, have you run any Gen AI systems over the entire knowledge base to make sure that it's consistent? That there aren't contradictions between answers?

• 36:16Speaker 00

  Yeah, I think that, you know, just the system in itself was, I think, really brilliant about what the founders of this company back in two thousand and eight set up was the very, very specific rule set to account for all of that. Right? So duplicates is a kind of an inside joke with Stack Overflow. So if your question is, you know, it's been asked before and maybe in a different form factor, it's going to get called out pretty quickly. Right? And it's, it is a school of hard knocks for that reason. And so the, the standard's very, very, very high, correct? So I think, I think almost to a place where it is a little bit tough for new users or has historically been until our new Gen AI incorporation, as I described to you.

• 36:53Speaker 00

  But, but it does account for the fact that, you know, within the programming knowledge base, so Stack Overflow itself, it's sort of about half of our traffic, half of our content, and that's one out of one hundred and eighty-five sites that we have. So the other one hundred and eighty-four sites are what we call Stack Ashane sites. And these are, you know, other sort of technology topics, so it includes artificial intelligence, server, you know, system administration sort of topics, DevOps topics and so on, security topics. And so, yes, those are more specialized communities where there may be an intersection, but we have kept them somewhat apart. But Stack Overflow on its own is huge because, you know, it's got every possible programming knowledge, every cloud technology.

• 37:37Speaker 00

  It's the massive, you know, a part of the technology knowledge base. But, you know, the other ones, I don't think there's like, there is, we are working on something called Unified Search across all of the stack machines across one eighty-five. But they've been, you know, somewhat distinct over time. But, you know, there's the core Stack Overflow knowledge base itself is quite cross-functional, obviously, because given the depth of the the knowledge base there and, you know, I think the system that's been set up prevents for the, for the issues that you're describing.

• 38:05Craig S. Smith?

  Yeah, uh, something else occurred to me while you were answering that. Uh, is there a concern that this knowledge base will be stolen uh by some agentic system that could be pinging? It wouldn't take very long uh to download sixty million answers.

• 38:32Speaker 00

  Yeah, yeah, no, that is, you know, I, I think the way we think about that is back in twenty twenty two, twenty twenty three, when we noticed that it had in fact been, let's say, leveraged uh by a lot of the L L M providers. Uh, you know, we, we, you know, took precautions to then sort of set up all the anti-scrapers as I described. Uh, we want to stay open and make sure our community has access to all this data because that's always been, you know, kind of our ethos, which is we're by the community for the community, so they should get access to all the contributions that they provide. Uh, we're governed by something called Creative Commons. Uh, and so in that, so we want to strike this balance of where our community users, you know, have to sort of sign up to get access to the data and tell us who they are.

• 39:15Speaker 00

  So, you know, we know they're not coming from a company as an example. Uh, but if you're a company, if you're a tech company, AI company, L L M company, AI agent company and so on, uh, you've got to work with us officially because I think we really can solve your problems as long as, you know, you work with us in a, uh, through sort of official manner and licenses content, uh, you know, in a real way. So, yeah, we'll take defensive, you know, precautions like the anti-scrapers and rule sets and all that kind of stuff. But, but we're really trying to say, and the good news is that most companies have embraced it. You know, that's, that's the good news. That's why we've now got relationships pretty much with every big AI and L L M and cloud provider that you can imagine. And there'll be, there's plenty more that we're working on, uh, you know, there's a long tail of companies that are interested in engaging with us.

• 40:00Craig S. Smith?

  Yeah, uh, the licensing this data to, um, to code generation systems. Uh, are there any metrics? Can you see the improvement in code generation once they have included Stack Overflow in their training data?

• 40:20Speaker 00

  Very much. I think if you look at some of the, let's call them league tables or the benchmark tables that are, you know, coming out on a weekly basis, the companies that work with us are at the top of the list, especially on code coding benchmarks, right? Because what's, by the way, also as a side point, Craig, the data that we have is not only used for, is really useful for coding related AI tasks, but also non-coding related AI tasks. Because the, the depth, the iceberg under the water, so to speak, of all the comment history, we, we underneath an answer, the curation of downboarding an older answer and a new, you know, more elegant answer to a question, all this, all the things that have happened on, on that very specific one primitive of Q and A in amongst

• 41:03Speaker 00

  60 million of them, that is really, really useful for things like reasoning and really sort of being able to drive logic. So, um, so yes, we do absolutely see the, the performance on the high end of the table associated with, you know, us being associated with them. So that's heartening to see. And we, we had no doubt of that, only because when we continue to run these tests, we even have a technical paper out called Stack eval, which is like a score that we actually have around various AI models. And, you know, when they use our data versus not, and so we can see the big differences between, uh, between using it or not using it.

• 41:38Craig S. Smith?

  Yeah. Um, but with, uh, these large models absorbing all of this knowledge, uh, and, and curating their training data, uh, more and more, are, are you concerned that the day will come where, when Stack Overflow is, is no longer unique, that that data will have leaked out into the public sphere, into these models?

• 42:11Speaker 00

  Well, I think that the amount of new, firstly, there are two things. One is to be able to get access to the data. You are, you know, these are multi-year agreements that are recurring in nature. So the only way you are able to use this data for future model training needs is if you work with us. Right? So if you don't work with us, then you can't use it going forward. That's basically the kind of the arrangement. And number two is that obviously we're generating new information. So if you want the new information to continuously to give you the latest and greatest of what's happening in the ecosystem, again, with human thought and and and judgment, then you should, that's the second reason for you to work with us. So I think both, one is more of a kind of a fundamental legal requirement to say,

• 42:53Speaker 00

  you got to work with us if you want ongoing access and do that on a, on a, on a yearly basis. And then secondly, if you need all the new information, you should also work with us for that.

• 43:03Craig S. Smith?

  Yeah. Okay, we're coming up to the hour. Is there anything I haven't asked that you talked about at HumanX or that you, you want listeners to know?

• 43:19Speaker 00

  Yeah, yeah, I think, I think maybe one thing we haven't spent time on a little bit is, if you think about, you know, the ecosystem of AI players, the big LLM providers and the, of course, they're all after the whole, you know, kind of from a, from a, creating like the industry-leading LLM that's specialized or just more generic. And then you've got the, you know, the long tail of, of AI companies that are all vying for this agentic AI future, especially within companies. And I think what we have noticed is that are every one of our business lines, Craig, and let me just explain that, which is, we have obviously our public platform, which we've spent a bunch of time on, which is our 60 million questions and answers with millions of people asking questions.

• 44:02Speaker 00

  And then we have the enterprise business, StackO for for teams, like we just described, which is used by thousands of enterprises for, inside that company for now, AI agentic needs. And then we have an advertising business because we have. You know, the fact that we we have a lot of traffic and folks, we are able to showcase their products. And then we, of course, have our data licensing business that that that we've already spent time on. So in every one of these, I think historically, people were, you know, you know, back in twenty twenty two, twenty twenty three. I think there's this question about, okay, this, we need to change our business model. The model of the Internet has changed. We've got to change with that. People are no longer just relying on advertising as the sole source. So data, quality, et cetera, becomes really, really important. So that's what we've focused on.

• 44:46Speaker 00

  But also, what's interesting is that a lot of these AI agent companies, what we noticed, they really want not only our data and to be able to build great AI agents for the enterprise, but also they want access to our distribution. You know, the fact that we have millions of people that have can try out their products, including the ones that you're trying out. Imagine you have access to the developers of the world to be able to showcase your product on Stack Overflow to say, here's the leading AI agent for XYZ. And, you know, it's a huge driver of, by the way, you know, really, really powerful demand gen driver for for cloud companies who, you know, advertise with us. You know, all the big companies do, Microsoft and the Amazons and the Googles as an example.

• 45:29Speaker 00

  And so it's a great place for AI agents, agentic companies to to be able to get distribution to to the developer ecosystem. So that's the other aspect we didn't spend much time on. So we literally are, you know, we see this as a tremendous opportunity for the company because literally all parts of our company can be useful and moving humanity forward with, you know, being able to access all this AI power in their favor.

• 45:56Craig S. Smith?

  Yeah, so so the the Gen AI future,

• 46:02Craig S. Smith?

  you guys are embracing it. I mean, a lot of people are afraid of it because, as I said, it's it's going to make a lot of operations obsolete. But but I think it's important, what you were saying about

• 46:18Craig S. Smith?

  the question and answer, that that pristine knowledge base. We need human.

• 46:26Craig S. Smith?

  Knowledge that isn't polluted by, uh, you know, agentic systems or generative AI answers that that is kind of the ground truth and then

• 46:39Speaker 00

  Indeed.

• 46:39Craig S. Smith?

  Build on that, you know?

• 46:41Speaker 00

  Absolutely, yeah. And I think, you know, this is, you know, as I often say, welcome to technology, right? There's it's nothing but change and progress. So companies absolutely have to adapt, you know, we've had to adapt over the past couple of years to this new future and we're still making our way and it's been fun to do all the things that we've all these partnerships that we've struck, as I mentioned. Um, so it just, I think it's going to open up plenty of opportunities and I think they know it could have, I would not be overly pessimistic about, you know, it's probably going to have some sort of, uh, you know, I would say it's it's probably harsh in the near term and good in the long term. Because, you know, it will unlock a lot more potential for people to be a lot more productive. I think more companies will get creative because, you know, again, there's an infinite amount of technology to go, you know, be written and innovation will keep happening.

• 47:26Speaker 00

  So I think if if people have an open mindset about it, this could be a very, very good thing in the long run. I think near term, we're right. I mean, it probably does create a lot of disruption for for teams and and hiring plans and because, you know, ultimately this the CFOs of the world, they're going to have to ask, okay, I've invested a lot in this technology. What am I getting out of it?

• 47:45Craig S. Smith?

  Yeah.

• 47:46Speaker 00

  So I think that's the reckoning for that will happen pretty soon.

• 47:49Craig S. Smith?

  There's a growing expense eating into your company's profits. It's your cloud computing bill. You may have gotten the deal to start, but now the spend is sky high and increasing every year. What if you could cut your cloud bill in half? And improve performance at the same time? Well, if you act by May thirty first, Oracle Cloud Infrastructure can help you do just that. OCI is the next generation cloud designed for every workload where you can run any application, including any AI projects faster and more securely for less. In fact, Oracle has a special promotion where you can cut your cloud bill in half when you switch to OCI.

• 48:35Craig S. Smith?

  savings are real. On average, OCI costs 50% less for compute, 70% less for storage, and 80% less for networking. Join Skydance Animation and today's innovative AI tech companies who upgraded to OCI and save. Offer only for new US customers with a minimum financial commitment. See if you qualify for half off at oracle.com/eyeonaai. That's oracle.com/eyeonaai. E Y E O M A I, all run together. oracle.com/eyeonaai.