Why StackOverflow usage is down 50% — with David Hsu of Retool

We are announcing the second edition of our Latent Space demo day event in SF on 2/23: Final Frontiers, a startup and research competition in “The Autonomous Workforce”, ​”Beyond Transformers & GPUs”, and “​Embodied AI”. RSVP here https://lu.ma/latent-space-final-frontiers! The first one https://www.latent.space/i/99472423/generative-ai-hackathon-sf-feb was aimed for 15-20 people and ended up blowing up to >200 https://www.latent.space/p/demo-day-2023 and covered in the Information https://twitter.com/swyx/status/1636868321571373057 - let’s see what a year of growth (and competition) does to the local events space in 2024. You can find all Latent Space events here https://lu.ma/ls, and of course get in touch with us to host your own AI Engineer meetups like AI Engineering Singapore https://lu.ma/aie-sg. In our December 2023 recap https://www.latent.space/p/dec-2023 we covered the Four Wars of the AI stack. But how do we know when it’s time to crown a winner? As we kick off 2024, we wanted to do a recap of the State of AI in 2023 to set a baseline of adoption for different products. Retool https://retool.com/ had a great report at the end of last year https://retool.com/reports/state-of-ai-2023 which covered a lot of it. David Hsu, CEO and co-founder of Retool, joined us to go over it together. We also talked about the history of Retool, why they were too embarrassed to present at YC demo day, and how they got to $1M ARR with 3 employees. If you’re a founder, there are a lot of nuggets of advice in here! Retool AI In our modeling of the “Software 3.0 Stack”, we have generally left a pretty wide open gap as to the “user interface” equivalent of the AI stack: Retool AI launched 4 months ago https://news.ycombinator.com/item?id=37419886 with some nifty features for SQL generation, and its own hosted vector storage service (using pgvector). However, as he explains on the pod, the more interesting potential of Retool is in helping developers build AI infused applications quickly, in combination with its Workflows https://retool.com/products/workflows feature. This moves Retool down the stack from just the UI for internal tooling to the business logic “piping” as well. There are a bunch of dedicated tools in this space like Respell https://www.respell.ai/, BuildShip https://buildship.com/, Flowise https://flowiseai.com/, and Ironclad Rivet https://rivet.ironcladapp.com/. "We think that practically every internal app is going to be AI infused over the next three years." - David on the pod RIP StackOverflow? In July 2023 we talked about the impact of ChatGPT and Copilot: This was then disputed by StackOverflow https://stackoverflow.blog/2023/08/08/insights-into-stack-overflows-traffic/, who pointed out (very fairly so) that there were privacy-related changes in their analytics instrumentation in 2022. StackOverflow no longer reports traffic, but based on StackOverflow’s continuing transparency we can see that organic declines have continued throughout 2023. Retool’s report comes over a year after those changes and has some self reported samples from users: * 57.6% of people said they have used StackOverflow less; almost all of them replaced it with ChatGPT and Copilot. * 10.2% said they no longer use StackOverflow. We also saw a lot more tools being released in the dev tools space such as (one of our oldest pod friends) Codeium https://www.latent.space/p/varun-mohan (which just raised a $65M Series B https://twitter.com/codeiumdev/status/1752354727411802257), SourceGraph https://www.latent.space/p/sourcegraph (and their newly released Cody), Codium AI https://www.latent.space/p/codium-agents (just released AlphaCodium which was picked up by Karpathy https://twitter.com/karpathy/status/1748043513156272416), Phind https://www.latent.space/p/phind#details (which beat GPT-4 with OSS models), and Cursor https://www.latent.space/p/cursor, one of the most beloved products in the dev community at the moment. Intelligence is getting closer and closer to the IDE, and the trend doesn’t seem to be reverting. We already said that “You are not too old (to pivot into AI)“ https://www.latent.space/p/not-old, and the advice still stands. When asked to rate “Preference for hiring engineers effective at using ChatGPT/Copilot for coding” on a scale of 1 to 10, where 10 is “Much more likely”, ~40% of companies voted 8-10. Having an AI Engineer https://www.latent.space/p/ai-engineer skillset is extremely important. 45% of companies between 1,000-4,999 employees said that they increased the difficulty of technical interviews to compensate for these new tools, so the gap between users and non-users will keep widening. Crossing the AI in Production Chasm Geoffrey Moore’s “Crossing the Chasm https://en.wikipedia.org/wiki/Crossing_the_Chasm” is one of the most quoted business frameworks. Every market has an initial group of Innovators and Early Adopters, who are willing to suffer through the rough edges of products initially, and eventually crosses into the Early Majority, which expects a full product. In the AI world, ChatGPT and Midjourney / DALL-E have crossed the chasm in the consumer space. Copilot is probably the only tool that did it in the enterprise, having crossed 1M paid users. ~$50B were invested in AI in 2023, and we still only have According to the survey, only 25% of companies had real production usage, but 77.1% said their company is making efforts to adopt more. Closing that gap could triple AI adoption in one year. The report also broke down adoption by use case. 66% of companies use it internally, while only 43% do so in customer-facing use cases. Internal usage of AI is much more varied than customer-facing one as well: One point that David made in the podcast is that this number isn’t a knock on AI as a tool, but rather about the demographics of businesses outside of our Silicon Valley bubble: We all work in Silicon Valley, right? We all work at businesses, basically, that sell software as a business. And that's why all the software engineers that we hire basically work on external facing software, which makes sense with most software companies. But if you look at most companies in the world, most companies in the world are actually not software companies. […] Most of the [work of] software engineers in the world actually goes towards these internal facing applications. Beyond code models, it’s clear that the big winners of the first wave of AI adoption are vector stores and RAG. Knowledge base Q&A, customer chatbots, recommendation systems, etc are all based on them. Retool even rolled out their own with Retool Vectors https://docs.retool.com/ai/concepts/vectors. Expect the battlefield to get even hotter in these areas, with Mongo and Chroma leading the charge on a NPS/popularity basis. It’s also clear that OpenAI won the first campaign in the AI models war, by far. Hopefully Mistral and LLaMA3 will shake up this chart when we look back at it in 2025: TLDR: We’re really early. If you want to build in AI, there’s a ton of work to be done, and a lot of problems to be solved. You can find the full report here https://retool.com/reports/state-of-ai-2023 to dive through all the numbers. Video podcast Watch along on our snazzy YouTube https://youtu.be/WTtXTlfm2rg! Show Notes Companies and Projects: * Retool https://retool.com/ * State of AI Report https://retool.com/reports/state-of-ai-2023 * Retool AI https://retool.com/products/ai * Retool Workflows https://retool.com/products/workflows * Raising less money at lower valuations https://retool.com/blog/series-c * Paul Graham's "playing house" essay https://paulgraham.com/before.html * Gödel, Escher, Bach (GEB) https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach Timestamps * [00:00:00] Introduction * [00:02:43] Retool's founding story and decision not to present at YC demo day initially * [00:09:08] Philosophy on fundraising - raising less money at lower valuations * [00:12:53] Overview of what Retool is * [00:15:41] Origin story of Retool AI product * [00:19:59] Decision to use open source vector database PG Vector * [00:21:29] Most underrated AI use cases * [00:25:56] Retool's AI UX and workflows * [00:30:38] Zapier vs Retool * [00:32:54] Updates from Retool's 2023 State of AI survey * [00:35:21] Who is adopting AI first? * [00:37:40] Evolving engineering hiring practices in the age of Copilot/ChatGPT * [00:40:02] Retool's views on internal vs external AI adoption * [00:41:50] OSS models vs OpenAI in production * [00:44:46] Additional survey questions to ask in 2024 * [00:47:04] Balancing enterprise sales vs bottom-up adoption * [00:51:54] Philosophical thoughts on AGI and intentionality Transcript Alessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO in Residence at Decibel Partners https://decibel.vc/, and I'm joined by my co-host Swyx, founder of Smol.ai https://smol.ai/. Swyx [00:00:16]: And today we are in the studio with David Hsu from Retool. Welcome. David [00:00:20]: Thanks. Excited to be here. Swyx [00:00:23]: We like to give a little bit of intro from what little we can get about you and then have you talk about something personal. You got your degree in philosophy and CS from Oxford. I wasn't aware that they did double degrees. Is that what you got? David [00:00:35]: It's actually a single degree, which is really cool. So basically you study content, you study philosophy, and you study intersection. Intersection is basically AI, actually, and sort of computers think, or computers be smart. What does it mean for a computer to be smart? As well as logic. It's also another intersection, which is really fun too. Swyx [00:00:51]: In Stanford, it might be symbolic systems or whatever. It's always hard to classify these things when we don't really have a word for it. Now I guess everything's just called AI. Five years ago, you launched Retool. You were in YC at winter 17 and just been a straight line up from there, right? David [00:01:09]: I wish. Swyx [00:01:10]: What's something on your LinkedIn that people should know about you? Maybe on their personal hobby or, you know, let's just say something you're very passionate about. David [00:01:17]: Yeah, sure. I read quite a bit. I probably read like two books a week around about. So it's a lot of fun. I love biking. It's also quite a bit of fun. So yeah. Swyx [00:01:25]: Do you use Retool to read? David [00:01:27]: No, I don't use Retool to read. No, that'd be funny. Swyx [00:01:30]: What do you read? How do you choose what you read? Any recommendations? David [00:01:35]: I'm mostly reading fiction nowadays. So fiction is a lot of fun. I think it helps me be more empathetic, if you will. I think it's a lot of fun. I actually just want to see what it's like to be in someone else's shoes. That's what I really good about philosophy as well. I find philosophy just so interesting, especially logic. We can talk more about that for probably hours if you want. Swyx [00:01:50]: So yeah, I have a casual interest in epistemology. And I think that any time you, you know, you're trying to figure out a way to solve a problem, you're going to have to figure out a way to solve it. David [00:02:05]: Yeah, totally. What does it mean to know? Alessio [00:02:13]: That's its own podcast. We should do a special edition about it. That's fun. Let's maybe jump through a couple of things on Retool that I found out while researching your background. So you did YC, but you didn't present a demo day initially because you were too embarrassed of what you had built. Can you maybe give any learnings to like founders or people who are built? I've seen a lot of people kind of give up early on because they were like, oh, this isn't really what I thought it was going to be to be a founder. They told me I would go to YC and then present and then raise a bunch of money and then everything was going to be easy. So how did that influence also how you build Retool today in terms of picking ideas and deciding when to give up on it? David [00:03:30]: Yeah. Let's see. So this is around about 2017 or so. So we were supposed to present at the March demo day, but then we basically felt like we had nothing really going on. We had no traction, we had no customers. And so we're like, okay, well, why don't we take six months to go find all that before presenting? Part of that, to be honest, was I think there's a lot of noise around demo day, around startups in general, especially because there's so many startups nowadays. And I guess for me, I'd always wanted to sort of under-promise and over-deliver, if you will. And then demo day, I mean, maybe you two have seen a lot of videos. It's a lot of, honestly, over-promising and under-delivering because every startup says, oh, I'm going to be the next Google or something. And then you peer under it and you're like, wow, nothing's going on here, basically. So I really didn't want that. And so we chose actually not to present on demo day, mostly because we felt like we didn't have anything substantial underneath. Although actually a few other founders in our batch probably would have chosen to present in that situation, but we were just kind of embarrassed about it. And so we basically took six months to just say, okay, well, how do we get customers? And we're not presenting until we have a product that we're proud of and customers that we're proud of. And fortunately, it worked out. Six months later, we did have that. So I don't know if there's much to learn from the situation besides I think social validation was something that I personally had never really been that interested in. And so it was definitely hard because it's hard to sort of, it's almost like you go to college and all your friends are graduating, but you failed or something, you failed the final and you have to like redo it here. It's like, well, it kind of sucks that all your friends are up there and on the podium presenting and they are raising a ton of money and you're kind of being left behind. But in our case, we felt like it was a choice. We could have presented if we really wanted to, but we would not have been proud of the outcome or proud of what we were presenting. And for us, it was more important to be true to ourselves, if you will, and show something that we're actually proud of rather than just raise some money and then shut the company down in two years. Alessio [00:04:45]: Any Sam Altman stories from the YC days? Could you tell in 2017 that Sam was going to become, like, run the biggest AI company in the world? David [00:04:49]: Wow. No one's asked me that before. Let me think. Sam was, I think he was, I want to, I forgot, I think maybe president of YC in our batch. We actually weren't in his group actually at the very beginning. And then we got moved to a different group. I think Sam was clearly very ambitious when we first met him. I think he was very helpful and sort of wanted to help founders. But besides that, I mean, I think we were so overwhelmed by the fact that we had to go build a startup that we were not, you know, honestly paying too much attention to everyone else's partner taking notes on them. Alessio [00:05:20]: That makes sense. Well, and then just to wrap some of the Retool history nuggets, you raised a series A when you were at 1 million in revenue with only three or four people. How did you make that happen? Any learnings on keeping teams small? I think there's a lot of overhiring we've seen over the last few years. I think a lot of AI startups now are raising very large rounds and maybe don't know what to do with the capital. David [00:05:42]: So this is kind of similar, actually, from sort of why we choose not to demo day. And the reason was, it feels like a lot of people are really playing startup. I think PG has an essay about this, which is like, you're almost like playing house or something like that. Like, it's like, Oh, well, I hear that in a startup, you're supposed to raise money and then hire people. And so therefore you go and do that. And you're supposed to, you know, do a lot of PR, because that's what, you know, startup founders do. And so you could do a lot of PR and stuff like that. And for us, we always thought that the point of starting a startup is basically you have to create value for customers. If you're not creating value for customers, like everything else is going to, nothing's going to work. Basically, you can't, you know, continuously raise money or hire people if you don't have customers. And so for us, we were always very focused on that. And so that's initially where we started. I think it's, again, maybe goes to like the sort of presenting something truthful about yourself or staying true to yourself is something to that effect, which is we didn't want to pretend like we had a thriving business. And so the only way to not pretend was actually to build a thriving business. And so we basically just, you know, put our heads down and, you know, grinded away for probably a year, year and a half or so, just writing code, talking to customers. And I think that at that point we had raised something like maybe a million dollars, maybe a million and a half, something out of YC. So I mean, to us, to people, you know, that was a huge amount of money. I was like, wow, like, how are we ever going to spend a million and a half? The runway was like, you know, five, six years at that point, right? Because we're paying ourselves 30, 40K a year. And so then the question was not like, oh, we're going to run on the runways. The question was like, we better find traction because if we don't find traction, we're going to, you know, just give up psychologically. Because if you run an idea for four years and nothing happens, you're probably psychologically going to give up. And I think that's actually true in most startups, actually. It's like most startups die in the early stages, not because I run out of money, but really because you run out of motivation. And for us, had we hired people, I think it would have actually been harder for us because we want to run out of motivation faster. Because when you're pre-product market fit, actually, trying to lead the team of like, you know, 10 people, for example, to Marshall's product market fit, I think it's actually pretty hard. Like it's, you know, every day people are asking you, so why are we doing this? And you're like, I don't know, man, like, hey, trust this. That's actually a very tiring environment to be in. Whereas this is just like, you know, the founders figuring out product market fit, I think it's actually a much sort of safer path, if you will. You're also schooling less with employees, like when you hire employees, you have an idea you're trying to work with your customers. That's actually, I think, a lot more stable of a place for employees to join as well. Swyx [00:08:00]: Yeah. I find that typically the sort of founder employee relationship is, employee expects the founder to just tell them what to do, and you don't really get critical pushback from the employee, even if they're a body and even if they like you as an early engineer. It's very much like the role play of like, once you have that founder hat on, you think differently, you act differently, and you're more scrappy, I guess. In trying to figure out what that product is. Yeah, I really resonate with this, because I'm going through this right now. David [00:08:26]: Awesome. One thing we did actually early on that I think has paid a lot of dividends, especially your rituals a lot larger now is we hired a lot of former founders. So I want to say like, when we were 20, 30, 40 people, we were probably like half former founders at each one of those stages. And that was actually pretty cool, because I think you infuse sort of a, you know, get things done kind of culture, a outcome oriented culture of like a very little politics, because you know, no one came from larger companies, everyone was just like, this is my own startup, let me go figure out how to achieve the best outcome for the customer. And so I think from a cultural perspective, even today, a lot of rituals culture is sort of very self startery. I think it's actually because of sort of these like, you know, early founders that we hired, which was really, really, you know, we're really lucky to have had them. Yeah. Swyx [00:09:08]: And then closing off on just a little bit of the fundraising stuff, something notable that you did was when in 2021, when it was sort of peak Zerp, and everyone was raising hundreds and hundreds of millions of dollars, you intentionally raised less money at lower valuations as your title. And I think it's a testament to your just overall general philosophy and building retool that you're just very efficient and you do things from first principles. Any updates on like, would you still endorse that? You know, would you recommend that to everyone else? What are your feelings sort of two years on from that? David [00:09:38]: Yeah. I think exactly you said is correct, where we raise less money at a lower valuation. And I think the funny thing about this is that when we first announced that, even, you know, internally and both externally, I think people were really surprised, actually, because I think Silicon Valley has been conditioned to think, well, raising a giant sum of money at a giant valuation is a really good thing. So like, you know, you should maximize both the numbers, basically. But actually maximizing both the numbers is actually really bad, actually, for the people that matter the most, you know, i.e. your employees or your team. And the reason for that is raising more money means more dilution. So if you look at, you know, a company like, let's say Uber, for example, if you join Uber at like, I don't know, like a $10 billion valuation, or let's say join for a huge route, which I think happened at a few billion dollars in valuation, you actually got diluted a ton when Uber fund raises. So if Uber raises, if Uber dilutes themselves by 10%, for example, let's say it raised $5.25 billion, for example, I think employees' stake goes down by 10% in terms of ownership. Same with, you know, previous investors, same with the founders, etc. And so, if you look at actually a lot of founders in sort of, you know, the operations statistics space, or, you know, those that fundraise, like, you know, 2013, 2017, a lot of the founders by IPO only have a few percentage points, actually, for a company. And if founders only have a few percentage points, you can imagine how, you know, how little employees have. And so, that I think is actually still really, you know, bad thing for employees overall. Secondly, sort of higher valuation, given the same company quality is always worse. So basically, what that means is, if you are fundraising as a company, you could commit a certain valuation in the market, you know, let's say it's, you know, x. For example, maybe if you're lucky, and you can raise two times x, for example. But if you choose two times x, your company itself is not fundamentally changed. It's just that, you know, for some reason, investors want to pay more for it. You know, maybe today you're an AI company, for example. And so, investors are really excited about AI and want to pay more for it. However, that might not be true in a year or two years time, actually. And if that's not true in two years time, then you're in big trouble, actually. And so, now, I think you see a lot of companies that are raising really high valuations about 2021. And now, they're like, man, we're at like 100x, or, you know, we raised 300x multiple, for example. And if we're at 300x, then, you know, maybe now we're at like 200x, man, we just can't raise money ever again. Like, you know, we have to grow like 50x to go raise money, reasonable valuation, I would say. And so, I think that is really challenging and really demotivating for the team. And so, I think a lower valuation actually is much better. And so, for us, in retrospect, you know, to answer your question, two years later, we did not predict, you know, the crash, if you will. But given that, I think we've done extremely well, mostly because our valuation is not sky high. If our valuation were sky high, I think we'd have a lot more problems. We'd probably have recruiting problems, for example, and probably have a lot of internal morale problems, etc. A lot of people would be like, you know, why is the valuation this way? We might have cash flow problems because we might have to go raise money again, you know, etc. But we can't because the valuation is too high. So, I would urge, I think, founders today to, quote unquote, like, leave money on the table. Like, there are some things that are not really worth optimizing. I think you should optimize for the quality of the company that you build, not like the valuation, you raise that or the amount you raise, etc. So, Swyx [00:12:34]: Highlight 2020, but it looks like, you know, you made the right call there anyway. Maybe we should also, for people who are not clued into Retool, do a quick, like, what is Retool? You know, I see you as the kings or the inventors of the low-code internal tooling category. Would you agree with that statement? You know, how do you usually explain Retool? David [00:12:53]: I generally say it's like Legos for code. We actually hate the low-code moniker. In fact, we have docs saying we will never use it internally, or even to customers. And the reason for that is, I think, low-code sounds very not developer-y. And developers, they hear the phrase low-code, they're like, oh, that's not for me. I love writing code. Like, why would I ever want to write less code? And so, for us, Retool is actually built for developers, like, 95% of our customers actually are developers, actually. And so, that is a little bit surprising to people. I'll generally explain it as, and this is kind of a funny joke too, I think part of the reason why Retool has been successful is that developers hate building internal tools. And you can probably see why. I mean, if you're a developer, you've probably built internal tools yourself, like, it's not a super exciting thing to do, you know, it's like piecing together a CRUD UI, you've probably, you know, pieced together many CRUD UIs in your life before, and there's a lot of grunt work involved. You know, it's like, hey, state management, it's like, you know, data validation, it's like displaying error messages, it's like the bouncing buttons, like, all these things are not really exciting. But you have to do it, because it's so important for your business to have high quality internal software. And so what Retool does is basically allows you to sort of, really fast, whether it's a front end, whether it's a back end or whatever else. So yeah, that's what Retool is. Swyx [00:14:02]: Yeah, actually, if you started hiring, and so I do a lot of developer relations and community building work, and you hired Krithika, who is now, who's also an AI, to start out your sort of DevRel function. And I was like, what is Retool doing courting developers? And then she told me about this, you know, developer traction. And I think that is the first thing that people should know is, which is that the burden and the weight of internal tooling often falls to developers, or it's an Excel sheet somewhere or whatever. But yeah, you guys have basically created this market, you know, in my mind, I don't know if there was someone clearly before you in this, but you know, you've clearly taken over and dominated. Every month, there's a new YC startup launching with that it's like, you know, we're the open source Retool, we're like the lower code Retool, whatever. And it's pretty, I guess it's endearing, you know, we'll talk about Airplane later on. But yeah, I think I've actually used Retool, you know, in my previous startups for this exact purpose. Like, we needed a UI for AWS RDS that they can, you know, like the rest of our non less technical people, like our sales operations, people could could interact with and yeah, Retool is perfect for that. David [00:15:04]: Yeah, it's a good example of like, that's an application that an engineer probably does not want to build, like building an app on top of Salesforce or something that is not exciting. And so it sucks. It's very limited. It's like not a fun experience at all. But piecing together a Retool is quite a bit easier. So yeah, let me know if you have any feedback, but also, yeah, no, of course, like more recently, Swyx [00:15:23]: I think about three, four months ago, you launched Retool AI, obviously, AI has been sort of in the air. I'd love for you to tell the journey of AI products ideation within Retool. Given that you have a degree in this thing, I'm sure you're not new to this, but like, when would you consider sort of this the start of the AI product thinking in Retool? David [00:15:41]: So we actually had a joke internally at Retool. We are part of roadmap for every year, it was like 2019 or something. We had this joke, which was like, what are we going to build this year? We're going to build AI programming, is what we always said as a joke. And so, but it was funny, because we were like, that's never gonna happen. But like, let's add it because it's like a buzzword thing that enterprises love. So let's look at it. And so it was almost like a funny thing, basically. But it turns out, you know, we're actually building that now. So this is pretty cool. So I would say maybe AI thinking on Retool probably first started maybe like, I don't know, a year and a half ago, something like that. And when we first started thinking about it, sort of in a philosophical way, if you will, it's like, well, what is the purpose of AI? And how can it help, you know, what Retool does? And there were two main prongs, if you will, value. One was helping people build apps faster. And so you've probably seen Copilot, you've seen sort of so many other coding assistants, P0 to them, you know, stuff like that. So that's interesting, because, you know, engineers, as we talked about, do some grunt work. And grunt work, you know, maybe could be automated by AI was sort of the idea. And it's interesting. So we actually, I would say, kind of proved or disproved the hypothesis a little bit. If you talk to most engineers today, like a lot of engineers do use Copilot. But if you ask them, like, how much time does Copilot save you? It's not like coding is 10x faster than before, you know, coding is maybe like 10% faster, maybe 20% faster, or something like that, basically. And so it's not like a huge step change, actually. And the reason for that, as we think, is because the sort of fundamental frameworks and languages have not changed. And so if you're building, let's say, you know, like the sales ops tool we're talking about before, for example, let's say you've got AI to generate you a first version of that, for example, the problem is that it probably generated it for you in like JavaScript, because you're writing for the web browser, for example, right. And then for you to actually go proofread that JavaScript, for you to go read the JavaScript to make sure it's working, you know, to fix the subtle bugs that AI might have caused, hallucinations, stuff like that, actually takes a long time and a lot of work. And so for us, the problem is actually not like the process of coding itself, it is more sort of the language or the framework, I think it's like way too low level. It's kind of like anything like punched cards, like, let's say, back in the day, who designed punched cards, and AI could help you generate punched cards, okay, you know, I guess that helps me punch the cards a little bit faster, because I have a machine punching them for me. But like, when there's a bug, I still have to go read all the punched cards and figure out what's wrong, right? It's like, it's a lot of work, actually. And so, for us, that was the sort of initial idea was, can we help engineers code faster? You know, I think it's somewhat helpful, to be clear, like, again, I think it's 10 or 20%. So we have things like, you know, you can generate school careers by AI, you can generate UIs by AI, and stuff like that. So that's cool, to be clear. But it's not, I think, the step change, that I think is, you know, the, we're investing somewhat in that. But the bulk of investment, actually, is a number two, which is helping developers build AI enabled applications faster. And the reason why we think this is so exciting is we think that practically every app, every internal app, especially, is going to be AI infused over the next, like, three years. And so every tool you might imagine, so like the tool you were mentioning, like a sales operations tool, for example, probably, you know, if you were to build today, one of the corporate subform of AI. And so, you know, we see today, like, for us, like, a lot of people build, you know, I'll say sales manager tools, or retool. An example is there's a fortune, like a company is building like, sales forecasting tools. So they basically have salespeople enter their forecast, you know, for the quarter, the beginning of the quarter, like, hey, I have these deals. And these deals are going to close, these deals are not going to close, you know, I think I'm upsiding these, downsiding these, stuff like that, basically. So you can imagine it's pulling in deals from your Salesforce database. And so it pulls in the deals that actually use AI to compute like, okay, well, you know, given previous deal dynamics, like, these are the deals that are more likely to close this month versus next month was this quarter, next quarter, etc. And so it could actually, you know, pre write you a draft of, you know, your report, basically. And so that's an example where I think all apps, whether it's, you know, a sales app, you know, until it looks like fraud app, a, you know, fintech app, you know, whatever it is, basically, especially internal apps, I think, like you said, Alessio, in order to make you more productive, it's going to incorporate some form of AI. So the other question is, can we help them incorporate AI faster? So that's why we launched like a vector database, for example, built directly into retool. That's why we know launches all these AI actions, you don't have to go figure out what the best model is and do testing and stuff like that, which gives you out of the box. So for us, I think that is really the really exciting futures. Can we make every app and also retools use AI a little bit and make people more productive? Alessio [00:19:59]: So for Wang, who's the co founder and chief architect of amplitude, he mentioned that you just use Postgres vector. When you were building retool vectors, how do you think about, yeah, leveraging a startup to do it, putting vectors into one of the existing data stores that you already had? I think like, you're really a quite large customer scale. So like, you're maybe not trying to get too cute with it. Any learnings and tips from that? David [00:20:23]: Yeah, I think a general philosophical thing I think we believe is, um, we think the open source movement in AI, especially when it comes to all the supporting infrastructure is going to win. And the reason for that is we look at like developer tools in general, especially for such a fast moving space. In the end, like, there are really smart people in the world that have really good ideas, and are going to go build companies and they're going to go build projects basically around these ideas. And so for us, we have always wanted to partner with maybe more open source providers or projects, you could say, like PG factory, for example. And the reason for that is it's easy for us to see what's going on under the hood. A lot of this stuff is moving very fast. Oftentimes, there are bugs, actually. And so we can go look and fix bugs ourselves and contribute back to them, for example. But we really think open source is going to win in this space. It's hard to say about models. I don't know about models necessarily, because it's going to be pretty complicated there. But when it comes to tooling, for sure, I think there's just like so much, there's an explosion of creativity, if you will. And I think betting on any one commercial company is pretty risky. But betting on the open source sort of community and the open source contributors, I think is a pretty good bet. So that's why we decided to get at consumer games. Awesome. Alessio [00:21:29]: So we're going to jump into the survey next, but we're going to put a bunch of links in the show notes about Result AI and whatnot. Is there any most underrated feature, like something that customers maybe love that you didn't expect them to really care about? I know you have a like text to SQL, you have UI generation, there's like so many things in there. Yeah. What surprised you? David [00:21:49]: Yeah. So what's really cool, and this is my sense of the AI space overall, you know, if you're on YouTube as well, is that, especially in Silicon Valley, where a lot of the innovation is happening, I think there's actually not that many AI use cases, to be honest. And AI to me, even as of January 19th of 2024, still feels like in search of truly good use cases. And what's really interesting, though, about Retool, and I think we're in a really fortunate position, is that we have this large base of sort of customers, and a lot of these customers are actually much more legacy, if you will, customers. And a lot of them actually have a lot of use cases for AI. And so to us, I think we're almost in like a really perfect or unique spot, we're able to adopt some of these technologies and provide them to some of these like older players. So one example that actually really shocked and surprised me about AI was, so we have this one clothing manufacturer, I think it's either the first or second largest clothing manufacturer in the world, who's using Retool. They're a ginormous company, very multinational, stores on pretty every mall in the world. And so they have one problem, which is, they need to design styles every year, for the next year, basically, for every season. So like, hey, just like summer 2024, for example, and we're going to design. And so what they used to do before is they were hiring designers, and designers would go to study data, they'd be like, okay, well, it looks like, you know, big floral patterns are really hot. And like, you know, California, for example, in 2023, and like, do I think it's going to be hot in 2024? Well, let me think about it. I don't know. Maybe. And if so, if I believe it's going to be hot, let me go design some floral patterns, actually. And what they ended up doing in Retool, actually, is they actually automated a lot of this process away in Retool. So they actually now built a Retool app that allows actually a non-designer, so like an analyst, if you will, to analyze like, you know, who are the hottest selling patterns, you know, particular geos, like this was really hot in Brazil, this is really hot in China, it's really hot, you know, somewhere else, basically. And then they actually feed it into an AI. And the AI, you know, actually generates with Dolly and other image generation APIs, actually generates patterns for them. And they print the patterns, which is really cool. And so that's an example of like, honestly, a use case I would have never thought about, like thinking about like, you know, how clothing manufacturers create their next line of clothing, you know, for the next season, like, I don't know, I never thought about it, to be honest, nor did I ever think, you know, how it would actually happen. And the fact that they're able to leverage AI and actually, you know, leverage multiple things in Retool to make that happen, is really, really, really cool. So that's an example where I think if you go deeper into sort of, if you go outside the Silicon Valley, there are actually a lot of use cases for AI. But a lot is not obvious, like you have to get into the businesses themselves. And so I think we're, we personally are in a really fortunate place. But if you know, you're working in the space and want to find some use cases, please come talk to us like, you know, we're really excited about marrying sort of technology with use cases, which I think is actually really hard to do right now. Swyx [00:24:38]: Yeah, you know, I have a bunch of like, sort of standing presentations around like, how this industry is developing. And like, I think the foundation model layer is understood. The chain vector DB rag layer is understood, I always have a big question mark and actually have you and Vercel V0 in that box, which is like sort of the UI layer for AI. And like, you know, you are perfectly placed to expose those functionalities to end users, you personally don't really know what they're going to use it for. And sometimes they'll surprise you with their creativity. One segment of this, I do see some startups springing up to do this is related to the things that to something that you've you also build, but it's not strictly AI related, which is retool workflows, which is the sort of canvassy boxes and arrows point and click do this then do that type of thing like which which every what are we calling low code? Every internal tooling company eventually builds, you know, I worked at a sort of workflow orchestration company before, and we were also discussing internally how to make that happen. But you are you're obviously very well positioned to it to that. Yeah, basically, like, do you think that there is an overlap between retool workflows and AI? I think that, you know, there's there's a lot of interest in sort of chaining AI steps David [00:25:55]: together. Swyx [00:25:56]: I couldn't tell if like that is already enabled within retool workflows, I don't think so. But you could you could sort of hook them together as kind of jankily, like, what's the interest there? You know, is it all of a kind, ultimately, in your mind? David [00:26:07]: It is 100% on time. And yes, you can actually already saw a lot of people actually are building AI workflows down retool, which is what we're gonna talk about in a second. But a hot take here is actually, I think a lot of the utility in AI today, I would probably argue 60 70% of the utility, like, you know, businesses have found an AI is mostly via chat GPT, and across the board. And the reason for that is, I mean, the chat GPT is sort of a UI, you could say, or interface and user experience is just really quite good, you know, you can sort of converse, you know, with an AI, basically. But that said, there are downsides to it. If you talk to like a giant company, like a J. P. Morgan Chase, you know, for example, they may be reticent to have people copy paste data into chat GPT, for example, even on chat GPT Enterprise, for example. Some problems are that I think chat is good for one off tasks. So if you're like, hey, I want a first version of representation or something like that, you know, and help me write this first version of a doc or something like that, chat is great for that. It's a great, you know, very portable, you know, if you will form factor, so you can do that. However, if you think about it, you think about some economic productivity, more generally, like chat, again, will help you like 10 or 20%. But it's unlikely that you're going to replace an employee with chat, you know, you're not gonna be like, oh, I'm a relationship manager at J. P. Morgan Chase, and I've replaced them with an AI chatbot. It's kind of hard to imagine, right, because, like, the employees are doing a lot of things besides, you know, just, you know, generating, you know, maybe another way of putting it is like, chat is like a reactive interface, like, it's like, when you have an issue, you will go reach out to chat and chatbot solve it. But like, chatbot is not going to solve 100% of your problems, it'll solve like, you know, 25% of your problems, like, pretty quickly, right. And so what we think the next like, big breakthrough in AI is, is actually like automation. It's not just like, oh, I have a problem, let me go to a chatbot and solve it. Because like, again, like, people don't spend 40 hours a week in a chatbot, they spend like two hours a week in a chatbot, for example. And so what we think can be really big, actually, is you're able to automate entire processes via AI. Because then you're really realizing the potential of AI, it's like, not, it's not just like, you know, a human copy pasting data into an AI chatbot, you know, pasting it back out or copying back out. Instead, it's like the whole process now was actually done in an automated fashion without the human. And that, I think, is what's going to really unlock sort of big canonical productivity, or that's what I'm really excited about. And I think part of the problem right now is, I'm sure you all thought a lot about agents is that the agents are actually quite hard. Because like, you know, the AI is wrong, like, you know, 2% of the time, but then you like, you know, a score, if you let's say, you know, raise to the power seven, for example, that's actually wrong, you know, quite often, for example. And so what we've actually done with workflows is we prefer, we've learned, actually, is that we don't want to generate the whole workflow for you by AI. Instead, what we want you to do, actually, is we want you to actually sort of drag and drop the workflow yourself. Maybe you can get a vSphere or something by AI, but it's coded, basically, you should actually be able to modify the steps yourself. But every step can use AI. And so what that means is like, it's not the whole workflow is created by AI, every step is AI automated. And so if you go back to, for example, like the users are talking about, you know, with a clothing manufacturer, that's actually a workflow, actually. So basically, what they say is, hey, every day, we see all the data, you know, from our sales systems into our database. And then we do some data analysis, and, you know, it's just raw SQL, basically, it's nothing too surprising. And then they use AI to generate new ideas. And then the analysts will look at the new ideas and approve or reject them, basically. And that is like a, you know, that's true automation. You know, it's not just like, you know, a designer, copy pasting things as a chat, you can be like, hey, you know, give me a design. It's actually designs are being generated and generated 10,000 designs every day. And then you have to go and approve or reject these designs, which I think is a lot, you know, that's a lot more economically productive than just copy pasting something. So we think sort of the AI workflow space is a really exciting space. And I think that is the next step in sort of delivering a lot of business value by AI. I personally don't think it's, you know, AI chat or AI agents quite yet, so. Swyx [00:29:50]: That's a pretty reasonable take. It's disconcerting, because like, I know a lot of people trying to build what you already have in workflows. So you have that sort of, you're the incumbent sort of in their minds, I'm sure it doesn't feel that way to you. But like, I'm sure, you know, you're the incumbent in their minds, and they're like, okay, like how do I, you know, compete with retool or, you know, differentiate from retool. As you mentioned, you know, all these connections, it does remind me that you're running up against Zapier, you're running up against maybe Notion in the distant future. And yeah, I think that there'll be a lot of different takes at this space and like whoever is best positioned to serve their customer in the way that they need to shape is going to win. Do you have a philosophy against around like what you won't build, like what do you prefer to partner and not build in-house? Because I feel like you build a lot in-house. David [00:30:38]: Yes, there's probably two philosophical things. So one is that we're developer first. And I think that's actually one big differentiator between us and Zapier and Notion, and we're so very rare we'll see them actually, and the reason is we're developer first. Because developers, like, if you're like building a sales ops tool, you're probably not considering Notion if you're a developer, you're probably like, I want to build this via React, basically, or use retool. And so are you we build for developers, it's pretty interesting, actually, I think one huge advantage of some of the developers is that developers don't want to be given an end solution. They want to be given the building blocks liquid to themselves to build the end solution. And so for us, like, interesting point that equilibrium we don't get to, it's basically to say, hey, retool is a consulting company, and we basically build apps for everybody, for example. And what's interesting is, we've actually never gotten to that equilibrium. And the reason for that is for some of the developers, developers don't want, you know, like a consultant coming in and building all the apps for them. Developers like, hey, I want to do it myself, just give me the building blocks, give me the best table library, give me, you know, good state management, give me an easy way to query the rest of the API. So I'll do it myself, basically. So that is pretty, so we generally end up basically always building building blocks that are reusable by multiple customers. We have, I think, basically never built anything specific for one customer. So that's one thing that's interesting. The second thing is when it comes to sort of, you know, let's say like, in the AI space, we're going to build and we're not going to build, we basically think about whether it's all core competency or whether there are unique advantages to us building it or not. And so we think about the workflows product, we think workflows actually is a pretty core competency for us. And I think the idea that we can build a developer first workflows automation engine, I mean, I think after we released, you know, workflows, virtual workflows, there have been a sort of few copycats that are, I think, quite, quite far behind, actually, they sort of are missing a lot of more critical features. But like, if you look at the space, it's like, Zapier on one side, and then maybe like, Airflow on the other. And so virtual workflows actually is fairly differentiated. And so we're like, okay, we should go build that. This is the one I was going to build, so I'm just going to build it. Whereas if you look at like vectors, for example, you look at vectors like, wow, there's a pretty thriving space already, if you know vector databases. Does it make sense for us to go build our own? Like, what's the benefit? Like, not much, we should go partner with or go find technology off the shelf. Narcissus is pretty effective. And so for us, I think it's like, how much value does that for customers? Do we have a different take on the space? Do we not? And every product that we've launched, we've had a different take on the space and the products that we don't have a different take, we just adopt what's off the shelf. Alessio [00:32:54]: Let's jump into the state of AI survey that you ran, and maybe get some live updates. So you surveyed about 1600 people last August, and I were this busy like five years ago. And there were kind of like a lot of interesting nuggets and we'll just run through everything. The first one is more than half the people, 52% said that AI is overrated. Are you seeing sentiment shift in your customers or like the people that you talk to, like as the months go by? Or do you still see a lot of people? Yeah, that are not in Silicon Valley, maybe say, hey, this is maybe not as world changing as you all made it sound to be. David [00:33:30]: Yes, we're actually on the survey again, actually, in the next few months. So I can let you know when it changes. It seems to me that it has settled down a bit in terms of sort of the maybe like, I don't know, signal to noise, you could say like, it seems like there's a little bit less noise than before. I think people are still trying to look for use cases. I'm saying, but honestly, last year, like United States, again, and I think there are slightly more use cases, but still not substantially more. And I think as far as we can tell, a lot of the surveys, especially some of the comments that we saw, do feel like the companies are investing quite a bit in AI, and they're not sure where it's going to go yet. But they're like, right, it could be big. So I think we should keep on investing. I do think that based on what we are hearing from customers, if we're not seeing recurrence of like a year or something, there will be more skepticism. So I think there is like a, it is time bound, if you will. Alessio [00:34:15]: So you finally gave us some numbers on Stack Overflow usage. I think that's been a Twitter meme for a while, whether or not Chad GVT killed Stack Overflow. In the survey, 58 people said they used it less. And 94% of them said they used it less because of Copilot and Chad GVT, which I think it kind of makes sense. I know Stack Overflow tried to pull a whole thing. It's like, no, the traffic is going down because we changed the way we instrument our website. But I don't think anybody. And then you add right after that expectation of job impact by function and operations, people, 8 out of 10, basically, they think it's going to, it's going to really impact their job. Designers were the lowest one, 6.8 out of 10. But then all the examples you gave were designers of a job being impacted by AI. Do you think there's a bit of a dissonance maybe between like the human perception is like, oh, my job is like, can possibly be automated? It's funny that the operations people are like, yeah, it makes sense. I wish I could automate myself, you know, versus the designers or maybe they love their craft more. Yeah, I don't know if you have any thoughts on who will accept the first, you know, that they should just embrace the technology and change the way they work. David [00:35:21]: Yeah, that's interesting. I think it's probably going to be engineering driven. I mean, I think you two are very well, maybe you two even started some of this wave and sort of the AI engineer wave. I think the companies that adopt AI the best, it is going to be engineering driven, I think, rather than like operations driven or anything else. And the reason for that is, I think the rise of this like profile with AI engineering, like AI is very philosophical, like AI is a tool in my head. Like it is not a, in my head, I think we're actually pretty far from AGI. But AI is not like a, you know, thing that it's not like a black box where like it does everything you want it to do. The models that we have today require like very specific prompting, for example, in order to get like, you know, really good results. And the reason for that is, it's a tool that, you know, you can use it a specific way. So if you use it the wrong way, it's not going to produce good results for you, actually. It's not like by itself taking a job away, right? And so I think actually, to adopt AI, it's probably going to be going to have to be engineering first, basically, where engineers are playing around with it, figuring out limitations of the models, figuring out like, oh, maybe like using vectorized databases is a lot better, for example, maybe like prompting in this particular way, it's going to be a lot better, etc. And that's not the kind of stuff that I think like an operations team is going to really be like experimenting with necessarily. I think it really has to be engineering led. And then I think the question is, well, what are the engineers going to focus on first? Like, are they going to focus on design first or like operations first? And that I think is more of a business decision. I think it's probably going to be more like, you know, the CEO, for example, says, hey, we're having trouble scaling this one function. So like, why don't we try using AI for that? And let's see what happens, for example. And so in our case, for example, we are really we have a lot of support issues. So what I mean by that is we have a really, really high performance support team. But we get a lot of tickets. And the reason for that is, you know, we're a very dynamic product, you can use it in so many different ways. And we'll have a lot of questions for us, basically. And so we were looking at, well, you know, can we, for example, draft some replies and support tickets, you know, by AI, for example, can we allow our support agents to be, you know, hopefully, you know, double as doubly productive as before, for example. So I guess I would say it's like business needs driven, but then engineering driven after that. So like, you know, we the business decides, okay, well, this is where AI can be most applied. And then we assign the project to an engineer, and the engineer goes and figures it out. I honestly am not sure if like the operation, we're gonna have much of a, like, if they accept or reject it, I don't know what's gonna change the outcome, if you will. Alessio [00:37:40]: So another interesting part was the importance of AI in hiring. 45% of companies said they made their interviews more difficult in the in the engineering side, made interviews more difficult to compensate for people using copilot and chat GPT. As they change every tool, like, have you? Yeah, have you thought about it? I don't know how much you're still involved with engineering hiring, I get the company, but I'm curious how we're scaling the difficulty of interviews, even though the job is the David [00:38:11]: same, right? Alessio [00:38:11]: So just because you're gonna use AI doesn't mean the interview should be harder. But I guess it makes sense. David [00:38:16]: Our sense, basically, the survey, and this is true, we believe, too, is we are most when we do engineering interviews, we are most interested in assessing like critical thinking or thinking, you know, on the spot. And I guess, you know, when you hire the employee, you know, in the end, the job of employees to be productive, which they choose whatever tools they want to be productive. So, you know, that's kind of our thinking, too. However, we do think that, you know, if you think about it from a first person's way, if your only method of like coding is literally copy pasting, you know, off of chat GPT, or like, you know, it's pressing tab and copilot, I think that would be concerning. And so, for that reason, we still do want to test for like, you know, fundamentals understanding of comp sci. Now, that said, I think if you're able to use chat GPT or copilot, let's say competently, we do view that as a plus, we don't view it as a minus. But if you only use copilot, and you aren't able to reason about like, you know, how to write a for loop, for example, or how to write fizzbuzz, that would be highly problematic. And so, for us, we do today is we'll base a screen share, or a rest is a hackpad, actually. So it's, sorry, this is no copilot there to sort of see what they're doing, or see what they're thinking. And we really want to test for thinking, basically. But yeah, I mean, we ourselves internally have embraced copilot, and we would encourage engineers to go over this copilot too. But we do want to test for understanding of what you're doing, rather than just copy pasting a copilot. Alessio [00:39:27]: The other one was AI adoption rate, only 27% are in production. Of that 27%, 66% are internal use cases. Shout out to retool, you know, do you have a mental model as to how people are gonna make the jump from like, using it internally to externally? Obviously, there's like all these different things like privacy, you know, if an internal tool hallucinates, that's fine, because you're paying people to use it basically, versus if it hallucinates to your customer, there's a different bar. Because for you, if people build internal tool with retool, there are external customers to you, you know, so I think you're on the flip side of it. David [00:40:02]: Yeah, I think it's hard to say, maybe a core retool belief was actually that most software built in the world is internal facing, actually, which actually sounds may sound kind of surprising, you know, for some of you hearing this, but effectively, like, you know, we all work at Silicon Valley, right? We all work at businesses, basically, that sell software as, you know, as sort of a business. And that's why all the software engineers that we hire basically work on external facing software, which makes sense with most software companies. But if you look at most companies in the world, most companies in the world are actually not software companies. If you look at like, you know, the clothing manufacturer that I was talking about, they're not a software company, like they don't sell software, you have to make money, they sell clothing to make money. And most companies in the world are not software companies, actually. And so most of the engineers in the world, in fact, don't work at Silicon Valley companies, they work outside of Silicon Valley, they work in these sort of more traditional companies. So if you look at the Fortune 100, for example, probably like 20 of them are software companies, you know, 480 of them are not software companies. That's the employable software engineers. And so most of the software engineers in the world, and most of the code engineers in the world actually goes towards these internal facing applications. And so, for all the reasons you said there, like, I think hallucination matters less, for example, because they have someone checking the output, and consumer, so hallucination is more okay, it's more acceptable as well. Yeah, it can be unreliable, because it's probabilistic, and that's also okay. So I think it's kind of hard to imagine AI being adopted in a consumer way without the consumer like opting in, like, Chachapiti is very obviously a consumer, the consumer knows that it's Chachapiti, they're using it. I don't know if it's going to make its way to like the banking app anytime soon. Maybe for like, even for support, it's hard. Because if it hallucinates, then, you know, it's actually quite bad for support if you're hallucinating, right? So it's, yeah, it's hard to say. I'm not sure. Alessio [00:41:50]: Yeah, I think a lot of people, like you said, we all build software. So we expect that everybody else is building software for other people. But most people just want to use the software that we build out here. I think the last big bucket is like models breakdown. 80% of people use it, just use OpenAI. Some might experiment with smaller models. Any insights from your experience at Retool, like building some of the AI features? Have you guys thought about using open source models? Have you thought about fine tuning models for specific use cases? Or have you just found GPT-4 to just be great at most tasks? David [00:42:24]: Yeah, so two things. One is that from a data privacy perspective, people are getting more and more okay with using a hosted model like a GPT-4, for example. Especially because GPT-4 or OpenAI often has to have enterprises who went to some companies already because I think a lot of CIOs are just like, let's get a second house. Like, you know, let's use Azure, for example. And, you know, let's make it available for employees to experiment with. So I do think there is more acceptance, if you will, today of feeding data into GPT. That's going to take some sensitive data. People might not want to do so. Like, you know, feeding in like earnings results data, you know, three days for you to announce earnings, like probably is a bad idea. You probably don't want people to be writing your like earnings statement for you. So yeah, there's still some challenges like that. But I think actually open source models could actually help solve like a lot of greed when it comes to, and that can be exciting. So that's maybe just one thought. The second thought is, I think OpenAI has been really quite smart with their pricing. And they've been pretty aggressive of like, let's get, you know, let's create this model and sell it at a pretty cheap price to make it such that there's no reason for you to use any other model. Just from like a strategy perspective, I don't know if that's going to work. And the reason for that is you have really well-funded players like Google or like Facebook, for example, that are actually quite interested. I think if it was creating startups, OpenAI would win for sure. Like at this point, OpenAI so far had from both a model and a pricing perspective that like there was no reason for it to go just really, I think, in my opinion, at least a startup model. But if like, you know, Facebook is not going to give up on AI, like Facebook is investing a lot in AI, in fact. And so competing against a large FANG company on making a model open source, I think that is challenging. Now, however, where we are right now is I think GPT-4 so far in terms of performance and I would say a model performance is so important right now because like the average, I'm not going to argue LLAMA-2 is actually so far behind, but like customers don't want to use LLAMA-2 because it's so far behind right now. And so that I think is part of the challenge. As AI progress slows down, so if we get like LLAMA-4 and LLAMA-5, for example, maybe it's a comparable at that point like GPT-5 or GPT-6, like it may get to the point where it's like, look, I just want to use LLAMA. Like it's safer for me to host it on-prem, it's just as fast, just as cheap, like why not basically? But I think right now we are in this state, we're opening up next year really well, I think. And right now they're thriving, but let's see what happens in the next year or two. Swyx [00:44:40]: What are you going to ask differently for the next survey? Like what info do you really actually want to know that's going to change your worldview? David [00:44:46]: I'll also ask you that, but if you have any ideas, let me know. For us, actually, we were planning on asking very similar questions because for us, the value of the survey is mostly seeing changes over time and understanding like, okay, wow, for example, GPT-4 Turbo MPS has declined. That would be interesting, actually. One thing that was actually pretty shocking to us was, let me find the exact number, but one change that we saw, for example, if you compare GPT-3.5 MPS, I want to say it was like 14 or something, it was not high, actually. The GPT-4 MPS thing was like 45 or something like that, so it was actually quite a bit higher. So I think that kind of progress over time is what we're most interested in seeing, is are models getting worse, models getting better? Are people still loving PG Vector? Do people still love Mongo? Stuff like that. That I think is the most interesting. Swyx [00:45:33]: It seems like you're very language model focused. I think that there's an increasing interest in multi-modality in AI, and I don't really know how that is going to manifest. Obviously, GPT-4 Vision, as well as Gemini, both have multi-modal capabilities. There's a smaller subset of open source models that have multi-modal features as well. We just released an episode today talking about IdaFix from Hugging Face, and I would like to understand how people are adopting or adapting to the different modalities that are now coming online for them. What their demand is relative to, let's say, generative images versus just visual comprehension versus audio versus text-to-speech. David [00:46:15]: What do they want? Swyx [00:46:15]: What do they need? And what's the sort of forced, stacked ranked preference order? It's something that we are trying to actively understand because there's this sort of multi-modality world, but really multi-modality is kind of... I've been thinking about this phrase, multi-modality is like cancer. It's this umbrella term for actually a whole bunch of different things that aren't quite honestly not really that related to each other unless in the limit. But it tends towards maybe everything uses transformers and ultimately everything can be merged together with a text layer because text is the universal interface. But if you're given the choice between, I want to implement an audio feature versus I want to implement an image feature versus video, whatever, what are people needing the most? Should we pay the most attention to what is going to be the biggest market for builders to build it? David [00:47:03]: I don't know. Swyx [00:47:04]: I think I would just kind of zoom out a little bit to just a general founder questions. You have a lot of fans in the founder community. I think you're just generally well-known as a very straightforward, painstaking person about just business. Something that is the perception from Joseph is that you have been notably sales-led in the past. That's his perception. I actually never got that, but I'm not that close to your sales portion. And it's interesting to understand your market, the internal tooling market versus all the competition that's out there. There's a bunch of open source retools and there's a bunch of... I don't know how you categorize the various things out there, but effectively what he's seeing and what he's asking is, how do you manage between enterprise versus ubiquity? Or in other words, enterprise versus bottom-up, right? I was actually surprised when he told me to ask that question, because I had always assumed that you were a self-serve, sign-up, bottom-up led. But it seems like you have a counter consensus view on that. David [00:48:04]: Yeah. So actually when Retwelf first started, we started mostly by doing sales, actually. And the reason we started by doing sales was mostly because we weren't sure whether we had product-market fit and sales seemed to be the best way of proving whether we had product-market fit out. Because I think this is true of a lot of AI projects. You can launch a project and people might use it a bit and people might stop using it and you're like, well, I don't know. Is that product-market fit? Is that not? It's hard to say, actually. However, if you work very closely with the customer in a sales-led way, it's easier to understand their requests, understand their needs, and stuff like that, and actually go build a product that actually serves them really well. And so basically, we viewed sales as like working with customers, basically, which is like, I think actually quite a, I think it's a better way to describe it, what sales is of an early-stage company. And so we did a lot of that, certainly, when we got started. I think we, over the last maybe five years, maybe like three years ago, four years ago, something like that, I think we have invested more on the self-serve ubiquity side. And the reason for that is when we started Retwelf, we always wanted, actually, some percent of software to get built inside of Retwelf, whether AI software or origin software or broadly UIs and whatnot, but like software, basically. And for us, we're like, we think that maybe one day, 10% of all the code in the world could be written inside of Retwelf, actually, or 10% of the software could be running on Retwelf, which would be really, really cool. And for us to achieve that vision, it really does require a broad-based option of the platform. It can't just be like, oh, only like 1,000 customers, but the largest 1,000 companies in the world use it. It has to be like all the developers in the world use it. And for us, there's like, well, I think 25, 30 million developers in the world. That's of course, how do you get to all the developers? And the only way to get to those developers is not by sales. You can't have a salesperson talk to 30 million people. It has to be basically in this sort of moms-up, product-led, Ubiquity kind of way, basically. And so for us, we actually changed our focus to be Ubiquity, actually, last year. So our gold star metric used to always be sort of revenue-generated or revenue-generated. We actually changed it to be number of developers building on the platform, actually, last year. And that, I think, was actually a really clarifying change because obviously, revenue was important. It funds a lot of our product and funds the business. But we're going to fail if we aren't able to get to something like 10, 20, 30 million developers one day. We can't convince all developers that Retool's a better way to build a sort of class of software, let's say, internal applications for today. And so I think that has been a pretty good outcome. I think about the last five years of Retool. I think the starting off with sales, so you can build revenue, and then you can actually build traction, and you can hire more slowly. I think it was really good. I do think the focus towards bottoms-up Ubiquity also was really important because it helps us get to our long-term outcome. What's interesting, I think, is that long-term Ubiquity actually is harder for us to achieve outside of Silicon Valley. To your point, I think at Silicon Valley, Retool is reasonable Ubiquitous. I think if you're starting a startup today and you're looking to build an internal UI, you're probably going to consider Retool, at least. Maybe you don't choose it because you're like, I'm not ready for it yet or something. But you're going to consider it, at least. And when you want to build it, I think it's actually a high probability you will actually end up choosing it. It's awesome. But it's that if you think about a random developer working at, let's say, like an Amazon, for example. Today at Amazon, actually, we have, I think, 11 separate business units that use Retool at this point, which is really awesome. So Amazon is actually a big Retool customer. But the average here at Amazon probably has never heard of Retool, actually. And so that is where the challenge really is. How do we get, like, I don't know, let's say 10,000 developers at Amazon building via Retool? And that, again, I think is still a bottom-up ubiquity thing. I don't think that's like a, I don't think we're going to like, you know, go to Amazon and knock on every developer's door or send out an email to every developer and be like, go use Retool. They're going to ignore us, actually. I think it has to be, use the product, and you love it, you tell your co-worker about it. And so for us, a big bottom-up ubiquity, but marrying that with enterprise or the community business has been something that's really near and dear to our hearts. Swyx [00:51:54]: Yeah, just like general market thoughts on AI. Do you spend a lot of time thinking about like AGI stuff or regulation or safety What interests you most, you know, outside of the Retool context? David [00:52:07]: There's a lot of hype in AI right now. And it's again, not too many use cases. So for us, at least from a Retool context, it really is, how do we bring AI and have it actually meet business problems? And again, it's actually pretty hard. Like I think most founders that I've met in the AI space are always looking for use cases, never have enough use cases, right? Sort of real use cases, people pay money for them. But I think really where the Retool interest comes from, me personally, I think philosophically, yeah, I've been thinking recently myself a bit about sort of intentionality and AGI and like, you know, what would it take for me to say, yes, you know, GPT-X for, you know, any sort of model actually is AGI. I think it's kind of challenging because it's like, I think if you look at like evolution, for example, like humans have been programmed to do like three things, if you will, like, you know, we are here to survive, you know, we're here to reproduce and we're here to like, you know, maybe this is just two things, I suppose. So basically, to survive, you have to go eat food, you know, for example. To survive, maybe like having more resources helps you want to go make money, you know, for example. To reproduce, you should go date, you know, or whatever, you get married and stuff like that, right? So like, that's, we have a program to do that. And humans that are good at that have propagated. And some humans that, you know, we're not actually surviving, probably have disappeared just due to natural selection. Humans that we're not interested in producing also disappeared because there are less of them, you could say, because they just, they just stopped carrying on basically. And so, so it almost feels like humans have sort of naturally self-selected for these like two aims. I think the third aim I was thinking about was like, does it matter to be happy? Like, maybe it does. So maybe like happier humans, you know, survival, it's hard to say. So I'm not sure. But if you think about that, and they're all just like AIs, if you will, right now, we're not really selecting AIs for like, you know, reproduction. Like, it's not like, you know, we're being like, hey, AI, you know, you should go make 30 other AIs. And you know, those that make the most AIs, you know, are the ones that survive. We're not saying that. So it's kind of interesting sort of thinking about where intentionality for humans come from. And like, I think you can argue the intentionality of the human space that comes out of these three things. You know, like, if you want to be happy, you want to survive, you want to reproduce. That's like basically your sort of goal, you know, in life. Whereas like, the AI doesn't really have that. But maybe you could program it in. Like, if you, you know, prompt inject, for example, like, hey, AI, you know, go do these three things. And you can even create a simulation, if you will, like all these AIs, you know, in the world, for example. And maybe you don't have AGI in the world, which I think is kind of interesting. So that's kind of stuff I've been thinking about when I talk about with some of my friends from a sort of philosophical perspective. But yeah, it's kind of interesting. Swyx [00:54:29]: Yeah, my quick response to that is we're kind of doing that. Maybe not at the sort of trained final model level, but at least at the data sets level, there's a lot of knowledge being transferred from model to model. And if you want to think about that sort of evolutionary selection pressure, it is happening in there. And, you know, I guess one of the early concerns about being in Sydney and sort of like bootstrap self bootstrapping AGI is that it actually exists if these models are sentient, it actually exists in their incentive to get as much of their data out there into our data sets so that they can bootstrap themselves in the next version as they get trained. That is a scary sobering that we need to try to be on top of. Alessio [00:55:13]: David, I know we're both fan of Hofstadter's GB and actually saw in one of your posts on the Segovia blog, you referred to the anteater. I don't even know if you call them chapters and GB is just kind of like this, this continuous rift. But basically, like our ants are like not intelligence, but like ant colony has signs of intelligence. And I think Hofstadter then use that to say, hey, you know, neurons are kind of like similar and then computers maybe will be the same. I've always been curious if like we're drawing the wrong conclusion from like neural networks where people like, oh, each weight is like a neuron and then you tie them together should be like a brain. But maybe like the neuron is like different models that then get tied together to make David [00:55:57]: the brain. Alessio [00:55:57]: You know, we're kind of looking at the wrong level of abstraction. Yeah, I think there's a lot of interesting philosophical discussions to have. Sean and I recorded a monthly recap podcast yesterday, and we had a similar discussion on are we using the wrong? What did you say Sean on the plane and the bird? I think that was a good analogy. Swyx [00:56:16]: The sour lesson, are we using the wrong analogies? Because we're trying to be inspired by human evolution and human development, and we are trying to apply that analogy strictly to machines. But in every example in history, machines have always evolved differently than humans. So why should we expect AI to be any different? David [00:56:33]: Yeah, it is interesting because it does feel like, yeah, if you sort of peer under the hood of AGI, if you insist that AGI, we'd have always used AGI for things like a human. And that is the Turing test, I suppose. But whether that is a good point, like if it works, no, it's not the Turing test. The Turing test is if the output is the same as a human, then I'm happy. I don't really care about what's going on inside. And so it feels like caring about the inside is like a pretty high bar. Like, why do you care? It's kind of like the plane thing for flies. It's not a bird. I agree. It does not fly necessarily the same way as a bird. Physically, it does, I suppose. But you see what I mean? It's not the same under the hood. But it's OK for the flies. That's what I care about. And it does seem to be like AGI probably doesn't think and can achieve outcomes that I give it. It can achieve its own outcomes. And if it can do that, I kind of don't care what it is under the hood. It may not need to be human life at all. It doesn't matter to me. So I agree. Awesome. Alessio [00:57:26]: No, we kept you long. Actually, I have GUB right here on my bookshelf. Sometimes I pick it up and I'm like, man, I can't believe I got through it once. David [00:57:34]: It's quite the piece of work. It's a lot of fun, though. Yeah. Alessio [00:57:38]: I mean, I started studying physics in undergrad. So, you know, it's one of the edgy things that every physicist starts going through. But thank you so much for your time, David. This was a lot of fun. And looking forward to the 2024 state of AI results to see how things change. David [00:57:54]: Yeah, I'll let you know. Thanks, both. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Alessio Fanelli?

  Hey everyone, welcome to the Late in Space Podcast. This is Alessio, Partner and CTO in Residence at Decibel Partners, and I'm joined by my co-host Swixx, founder of Small AI.

• 0:16Speaker 01

  And today, we are in the studio with David Su from Rito. Welcome.

• 0:21Speaker 02

  Alex, excited to be here.

• 0:22Speaker 01

  We like to give a little bit of intro from what little we can get about you and then have you talk about something personal. You got your degree in Philosophy and CS from Oxford. I wasn't aware that they did double degrees. Is that what you got?

• 0:34Speaker 02

  It's actually a single degree, which is really cool. So basically, I used to study calculus, I used to study philosophy, and I used to do the intersection. Intersection is basically AI, actually, in sort of computers think, are computers be smart? What does it mean for a computer to be smart? As well as logic, it's also another intersection, which is really fun too.

• 0:51Speaker 01

  In Stanford, it might be symbolic systems or whatever. And it's always hard to classify these things when we don't really have a word for it. Now, I guess everything's just called AI. Five years ago, you launched Rito. You were in YC, Winter 17, and it's just been a straight line up from there, right?

• 1:09Speaker 02

  I wish.

• 1:10Speaker 01

  What's something now you're linking that people should know about? You may be on their personal hobby, or you know, it's just something you're very passionate about.

• 1:17Speaker 02

  Yeah, sure. I read quite a bit, I probably read like two books a week, or about that. So it's a lot of fun. I love biking, it's also quite a bit of fun, so yeah.

• 1:24Speaker 01

  Do you use Rito to read? Like, what the hell?

• 1:27Speaker 02

  No, I don't use Rito to read. That'd be funny.

• 1:30Speaker 01

  What do you read? How do you choose what you read? Any recommendations?

• 1:33Speaker 02

  I'm mostly reading fiction nowadays. Fiction's a lot of fun. I think it maybe helps to be more empathetic, if you will. I think it's a lot of fun. I actually just want to see what it's like to be in someone else's shoes. So that's a lot of fun. I've heard that I read a good amount of philosophy as well. I find philosophy just so interesting, especially logic. We can talk more about that for probably hours if you want, so.

• 1:51Speaker 01

  Yeah, I have a casual interest in epistemology and I think that anytime you try to think about machine learning in a, on a philosophical angle, you have to start wrestling with these like very fundamental questions about how do you know what you know?

• 2:04Speaker 02

  Yeah, totally. What does it mean to know?

• 2:07Alessio Fanelli?

  That's its own pockets. We should do a special edition about it, but that's a, that's fun. Let's just maybe jump through a couple of things on Rito that I found out while researching your background. You did Y C, but you didn't present a demo day initially because you were too embarrassed of what you had built. Can you maybe give any learnings to like founders on like jumping back from that? I've seen a lot of people kind of like give up early on because they were like, oh, this isn't really what I thought it was gonna be to be a founder. They told me I would go to Y C and then present and then raise a bunch of money and then everything was gonna be easy. So how did that influence also how you build ritual today, you know, in terms of like picking ideas, deciding when to give up on it?

• 2:46Speaker 02

  Yeah, let's see. So this is around about 2017 or so. So we were supposed to present at the March Demo Day, but then we basically felt like we had nothing really going on and no traction, we had no customers. And so we're like, okay, well, why don't we take six months to go find all that before presenting? Part of that, to be honest, was I think there's a lot of noise around Demo Day, around startups in general, especially because there's so many startups nowadays. And I guess for me, I'd always wanted to sort of under promise and over deliver, if you will. And in Demo Day, I mean, maybe you two have seen a lot of videos, like it's a lot of honestly over promising and our sort of delivering. Because every startup, you know, says, oh, we are, you know, I'm gonna be the next Google or something. And then you peer on under and you're like, well, nothing's going on here.

• 3:28Speaker 02

  So I really didn't want that. And so we chose actually not to present the Demo Day mostly because we felt like we didn't have anything substantial underneath. Although actually a few other founders in our batch probably would have, you know, chosen to present in this situation, but we were just, you know, kind of embarrassed about it. And so we basically took some months to just say, okay, well, how do we get customers? And we're not presenting until we have, you know, a product that we're proud of and customers that we're proud of. And fortunately it worked out, you know, since it was later, we did have that. So I don't know if there's much to learn from this situation besides I think social validation. Something that I personally had never really been that interested in. And so it was definitely hard because it's almost like you go to college and all your friends are graduating.

• 4:10Speaker 02

  When you failed or something, you failed the final, you have to like redo it here. And so, well, it kind of sucks that all your friends are up there and on the podium now presenting. And they are raising a ton of money and you're kind of being left behind. But in our case, we felt like it was a choice and we could have presented if we really wanted to. But, you know, we would not have been proud of the outcome or proud of what we're presenting. And for us, it was more important to be true to ourselves, if you will, and show something that we're actually proud of. Rather than just, you know, raise some money and then shut the company down in two years.

• 4:37Alessio Fanelli?

  Yeah. Yeah. Any Sam Altman stories from the Y C days? Could you tell in 2017 that Sam was gonna become like run the biggest A I company in the world?

• 4:48Speaker 02

  Wow. No one's asked me that before. Ah, let me think.

• 4:53Speaker 02

  Sam was, I think he was, I want to, I forgot, I heard maybe president of Y C in our batch. We actually were in his group actually at the very beginning. And then we got moved to a different group. I think Sam was clearly very ambitious when we first met him. I think he was very helpful and sort of wanted to help founders. But besides that, I mean, I think we were so overwhelmed by the fact that we had to go build a startup that we're not honestly paying too much attention to every Y C partner taking notes on them. So.

• 5:20Alessio Fanelli?

  That makes sense. Well, and then just to wrap some of the Ritual history nuggets, you raised a Series A when you were at one million in revenue with only three or four people? How did you make that happen? Any learnings on keeping teams small? I think there's a lot of over hiring we've seen over the last few years. I think a lot of A I startups now are raising very large rounds and maybe don't know what to do with the capitals.

• 5:41Speaker 02

  So this is kind of similar actually from sort of why we chose not to present a demo day. And the reason was it feels like a lot of people are really playing startup. I think P G has an essay on this, which is like, you're almost like playing house or something like that. Like, it's like, oh, well, I hear that in a startup, you're supposed to raise money and then hire people. And so therefore you go and do that. And you're supposed to, you know, do a lot of P R because that's what, you know, startup founders do. And so you could do a lot of P R and stuff like that. And for us, we always thought that the point of starting a startup is basically you have to create value for customers. If you're not creating value for customers, like everything else is gonna. Nothing's gonna work, basically. You can't continuously raise money or hire people if you don't have customers.

• 6:16Speaker 02

  And so for us, we were always very focused on that. And so that's initially where we started. I think it's, again, maybe goes to like the sort of presenting something truthful about yourself or staying true to yourself, something to that effect, which is, we didn't want to pretend like we had a thriving business we could actually. And so the only way to not pretend was actually to build a thriving business. And so we basically just, you know, put our heads down and, you know, grind it away for probably a year, year and a half or so, just writing code, talking to customers. And I think that at that point, we had raised something like maybe a million dollars, maybe a million and a half, something out of Y C. So, I mean, to us, you know, two people, you know, that was a huge amount of money. I was like, wow, like, how are we ever gonna spend a million and a half?

• 6:56Speaker 02

  Our runway was like, you know, five, six years at that point, right? Because we're paying ourselves thirty, forty K a year. And so then the question was not like, oh, we're gonna run out of runways. The question was like, we better find traction, because if we don't find traction, we're gonna, you know, just give up psychologically. Because if you run an idea for four years and nothing happens, you're probably psychologically gonna give up. I think that's actually true in most startups, actually. It's like, most startups die in the early stages, not because you run out of money, but really because you run out of motivation. And for us, had we hired people, I think it would have actually been harder for us, because we would have ran out of motivation faster. Because when you're pre-part of market fit, actually, trying to lead a team of like, you know, ten people, for example, to marshal a spot on market fit, I think it's actually pretty hard. Like, it's, you know,

• 7:37Speaker 02

  every day people are asking you, so why are we doing this? And you're like, I don't know, man. Like, hey, trust this. And that's actually a very tiring environment to be in. Whereas it's just like, you know, the founders figuring out product market fit, I think, obviously, a much sort of safer path, if you will. You're also schooling less with employees. Like, when you hire employees, you have an idea, you have product market, you have customers. That's actually, I think, a lot more stable for a place for employees to join as well, so.

• 8:00Speaker 01

  Yeah, I find that typically the sort of founder-employee relationship is employee expects the founder to just tell them what to do, and you don't really get critical pushback from the employee, even if they're a body and even if they like you as an early engineer. It's very much like the, the role play of like, once you have that founder hat on, you think differently, you, you act differently, and you're more scrappy, I guess, in trying to figure out what that product is. Yeah, I really resonate with this because I'm going through this right now.

• 8:27Speaker 02

  Awesome. One thing we did actually early on that I think has paid a lot of dividends, especially when it was a lot larger now is we hired a lot of former founders. So I want to say like when we were 20, 30, 40 people, we were probably like half former founders at each one of those stages. And that was actually pretty cool because I think you can fuse sort of a, you know, get things done kind of culture, a outcome oriented culture of like very little politics, because you know, no one came from larger companies. Everyone was just like, this is my own startup. We'll go figure out how to shoot the cycle for the customer. And so I think from a cultural perspective, even today, a lot of those cultures sort of very soft stargery. I think it's actually because of sort of these like, you know, early founders that we hired, which was really, really, you know, we really liked to pat them.

• 9:07Speaker 01

  Yeah. And then closing off on, on just a little bit of the fundraising stuff, something notable that you did was when in 2021, when it was the sort of peak Zerp and everyone was raising hundreds and hundreds of millions of dollars, you intentionally raised less money at lower valuations is your title. And I think it's a testament to your just overall general philosophy in building Retool that you are, you're just very efficient and you do things from first principles. Any updates on like, would you still endorse that? You know, would you recommend that to everyone else? What, what are your feelings sort of two years on from that?

• 9:38Speaker 02

  Yeah. So exactly, you said is correct. We raised less money at a lower valuation. And I think the funny thing about this is that when we first announced that, even, you know, internally and both externally, I think people were really surprised actually because I think Silicon Valley has been conditioned to think, well, raising a giant sum of money at a giant valuation is a really good thing. So like, you know, you should maximize both the numbers, basically. But actually maximizing both the numbers is actually really bad, actually, for the people that matter the most, you know, i.e. your employees or your team. And the reason for that is raising more money means more dilutions. If you look at, you know, a company like, you know, let's say Uber, for example, if you join Uber at like, I don't know, like a $10 billion valuation, or let's say join before the huge round, which I think happened at a few billion dollars valuation, you just got diluted

• 10:21Speaker 02

  a ton when Uber fundraised. So if, you know, Uber raises, if Uber delivers themselves like 10%, for example, let's say it raises 5.2 to 5 billion, for example, I think employee stake goes down by 10% in terms of ownership. Same with, you know, previous investors, same with the founders, et cetera. And so if you look at actually a lot of founders in sort of, you know, the operations logistics space, or, you know, those that fundraised like, you know, 2013, 2017, a lot of the founders by IPO only have a few percentage points actually for the company. And if founders only have a few percentage points, you can imagine how, you know, how little employees have. And so, but I think this is actually just a really, you know, backing for employees overall. Secondly, sort of higher valuation, given the same company quality, is always worse. So basically what that means is if you are fundraising as a company, you could command a certain valuation in the market. You know, let's say it's, you know, X, for example.

• 11:06Speaker 02

  Maybe you get lucky and you can raise two times X, for example. But if you choose two times X, your company, it's almost not fundamentally changed. It's just that, you know, for some reason investors want to pay more for it. You know, maybe today you're an AI company, for example, and so investors aren't really excited with AI, will pay more for it. However, that might not be true in a year or two years' time, actually. And if that's not true in two years' time, then you're in big trouble, actually. And so now I think you see a lot of companies that raised really high valuations about 2021. And now they're like, man, we're at like 100X or, you know, we raised 300X multiple, for example. And if we're at 300X then, you know, maybe now we're at like 200X. Man, we just can't raise money ever again. Like, you don't have to grow like 50X to go raise money, reasonable valuation, I would say. And so I think that is really challenging and really demotivating for the team. And so I think a lower valuation actually is much better.

• 11:51Speaker 02

  And so for us, in retrospect, to answer your question, two years later, we did not predict, you know, the crash, if you will. Uh, but given it, I think we've done extremely well, mostly because our valuation is not sky high. Because if our valuation was sky, were sky high, I think we'd have a lot more problems. We'd probably have recruiting problems, for example, we'd probably have a lot of internal raw problems, et cetera. People will be like, you know, why is the valuation this way? We might have cash flow problems because we might have to go raise money again, you know, et cetera. But we can't because the relation is too high. So I would urge, I think, founders today to, quote unquote, like, leave money on the table. Like, there are some things that are not really worth optimizing. I think you should optimize for the quality of the company that you build, not like the valuation you raise that or the amount you raise, et cetera. So.

• 12:34Speaker 01

  Hindsight's twenty twenty, but it looks like, you know, you, you, you made the right call there anyway.

• 12:39Speaker 01

  Maybe we should also, for people who are not clued into Retool, do a quick, like, what is Retool? You know, I, I see you as the kings or the inventors of the low-code internal tooling category. Would you agree with that statement? You know, how do you usually explain Retool?

• 12:53Speaker 02

  I generally say it's like legless for code. We actually hate the low code moniker. We actually never, in fact, we have docs saying we will never use it internally or even to customers. And the reason for that is I think low code sounds very dot developer. And developers that hear the phrase low code are like, oh, that's not for me. Like, I love writing code. Like, why would I ever want to write less code? And so for us, we're just actually built for developers. Like 95% of our customers actually are developers, actually. And so that is a little bit surprising to people. I'll generally explain it is, um, this is, you know, kind of a funny joke too. I think part of the reason why Retool has been successful is that developers hate building internal tools. And you can probably see why. I mean, if you're a developer, you're probably building internal tools yourself. Like, it's not a super exciting thing to do.

• 13:35Speaker 02

  You know, it's like seeing together a quid UI, probably, you know, piece together many quid UIs in your life before. And there's a lot of grunt work involved. You know, it's like, hey, stake management, it's like, you know, data validation, it's like displaying error messages, like the bouncing buttons. Like, all these things are not really exciting, but you have to do it. Because it's so important for your business to have high quality internal software. And so what Retool does is basically allows you to sort of piece together an internal app really fast. Whether it's a front end, whether it's a backend or whatever else. Um, so yeah, that's what Retool is.

• 14:03Speaker 01

  Yeah, actually, you started hiring, and so I do a lot of developer relations and community building work. And you hired Cryptica, who has now moved on to OpenAI, to start out your sort of DevRel function. And I was like, what is Retool doing coding developers? And then she told me about this, you know, developer traction. And I think that is the first thing that people should know is, which is that the burden and weight of internal tooling often falls to developers. Or it's an Excel sheet somewhere, whatever. But yeah, you guys have, it basically creates this market. You know, in my mind, I don't know if there was someone clearly before you in this, but you know, you've clearly taken over and dominated. Every month, there's a new YC startup launching with, that it's like, you know, we're, we're the, you know, open source Retool. We're like the low, lower code Retool, whatever. And it's, it's pretty, I guess it's endearing, you know?

• 14:47Speaker 01

  We'll talk about Airplane later on. But yeah, I think I've actually used Retool, you know, in my, in my previous startups for this exact purpose. Like, we needed a UI for, you know, it was RDS that they can, you know, like the rest of our non, less technical people, like our sales operations people could, could interact with. Then yeah, Retool was perfect for that.

• 15:04Speaker 02

  Yeah, it's a good example of like, that's a, you know, application that an engineer probably does not want to build. Like building an app on top of Salesforce or something that is not exciting. It's almost unfair, sucks, it's red limited, you know, it's like not a fun experience at all. But piecing it together at Retool is, you know, quite a bit easier. So yeah, let me know if you had any feedback, but I'll see you.

• 15:21Speaker 01

  Yeah, no, of course. Like more recently, I think about three, four months ago, you launched Retool AI. Obviously, AI has been sort of in the air. I'd love for you to tell the journey of AI product ideation within Retool. Given that you have a degree in this thing, I'm sure you're not new to this, but like, when would you consider sort of this, the start of the AI product thinking in Retool?

• 15:41Speaker 02

  So we actually had a joke internally at Retool. We are part of roadmap for every year. It was like 2019 or something. We had this joke, which was like, what are we going to build this year? We're going to build AI programming is what we always said as a joke. And so, but it was funny because we were like, ah, that's never going to happen. But like, that's out of the ship because it's like a buzzwordy thing that enterprises love. So let's look at it. And so it was almost like a funny thing, basically. But it turns out, you know, we're actually doing that now. So it is pretty cool. So I would say maybe AI thinking on Retool probably first started, maybe like, I don't see, maybe, you know, a year and a half ago, something like that. And when we first started thinking about it, sort of in a, you know, philosophical way, if you will, it's like, well, what is the purpose of AI? And how can it help, you know, what Retool does? And there were two sort

• 16:25Speaker 02

  of main prompts, if you will, value that one was helping people build apps faster. And so you've probably seen Copilot, you've seen sort of so many other coding assistants, Pisa, and stuff like that. So that's interesting because, you know, engineers, as we talked about, do some grunt work. And grunt work, you know, maybe could be automated by AI was sort of the idea. And it's interesting. So we actually, I would say, kind of proved or disproved the hypothesis a little bit. If you talk to most engineers today, like a lot of them is to use Copilot. But if you ask them like, how much time does Copilot save you? It's not like coding is 10x faster than before. You know, coding is maybe like 10% faster, maybe 20% faster, or something like that, basically. And so it's not like a huge step change, actually. And the reason for that is, we think, is because this sort

• 17:08Speaker 02

  of fundamental frameworks and languages have not changed. And so if you're building, let's say, you know, like the sales ops tool we were talking about before, for example. Let's say you've got AI to generate you a first version of that, for example. The problem is that it probably generated it for you in like JavaScript, you know, writing for the web browser, for example, right? And then for you to actually go proofread that JavaScript, for you to go read the JavaScript to make sure it's working, you know, to fix the subtle bugs that AI might have caused. Uh hallucinations, stuff like that, actually takes a long time and a lot of work. And so for us, the problem is actually not like the process of coding itself. It is more sort of the language or the framework. I think it's like way too low level. It's kind of like, you know, like punched cards, like let's say back in the day, you could just like punch cards. And AI could help you generate punched cards. Okay, you know, I guess that helps me punching cards is a little bit faster now because I have a machine punching them for me.

• 17:54Speaker 02

  But like when there's a bug, there's something to go read all the punched cards and figure out what's wrong, right? It's like, it's a lot of work actually. And so for us, that was the sort of initial idea was can we help engineers code faster? You know, I think it's somewhat helpful to be clear. Like again, I think it's 10 or 20%. So we have things like, you know, we can generate school careers by AI and generate UIs by AI and stuff like that. So that's cool to be clear. But it's not, I think, the step change. That I think is, you know, the, we're investing somewhat in that, but the bulk of investment actually is a number two, which is helping developers build AI-enabled applications faster. And the reason why we think this is so exciting is we think that practically every app, every internal app especially, is going to be AI-infused over the next like three years.

• 18:35Speaker 02

  And so every tool you might imagine, so like the tool you were even mentioning, like a sales operations tool, for example, probably, you know, if you were to build it today, one of corporate self-form of AI. And so, you know, we see today, like for us, like a lot of people build, you know, let's say sales manager tools of retail. An example is there's a fortune, like a financial company is building like a sales forecasting tools. So they basically have salespeople enter their forecast, you know, for the quarter, uh beginning of a quarter, like, hey, I have these deals. And these deals are going to close, these deals are not going to close, knocking up upside in these, downside in these, stuff like that basically. So you can imagine it's pulling in deals from your Salesforce database. And so it pulls in the deals that actually use AI to compute like, okay, well, you know, given previous deal dynamics, like these are the deals that are more likely to close this month versus next month versus this quarter, next quarter, et cetera.

• 19:19Speaker 02

  And so it could actually, you know, pre-write you a draft of, you know, your report basically. And so that's an example where I think all apps, whether it's, you know, a sales app, you know, until, uh, it looks like fraud app, a, you know, fintech app, you know, whatever it is basically, especially internal apps, I think, like you said, Alessio, in order to make you all productive, it's going to incorporate some form of AI. And so another question is, can we help them incorporate AI faster? And so that's why we launched like a vector database, once and we'll build directly into retool. That's why we, you know, launches all these AI actions. So you don't have to, you know, go figure out what the best model is and do testing and stuff like that. We'll just, you know, give it to you out of the box. So for us, I think that is really the really exciting future is can we make every app, you know, retool, use AI a little bit and make people a little productive. So.

• 20:00Alessio Fanelli?

  We talked with Jeffrey Wang, who's the co-founder and chief architect of Amplitude. He mentioned that you just use PostCraft's vector. When you were building retail vectors, how do you think about leveraging a startup to do it? Putting vectors into one of the existing data stores that you already had? I think you're really a quite large customer scale. So like you're maybe not trying to get too cute with it. Any learnings and tips from that?

• 20:24Speaker 02

  Yeah, I think generally the philosophical thing I think we believe is we think the open source movement in AI, especially when it comes to all the supporting infrastructure, is going to win. And the reason for that is we look at like developer tools in general, especially such a fast-moving space. In the end, like there are really smart people in the world that have really good ideas. And I'm going to go build companies and I'm going to go build projects basically around these ideas. And so for us, we have always wanted to partner with maybe more open source providers or projects, you could say, like PG Vector, for example. And the reason for that is it's easy for us to see what's going on under the hood. A lot of this stuff is moving very fast. All the time there are bugs, actually, and so we can go look into explosives ourselves, like Cottrell's back, for example.

• 21:07Speaker 02

  But we really think open source is going to win in this space. It's hard to see about models. I don't know about models necessarily, because it's still pretty complicated there. But when it comes to tooling, for sure, I think there's just like so much. There's an explosion of creativity, if you will. And I think betting on any one commercial company is pretty risky. But betting on the open source sort of community and the open source contributors, I think is a pretty good bet. So that's why we have a different computer.

• 21:30Alessio Fanelli?

  Awesome. And we're going to jump into the survey next, but we're going to put a bunch of links in the show notes about retail AI and whatnot. Is there any most underrated feature, like something that customers maybe love that you didn't expect them to really care about? I know you have a like text to SQL, you have UI generation, there's like so many things in there. Yeah, what surprised you?

• 21:50Speaker 02

  Yeah, so what's really cool, and this is my sense of the AI space overall, you know, if you're scared to take some YouTube as well, is that especially in Silicon Valley or a lot of the innovation is happening, I think there's actually not that many AI use cases, to be honest. And AI to me, even as of what, like January nineteenth, yeah, nineteenth of twenty twenty four, still feels like in search of truly good use cases. And what's really interesting though about Retool, and I think we're in a really fortunate position, is we have this large base of sort of customers, and a lot of these customers are actually a much more legacy, if you will, customers. And a lot of them actually have a lot of use cases for AI. And so to us, I think we're almost in like a, you know, really, you know, perfect or unique spot.

• 22:34Speaker 02

  We're able to adopt some of this technology and then provide them to, you know, some, some of these like older players. So one example that actually really shocked and surprised me about AI was some, so we have this one clothing manufacturer, I think it's, I think it's either the first or second largest clothing manufacturer in the world who's using Retool. And you know, they're a ginormous company with, you know, uh, really multinational, you know, stores on, you know, pretty every model in the world. And so they have one problem, which is they need to design styles every year for the next year, basically, for every season. So like, hey, just like summer twenty twenty four, for example, and what are we going to design? And so what they used to do before is they would hire designers, and designers would go to study data, they'd be like, okay, well, it looks like, you know, the floral patterns are really hot. And like, you know, California, for example, in twenty twenty three, and like, do I think it's going to be hot in twenty twenty four?

• 23:17Speaker 02

  Well, let me think about it, I don't know, you know, so let me, maybe, and if so, if I believe it is going to be hot, let me go design some floral patterns, actually. And what they ended up doing in Retool, actually, is they actually automated a lot of this process away in Retool. So they actually now built a Retool app that allows actually a non-designers, like an analyst, if you will, to analyze, like, you know, who are the hottest selling patterns, you know, particular geos. Like this was really hot in Brazil, it was really hot in China, it's really hot, you know, somewhere else, basically. And then they actually feed it into an AI and the AI, you know, actually generates. With Dolly and other image generation APIs, I actually generates patterns for them. And they put the patterns, which is really cool. And so that's an example of like, honestly, the use case I would have never thought about. Like, thinking about like, you know, how clothing manufacturers create their next line of clothing, you know, for the next season.

• 24:01Speaker 02

  Like, I don't know, I, I never thought about it, to be honest. Nor did I ever think, you know, how it would actually happen. And the fact that they're able to leverage AI, they actually, you know, leverage multiple things in retail to make that happen, is really, really, really cool. And so that's an example where I think if you go deeper into sort of, if you go outside of Silicon Valley, there are actually a lot of use cases for AI. But a lot of this is not obvious, like you have to get into the businesses themselves. And so I think we're, we personally are in a really fortunate place. But if, you know, you're working in the AI space and want to find some use cases, please come talk to us. Like, you know, we're really excited about marrying sort of technology with use cases, which I think is actually really hard to do right now, so it's bad, so.

• 24:41Speaker 01

  Yeah, you know, I have a bunch of like sort of standing presentations around like how this industry is developing. And like, I think the foundation model layer is understood. The langchain vector DB reg layer is understood. I always have a big question mark and actually have you and Vercel V0 in that box, which is like sort of the UI layer for AI. And like, you know, you are perfectly placed to expose those functionalities to end users, even if you personally don't really know what they're going to use it for. And sometimes they'll surprise you with their creativity. One segment of this, I do see some startups springing up to do this. It's related to the things that, to something that you also build, but it's not strictly AI related, which is retool workflows,

• 25:25Speaker 01

  which is the sort of canvassy boxes and arrows, point and click, do this, then do that type of thing, like, which, which every, what are we calling low code, uh, let's, let's, every internal tooling company eventually builds. You know, I, I worked at a sort of workflow orchestration company before, and we were also discussing internally how to make that happen. But you, you are, you are obviously very well positioned to, to that. Yeah, basically, like, do you think that there is an overlap between retool workflows and AI? I think that, uh, you know, there's, there's a lot of interest in sort of chaining AI steps together. I couldn't tell if like that is already enabled within retool workflows, I don't think so. But you could, you could sort of hook them together, it's kind of jankily. Like, what's the interest there, you know, is it all of a kind ultimately in your, in your mind?

• 26:09Speaker 02

  It is a hundred percent our time. And yes, you could actually already, so a lot of people actually are building AI workflows down very tall, which is, we could talk about in a second. But a hot take here is actually, I think a lot of the utility in AI today, I would probably argue sixty, seventy percent of the utility, like, you know, businesses I found in AI, is mostly via ChatGPT and across the world too. And the reason for that is I think the ChatGPT sort of UI, you could say, or interface, or user experience is just really quite good. You know, you can sort of converse, you know, with an AI basically. But that said, there are downsides to it. If you talk to like a giant company like a JP Morgan Chase, you know, for example, they may be reticent to have people copy paste data into ChatGPT, for example, even on ChatGPT enterprise, for example.

• 26:51Speaker 02

  Some problems are that I think chat is good for one-off tasks. So if you're like, hey, I want a first version representation or something like that, you know, and help me write this first version of Doc or something like that. Chat is great for that. It's a great, you know, very portable, you know, if you will, form factor. So you can do that. However, if you think about it, you think about some kind of productivity more generally, like chat, again, will help you like ten or twenty percent, but it's unlikely that you're going to replace an employee with chat. You know, you're not going to be like, oh, I have a relationship manager at JP Morgan Chase and I've replaced him with a, you know, AI chatbot. Like, it's kind of hard to imagine, right? Because like, the employees are actually doing a lot of things besides, you know, just, you know, generating, you know, maybe another way of putting it is like, chat is like a reactive interface. Like, it's like when you have an issue, you'll go reach out to chat and chat might solve it.

• 27:34Speaker 02

  But like, chat is not going to solve a hundred percent of your problems. It'll solve like, you know, twenty-five percent of your problems, like, pretty quickly, right? And so what we think for the next, like, big breakthrough in AI is, is actually like automation. It's not just like, oh, I have a problem when they go to a chatbot and solve it. Because like, again, like, people don't spend forty hours a week in a chatbot. This time they've been like, two hours a week in a chatbot, for example. And so what we think can be really big, actually, is you're able to automate entire processes via AI. Because then you're really raising the potential of AI. It's like, not, it's not just like, you know, a human copying, pasting data into an AI chatbot and, you know, pasting it back out or copying it back out. Instead, it's like, the whole process now is actually done in an automated fashion without the human. And that, I think, is what's going to really unlock sort of big kind of productivity, or, um, that's what we're really excited about.

• 28:18Speaker 02

  And I think part of the problem right now is, you know, I'm sure you all thought a lot about agents. Agents are actually quite hard because like, you know, the AIs were only, you know, 2% of the time. But then you like, you know, a few, let's say, you know, raise it to the power of seven, for example. That's actually wrong, you know, quite often, uh, for example. And so what we've actually done with workflows is we prefer, we've learned actually is that we don't want to generate the whole workflow for you. By AI, instead, what we want you to do actually is what we want you to actually sort of drag and drop the workflow yourself. And maybe you can get a V zero or something by AI, but it's coded, basically. You should actually be able to modify the steps yourself, but every step can use AI. And so what that means is like, it's not the whole workflow is created by AI. Instead, like every step is AI automated. And so if you go back to, for example, like the users are talking about, you know, with a coding manufacturer, that's actually a workflow, actually.

• 29:03Speaker 02

  So basically what they say is, hey, every day we eat all the data now from our sales systems into our database. And then we, you know, do some data analysis and, you know, that's just, you know, raw SQL, basically, it's nothing too surprising. And then they use AI to go generate new ideas. And then the analysts will look at the new ideas to prove or reject them, basically. And that is like a, you know, that's true automation, you know? It's not just like, you know, a designer copying, pasting things as a chat GPT, be like, hey, you know, give me a design. It's actually designs are being generated, they're generating kind of designs every day. And then you have to go and approve or reject these designs, which I think is a lot, you know, that's a lot more economically productive. They're just copying, pasting stuff in chat quickly. So we think sort of the AI workflow space is a really exciting space. And I think that is the next step in sort of delivering a lot of business value by AI.

• 29:49Speaker 02

  I personally don't think it's, you know, higher chatter, you know, of agents quite yet, so.

• 29:53Speaker 01

  That's a pretty reasonable take. It's, it's, um, disconcerting because like, I know a lot of people are trying to build what you already have in workflows.

• 30:03Speaker 01

  You're, you're the incumbents sort of in their minds. I'm sure it doesn't feel that way to you, but like, I'm sure, I'm sure, you know, you're the incumbent in their minds and they're like, okay, like, how do I, uh, you know, compete with retool? Or, you know, differentiate from retool? As you mentioned, you know, all these connections, it does remind me that you're running up against Zapier. You're running up against maybe Notion in the distant future. And yeah, I, I think that there'll be a lot of different takes at this space and like whoever is best positioned to serve their customer in, in the way that they, they, they sort of need to shape this, uh, is going to win. Do you have a philosophy against around like what you won't build? Like what do you prefer to partner and, and not build in-house? Because it seems, I feel like you build a lot in-house.

• 30:41Speaker 02

  Yes, this will be two philosophical days. So one is that we're developer first, and I think that's actually one big difference. You know, like Austin Zachary, you know, we're so very rare to see them, actually. The reason is we're developer first. Because developers, like, if you're like building a sales ops tool, you're probably not considering notion if you're a developer. You're probably like, I want to build this by reacting to clean or user tool. And so are you we build for developers? It's pretty interesting, actually. I think one huge advantage of some of the developers is that developers don't want to be given an institution. They want to be given the building blocks so they can do themselves go build the institution. And so for us, like, interesting point that equilibrium we don't get to is basically if you say, hey, we're just a consulting company. And we basically build apps for everybody, for example. And what's interesting is we've actually never gotten to equilibrium. And the reason for that is with some of the developers.

• 31:24Speaker 02

  Developers don't want, you know, like a consultant coming in and building all the apps for them. Developers like, hey, I want to do it myself. Just give me the building blocks. Give me the best table library. Give me, you know, good state management. Give me an easy way to create a rest in peace. And I'll do it myself, basically. So that's what I think is pretty. So we generally end up basically always building building blocks that are reusable by multiple customers. We have, I think, basically never built anything specific for one customer. So that's one bit interesting. A second thing is when it comes to sort of, you know, let's say like in AI space, we won't be able to build, we're not going to build. We basically think about whether it's all core competency or whether there are unique advantages to us building this or not. And so we think about the workflows product. We think workflows actually is a pretty core competency for us. And I think the idea that we could build a developer-first workflows automation engine.

• 32:07Speaker 02

  I mean, I, I think after we released, you know, workflows, we do workflows. There have been a sort of few copycats that are, I think, quite, quite far behind, actually, be sort of are missing a lot of, I think, more critical features. But like, if you look at the space, it's like Zapier on one side, and then maybe like Airflow on the other. And so virtual workflows actually is fairly differentiated. And so we're like, okay, we should go build that. This is the one else we built, so let's go build it. Whereas if you look at like vectors, for example, you look at vectors, like, wow, this is a pretty thriving space already of, you know, vector databases. Does it make sense for us to go build our own? Like, what's the benefit? Not much. We should go partner with or go find a technology off the shelf. In our case, it's pitch back. And so, plus, I think it's like, how much value does that for customers? Do we have a different take on the space? Do we not?

• 32:49Speaker 02

  And every product that we've launched, we've had a different take on the space. And the products that we don't have a different take, we just adopt what's off the shelf.

• 32:56Alessio Fanelli?

  Let's jump into the state of AI survey that you ran and maybe get some live updates. So you surveyed about sixteen hundred people last August and I were this busy like five years ago. And there were kind of like a lot of interesting nuggets and we'll just run through everything. The first one is more than half the people, fifty-two percent, said that AI is overrated. Are you seeing sentiment shift in your customers or like the people that you talk to like as the months go by? Or do you still see a lot of people, yeah, that are not in Silicon Valley? Maybe say, hey, this is maybe not as worth changing as you all made it sound to be.

• 33:32Speaker 02

  Yeah, so actually, we were on the survey again actually the last few months, so I can let you know if you don't want it changes. It seems to me that it has settled down a bit in terms of sort of the maybe like, I don't know, signal of noise, you could say like, it seems like there's a little bit less noise than before. I think people are still trying to look for use cases, or I'm just saying, but all this is last year, like net states again. And I think there are slightly more use cases, but still not so much as before. And I think as far as we can tell, a lot of the engineers surveys, especially some of the comments that we saw, do feel like the companies are investing quite a bit in AI. And they're not sure where it's going to go yet, but they're like, it could be big, so I think we should keep on investing. I do think that based on what we are hearing from customers, if we're not seeing returns of like a year or something,

• 34:14Speaker 02

  there will be more skepticism. So I think there is like a, it is time bound, if you will, so.

• 34:19Alessio Fanelli?

  You finally gave us some numbers on Stack Overflow usage. I think that's been a Twitter meme for a while, whether or not ChatGPT kills Stack Overflow. In the survey, fifty-eight people said they use the less and ninety-four percent of them said they use the less because of Copilot and ChatGPT, which, yeah, I think it kind of makes sense. I know Stack Overflow tried to pull a whole thing. It's like, no, the traffic is going down because we changed the way we instrument our website, but I don't think anybody bought that. And then you add right after that expectation of job impact by function and operations people, eight out of ten, basically, they think it's gonna, AI is gonna really impact their job. Designers were the lowest one, six point eight out of ten, but then all the examples you gave were designers of a job being impacted by AI.

• 35:03Alessio Fanelli?

  Do you think there's a bit of a dissonance maybe between like the human perception of like, oh, my job is like, can't possibly be automated? It's funny that the operations people are like, yeah, it makes sense. I wish I could automate myself, you know, versus the designers or maybe they love their craft more. Yeah, I don't know if you have any thoughts on who will accept it first, you know, that they should just embrace the technology and change the way they work?

• 35:25Speaker 02

  Yeah, that's interesting. I think it's probably gonna be engineering driven. I mean, I think you two are very well, maybe you two even started some of this wave, sort of the AI engineer wave. I think the companies that adopt AI the best, it is going to be engineering driven, I think, rather than like operations driven or anything else. And the reason for that is I ended up rising this like profile with AI engineer. Like AI is very, maybe it's kind of philosophical, like AI is a tool in my head. Like it is not a, in my head, I think we're actually pretty fond of AGI, so you know what happens. But AI is not like a, you know, thing that, it's not like a black box where like it does everything you want it to do. The models that we have today require like very specific prompting, for example, in order to get like, you know, really good results.

• 36:06Speaker 02

  And the reason for that is it's a tool that, you know, you could use it in specific ways. If you use it the wrong way, it's not gonna produce good results for you, actually. It's not like by itself taking job of it, right? And so I think actually to adopt AI, it's probably gonna be, gonna have to be engineering first, basically, where engineers are playing around with it, figuring out limitations of the models. Figuring out like, oh, maybe like, use that for our database is a lot better, for example. Maybe like prompting this particular way is gonna be a lot better, et cetera. And that's not the kind of stuff that I think like an operations team is gonna really be like experimenting with necessarily. I think it really has to be engineering led. And then I think the question is, well, what are they just gonna focus on first? Like I think they're gonna focus on design first or like operations first.

• 36:45Speaker 02

  And that I think is more of a business decision. I think it's probably gonna be more like, you know, the CEO, for example, says, hey, you know, we're having trouble scaling this one function. So like, why don't we try using AI for that and we'll see what happens, for example. And so in our case, for example, we are really, we have a lot of support issues. So what I mean by that is we have a really, really high performance support team, but we get a lot of tickets. And the reason for that is, you know, we're a very dynamic product, you can use it in so many different ways, and we'll have a lot of questions for us, basically. And so we were looking at, well, you know, can we, for example, draft some replies and support tickets, you know, by AI, for example? Can we allow our support agents to be, you know, hopefully, you know, double as doubly productive as before, for example? And so I guess I would say it's like, this is just needs driven, but then engineering driven after that. So like, you know, we, the business decides, okay, well, this is where AI can be most applied.

• 37:31Speaker 02

  And then we assign the project to an engineer and the engineer goes and figures it out. I honestly am not sure if like the operations will have much of a, like if they accept or reject it, I don't know if it's gonna chase the outcome, if you will, so.

• 37:45Alessio Fanelli?

  Another interesting part was the importance of AI in hiring.

• 37:50Alessio Fanelli?

  Forty-five percent of companies said they made their interviews more difficult in the engineering side. Make interviews more difficult to compensate for people using Copilot and ChatGPT. As they change it, Ritu, like, have you, yeah, have you thought about it? I don't know how much you're still involved with engineering hiring the company, but I'm curious how we're scaling difficulty of interviews, even though the job is the same, right? So just because you're gonna use AI doesn't mean the interview should be harder, but I guess it makes sense.

• 38:20Speaker 02

  Our sense, basically, of the survey, and this is true of what we believe too, is we are most, when we do engineering interviews, we are mostly assessing like critical thinking or thinking, you know, on the spot. And I guess, you know, when you're hiring the employee, you know, in the end, the job of employees to productive, which they should use whatever tools they want to be productive, so, you know, that's kind of our thing too. However, we do think that, you know, if you think about it first, most, most way, if your only method of like coding is literally copy pasting, you know, off of ChatGPT, or like, you know, it's pressing tab in Copilot, I think that would be concerning. And so for that reason, we still do want to test for like, you know, fundamentals, understanding of CompSight. Now that said, I think if you're able to use ChatGPT or Copilot, let's say, competently, we do view that as a plus, we don't view it as a minus.

• 39:02Speaker 02

  But if you only use Copilot and you aren't able to reason about, like, you know, how to write a for loop, for example, or how to write fizz buzz, that would be highly problematic. And so for us, what we do today is we basically screen share, or the rest is a hackpad, actually. So it's, uh, sorry, I guess there's no Copilot there, just want to see what they're doing, or see what they're thinking. And we really want to test for thinking, basically. But yeah, I mean, we ourselves have totally embraced Copilot, and we would encourage engineers to go with this Copilot too. But we do want to test for understanding of what you're doing, rather than just copy pasting on Copilot, so.

• 39:31Alessio Fanelli?

  The other one was AI adoption rate. Only twenty-seven percent are in production. Of that twenty-seven percent, sixty-six percent are internal use cases. Shout out to Ritu, you know? Do you have a mental model as to how people are gonna make the jump from like using it internally to externally? Obviously, there's like all these different things, like privacy, you know? If an internal tool hallucinates, that's fine, because you're paying people to use it, basically, versus if it hallucinates to your customer, there's a different bar. Because for you, if people build internal tool with Ritu, there are external customers to you, you know? So I think you're on the flip side of it.

• 40:06Speaker 02

  Yeah, I think it's hard to say. Maybe a core eternal belief was actually that most software built in the world is internal facing, actually, which actually sounds, may sound kind of surprising, you know, personally hearing this, but effectively, like, you know, we all work in Silicon Valley, right? We all work at businesses, basically, that sell software as, you know, as sort of a, as business. And that's why all the software engineers that we hire basically work on external facing software, which makes sense with software companies. But if you look at most companies in the world, most companies in the world are actually not software companies. If you look at like, you know, the clothing manufacturers I was talking about, they're not a software company, like, they don't sell software, you know, to make money. They sell clothing to make money. And most companies in the world are not software companies, actually. And so, most engineers in the world, in fact, don't work at Silicon Valley companies.

• 40:51Speaker 02

  They work outside of Silicon Valley. They work in these sort of more traditional companies. So if you look at Fortune 100, for example, probably like 20 of them are software companies. You know, there are 488 that are not software companies, and actually, they employ those software engineers. And so, most of the software years in the world, and most of the code written in the world, actually goes towards these internal facing applications. And so, like, for all the reasons you said there, like, I think hallucination matters less, for example, because they have someone checking the output, and the consumer, so hallucination is more okay, it's more acceptable as well. The AI can be unreliable, this is probabilistic, and that's also okay. So, I think it's kind of hard to imagine AI being adopted in a consumer way without the consumer, like, opting in, like, ChatGPT is very obviously a consumer, the consumer's,

• 41:35Speaker 02

  you know, knows that it's ChatGPT, it's, you know, using it. I don't know if it's gonna make its way to like the banking app anytime soon. I, maybe for like, even for support, it's hard, because if it hallucinates, then, you know, it's actually quite bad for support if you're hallucinating, right? So, it's, yeah, it's hard to say, I'm not sure.

• 41:55Alessio Fanelli?

  Yeah, I think a lot of people, like you said, we all build software, so we expect that everybody else is building software for other people, but most people just wanna use the software that, that we build out here. I think the last big bug is like, models breakdown. Eighty percent of people you survey just use OpenAI. Some might experiment with smaller models. Any insight from your experience or visual, like, building some of the AI features? Have you guys thought about using open source models? Have you thought about fine tuning models for specific use cases? Or have you just found GPT-4 to just be great at most, most tasks?

• 42:29Speaker 02

  Yeah, so two things. One is that...

• 42:33Speaker 02

  From a data privacy perspective, people are getting more and more okay with using a hosted model like a GPT-4, for example. Especially because GPT-4 or OpenAI often has enterprise, we went to some companies already because I think a lot of CIOs was like, let's get a second house, like, you know, let's use Azure, for example, and, you know, let's make it available for employees to experience. So I do think there is more acceptance, if you will, today, feeding data into GPT. That's how to take some sensitive data, people might not want to do so, like, you know, feeding in like earnings results data, you know, three days for you to announce earnings, like, probably is a bad idea, probably don't want GPT writing your, like, earnings statement for you, so. We have, there's still some challenges like that, but I think actually, most models could actually help solve like a lot of three, you know, when it comes to, I think that could be exciting.

• 43:17Speaker 02

  So let's, maybe just one thought. The second thought is, I think OpenAI has been really quite smart with their pricing, and they've been pretty aggressive. Of like, let's get, you know, let's create this model and sell it at a pretty cheap price and to make it such that there's no reason for you to use any other model. Just for like a, you know, strategy perspective, I don't know if that's gonna work, and the reason for that is you have really well-funded players, like a Google or like a Facebook, for example, that are actually quite interested, like, I think if anybody's competing startups, OpenAI would win for sure. Like, at this point, OpenAI so far had from both a model and a placing perspective, that like, there is no reason for it to go just really, I think, in my opinion, at least, a startup model. But if like, you know, Facebook is not gonna give up on AI, like, Facebook is investing a lot in AI, in fact.

• 43:58Speaker 02

  And so competing against a large FAANG company, making a model open source, I think that is challenging. Now, however, where we are right now is I think GPT-4 so far in terms of performance that, and I would say a model before was so important right now, because like the average, you can argue Llama 2 is actually so far behind, but like, customers don't want to use Llama 2 because it's so far behind right now. And so that, I think, is part of the challenge. As AI progress slows down, so if we get like Llama 4, Llama 5, for example, maybe it's comparable to that point, like GPT-5 or GPT-6, like, if they get to the point where it was like, look, I just wanna use Llama, like, it's, you know, safe for me to, you know, host it on-prem, it's just as fast, just as cheap, like, why not, basically?

• 44:36Speaker 02

  So, but I think right now we are in this state, we're opening it next year really well, I think, and right now they're thriving, but let's see what happens in the next year or two, so.

• 44:44Speaker 01

  What, what are you gonna ask differently for the next survey? Like, what, what info do you really actually want to know that's, that's gonna change your worldview?

• 44:51Speaker 02

  I wasn't ask you that, but if it is, let me know. For us, actually, we were planning on asking very similar questions because for us, the value of the survey is mostly seeing changes over time. And understanding like, okay, wow, like, let's like, for example, GPT-4 Turbo MPS has declined. You know, that would be interesting, actually. One thing that was actually pretty shocking to us was, let me find the exact number, but like, one change that we saw, for example, was like, if you compare GPT-3.5 MPS, I want to say it was like 14 or something like, it was like not high, actually. On the GPT-4 MPS thing was like 45 or something like that, so it was actually quite a bit higher. So just, I think that kind of progress over time is what we're most interested in seeing is, you know, are models getting more small, is getting better? Are people still loving PG back there? Do people still love Lingo, you know, stuff like that?

• 45:35Speaker 02

  That I think is the most interesting.

• 45:38Speaker 01

  It seems like you're very language model focused. You know, I think that there's an increasing interest in multimodality in AI. And I don't really know how that is going to manifest. Obviously, GPT-4 Vision as well as Gemini both have multimodal capabilities. There's a smaller subset of open source models that have multimodal features as well. Like, we just released an episode today talking about IdaFix from Hugging Face. And I would like to understand how people are adopting or adapting to the different modalities that are now coming online for them? What their demand is relative to, for like, let's say, generative images versus just visual comprehension versus audio versus text to speech? Like, what do they want? What do they need?

• 46:21Speaker 01

  And what's the sort of forced, like, stack ranked, you know, preference order? It's something that we are trying to actively understand because, you know, there's this sort of multimodality war. But really, like, multimodality, it's this term that like, this is like umbrella term for like, actually a whole bunch of different things that are quite honestly, like, not really that related to each other unless, you know, in the limit. But it tends towards like, maybe everything uses transformers and ultimately everything can be merged together with a text layer because text is the universal interface. But if you're given the choice between, like, if I want to implement an audio feature versus I want to implement an image feature versus video, whatever, what do, what are people needing the most? What should we pay the most attention to? What is going to be the biggest market for builders to build in? I don't know.

• 47:05Speaker 01

  I figure we'll just kind of zoom out a little bit to just the general founder questions. You have a lot of fans in the founder community. You know, I think you're just generally well known as a very sort of straightforward, plain spoken person about about just business. Something that is the perception from Joseph is that you have been notably sort of sales led in in the past. That's his perception. I actually never got that, but I I'm not that close to to sort of your sales function. And it's interesting to understand your market, like the internal tooling market versus all the competition that's out there, right? There's a bunch of open source retools and there's this bunch of like, you know, I don't know how you sort of categorize that, you know, the the various things out there. But effectively, what he's seeing and what he's asking is how do you manage between sort of enterprise versus ubiquity?

• 47:49Speaker 01

  Or in other words, enterprise versus bottom up, right? I was actually surprised when he told me to ask that question because I had always assumed that you were self-serve signup, like bottom ups, bottom up led. But it seems like you have a counter consensus view on on that.

• 48:05Speaker 02

  Yeah, so actually, when we talked for started, we started mostly by doing sales, actually. And the reason we started by doing sales was mostly because we weren't sure whether we had product market fit. And sales seemed to be the best way of proving whether we had product market fit out. Because I think this is true of a lot of AI projects, you can like watch a project and people might use it a bit and people might stop using it. You're like, I don't know, is the product market fit, is that not? It's hard to say, actually. However, if you work very closely with the customers or on a sales-led way, it's easier to understand, you know, their sort of requests, understand their needs and stuff. And then actually go build a product that actually serves them really well. And so basically, we viewed sales as like working with customers, basically, which is like, yeah. I think actually quite a, I think it's a better way to describe it, what sales is, a real estate company. And so we did a lot of that also when we got started.

• 48:50Speaker 02

  I think we, over the last maybe five years, maybe like three years ago, four years or something like that. I think we have invested more on the self-serve ubiquity side. And the reason for that is when we started retool, we always wanted actually some percent of software to be built inside of retool. Whether AI software or origin software, broadly UIs and whatever, but like software, basically. And for us, we're like, we think that maybe one day, you know, 10% of all the code in the world could be written inside of retool, actually. Our 10% of the software could be running on retool, which would be really, really cool. And for us to achieve that vision, it really does require like broad basis option of the platform. It can't just be like a, oh, you know, only like a thousand customers, but the largest 1,000 companies in the world use it. It has to be like all the developers in the world use it.

• 49:36Speaker 02

  And for us, you know, there's like, well, I think 25, 30 million developers in the world. This is how do you, you know, get to all the developers? And the only way to get to those developers is not by sales. You can't have a sales force to talk to 30 million people. You know, it has to be basically in this one of bottoms up product land ubiquity kind of way, basically. And so for us, we actually changed our focus to be ubiquity actually last year. So our whole star metric used to always be sort of revenue generated or an new ARR generated. We have to change it to be number of developers building on the platform actually last year. And that I think was actually a really clarifying change because obviously revenue was important. You know, with funds, you know, a lot of, you know, our product and funds, you know, the business. But we're going to fail if we aren't able to get to like something like, you know, 10, 20, 30 million developers one day.

• 50:17Speaker 02

  We can't convince all developers that there was a better way of building a sort of class of software. Let's say in total applications for today. And so I think that has been a pretty good outcome. Like if I think about, you know, the last like, I don't know, five years of retool. Like I think the starting off with sales, so you can build revenue and then you can actually build traction. And you can hire them more slowly. I think it was really good. I do think the focus towards like, you know, bottoms up ubiquity also is really important because it helps us get to our long-term outcome. What's interesting, I think, is that long-term ubiquity actually is harder for us to achieve outside of Silicon Valley. Like to your point, I think a Silicon Valley retool is like reasonable ubiquity. So I think like, if you're starting a startup today and you're looking to build a internal UI, you've, you're probably going to consider retool at least.

• 51:01Speaker 02

  Maybe you don't choose it because you're like, I'm not ready for it yet or something, but you're going to consider it at least. And when you want to build it, I think it's actually high probability you will actually choose it. It's awesome. But it's that, you know, if you think about, you know, a random developer working at, let's say like an Amazon, for example. Today at Amazon, actually, we have, I think, 11 separate business units that use retool at this point, which is really awesome. So Amazon's actually a big retool customer, but like the average year at Amazon probably has never heard of retool actually. And so that is where the challenge really is. How do we get like, you know, I don't know, let's say 10,000. Developers at Amazon building by a retail. And that again, I think is still a bottom of ubiquity thing. I don't think that's like a, I don't think we'd like, you know, go to Amazon and knock on every developer's door or send out on either of your developer be like, go use retail.

• 51:42Speaker 02

  They'll ignore us actually. I think it has to be, use the product, you love it, you tell your coworker about it. And so for us, the bottom of ubiquity by marrying that with enterprise or the community business has been something that's really near to our hearts.

• 51:56Speaker 01

  Yeah, just like general market thoughts on AI. Do you think spend a lot of time thinking about like AGI stuff or regulation or safety? Or like, what interests you most, you know, outside of the retail context?

• 52:08Speaker 02

  Yeah, my opinion, I mean, I think there's a lot of hype at AI right now and just again, not too many use cases. So for us, at least from retail context, it really is, how do we bring AI and have it actually meet business problems? And again, it's actually pretty hard. Like I think most founders that I've met in the AI space will always leave work use cases, never have enough use cases. Sort of real use cases, people will pay money for it. So that's I think really where the retail interest comes from. Me personally, I think philosophically, yeah, I've been thinking recently myself a bit about sort of intentionality and AGI and like, you know, what would it take for me to say, yes, you know, GPT X or, you know, any sort of model actually is AGI?

• 52:48Speaker 02

  I think it's kind of challenging because it's like, I think if you look at like evolution, for example, like humans have been programmed, you know, to do like three things, if you will, like, you know, we are here to survive, you know, we're here to reproduce and we're here to like, you know, maybe this is just two things, I suppose. So like basically, to survive, if you go eat food, you know, for example. Uh, to survive, maybe like having the resources is helpful. You want to go make money, you know, for example. Uh, to reproduce, maybe you should go date, you know, or whatever, you should get married and stuff like that, right? So like, that's, we have a program to do that. And humans that, you know, are good at that have propagated. And so humans that, you know, were not interested in surviving, probably have disappeared. Or just due to natural selection, humans that were not interested in producing also disappeared because, uh, or, you know, they're less of them, you could say, because they just, they just don't care enough.

• 53:32Speaker 02

  So it almost feels like humans have been sort of naturally self-selected for these like two aims. I think the third day I was thinking about was like, does it matter to be happy? Maybe it does. So maybe like. Happier humans, you know, survival, it's hard to say, so not sure. But if you think about that in a role, it's like AIs, if you will. Right now, we're not really selecting AIs for like, you know, reproduction. Like, it's not like, you know, we're being like, hey, AI, you know, you should go make thirty other AIs. And, you know, those that make the most AIs, you know, are the ones that survive. We're not saying that. So it is kind of interesting, sort of thinking about where intentionality for humans come from. And like, I think it argues intentionality, like, humans basically come from these three things. You know, like, you know, if you want to be happy, you want to survive, you're, it's like, basically, your sort of goal, you know, of life. Whereas, like, they definitely have that.

• 54:13Speaker 02

  But maybe you could program it in, like, if you, you know, prompt inject, for example, like, hey, AI, you could just, you know, go, go do these things. And, you know, you're getting a critical simulation, if you will, like, of the AIs, you know, in the world, for example. And maybe you don't have AGI in that world, which I think is kind of interesting. So that's kind of the find, I think, that's what I talk about with some of my friends, uh, from a sort of philosophical perspective, but that's kind of interesting.

• 54:32Speaker 01

  Uh, yeah, my, my quick response to that is we're kind of doing that, maybe not at the sort of trained, uh, final model level. But at least at the data sets level, there's a lot of knowledge being transferred from model to model. And if you want to think about that sort of evolutionary selection pressure, it, it is happening in there. And, you know, I guess one of the early concerns about being in Sydney and it, it's sort of like bootstrap, self bootstrapping AGI is that, uh, it actually is, if, if these models are sentient, it actually is in their incentive to get as much of their data out there into our data sets so that they can bootstrap themselves in the next version as they get trained.

• 55:12Speaker 01

  There is a scary silver lining that we, that we need to, um, try to be on top of.

• 55:16Alessio Fanelli?

  David, I know we're both fans of, um, Hofstadter's, uh, GB. Um, and, and actually, so in, uh, one of your posts on the Segovia blog, you referred to the ant eater. I don't even know if you call them chapters, and GB is just kind of like this, this continuous riff. But is he like, I'll answer like, not intelligence, but like, an colony has science of intelligence. And I think Hofstadter then used that to say, hey, you know, neurons are kind of like similar, and then computers maybe will be the same. I've always been curious if, like, we're drawing the wrong conclusion from, like, neural networks, where people are like, oh, each weight is like a neuron, and then you tie them together, it should be like a brain. But maybe like the neuron is like different models that then get tied together to make the brain.

• 56:01Alessio Fanelli?

  You know, we're kind of looking at the wrong level of abstraction. Yeah, I think there's a lot of interesting. Philosophical discussions. So I've been, Sean and I recorded a monthly we got pockets yesterday and we had a similar discussion on, what did you say Sean on the plane and the bird? I think that was a good analogy.

• 56:18Speaker 01

  The sour lesson. Are we using the wrong analogies? Because we're trying to be inspired by human evolution and human development and we are trying to apply that analogy strictly to machines. But every, in every example in history, machines have always evolved differently than humans. So why should we expect AI to be any different?

• 56:35Speaker 02

  Yeah, it is interesting because it does feel like, yeah, if you're sort of pure under the hood of AGI, if you insist on AGI, we have always used AGI for things like a human. That is the Turing test, I suppose, but whether that is a good point, like if it works, that's not the trick. Because the Turing test is if the output is the same as a human that I'm had, I don't really care about what's going on inside. And so it feels like caring about the inside is like a pretty high bar. Like, why do you care? It's kind of like the plane thing for flies. It's not a bird. I agree. It does not fly necessarily the same way as a bird. Physically, it does, I suppose. But you see what I mean? Like, it's not the same under the hood, but it's okay. The flies, that's what I care about. And I, it does seem to be like AGI is probably like, doesn't think and could achieve like,

• 57:19Speaker 02

  you know, outcomes that I give it and can achieve its own outcomes. And I can do that. Like, I kind of don't care what it is. Like, it may not need to be human-like at all. Doesn't matter to me. So I agree.

• 57:28Alessio Fanelli?

  Awesome. No, we kept too long. I actually have G B right, right here on, on my bookshop. Sometimes I pick it up and I'm like, man, I can't believe I got through it once. It's quite, it's quite the piece of work.

• 57:38Speaker 02

  It's a lot of fun, though, so.

• 57:40Alessio Fanelli?

  Yeah, no, I mean, I, I started studying physics in undergrad. So, you know, it's one of the edgy things that every physicist starts, starts going through. But thank you so much for your time, David. This was a lot of fun and looking forward to the 2024 set of AI results to see how, how things change.

• 57:56Speaker 02

  Yeah, we'll let you know. So thanks, love.

• 57:58Speaker 01

  All right.