Ep 25: Intercom Co-Founder Des Traynor on Building Winning AI Strategy and Where We Are in the AI Adoption Curve

Intercom is one of the earliest adopters of AI - its AI product Fin has generated over two million answers and been used by thousands of users since it was launched last March. On this week’s episode of Unsupervised Learning, we sat down with Intercom Co-Founder and Chief Strategy Officer Des Traynor to talk about how AI is incorporated into Intercom, structuring its AI team, using RAG vs. fine-tuning techniques, where we are in the AI adoption curve, and his advice for startups building on top of AI.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Jacob Effron?

  This is unsupervised learning, Red Points AI podcast. I'm Jacob Effron and I'm joined by my co-host Jordan Siegel. And today we had a fascinating episode with Dez Trainer, the co-founder and chief strategy officer of Intercom. Dez gave us a really interesting perspective on how Intercom shipped Finn, its AI products, super fast after ChatGPT, near months later. We talked about the future of customer support and what role AI will play. And then we also hit on how Intercom has actually rolled out these products to their customers and the crawl walk run approach they've taken. I think folks are really going to enjoy this. Anyone building in AI, there's just some fascinating product lessons from a company really deploying these models at scale. Well, to start, I'd love to hear from you. What was it like at Intercom the day ChatGPT came out?

• 0:43Speaker 01

  Yeah, it was fun. So I'm in Dublin, Ireland, and so is our AI and ML team. And I think it came out, I want to say like 5:00 PM over here or something like that. And I got home and I got a Slack message from Fergal, who's our VP of AI. And he was like, have you seen this shit? I was like, oh, hang on, let me say. So I kind of like looked at it, you know, I played on my phone, but if you have to recall, before we all knew ChatGPT, what happened with these things was you just asked it some bullshit question that you knew nobody answered, like who's the top scorer in the league or whatever? And of course it answered all those questions. So then I was like, at the time, the thing that blew everyone's minds was like, they're like, write me a song in the style of Rage Against the Machine that talks about installing, you know, a Windows driver or something like that.

• 1:24Speaker 01

  And like, when it did that really convincingly, I was like, oh shit, this is actually something interesting. And then I got another message from, again, from Fergal, probably like 11:00 PM at night, saying, I think we should chat about this tomorrow. And so I think we all in Intercom were just playing with it like all that evening. And then the following morning, we had a meeting and then I texted Owen, our CEO, like I think that evening. And we spoke on Sunday for a couple of hours and we're like, just trying to discuss like, Are we really about to rip up the entire AI and ML roadmap and everything they're about to do to go all in on this thing where all we've really seen is some really cool impersonations of Taylor Swift songs and shit like that.

• 2:03Speaker 01

  But it did seem quite convincing. So we got to work straight away and we shipped something before Christmas and we had a reasonable release in January and then we launched Finn in March initially and then broadly in July. So it was definitely an all hands on deck moment because I think for us, you have to realize like computer, sorry, sorry, customer support is like so, so in the, the kill zone of AI and these large language models. Cause if you think about what does, what did large language models do? They're very conversational. They can look up facts. They can read things, understand them and summarize them. Like it, it, it by default out of the box, they do such a good first stab at the job of a, of a CS rep specifically for like the undifferentiated aspect of CS where it's just like fact finding and answer giving.

• 2:48Speaker 01

  That we're like, if we aren't the people who like lead here, somebody else is. And I'm not sure there's going to be a lot of support left for the rest of us, you know? So it really was a kind of an all hands on deck moment and thankfully we got it done and we're out the far side. I don't think we're like now in a position to be like leading the customer support in, in industry and its adoption of AI.

• 3:07Jacob Effron?

  Yeah, I guess on, on that first day, you could finally fulfill those customer requests of, you know, make our support responses sound more like Rage Against the Machine songs or, you know, what had been on the roadmap forever, I'm sure.

• 3:15Speaker 01

  Yeah, yeah, absolutely. Yeah, exactly. Out of the box capabilities were fantastic. Yeah.

• 3:20Speaker 02

  Well, definitely want to understand sort of that transition of the company to this, you know, all in on the AI part, but I think it'd be great for our listeners to understand a little bit more about how AI is actually incorporated in the intercom product today. You've had a ton of announcements this year from Finn to Inbox AI and others. So I'd love to hear a little bit more about those.

• 3:38Speaker 01

  Yeah. So the first, um, the first thing we did, which was like, I, I think at the time, I, I, I still think this is probably a good advice for everybody. So the first sort of tracer bullet we fired of like adopting GPT. What was at the time, three point five turbo, right after ChatGPT launched. Was to simply build some very, I would call it like a, you know, zero downside AI features into our inbox. So this was things like summarize this conversation, translate this message if you're, if you do multilingual support, expand on this, you know, collapse this, you know, what I would call a lot of, a lot of the, you know, every big enterprise at work is be like, oh, we now have AI. And it turns out what they can do is rephrase

• 4:22Speaker 01

  or like summarize or whatever. Like we kind of did that within the first few weeks because we wanted, we knew that things like summarization is an actual real job because for support, like summarize this issue and create a ticket or whatever. So we wanted to build those things out and they worked really, really well. But that was our first sort of release. And I think that the great starting point for startups is to find the things where it's, it's, you know, there's no downside, but there's an upside. So if people don't like the summary, just don't click the summary button, easy. Whereas if the summary is actually useful, then all of a sudden people, you know, immediately we got from, we went from like, you can summarize to people immediately be like, how do I make it automatically summarize? And as a side note, this is what, this is what took us on our little path of learning how costs work because Intercom per is like 500 million conversations a month. If we

• 5:07Speaker 01

  were to automatically summarize all of them with, you know, put it this way, what do we, OpenAI's valuation would be a lot higher and Intercom would no longer exist. So, so we had to, we had to kind of box clever in terms of what we should, what made, what made sense to do. So our first release was just that. Our next release was really backed by when we got access to the beta of GPT-4. That was the first point. I, you probably don't remember these days, but we used to talk a lot about hallucinations and how that was a huge concern with GPT 3.5. With four, it was the first time we really got the ability to kind of contain, to contain the bot. So we released the first version of a user facing chatbot, which we called Finn, which would actually go ahead and actually answer questions based on a, on a high confidence threshold.

• 5:51Speaker 01

  And there was a lot of stuff we had to do to make our customers excited to adopt this. We had to make sure it was trustworthy, reliable, other stuff like it would stay on topic. Like a lot of our customers are like, hey, I don't want my chatbot, you know, giving political opinions. I don't want it recommending my competitors. I don't, you know, there's a lot of things I don't want to do. And that like, you know, I pay my support humans a lot of money to not do. Um, so, uh, so we had to do a lot of like, uh, work on that side of things. But Finn was in, uh, the next release, uh, Inbox AI came afterwards. It was kind of like the original set of features that we added to the inbox kind of plus plus from there, including things like, uh, put this into standard intercom tone of voice, make it match my tone, et cetera. And there'll be a lot, there's a lot more to come in that area that you'll see in the next quarter. So, uh, where we have like a significant, uh, new look at like what, um,

• 6:35Speaker 01

  what could Finn do in the inbox basically as well? We'll be talking about, uh, nothing to say yet beyond that.

• 6:41Jacob Effron?

  Yeah, no, you talked about, I mean, obviously a big enabler of rolling out a broader set of features has been the work you've done around guardrails and hallucinations. And I feel like every enterprise we talk to is thinking about those, those two areas, right? And, and I'm sure, you know, they're curious for, you know, how did you guys go about actually building that? What are some of the things you've learned around, you know, uh, how to make guardrails and hallucinations prevention work?

• 7:00Speaker 01

  A lot of it is, um, so like the, the, the core ingredient here is to have a really good torture test of all the scenarios that you don't, all the types of misbehaviors and all the good behaviors. And you, and then have a, a kind of an internal, like not an ML facing, but like an internal sense of waiting. Like, uh, like if the answers can be that good, are we okay with it occasionally giving you a political opinion or whatever? You know, that sort of spectrum of like, uh, cause I, you know, a lot of the times these error bars shrink about their actions, right? The, the more constrained you make it, the less, the less creative it is. It means you're going to have a few times where I would have got the right answer, but it misses. Um, so the torture test is really like a long set of scenarios, questions, contexts,

• 7:43Speaker 01

  where we see its behavior and you can like, you know, loop back, refer, merge contexts, try and trick it effectively, see what's the worst case scenario. So if you go to like a pizza company and ask it like, you know, hey, what's the best way to cook a pizza? Maybe you want to answer that question. What if it comes back and it says, well, the best way is to like, you know, you know, like. You can imagine like stonebaked. Okay, grab. What if this company doesn't do that? So there's like loads of ways where like this can go quite wrong. And then, and then there's other examples of this, like, which is like what the L L M knows versus what you wanted to forget. So the L L M might know, for example, like they're like, you know, sunbeds are totally legal in most of the world. But you might be in, say, a company like a country like, say, I think Australia where they're illegal.

• 8:27Speaker 01

  And you might get very much confused at saying, given this context, plus what you already know, we need you to appropriately understand that the context you're, we're giving you is supersedes everything else. So you have a lot of this kind of like, um, a lot of this, how would you say, like, just, uh, prompting that you're, you're basically doing, which is given this, understand that, prioritize this over that. Uh, you know, consume these four docs. But if there's a conflict between those four docs, we may have to resolve it with the following weighting, et cetera. Um, so what we have been doing kind of since three point five launched, um, you know, we continue to do this to, to, uh, today is look at three point five, there are about four. Uh, latest versions of four latest builds, uh, then also anthropics, Claude, or, you know, all, all the rest of the gang, we stral, uh, Lama, you, you know.

• 9:13Speaker 01

  And put them all through, uh, the same scenarios to see where do we get the highest scores on. And it's not just like, trust isn't obviously a thing, but trust isn't the only thing there are like, you know, we have to think about like cost, uh, reliability, stability, uptime, um, and then like, yo, um, malleability, how much can we, can we control these things? Uh, speed is a huge one. Uh, like, that's probably a good, if I don't need magic wand, it would be for speed. Uh, so you're, you're kind of like, you're trying to pick a model based on all of that. Uh, and, and ultimately it's not clear to me that we might, we won't have to go to a world where we actually pass this decision on to our customers in some future world where we say, you know what? Our default out of the box works like this, but if you prefer, you can go that way instead.

• 9:56Speaker 02

  You mentioned, uh, if you had done five hundred million summaries, uh, intercom would probably, you know, close to bankrupt and opening I would be a lot higher valuation. Um, you've mentioned before that you, you were in the exploration phase of AI. It sounds like you've transitioned a little bit more to the cost optimization phase. How do you sort of think about that transition? And when do you think startups should sort of think about more on the cost side versus the exploration side?

• 10:18Speaker 01

  If I was to revise my comment that open, I don't actually bankrupting your customers is a great way. So it's a great way to say business. So maybe, maybe they wouldn't have got the valuation book that helped her. But really, what I what I mean is, if you recall, like, right after ChatGPT launched, it was more expensive back then, it was significantly more expensive. It's come down like money, money, like, probably like 10x since then.

• 10:38Speaker 01

  So 500 million summarizations would have been quite expensive uh uh to do per month, and we just couldn't afford to do it. So like, it was the first time I've kind of felt. Maybe you get a little bit of this with speed when you're building software, but like the first time I was like, that's a really good feature that we should do by default, but we can't afford to. Okay, you know, and the problem there is that like our own customers probably wouldn't pay every single time. Uh, so like, so you have to, you end as like, which ones should we summarize and all that? So that's why you put a button on the UI instead. Anyway, um, we're not really, I wouldn't describe this as being in cost optimization. I'd say we're still in deep exploration mode. We know where there is more areas where we could, you know, be smarter with how we spend, you know, and, you know, there's certainly calls that you can move from four down to 3.5.

• 11:22Speaker 01

  Or even, I don't think we've done this yet, but like from 3.5 down to like an open source, like a Llama or something like that, it said. We have, we're not really there yet, purely because I don't think it's the, um, it's, it's probably just not the most effective use of our AI team's time right now. We're still very much in like, there is opportunities for AI everywhere. Our reporting should be powered by AI, our like, our human agents should be augmented by AI. You know, lots of other decisions around how Intercom works, the messages we send, like Intercom also starts conversations, we could power them by AI. So we're very much like, um, we're still, I think, squeezing a lot of juice out of what's currently possible, let alone what's coming down the line. When we do get the cost optimization.

• 12:05Speaker 01

  I don't have a full grasp of this, but I'll sort of say like, the way we, I think we think about it, uh, is like, you know, model is obviously like the foundational thing. But beyond that, it's just that then how do you ping it, where, you know, how many times you have to go, how many API calls, where, when, et cetera. Uh, and then I will still say, um. Again, I still think there's like room for improvement and latency here, both on their side and on our side. Uh, where, cause I think like most of the time you click one of these magic ones. It does feel a little bit like dial up modem internet days where you're like, oh, I didn't really, we're literally walking over to the server. Are we? Oh, okay. All right. Well, let me know what you come. I know, you know, like it, there is that sort of uh thing, which I suspect will go away pretty quick.

• 12:48Speaker 01

  I'd be, I'd be curious and excited to see, say, what uh, what Apple do if they actually onboard uh, like, you know, the actual, you, you, I could imagine an L L M that'll fit in the phone. In fact, I think Google's most recent Gemini builds could. And so I, I suspect, you know, we're going to start to see what instant AI will look like. You can already get a sniff of it with G P, uh, chat G P T when you put it in three point five mode. You can get a sense of shit. This could be a lot faster. Uh, so anyway, I'm excited for that, but like, I think cost optimization will happen at some stage in parallel with us needing to think about margin and our customers willing this to pay. But speed optimization is the thing too.

• 13:21Jacob Effron?

  Yeah, no, it's really interesting. I mean, I feel like a lot of, uh, you know, what you said echoes a lot of what we hear from companies, right? Where just there's so many potential use cases you can build on top of these models that, you know, it feels silly to, to try to optimize before you've explored all of them. You know, I wonder what, what does end up becoming, you know, when do you think you make that shift? If only because, you know, by the time you've explored all these G P T four cases, you know, G P T five comes up, there's always kind of new capabilities of these models. And it's interesting to hear, it sounds like for you, the latency requirements actually may be more of a forcing function for, for some of the uh exploration of smaller models than, than the cost side.

• 13:53Speaker 01

  Yeah, I mean, it's latency right now, because I, I think in general, like what, what our approach is, and I think what most companies were trying to do is like, build what you believe to be the best possible product. And then, you know, make it work, uh, like, you know, you know, take your various silicon valley phrase that you like nail it before you scale it type thing. But like for us, it's uh very much build the best possible customer support platform empowered and enriched with AI. And then if the only problem we have is like, we're like, you're, you know, our tens of thousands or hundreds of thousands of customers and our hundreds of millions of revenue, that we just kind of get the right margin, I'm sure we can then be like, all right, maybe we'll take a slightly smaller hit on, on latency over here.

• 14:36Speaker 01

  Or we'll, we'll do whatever we have to over there to try and make it, make the economics make sense. I think, um, so that's why it's kind of like, our, our job is to first have the best product and then second, uh, make everything else work. Do you, do you, in kind of implied in that is a belief that like, you know? Technology generally gets cheaper and faster. So like even if like even if we took our hands off the wheel for a year, like Finn would get better because the models get better and faster and cheaper and all that. So so there's a kind of um. It's kind of like solve the right problem at the right time. I think it's interesting if you think about like S curves like. I I think it's like open eyes clearly like still in the acceleration of the S curve, like the area where like the models are still rapidly improving.

• 15:14Speaker 01

  And we are still, uh, cause we're building on top of them, we're on that too. Once we feel they're starting to plateau, like with G P T seven is only incrementally better than G P T six. That might be the point where we were like, all right, it doesn't look like Santa Claus is about to bring us any new cool tech. So let's let's make the most of what we have right now. And I think that's when it would make sense to uh to start to like really just like, right, we're not gonna, you know, for the ninth time this decade reimagine support, we're just gonna make this one work, you know. So I think that's where we're kind of.

• 15:43Speaker 02

  That makes makes a lot of sense. Um, aside from the latency side, that makes a ton of sense. Is there anything else that you think is missing right now or was missing as you made this transition to an A I company that would have made it a lot easier?

• 15:55Speaker 01

  Um, like I'd say two things. One, it's hard for us to say right now what's um like what's. Missing because we end up building a lot of stuff ourselves uh because of because it's not there, right? Uh, but when you're doing that, you often don't know, like, do you want to have control over that unit or are you building it just because there wasn't one already on the shelf for you? Um, but there is probably tooling that should need to exist in like that will need to exist in the future, like the. The most obvious example would come to me would be something like um, something around prompt management is an obvious one, right? Like a lot of what we're doing is like. Very subtle changes to prompts and then rerunning tests and all that sort of stuff. And uh, I like it's, you know, versioning prompts versus different models and all that sort of stuff like. So I think, you know.

• 16:40Speaker 01

  It doesn't, I I'm not saying this is a billion dollar opportunity. This could be like a, this could end up being like a reasonably enriched crud app that might only do like three hundred million revenue, like if if you can stomach it. Um, but like, uh, I think there's probably some stuff like that's just the first area that comes to mind. A separate, a separate one, which is kind of like not as exciting an answer for a V C there is just like. It'd be nice if there was just more robust stuff around this area. So like, you know, getting Finn to work in EU was like much messier than you'd like, but you know, just because why? I'll guess all the servers are over in America and like blah, blah, blah. And like next thing you know, all of a sudden now we have a relationship with Microsoft as your never thought that would happen, you know,

• 17:21Speaker 01

  there's that type of thing. And then like, you know, cheaper and faster and all that was just all like, you know, they're all things that I expect will happen. And just the sooner they happen, the better we, the quicker we went. There will probably be more tool, like they're prompting sort of, you know, if you like version control for prompting combined with like prompt manager than A B testing and all that sort of stuff. I think that's an area that's probably worth digging in. Obviously, there's tools like Pinecone where, you know, I think one reason we don't use Pinecone is like, no, I'm not going to do it at all. It's like, we didn't like start AI last November, like a lot of our competitors might have, but we actually had a significant AI product called Resolution Bot in the space before, for which we'd already built a complete sort of vector search capabilities. So we had a kind of a running start that a lot of our,

• 18:05Speaker 01

  our sort of, you know, competitors and like just other companies would not have had. But I do think like, there's got to be areas that are rich to go digging all of around the developer experience, where I think there will possibly be, you know, a good few, in the same way, cloud, you know, gave birth to so many different new multi-billion dollar categories, whether it's like ops or engineering, or ops or analytics radios. I suspect there'll be similar here. It's just, it's not obvious to me what they are yet. And the other thing that is just always tricky to work out is with OpenAI, like they're both, like a lab that's working on the future of AGI, they're like the AWS of AI, they're also a consumer company, which is like ChatGPT with its 100 million users and possibly beyond that, like I've heard they might be

• 18:49Speaker 01

  building a phone and stuff like that. So it's, you know, it wouldn't shock me if, if, if they also build the ops tooling around the effectively AWS for AI layer that they're building. So it's not obvious to me. You might end up getting like in the in Apple terms, you might end up getting Sherlocked if you decide to go after one of these areas only to find that they do it better.

• 19:07Jacob Effron?

  You know, I think one thing that will be super interesting to our listeners is just the pace at which you guys shipped products here. I mean, I feel like you were one of the first companies to have like this pretty broad suite of AI products. Um, I'm sure folks are eager to know, like, how many people does it actually take to to to get these kind of products out? And, you know, I think a lot of folks are thinking about as they, you know, think about structuring teams, you know, do you need a separate AI ML team that lives, you know, independent of your product orgs? Or do you just kind of embed it within each of the product teams? You know, how did you guys think about that? And what was the effort like?

• 19:37Speaker 01

  Generally centralized, like the sort of central agency versus distributed model. That question that comes up all the time when I talk to other companies. We have a central AI ML team. It's staffed with like data scientists, AI and ML engineers, people, a lot of sort of rich, deep domain expertise who like, you know, everything from like, you know, building models, running models, trading models, using models, that's their expertise. And there's about like our our team today in total is about seventeen, maybe twenty people.

• 20:06Speaker 01

  And but when we started on this journey, it was probably like nine or something like that, to be clear. I think, you know, there's a few threads I'd put on like, it's not, you know, people often forget this, but small teams can do an awful lot. So you I wouldn't necessarily say that the size is the first area of concern. But around that team of, say, fifteen to twenty people today is probably like one hundred and fifty or so, what we would call quote unquote, regular product engineers, who who who build on the things that like that the the AI team enable. So they'll say, hey, we have these two new endpoints. One is a given a question that'll give you an answer. And one is like, given a conversation, it'll give you a suggested next step or whatever.

• 20:47Speaker 01

  And it's up for like the teams, whether you're in the inbox or in the messenger to say, right, how can we use that to build kick ass AI parrots customer support platform? And that's, I think, like, the way we sort of think about it. The like, the question as to how, you know, how much should you have a central team? Should you should you just have one good person who's a bit handy with OpenAI on every product team?

• 21:10Speaker 01

  I I often draw to the distinction, really, like businesses or companies that like you, like there are some startups that are AI, as in they're literally working on the bleeding edge of AI and they're constantly like finding new areas to push and making new things possible. There's companies that like, you know, use AI, which is they like they're rebuilding or building new product categories existing that like whose existence depends entirely on things like Anthropic or OpenAI or whatever. So like they're very much an AI first sort of startup. And then there's companies that just like have a bit of AI, which is like, you know, we can summarize this doc, or if you click the magic wand, we'll tell you what the email says, you know, that type of thing. And I think like, if it's the latter category,

• 21:53Speaker 01

  where it's like you're applying AI like, you know, like salt and pepper around your product, just like makes a few bits taste nicer. I think you can get away with just product engineers who, who, you know, who are a bit of a dab hand at like picking out OpenAI specs. I think if you're in the other two categories, you're going to need like data scientists, you're going to need people who have like decent experience in engineering in the space. You're going to need people who have a deep understanding of what's possible, what's probable. Like, the, the one area I always try to caution, like, I'm not an engineer, as you can probably tell by my baffled explanations here, but like, a lot of the product manager folks or designer folks that I speak with, I always have pains to remind them, the difference

• 22:35Speaker 01

  between a AI ML powered project, like where you're trying to output some software versus the olden days where we're just building normal like, you know, crude apps, is that we're, we're used to a world where all the uncertainty might be in the design stage. I was saying, hey, what's the best way to build a new, a new email client? And we're like, oh, well, here's nine different ideas, and I've got seven different Figma files, and you can blow them all up and play with them. And like, you're trying to explore the risk and de-risk the project and maximize the sense. But you're doing that all through the power of the pixel. Like, the difference with an AI ML project is, you do all that stuff, and then you have a second wave, which is like, is any of this shit even possible? And, and the worst part about the, is any of this shit even possible is that you don't

• 23:18Speaker 01

  even know if you'll ever know the answer to that question. Like, you know, all you know is it hasn't started working yet. And you'll never actually have a clean, no, it's firmly not possible. So you can end up kind of like dry humping failure for quite a long time, before you realize, oh shit, at the end of all this, we're actually not going to have a product to show for it. And that's why, to some degree, you need to think about your AI and ML work. For startups who are moving that way, along with like how you staff, it's also how you think you need to think about the work you do as like a portfolio of bets. Some of which are pretty like high probability, isn't I'm pretty sure like expanding on text and reframing text and like, you know, that's what it's like, you can call that like a 99% probability it'll work.

• 23:59Speaker 01

  But there's other stuff where the real meat and the real like where the billion dollar companies look at sort of made like. Can we use DALL-E style technology to create a vector based graphic that the user can then import it to like, say, like Figma and edit? Well, that's actually, we don't know. That's maybe a 20% probability or 40% probability. If it was 90, everyone would be doing it. But when you, when you realize you're playing those low probabilities, the stinger isn't just that it's like unlikely to work. It's like, you'll never actually know. So you kind of have to think about it like, like the same way you guys would think about like your portfolio, which is like. Some of these things look like they're definitely going to work and we can probably like, you know, take that to the bank.

• 24:39Speaker 01

  And some of this shit, we're going to have to cross our fingers for two years and see. And that's sort of, it's not just how you staff, it's how you run and think about your software projects entirely.

• 24:46Speaker 02

  And what, what is an example, maybe of something that you guys tried and, you know, it failed and something that you think maybe GPT-5 will solve in a couple of years?

• 24:55Speaker 01

  Completing the sentence is something that like you see in email clients all the time, right? Like, so, you know, this idea of tab to complete. The stuff that makes it hard isn't necessarily like that the models aren't there yet, but it's like telling the difference between a personal answer. And, and answers that contain PII or like, you know, context that only made sense for the, in for that one, like, you know, like, you know, the AI might say, hey. It sounds like this ends off like the conversation we had last Tuesday with a different person.

• 25:28Speaker 01

  And it might actually be correct, but it still might not suggest the right completion because it doesn't abstract away all of the irrelevance and actually focus on the task at hand. So. There's some work like that where I think, like I was talking to our head of AI about this, like there's some work like that where. In your head, like, given everything you've seen, you'd be like, yeah, this, this should work. But then when you actually go to build it, and this might not be a limitation of the AI, it might just be complexity of the problem domain, like. It's very tricky to sort of make it work in a way that that somebody is likely to accept the the sentence completion, if you like. So that's like, that's one area where where I was like, oh, I assume like that's, you know, if we can answer the questions directly for the end user, surely we can help agents auto complete the sentence or whatever. And actually, I'm

• 26:12Speaker 01

  right. And again, as I just said, I'm not even saying it's not possible. I'm just saying that we haven't like, you haven't seen it nailed yet. Now, I could come back to you in six weeks' time, I'd be like, actually, surprise, surprise, we haven't now. But like, that's the, that's the sort of the weird nature of this world. It's not, you're kind of always climbing the slope of uncertainty. And at some point, it doesn't go from like, oh, we think we have it still. It just goes from probably not, probably not, probably not. There it is. You know, that that's kind of how the pattern often goes, which is just a bit tricky.

• 26:36Jacob Effron?

  I mean, one thing I've also heard you talk about is how a lot of your customers that are using these features in Intercom are kind of in, you know, pilot mode, or they they start using them on like, maybe for weekend support, or for, you know, simple things like password reset. And I feel like that's generally true of the industry right now. We have a lot of pilots going on, and folks are kind of waiting, when when do the broad rollouts happen? I'm curious from what you've seen, like, when do you think that broad switch over happens? And kind of, what's required to enable that?

• 27:01Speaker 01

  For us, like, we're starting to see people go from like, AI curious to all in on AI, if you like. And I'm like, usually the pattern, like, what we've been doing a lot of work on is like, is the question of how do we make it less of a trust fall, and more of a dip your toe to get started with AI? And we have a lot of like, because we're, you know, because, you know, prior to like, even prior to Finn, we were a complete platform that had like, full customer data. So we could do things like, say, hey, why don't we only try the AI on your free users? And they're like, oh, well, you know, some people, some people are like, yeah, that sounds great, because it's not going to cost me any money. Or they might, or they might say, oh, no, our free users are our biggest source of signups. So we don't risk them. And I say, okay, how about

• 27:43Speaker 01

  free, and have been using your product for more than a year, right? So they're definitely not going to convert, right? We can take some risks. But we can always usually find a place, or we'll say like, free, how about we only try it on the weekends? How about we only try it when the word, when the sentence is contained this, or doesn't contain that? So what we spend a lot of time doing is trying to make it easy, like, to dip your toe in and get a sense of the value. Because generally speaking, the value is there in all circumstances. So and what happens is people say, oh, well, you know, oftentimes, we see our customers mindset go from, okay, I'm interested, but only for my free users. And then we're like, all right, good. And then very quickly, they're like, shit, my free users are getting better support than my page users, because they're getting instant answers, and most of them are correct. Then like very quickly it inverts and they're like, right, like, let me go all in on this.

• 28:27Speaker 01

  So I think like, um, we have to do a lot of work here and like, obviously, you know, the nature of the pioneer is that you get all the arrows and then hopefully you live to get the gold. But we're doing a lot of work explaining to our customers how this will work. And it's worked, you know, maybe our competitors like Zen Desk or Fresh or whatever, don't have to do, uh, cause we'll hopefully do it for them, I guess. But what we're trying to like do is land this message of like, this is not, we have moved well beyond the future hand wavy, what if TEDx style talk, right? We're past that. Finn has handed out like over two million answers. Uh, we've got thousands of people using it. Like the answers are high quality. It's like, uh, you know, you could ask like a seven part question and it will give you a seven part answer.

• 29:10Speaker 01

  You know, Roman numerals included, like, you know, it's, you know, it's, it's very, very good. And in fact, we're actually, we're pretty soon, we hope to put live the page that will sort of exemplify this in a, in a kind of big way by showing you like, here's the sort of depth and quality of conversation you can actually have with Finn. And here's the sort of satisfaction levels you'll expect. And, uh, and I think people are slowly getting it, but right now, um, it's still not there yet. Now the biggest, I think, thing that will help us here genuinely is if and when, say, the likes of Apple, and I think they just announced it's going to happen soon, adopt, properly adopt and ship an L L M. What I suspect we'll see is much like when, when iPhone launched, right? It created this idea that software doesn't have to look like shit.

• 29:53Speaker 01

  And so people realize, oh my God, the stocks app on my phone is nicer than every single business productivity product that I use every day. And I don't even care about this, right? But that, so because of that post 2007, like I used to be a design consultant. We had a little design before we had Intercom, we actually had a, we used to design products for the people. Our business ticked up because everyone was just like, oh, I guess it turns out B two B software can't be O B anymore because Steve Jobs held up an iPhone. And because of that, everyone got interested in design. I suspect when Apple and to some degree, Google, Google are a little bit ahead right now. But when, when like a Bard style or I don't know if they call it like a, if they call it Bard, the okay Google thing or whatever.

• 30:33Speaker 01

  And when Siri ultimately both are enriched, I think it'll normalize the idea of talking to software and just saying the thing you damn want and getting it. And then people will just become accepting of AI as a thing, much like people started using words like typography and like font face and all that. In 2010, when they learned that software design mattered and it was a competitive battleground. I just think AI adoption and like, you know, how AI enriched your software is will become a competitive battleground over the next two or three years. And in that world, people will be less AI skeptical and more like, you know what? It's pretty clear all this shit works. I just asked Siri to call me a taxi and it worked. You know, so I think when we get to that point, you'll, it'll be one of those gradually and then suddenly moments. So that's really what I'm hoping for.

• 31:16Speaker 01

  Until then, our job is to go from customer support team to customer support team, explain it to them. This is the world you could be living in. Would you like to give it a try?

• 31:24Speaker 02

  It's super cool. I'm curious for, for Finn and some of the other AI products today, are you fine tuning? You know, for each specific customer based on their, their voice and their brand and things like that? Or are you doing rag for those particular use cases?

• 31:39Speaker 01

  Right now, it's gonna be, right now it's rag. We're not doing fine tuning per customer. We are doing fine tuning for, for tone of voice, but it's done through prompt. It's not done through like anything beyond that. The, they're like, we will get there, like, you know, every new hire who joins Intercom has like, wow, here's one other thing you forgot to do with AI. And we're like, we haven't forgotten. We just have a really long list, right? But like, like the stuff that like our customers most cared about over, of late was like, you know, it was like Finn in the EU. Or right now it's like Finn by email. So Finn will be able to answer your support emails as well. We generally speaking, like we have to, we, we follow our customers for the most part. That doesn't mean like, you know, we still have to do our own explorations. Like we have an actual AI lab ourselves. We're like, maybe like 30, 40% of the AI team are actually not working on customer facing products.

• 32:24Speaker 01

  They're entirely working on. Well, you know, what can we do at Llama? What sort of like, should we build our own model? What would happen if we trained a model on our own, on our own sort of customer support teams data, et cetera? Like, so we are doing a lot of exploration there. But for the most part, like, a lot of what we're doing with the product is like, listening to our customers and building the things that will unlock their usage of AI.

• 32:45Speaker 01

  And when, like, say, oh, I want to be able to control the branded tone of voice. When that kicks up to the top, yeah, it'll definitely get there. In practice, it kind of Finn already on the tone of voice thing specifically. Because Finn reads your docs and it reads your customer support teams conversations, like it already picks the tone of voice organically. So being able to like inject it at the top isn't as, isn't as valuable as you might think, just because if your content and your team already speak that way, that's how Finn's going to speak on your behalf, because it's slurping in all that content and using it, so.

• 33:14Speaker 02

  And then when you think about, you know, five years from now, say you do build your own LLM or, you know, just as OpenAI and others, the models get better. What percent of requests do you think will be handled by AI in the future?

• 33:26Speaker 01

  I mean, in some cases, definitely 100, like, I mean, you can't go above 100, but in some cases, I could well see it being 100. Like, it comes down to, like, we see this, you know, vertical by vertical customer support is very different. So if you were in an e-commerce store or e-commerce platform, generally speaking, you've got like seven or eight types of queries, like, it's like, where is the thing? Do you have the thing in stock? You know, blah, blah, blah, returns, refunds, gifts, coupons, blah, blah, like, it's, it's very, it's a very like, it's like an extreme parito, like, you know, like five questions again for 95% of the support. And you could nearly hand roll a support bot for the other five and you'd be done. So I think that's one vertical, a different entirely different vertical, be like something like, I don't know, Google Docs or something like that.

• 34:05Speaker 01

  Support for Google Docs, if they did it, would be like, you know, guarantee you, they probably get like thousands of different types of questions to the point where like, you know, I don't think you'll ever get to 100%. You might get to like, you know, 80, 90. So I think like the some in general, there's a I'm sure you see this on the other side as VCs, but like the more complicated the product, the higher the volume of support, but also the more diverse the support queries are and just the harder to automate the entire thing is. But also usually if it's successful, the bigger the business, right? And then on the flip side, you want to have some like Apple Notes or one of those like really, really nice, neatly contained and note taking ops where there's only like seven features and six of them are turned on by default

• 34:48Speaker 01

  or whatever, and you only ever get four support queries, right? And like, that's a totally different world where I think you could definitely get to 100% support automation. So there's some sort of spectrum of like automatability of support that's usually proportionate to like product footprint plus quality of service in a lot of cases. The only so that that's like the different types, the only other variable you keep an eye on is like, and we're doing work in this right now, not all like support queries are answered by just pure text. A lot of the time it's like, oh, I have to go and do a thing. So that thing might be, I have to go and log into Stripe and issue you a refund. Well, you know, who's to say Finn can't do that? So like, you know, the implication there is that my team will come over this, but like it's yeah, that's coming in the roadmap too, you know, like so the idea of like taking actions and all that sort of stuff is obviously underway.

• 35:33Speaker 01

  And then like, how much further do we go like, should we, you know, should we follow a support rep around a browser and see all the actions they take and just learn organically? Is that an option? Who knows? But like we, you know, the point being, I guess, it's not always like every support query isn't just, oh, here's a paragraph of text I found, ta-da. Sometimes it is, I have to go and take three significant destructive actions in the background to make something happen. Like as in cancel the order, you know, reissue the blah, do this, right? So you have to go and do all those things. In that world, I think you'll have some people who are like brilliant, do it all for me. Thank you. And other people will be like, no, no, here's what I want. I want my support reps to be like line managers who, I don't know, you know, so in Europe,

• 36:15Speaker 01

  if you go to one of the self-serve checkouts, you're trying to buy alcohol, you beep the thing through and it's a, nah, you have to show ID and some human walks up to you and says, show me your ID. And they look and they manually say, yes, this dude's clearly overt. Funny or whatever. Um, I think there'll be a version of that that we might want to enable with Finn, which is just like this person asked for a cancel blah. Uh, so Finn might turn around to the manager and say, hey, I'm proposing we cancel the last transaction, issue a credit, still send the product and follow up with a, with an apology email. Does that sound good? And the manager will just say yes, or they'll say no, because this is the fourth time this person's, you know, they'll have their own, they'll have their own logic. So there might, in some cases, people will be all in on full automation. Other cases, people might be like, yeah,

• 36:59Speaker 01

  I would rather have some human role in there, the decision making. And like, that's the, I mean, I'm kind of dumping my whole room up on you here, but like, that's, that's the, like, the, the spectrum of how we have to think about support, because we want to make it work for everyone.

• 37:11Jacob Effron?

  You mentioned actions. Like, I mean, is that, that's one of those categories we hear a lot about, and then people mess around with things and it turns out it's, it's still, it seems like it's still quite a hard problem. Do you feel like, you know, that's ready for prime time? Or is that one of those things that you're kind of waiting for, uh, progress and models to actually enable?

• 37:25Speaker 01

  I don't think it's going to be progress and models. I just think it's a hard problem. And by hard problem, what I mean is like, there is perhaps just lots of codes to be written. Um, there's two, like generally, there's two ways I could see this getting solved. One is you just, you just suck it up on like hand code a lot of stuff. So like, if you're building, uh, seeing where, you know, you want to take an action, like let's say a refund in Stripe, whatever, you build out your conditions and you build out your alls and all that sort of stuff and you go and do it. Another version is you ask the AI to go and read Stripe's developer docs and, and work it out itself. I think either of those could be an, an actual path where like, uh, you know, I, I, either could work. I, I just think, um, there is more like in terms of like

• 38:06Speaker 01

  authentication, uh, and then how to say, uh, monitoring and like, uh, data logging and stuff like that. There's more stuff to be done there, which is just, there's like just a lot of software to be built. Uh, it is my, is my guess. And I think that's okay. Like that, that nearly suits us as being the people who are at the forefront here, because it just, it means, oh yeah, here's an even, here's an even bigger moat for our competitors to have to like run through or whatever. But like, I, like, I, I'm very, like, we have, I have seen examples of, of actions happening like, It's totally doable. It's just peeling away all the special casing to make the demo work great for like for like show and tell. It's like that's possibly just a lot of code to be written.

• 38:45Speaker 02

  You talked a lot in the past about, you know, the shifting product landscape as a result of AI. You had a really great episode with your CPO, Paul Adams. Can you talk about some of the lessons that you've gained from intercom transitioning to an AI company? And both for startups and other incumbents alike implementing it?

• 39:03Speaker 01

  There are some areas where I really believe that the entire product stack is gonna be thrown out because of AI. So I'll give you one example. Like I have a friend who runs, I won't name the startup because this story doesn't end well for him. But um, like, he runs a advertising optimization startup, right? Where like you log in and you you pick out which of your ads are performing well. You add up, you create new versions of the ones that are already there. And you put them live and then you find your one that's LTV CAC positive and you default to that as the winner and you log out. Job done, right? You optimized your ads today.

• 39:42Speaker 01

  And I think in the future, that entire product doesn't exist. Or at least if it does exist, it's just you pay like you pay a fixed fee to some service that you don't even log into. Cause it goes and it generates versions of the ads. It puts the ads live, it measures the LTV CAC, it picks the winner. It closes all the other ads and it goes again, you know? And it's not even clear to me what your actual touch point with such a service would be. Maybe you log in to tell it what your new strategy is, so it knows what new ads to run or something like that. But like, you know, nothing about like every single screen. When I, when I think about like, how does AI apply to a startup name? I look at a screen and I'm like, how much of this could AI replace? And in some cases, it's all, it's quite maximal.

• 40:23Speaker 01

  And then there are other product areas where I think it's not obvious to me like that AI can easily kill it overnight. The example I think I gave before was like Mailchimp, say. And like, I'm not claiming Mailchimp is like beyond and all, in fact, maybe I should say Klaviyo or Braise or something a bit more modern. But like the idea being right. You want to send hundreds of millions of emails and we think that AI is going to make it really easy to design these emails and really easy to write these emails. I'm like, that's grant. You still need to send hundreds of millions of emails and that's a whole tech stack challenge is reputation. There's like, there's D marks, there's like, there's, how would you say, like, there's account spam compliance, all that sort of shit you have to deal with. So like, you have to build, and then you need a reputation that says you can

• 41:06Speaker 01

  do it all. So you're not just blacklisted by everyone overnight. So you need to do so much work to make sure that you actually have a credible business here. And then sure, add in your cutesy little AI for like designing the email and writing the email for you. And when I look at the roadmap of like, how much work you have to do to get to the starting line in those spaces, it's huge. Let's say it's a year's worth. And then I look at the incumbent and I say, okay, so that means that Clavio or Braze have a better year to copy your cutesy little email design editor feature. And I think you don't have an angle there. So I think generally for startups specifically, what I would say is try to pick areas where the incumbent tech

• 41:47Speaker 01

  stack is pretty much irrelevant, such that, and by relevant, what I mean is, if they started again, they would build it entirely differently, right? Entirely differently. And none of the features would be around, none of the UI would survive. Everything would be substantially different. And not because they wanted to rewrite it all and they're relying to be cool, but I mean genuinely different because meaningfully you would do it in a very, very different way. So that, that's my advice to sort of startups is find those areas where like it's, you know, it's the incumbents have no advantage, right? And then for incumbents, like, for like bigger companies, I think, first thing I'd say is I do try and find a sort of asymmetric upside, like just, there's usually areas where you're like, hey, I bet you we could generate the project

• 42:32Speaker 01

  title automatically, or hey, I bet you, you could do some simple stuff. That's really just a run water through the pipes to make sure that you actually know how to use these features and you're aware of costs and latency and all that sort of stuff. And then I think you'll have to go, like, there's a kind of an algorithm I propose, which is like, You have to first break down your entire product. It's like workflows. Like so like if you're like, I don't know if you're a Santa, like creating a project as a workflow, assigning tasks to workflow, planning a calendar, you know, reviewing the Gantt, completions, project updates, they're all different workflows. And I think you have to go workflow by workflow and say, can AI reliably do this accurately X percent of the time where X is above the threshold that your customers require.

• 43:10Speaker 01

  And um, and if it can, then you have to make AI do it and you have to delete all that sort of stuff, right? Just make it right. Yeah. Hey, I know you used to have to complete all your hashtags when you're uploading a video to YouTube. Now you don't. We're just going to do it all for you because we we've we've watched the video. We've learned the words. We've inferred what things is. So now all of that, uh, you know, uh, all of that, say augmentation of the video, automatic, right? Great. So I think you have to go through every single workflow. And if you if AI can remove it, AI has to remove it because that's what the startups are going to do. And then if AI can't remove it, but can augment it or reduce it down to a simple decision set, like if you're a call, say like the um, should we refund this customer or not?

• 43:46Speaker 01

  That might be an idea of reducing a workflow to a decision um, where you've massively simplified it. So it's a it's not a you haven't removed it, but you have massively increased the efficiency of the person. You should do that. After that, if all you can do is like a bit of salt and pepper, like sprinkling of AI, you probably should do that anyway, just for the sake of like looking like a proper AI complete uh, sort of offering. But um, but that's not as important, right? I think the algorithm in my head is like, remove what you can remove, optimize what remains. Sure. Apply some sort of sprinkling after after all that. And then like get busy learning how to sell this to your customers so that they understand the value.

• 44:23Jacob Effron?

  I love that. I mean, this has been a fascinating conversation. We always like to end our our uh, conversations with a quickfire round where we get your your take on a a few things. Um, and so maybe to kick that off, would love to hear, you know, one thing you think's overhyped and underhyped in the AI world today.

• 44:38Speaker 01

  Overhyped for me um, is the sort of productivity tools, doing things like let me write your email, let me write like, let me write this. Sales app and pitch or whatever. I think all of that's gonna get wiped. People are gonna learn to smell AI content, both uh, like the filters will, and I just see humans will as well. So I think. People have forgotten what writing is and what good writing actually is. And I think um, I think that's overhyped. I think underhyped is like uh, the the changes coming to creativity.

• 45:09Speaker 01

  Like, think about all Instagram did was shipped with like nine damn filters for a photo you took. Like, make it dark, make it black and white, turn up the color. It was not, and like, think about what that did for, we all of a sudden, we all thought we were photographers, right? We literally all thought we were amazing photographers, and they did that with just a few filters. So when I look at, say, like, what Kettle, I don't know if you've seen Kettle, it's like, yeah, awesome tool, I'm an investor. Or like Refusion, I don't know if you see, like, you give descriptions of sounds and you can augment it. Or Synthesia, I'm sure you've seen, because they're a hot shit. And when you think about what they can do for like artwork, or for music, or for video, I just think that, like, and like, you know, I'll draw a line and say it's not the same creativity as like people who actually picked up a guitar, or people who actually picked up a camera.

• 45:54Speaker 01

  It's totally different, I get that, but it's a new type of creativity that I think we've yet to really understand. So I think that's quite underhyped right now.

• 46:00Speaker 02

  And then when you think about the the incumbent side of things, you know, besides Microsoft and Google and obviously you guys, you know, what is an incumbent that you think you're most impressed by and then most disappointed by with their adoption of AI?

• 46:14Speaker 01

  The biggest company I've been surprised by, well, Microsoft aside, right? So maybe Adobe have been really good, getting out of the blocks quick with some interesting AI stuff, and they needed to be, and they were, and like, credit to them. Figma have been pretty good, Miro have been pretty good. In terms of just finding useful use cases, like, a lot of companies found like a very quick misapplication of AI, just so they could like put AI on their homepage and, you know, that sort of shit. But like, I think Figma and Miro have both been good at finding valuable things that like actually make sense. That they're probably ones that come to mind most. I think actually Notion have done some nice stuff, so have Coda as well. On the downside, it's not so much, it's not a lack of faith, it's more frustration, but I've been really surprised by how long it's taken Apple to do anything here.

• 46:59Speaker 01

  And same with Amazon as well, like, as in, like, it's so funny, like, turn the volume up or like play the next song, used to be impressive. Whereas like, and it's so hard, like, I have to explain to my daughter, right? When we go to bed, I take out my phone and I say, hey, tell me a story about a dragon who was late for school and didn't eat their dinner and goes on a mystical journey. And I get this from like ChatGPT, I get this beautiful voice, super emotive, telling you like a 10 minute story appropriate for a five year old. And then I have to explain to her why when she says to like Alexa, something like, you know, hey, I didn't like that song. Could you play something that's from the earlier album?

• 47:36Speaker 01

  Alexa's like, what the hell are you talking about? Like, you know, like, it's so weird to try and explain how some things are seemingly so primitive in the world now, like Siri or Alexa. And some things are so extremely advanced, like, say, ChatGPT. So I think like, my hope for 2024 is that like, we see a leveling out there. Cause I think that will be a big sort of moment for AI.

• 47:55Jacob Effron?

  Well, I mean, I feel like there, there's so many threads that our listeners are going to want to pull on here. And, and I mean, really, it's just been an amazing conversation. I'll leave the floor to you. Like, where can folks kind of go to learn more about you, about the kind of these exciting AI features at Intercom?

• 48:07Speaker 01

  Sure. Like, obviously, intercom.com, we have a blog there, intercom.com/blog. We talk, you know, that's where we did a podcast where we talk about AI a lot, like me, Fergal, Paul, et cetera.

• 48:18Speaker 01

  And yeah, like, you know, our, our product is obviously if you want to move to a sort of a customer support world that actually is part of the future, that's what Intercom does. And then I'm just as trainer everywhere as well, if you want to follow me.

• 48:27Jacob Effron?

  Amazing. Well, seriously, Des, thank you so much. This has been just a fascinating conversation. I think our listeners are really going to enjoy it.

• 48:32Speaker 01

  Awesome. Thanks. I enjoyed it too.

• 48:35Jacob Effron?

  Jordan, what did you think of that episode?

• 48:36Speaker 02

  I think it was awesome. I thought he was, he was great. I think he had a lot of interesting points. I think there's a lot for us to talk about now.

• 48:44Jacob Effron?

  It's amazing how fast they moved. You know, it's funny. I think if you look at the customer support landscape, it always felt like one of those spaces where it's such a natural first place to apply a lot of these models. But I think you've had a lot of companies like announce that they're going to do things or like, you know, put it on the roadmap. And it's like Finn was rolled out like, you know, a few months after ChatGPT. It's really impressive just the pace at which that team moved.

• 49:07Speaker 02

  Totally. I think there's, there's two really interesting things you said. One was just like, the day ChatGPT was announced, they just had an all hands and they were like, we're going to rip up the entire roadmap and just go all in on this. And I think a lot of companies, they think AI is really important. But they're like, oh, I'll hire one person to really like, think about it and see what's out there. But Intercom, I think, really just grabbed the bull by the horns and just went all in. And I think that's what got them, you know, the first mover advantage. One other thing I was going to say is, you know, I really liked his point around sort of go with like an initial product that is just like zero downside. And, you know, he was saying like, it's okay if they don't use the AI functionality, it's not going to break their experience. And It reminded me a lot of like what Notion did, for example. I think Notion was really quick to AI.

• 49:51Jacob Effron?

  I love callbacks to previous episodes we did. That is always, always the best.

• 49:55Speaker 02

  Exactly, right. But no, I think it's true, you know, Notion AI for brainstorming. It was like, hey, here's some topics to think about. Auto fill this table, but if the user doesn't embrace AI in that respect, it's fine. They're still using Notion.

• 50:09Jacob Effron?

  Totally. I always loved that when people were talking about marketing and brainstorming as a first use case, it's like hallucination is a feature, not a bug. Like you are supposed to just throw out a bunch of random stuff and hope that some of it sticks. And I feel like to that point of, you know, lower stakes implementations, it feels like, you know, I think a lot of the way Finn initially got started was, oh, we'll use that on weekends when maybe there's less support volume, or we'll use it, uh, you know, just for this subset of folks. And, you know, I think it's really interesting here, Des's carriage station, that basically there's like, everybody wants to do something in AI, but I think there's still that gap between the folks that are like, okay, we'll do it in part of our business and willingness to just kind of, rip the Band-Aid and implement it completely.

• 50:46Speaker 02

  Yeah. One thing he also talked about when we asked about gaps and sort of, uh, developing AI and the tooling. He thought the prompt management side was really like, there was a gap there, um, which is interesting because we've seen so many companies pitched to us in the last, you know, six to 12 months on that. Um, he also said that he didn't think it was a $1 billion sort of market opportunity. Uh, do you have any thoughts on that?

• 51:07Jacob Effron?

  Yeah, it's, it's this weird thing where I feel we've seen a lot of seed pitches in the prompt management space. And yet every company I talk to that's doing this is doing it in a spreadsheet. Like, I think it's, uh, you know, it's one of those things where I imagine there, there's no way that holds long-term. But I think in the, uh, in the, in the short term, it's, it's, it's interesting just how common that problem ends up being.

• 51:27Speaker 02

  I don't know if it's a billion dollar opportunity just by itself, but I think we've seen a lot of interesting companies like Brain Trust, Gentrace and Human Loop and others that are sort of taking like a holistic end-to-end approach where, you know, you have the understanding regressions, the experiment tracking, um, you know, integrating into CICD, stuff like that. And so, you know, from our perspective, it's interesting for him to call out that, oh, maybe OpenAI is just going to be the AWS of, of AI and get into all of this themselves. So it'll be interesting to see what happens there, but I don't know. I, I don't know if OpenAI is going to be as focused on, on these areas as, as the startups are able to.

• 52:03Jacob Effron?

  Yeah, when I you called back successfully to another episode, we just did two on the OpenAI side with Logan. So I think that's but always much, much appreciated. But I think, you know, it's funny. I feel like when the prompt management stuff first came out, people were like, oh, prompting is such this temporary thing. Like, you know, it's just a feature of these like earlier models. And, you know, I think the the dialogue at least seems to have shifted a lot in the last months where it's like, no, I think like effective prompting. Obviously, it's so key to successful AI implementations today. But I think even going forward, there's kind of this understanding that like, yeah, that's probably ends up still being, you know, a key part of these systems going forward. And so maybe in the past, it felt like some of these things were temporary with the current set of models. And I feel like there's much more consensus that these tools will continue to exist, you know, in GPT five, six, seven.

• 52:47Speaker 02

  What did you think was the most interesting part of the episode?

• 52:50Jacob Effron?

  I thought that like, you know, one really interesting part is just, you know, the idea of when do you focus on cost optimization? And I think that like what he was basically saying is there's still so much to go explore of what these models can do, that they're not yet on the cost optimization side. And yet you obviously see it manifest itself in terms of what they can and can't do. Like, I thought this idea of, you know, look, would it be interesting to maybe summarize conversations like in the middle of them for for an agent to be able to look back on and be like, oh, you know, that's what we've talked about so far? Sure, maybe. And I think he said, you know, bankrupt the company if they were running summarization models that often. And so, you know, I think a big question we have in investors right now is like, when do people move? I think generally, like what Des is saying is true of most folks, like, we're still such in this exploration phase of like, what are the capabilities of these models and what can they do?

• 53:35Jacob Effron?

  But I wonder at some point, like, there's going to be, you know, okay, we know we want X use case or Y use case. How do we do that in like the most cost effective way? And maybe by having a summarization model that's 100 times cheaper, we can actually do something that we couldn't do, you know, otherwise. And so I wonder when that like shift starts to happen. But it's interesting, you know, a company that's been doing this one of the, you know, since the earliest days of post-JATGPT, still hasn't quite made the move to like the cost optimization side.

• 53:59Speaker 02

  Totally. He had a lot of interesting investment analogies, actually, which was, you know, the AI endeavors you should view as like a portfolio of bets where each one has a certain level of likelihood and you sort of think about how to do that in a cost-effective manner. Something that I really liked he said was, you know, don't potentially release this to every single user, but maybe you just do it for a subset of users on the weekend that have been with you for a year. So they're not likely to churn if they don't like the experience. And so it was very thoughtful how Intercom is actually thinking about rolling each functionality out and doing it to a subset of users where you're going to control churn, you're going to control costs, and you can really do it in an iterative, experimental manner. I thought that was super interesting and a really good lesson for companies as they think about deploying these out across

• 54:44Speaker 02

  their user base.

• 54:45Jacob Effron?

  Yeah, it just does feel like customer support is one of those areas that is going to see the first like wide-scale adoption. So I think it'll be really interesting to check in with him in a year. One, to see like, is this stuff, you know, have enterprises kind of said, hey, we're going to roll this out in a dramatically large fashion. And then two, are they still on like discover capabilities on the model side or have they moved to optimization? I feel like we're going to have to do a follow-up episode here because I, you know, it's just one of the most, you know, apparent and kind of probably one of the first movers in being widely implemented use cases.

• 55:14Speaker 02

  Totally. No, I mean, I thought he was great and a lot of cool stuff.

• 55:18Jacob Effron?

  Not sure if any news caught your eye, Jordan, in the last few weeks. Like, I definitely think the big, the big news in my mind was like the Meta announcement from Mark, you know, and I think the biggest part of that was just the sheer amount of like compute they plan to use. If we ever had doubt that open source models would be pretty damn good going forward, I think that doubt should be assuaged by now.

• 55:38Speaker 02

  Totally. I mean, it's absolutely incredible. I mean, it, I don't know, it sums up to, people are saying $10.5 billion of cost across the 350,000. I mean, you know, when we talk about sort of companies embracing AI like Intercom all in, I mean, that is one statement to really show that you're embracing it. And so I think, you know, to your point, the future of open source, I mean, they are all in on that and they said they're developing Llama 3 and it'll be really interesting to see what happens there.

• 56:04Jacob Effron?

  Totally. And it just sets a bar. I mean, I think for, you know, there's always, it feels like these open source models. You know, they catch up fast to the latest closed source. They're always maybe half a generation behind. And, you know, as the closed source providers throw more and more compute, it's like, could the open source models still stay caught up? And it's like, if Meta is this devoted to doing this, like, the answer seems to be yes. And it obviously won't be just them, but like, at a minimum, they will be in this game doing the open source approach with, with, with what is a really impressive amount of compute. I think on the closed source side, I mean, you know, I don't even think Anthropic at this point is raised enough to, to, to use that much compute. Though, you know, knowing our ecosystem, that may change in two weeks.

• 56:42Jacob Effron?

  Well, Jordan, never a dull week in the, in the AI world. I guess we'll, we'll look forward to whatever crazy announcements we have next episode in two weeks.

• 56:52Speaker 02

  Totally. We have a bunch of great folks coming up on the docket here, so stay tuned on supervised learning.