The Customer Service Revolution: Building Fin, with Eoghan McCabe & Fergal Reid of Intercom

Today Eoghan McCabe and Fergal Reid of Intercom join The Cognitive Revolution to discuss building their AI customer service agent Fin, exploring how they achieved a 65% resolution rate through rigorous optimization and custom model training rather than relying on base model improvements, while pioneering outcome-based pricing at $0.99 per resolution. Shownotes brought to you by Notion AI Meeting Notes - try one month for free at: https://notion.com/lp/nathan Sponsors: Linear: Linear is the system for modern product development. Nearly every AI company you've heard of is using Linear to build products. Get 6 months of Linear Business for free at: https://linear.app/tcr AGNTCY: AGNTCY is dropping code, specs, and services. Visit AGNTCY.org. Visit Outshift Internet of Agents Claude: Claude is the AI collaborator that understands your entire workflow and thinks with you to tackle complex problems like coding and business strategy. Sign up and get 50% off your first 3 months of Claude Pro at https://claude.ai/tcr Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) About the Episode (03:43) Keeping Up With AI (09:56) Evaluating Models and Evals (13:04) Incumbents vs. Startups (18:54) Product Risk and Judgment (Part 1) (19:00) Sponsors: Linear | AGNTCY (21:34) Product Risk and Judgment (Part 2) (23:42) The Klarna Layoff Story (Part 1) (32:11) Sponsors: Claude | Shopify (36:13) The Klarna Layoff Story (Part 2) (36:14) Driving Resolution Rate (45:00) Intelligence Isn't the Bottleneck (50:10) Closing the Automation Gap (56:20) Human vs. AI Accuracy (01:01:03) The Nuance of Speed (01:04:48) Considering Paradigm Changes (01:09:31) Outcome-Based Pricing Model (01:19:12) Casual Hacking and Insights (01:26:05) AI Adoption and Ambition (01:36:00) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share my conversation with Owen McCabe and Fergal Reid, CEO and Chief AI Officer at Intercom, makers of Finn, the AI customer service agent that's been a market leader since its launch some two and a half years ago. Regular listeners will know that Intercom has recently been a sponsor of the podcast, so it's worth noting that this episode was not part of that sponsorship deal. On the contrary, because I've been an Intercom customer for years at Waymark and also noticed that leading AI companies, and past guests Anthropic, Gamma, and Lovable, all have testimonials on the Finn website, I wanted to understand what's really working and what remains a challenge for a company that's been among the most successful at creating practical

• 0:45Nathan Labenz?

  business value with large language models. And I'm glad to say that this conversation really delivers. With a diverse customer base of more than 400,000 businesses, and Intercom's ability to measure successful resolution rate differences as small as a tenth of a percentage point, Finn is one of the most intensively tested large language model applications in the market today. And as you'll hear, Owen and Fergal are both remarkably candid, both about what they've learned and about what they still don't know. One, perhaps surprising finding that stood out to me, especially considering how much the AI discourse tends to focus on new model releases and frontier capabilities, was Fergal's assessment that intelligence is no longer the limiting factor for customer service automation.

• 1:29Nathan Labenz?

  On the contrary, he says that GPT-4 was already intelligent enough for the vast majority of customer service work, and that model improvements have only contributed a few of the more than 30 percentage point increase in resolution rate that the Finn team has delivered since launch. The vast majority of gains have actually come from better context engineering, which they've achieved through many rounds of careful optimization, of retrieval, re-ranking, prompting, and workflow design. Of course, we cover a lot more than that, including the fact that most customer service teams are currently underwater, which means that for now at least, Finn is allowing companies to support more customers and beginning to affect their hiring plans, but generally not yet leading to layoffs.

• 2:11Nathan Labenz?

  How Intercom thinks about the importance of speed and how they balance the desire to be first to market with the critical need to maintain customers' confidence. The culture of awareness, engagement, and constant experimentation that's allowed them to deliver that 1% improvement month after month for 30 months in a row. The intricate workflows that power Finn and why Intercom is now training custom models for some tasks, including a custom rerankor. How Intercom dogfords Finn and why their resolution rate, while above their customer average, is actually still quite a bit lower than top performers. Fergal's observation that no matter how sophisticated your offline evals, the messiness of real human interaction means there's no substitute for large-scale A/B tests in production.

• 2:52Nathan Labenz?

  How the 99 cents per resolution pricing model, which they pioneered, while initially unprofitable, has created strong alignment between Intercom and their customers and has become profitable thanks to improved success rates and lower inference costs. The 2X productivity goal that Intercom's CTO has set for their technology teams in light of AI coding assistance. And finally, how their vision is now expanding from service agents to what Owen calls customer agents that can work across the entire customer lifecycle, including sales and onboarding. Bottom line, if you're building AI products, you'll find in this conversation a bunch of valuable insights from a team that has brought real rigor and sustained discipline to the challenge of making large language models work reliably for businesses and their

• 3:36Nathan Labenz?

  customers at scale. This is Owen McCabe and Fergal Reid of Intercom. Owen McCabe and Fergal Reid, CEO and Chief AI Officer at Intercom, makers of Finn. Welcome to the cognitive revolution. Thank you.

• 3:54Nathan Labenz?

  So I'm excited for this conversation. My company, Waymark, has been a customer of Intercom for years. And so I've been following what you guys have been doing with AI with interest, both intellectual and applied over the last couple of years. And you've done some really interesting stuff and been in some ways really innovative leaders in the market. So excited to dig into all of that with you. I thought, first question, just because AI is moving so fast and obviously you guys are running and sitting in the leadership position of an 1100 person organization that's distributed across all the time zones of the world. How are you going about keeping up with AI?

• 4:38Nathan Labenz?

  What is it that you're doing, mix of hands on sources, whatever, to stay current and make sure that you know where we are in the development of the technology?

• 4:48Speaker 01

  Yeah, I mean, I'm sure Fergal and I will each have different answers. For me, I rely on Fergal and Dez and others. Dez is one of my co-founders and runs all of R and D. And many people who are very close to the action. You kind of have to pick your battles a little. And when you have those trusted longstanding relationships that I have with Dez and with Fergal, they know what I need to know and I know I can trust what they say. And so that's a big, big part of it for me. I actually got really disinterested in tech news many years ago.

• 5:31Speaker 01

  I found it to be super boring and very, very samey. So I don't follow all the fundraising announcements and who the latest hottest companies are. And so, yeah, that's kind of what works for me. I myself have, I guess it Technical training, if you like to call it that. I studied computer science, but I never

• 5:55Speaker 01

  practiced. And so I could pick up these things pretty damn fast. And funny enough, I mean, I graduated college in two thousand and six. I studied AI in two thousand and four. You know, like machine learning and AI was like the thing I specialized in. And, you know, truth be told, they had, they did not anticipate this moment, but it primed me a little bit. But maybe Fergal can tell you how he learns the things he teaches me.

• 6:22Speaker 02

  Yeah, absolutely. And, you know, just to say, like, look, you know, we've been in this game for a while. We've had AI products in production. I think shortly after I joined, I remember working with Owen on the first brief of what later became Resolution Bot. And so we have those working relationships and we have a certain organizational or institutional knowledge in the company for a long time of like dealing with and communicating with these things. And so that kind of gives us an ability to communicate rapidly inside when something new happens or changes. It's just that shared context and vocabulary and state in the leadership org. And I would say that, like, in

• 7:02Speaker 02

  terms of the external environment, it's really difficult. It's really a full-time job to kind of keep track of everything that goes on out there. And there's so much hype, there's so much rubbish, and you almost have to like test and verify everything yourself. So anytime there's a new benchmark or some new model, it sounds amazing, and you're like, wow, this is really good, if it bears out. But we now have to go through a period of like testing and analysis ourselves to be kind of sure that there's something really interesting here. So it's busy. Of course, there's, you know, there's like Twitter or X, there's like papers, there's, you know, you kind of learn who the top labs are and the people doing the top work in the space over time.

• 7:44Speaker 02

  But yeah, it's busy, it's hard. I don't know anyone in this space who isn't struggling to just keep track of the biggest developments.

• 7:51Speaker 01

  I would add one thing that Fergal did well is that, you know, we have a fifty-plus person team under Fergal that are, we call them the AI group and there's, you know, real scientists and researchers amongst that team. And part of what he has done well is create space for experimentation. Whenever there are new models or new technologies, it's not long before someone on the team has hacked together a new version of Finn that experiments with it. So there's, you know, fundamental research as part of their work, rather than keeping up with what's happening on the side. I think that's important.

• 8:27Speaker 02

  Absolutely. I completely agree. Like, you know, we have a very experimental and scientific mindset, which enables us to kind of quickly integrate information. So if something changes in the external world, we get to validate that. And, you know, it's pretty common where it's like, hey, some new model has dropped. And, you know, the next day or five hours later, we have like the results of it on back tests and then we can kind of integrate that and like, do we change anything here? Does this look promising or not? So yeah, we definitely have that pipeline and that kind of, that setup of people to kind of do that. We've invested in that and it's paid off numerous times.

• 9:05Nathan Labenz?

  Yeah, that's cool. You mentioned it's a full-time job. It's funny you say that because I've basically found a way to make it my full-time job, just to generally keep up with what's going on with AI. Have you actually hired somebody with that specific job description to basically say like, your job is to go track external developments and synthesize them, report them to the team? Or is that kind of a distributed responsibility across the team?

• 9:31Speaker 02

  It's definitely a distributed responsibility. And I would say it is a core competence of the AI group. And it is a core competence of any AI org in today's world. You just, otherwise you're just going to get outdated, you're going to fall behind. It's a fast game. You have to move fast. So no, we distribute that throughout the group. It's not any one person's job, but it is absolutely a core competence of the org.

• 9:56Nathan Labenz?

  Yeah, gotcha. What does that eval stack look like when you say, you know, you can take a new model, plug it in, backtest it with it a day or so, you know, that and you could break that down across a lot of different dimensions. Like, we always welcome shout outs to vendor, you know, companies that you particularly like frameworks, also just conceptual stuff like, to what degree do you trust LLM as judge paradigms, you know, to help you accelerate the evaluation. You know, a lot of companies, there's been this argument recently that we don't need evals, we can just do it all on vibes, but it sounds like you're not, not on that side of that argument.

• 10:36Speaker 02

  I mean, evals is a tricky thing. I would say we're a little skeptical about evals. And, you know, six months ago, it was this sort of meme going around, like, every product team needs to specialize in evals and be great at evals. And like, people started bigging up evals to be just like, amazing thing that will solve all your problems. And you should always be skeptical of that. So, you know, I'd say a number of things. I'd say, thing number one is, yeah, we're pretty good at like backtesting. We've always had a good backtesting framework for Finn, for our core things that we're interested in from an LLM. You know, does it hallucinate much? How often does it answer questions? And, you know, we have, we have a setup where, you know, given sort of a rag style framework,

• 11:21Speaker 02

  and with like, known good answers to particular questions, hey, does a particular LLM that we evaluate, does it do a good job at finding the right content? So, I would say like, we have pretty mature backtests, but we also have this, this kind of battle tested wisdom of like, always test in production, always test with an A B test at scale in production. And we can move quite fast to that posture with a new piece of technology. And the reason we've done that is because We have seen so many times, hey, there's something that's new and it's very exciting and it does well on our bank tests. But then in production, it actually underperforms.

• 12:02Speaker 02

  Just the real world of humans and the real messiness of human communication is like so messy that you can't build a perfect eval for it. You can build like something that's good enough to tell you to signal here, but you have to test in production. And like, we are quite sensitive to changes of like a tenth of a percentage point in resolution rate is something that we care about and we sweat. And so we have to test in production in order to really see those things with like massive at scale, massively overpowered A B tests. And we really, we consider that to be an edge. We consider that to be something that, that we can do, that like maybe smaller competitors or competitors with smaller numbers of customers just can't because we have this very large

• 12:46Speaker 02

  deployment of Finn with a many very large diverse apps. And so we lean in, lean into that. And anytime we've kind of gotten too down, you know, get too far into the evals and you get too scientific, uh, you, you gotta just test in production. Don't fall in love with your offline style.

• 13:04Nathan Labenz?

  Yeah, I want to come back to that notion of scale, uh, that you've mentioned as an edge, because that ties into some macro theses that I'm developing and maybe want to test against your worldview. Before going there though, how much pressure do you feel in terms of, you know, from customers, from, I don't know, other stakeholders to be really quick to market with kind of qualitatively new A I capabilities? I, I think one thesis that I've had is that there's probably enough time for incumbents to implement the best new stuff before, you know, a brand new startup will spin up and rebuild everything and eat your lunch.

• 13:48Nathan Labenz?

  Um, and we've got like quite a, you know, people are, are testing all parts of that spectrum, right? With Apple kind of bringing up the very rear of, you know, being, uh, clearly last to market, but they can hope, I guess, still maybe to be best one day. Where do you guys try to be in terms of like the bleeding edge of capability, somewhere in the middle, you know, not last, but later, but best? Like, what's your sort of philosophy of that? And, and what do customers, you know, demand or expect from you?

• 14:16Speaker 01

  I think we feel pressure from anyone in this space who would want to do the job that we want to do. The larger the incumbent, the more they have these incumbency advantages where they have prior customer relationships and some degree of lock-in. Apple has got the ultimate lock-in. And I've been pissed off with Apple for years now, not just on the AI stuff, but every third new interaction on my iPhone has a bug. I mean, at present,

• 14:55Speaker 01

  if I try to type in a contact's name in iMessage, then it just doesn't return a list of names. And I'm not going to leave anytime soon, anytime soon, because I use the entire Apple ecosystem. And if I thought that that insanely annoying bug in iMessage slowed down my day, wait until I try to integrate Android and all the Google products with my iMac and other Apple things. Like, they'd have to push it, and I would eventually switch at some point, but it's not about to happen soon. We don't have that same level of lock-in.

• 15:35Speaker 01

  And you could compare us with someone like Salesforce. So the bigger guys obviously have substantially more lock-in. Salesforce, for example, they have large customers. Those customers move slower. Salesforce have multi-year contracts, and they have a broad platform and so many other things that work with it. And they also have multiple products that they sell, and people become like a Salesforce shop. And if you kind of remove one thing, it makes the whole commercial contract less, you know, attractive, et cetera. So they have a lot going for them. So they do have more time, but. What happens is that there's a certain momentum in categories.

• 16:17Speaker 01

  And once a company kind of breaks out, particularly with the early adopters, it's very hard for the incumbents to kind of show up anew and kind of take away that sheen. Once a new disruptor has broken out and they've established a brand, they've got momentum with their technology, it's untypical for the incumbent to just completely unseat them. It's worth looking at the Dropbox story. And there's probably nuance in it, too. If you recall, many years ago, this is coming from the guy who just told you he doesn't follow tech news, but this is back when I guess I did.

• 17:00Speaker 01

  Famously, Steve Jobs offered to acquire Dropbox, and Drew Houston said no. And my understanding at the time was that Steve Jobs said, you know, we're going to build this. And Drew still said no. And it took them many years, but they did build it. And now iCloud exists, and it actually works. It works just fine. But Dropbox still exists. I'm not saying that Dropbox are totally killing it. I don't know if it's their fault, though. It could be the whole category. But the ways in which we reduce all of these narratives or concepts

• 17:40Speaker 01

  down to like winning or losing, catching up, falling behind, is too blunt. And so if I was to kind of summarize it all, I would say that like, the bigger the incumbent, the more established they are, the more time they have. Salesforce and others will probably catch up. But I think that the newcomers, once they actually catch some momentum, they will have established themselves very really in the market. And then some of them totally run away with it, and the incumbents can never catch up. And I would say that it's like kind of a meme to say this, and I've been saying it a lot. But I do think that this time is different in that. It's not just, hey, let's build A, B, C instead of X, Y, Z.

• 18:22Speaker 01

  It's more like building AI requires a fundamentally different culture, set of talents. You just heard Fergal talk about the fact that keeping up with the changes and using a highly qualitative and scientific approach was of fundamental importance. The older companies need way more of a fundamental reset to catch up than just deciding to build something else. So I would be relatively bearish on any incumbent catching up where there's now some significant momentum in any of the AI categories.

• 18:53Nathan Labenz?

  Yeah, interesting. Hey, we'll continue our interview in a moment after a word from our sponsors. So how does that lead you to make product decisions when there's sort of a cool but immature, perhaps, you know, new feature or extension of the product that you could offer versus holding off until maybe the next generation of model gets a little bit better? How do you think about like how much risk to take in terms of things not necessarily working as well as you might dream but being there versus, you know, the risk of somebody else beating you to be the first to deliver that?

• 19:30Speaker 01

  I think in previous years, we've had a very strong product inclination. In previous years, we would just try and build all the sexy things first. But now we know that while we're super product oriented and we're very careful to make sure that in our category that we've been leading, which is, you know, service agents or kind of AI customer agents, we wanted to make sure that we'd always be the company that people could trust on to get the new innovation. We've realized that if a smaller company no one has heard of or even has heard only

• 20:10Speaker 01

  a little about builds a sexy shiny feature before us, it's going to be okay. So for example, in our space, we have these agents that can talk on the phone and by text and chat and email, and they're incredibly effective at that. I mean, it's surprising and shocking.

• 20:30Speaker 01

  An obvious thing to do that will be really sexy and cool and that a big part of me would like to build before anyone else is, you know, 3D avatars and little talking heads in your app and not just a chat bubble, but an actual little digital AI person. It'd be fucking cool. It's just so exciting to me as a builder. But does it really move the needle today? Not really. People are still absorbing the other stuff. Might someone else build it first? It probably will. Will that hurt our business? Not at all. So apart, there's a lot of art and judgment to this and knowing what the customer wants and needs and where you're at.

• 21:10Nathan Labenz?

  Yeah. Um, on just business results, the, I look back, it's been 18 months since Klarna made a bunch of headlines by saying, you know, that they're going in all, all in on AI and, you know, look at these amazing results. Again, this is February 2024, right? They said that they had cut 700 full-time jobs. They were a little cagey about that. I think they said that nobody lost their job because they were actually working in a search firm that was able to reallocate those people, but they, you know, were employing 700 fewer. They cite, you know, millions of conversations, improvements in all the metrics, faster resolution, higher resolution rate, et cetera, et cetera. Then that became a big debate, like, is that real?

• 21:52Nathan Labenz?

  You know, are they really good at this when other people aren't? Are they kind of, is there some, you know, uh, steak oil there that we're being peddled ahead of an IPO or whatever? How would you look back on the last year and a half or so and tell the story of what the real impact has been? Like, how consistent is that Klarna story, even if it was leading, relative to like the broad customer base that you guys serve? How does, how does the impact that your customers are seeing line up against that story?

• 22:19Speaker 01

  Yeah, I, I think Fergal will have some fun things to say about this, but I'll first start by saying the things that Fergal won't say.

• 22:27Speaker 01

  Like, I never, you know,

• 22:32Speaker 01

  shit on other technology companies, it's really hard. And I don't know Klarna, the Klarna people at all. So I don't know anything about their story. But as a man who likes marketing moments and opportunities to tell brave stories, you know, that looked like a good one. And, um, you know, they say it takes one to know one. And that looked like a bit more of a show than a reality. Like, I for one, and maybe I'm completely wrong. And I'm, I'm totally open to being called out. I for one don't believe, didn't believe that story because I didn't see anyone else do it.

• 23:16Speaker 01

  To your, to your, um, uh, question. And I think I heard that they backtracked on that. I could be wrong, but I think they announced that they backtracked on that. And they hired people or they didn't fire people, I don't know. But that's about Klarna. The fascinating thing about this moment in time is that, and this, this has happened in all disruptive technologies, is that, at least in our category, but I expect this to win every category, that the new economics and the accessibility and ease of deployment

• 23:47Speaker 01

  means that before it replaces a bunch of humans, it actually increases the supply for the things the humans were doing that one could never afford to deliver in the past. And so, for example, in our category, service, service agents, people are deploying their service agents to their free customers that they never gave support to before. They're responding to their customers more quickly, which means that the customers will ask more questions and they'll get more service. They'll put it on email addresses, or maybe they'll deploy a chat feature that they never had before. And so they're just net doing more service, which means that their customers are more satisfied,

• 24:32Speaker 01

  more effective at doing the things that they want their customers to do, um, et cetera. I will say that for us, as a, you know, a prime customer of Finn, we deploy it in all the ways, we certainly have dramatically slowed the rate at which we hire service agents and haven't substantially grown our team since we launched Finn over two years ago. And so I think that that's where the first human disruption is going to be, where it's going to eat the supply, the future supply that was going to come from new headcount.

• 25:14Speaker 01

  But, you know, I'll finish by saying this, which is that, you know, we have this metric we look at, resolution rate, it's the percentage of customer queries that Finn can resolve according to the customer. And that resolution rate has been increasing on average by 1% every single month. We did a couple 2% months in the last few months. And it's currently, I think, in the low to mid 60s, I'm looking for Fergal's nod, which means that it's some years before it gets to even like the high 80s or 90s. And that assumes that people deploy it to all of the places where they get service requests and they don't typically do that.

• 25:56Speaker 01

  And then the other dynamic is that even as the resolution rate increases and we do a higher percentage of work, each additional point is slightly harder work. All the easy work was done a long time ago. And so even if it takes 10 months to add 10 resolution rate points, that's probably not 10 points of work. And so even if in a number of years, when we're in the high 90s of resolution rate, it still might be 20 or 30 or 40% of the work left to humans. And so I'm trying to lay on the idea that

• 26:36Speaker 01

  Even when it starts to disrupt the work and doesn't just serve on its demand, there'll still be substantial human work required. And I think that that's something that people in AI and in this space, you know, be commenting on it, have got wrong where they've flipped the future really fast. Imagine this brave new, if not scary world. And I've just not realized that all technology adoption takes time and that disruptive technology serves on its demand. And that it takes a phenomenally long time to kill the disrupted categories. Like if text disrupted email, if email disrupted letters, if letters disrupted fax, guess what?

• 27:21Speaker 01

  Fax still exists. So that's a very, very, very long way to say that.

• 27:28Speaker 01

  I don't see these big layoff things happening and I don't see them about to happen.

• 27:35Speaker 02

  Yeah, if I'll maybe come in on that as well, like I definitely agree with what Owen said there. Like absolutely Finn is resolving like a really meaningful sizable chunk of the overall volume of people's businesses. Most support teams are like underwater. Like I haven't met a support team that isn't underwater by like thirty percent versus the capacity they wish they had. And so when you come along on day one, you resolve thirty, fifty percent of their their queries, maybe thirty percent of their workload. You just take them from being like underwater to being like roughly a parity. And then, you know, this is always the good news story. This is the way we used to hope it would play out before it happened.

• 28:16Speaker 02

  And we're like, oh, we kind of hope that it doesn't really, you know, have massive job losses and we hope people move up the value chain. But that's what we see, people move up the value chain. Now, there's one major exception to that, which I would say is BPOs. If there are customers that have outsourced their like tier one or their frontline customer support to BPOs, they do very frequently deploy Finn and instantly get rid of the BPO. But most of the time, the internal team like pivots and goes up the stack and just isn't replaced, you know, we saw that internally in Intercom. I think we, we, we had like our support team had needed to grow by 25% the first year of Finn. That was the, the projected

• 28:58Speaker 02

  headcount and it never happened. It just stayed static the first year we deployed Finn and then Finn has continued to grow and improve its resolution, right? So yeah, it's complicated. I, I do also wonder if there's like an economic downturn or something like that. Like I think the support teams of the world are starting to realize that this is valuable and it's real. But I think the CFOs haven't kind of cracked the whip yet. And maybe if there's like a downturn or something like that, you'll see, then you'll see impacts, then you'll see a much harder conversation about operational efficiency post AI. But right now it's a good news story and we don't know the future.

• 29:34Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 29:39Nathan Labenz?

  Yeah, I have done two ad reads for Intercom over the last half a few months. I don't know if you guys were even aware that you've been sponsors of the show, but thank you whether you were or weren't. In that time, I think it's gone from 56% advertised resolution rate to 65.

• 29:57Speaker 01

  Right.

• 29:57Nathan Labenz?

  And I guess I'd like to dig in a little bit on like, first of all, how does that published resolution rate compared to your own resolution rate as Intercom? Second, what has driven that? You said, you know, it's 1% a month and obviously there's a lot of work and testing and whatever. But like in terms of the tailwinds that have made that possible, what are they? And then what do you think is kind of most likely to come next, you know, that will take you from 65 to 75 over, you know, whatever the next however many months?

• 30:29Speaker 01

  I'll just answer the very first of that part of that question, Fergal, and then would love you to jump, jump in. I think we're in the high 60s for, for Intercom and that's pretty damn good because Intercom is this sprawling product, like so many features, frankly, too many.

• 30:47Speaker 01

  So the fact that it's actually able to provide that level of coverage is incredible.

• 30:53Speaker 01

  And we have deployed it in all the places. So a really large chunk of our customer service is now done by Finn. And we certainly are one of the, you know, biggest deployments. But we actually have customers in the high eighties and in nineties. So if you've got a kind of narrower set of questions that people might ask, it can get very high. Over to you for a go.

• 31:15Speaker 02

  Yeah, absolutely. And then in terms of the actual, you know, the process by which we get that resolution rate up, like it's a lot of work and, you know, it's really weird how there's sort of a scaling law or Moore's law like phenomenon, where it has really consistently improved at about one percentage point month on month. And we have this thing internally where so often we're like, okay, well, we have three more things to try over the next cycle, over the next six weeks. I'm not too confident. And about like two of them work and one of them doesn't, or one of them works and two of them don't. And we get this net increase of about a percentage point month on month. We have this like ever growing machine in the AI group, always trying and testing more things.

• 31:59Speaker 02

  And that's roughly what it nets out at. And like, you know, that is this constant process of like optimization, trying to like, let's refine the retrieval model, let's go and work on the re-ranker, let's go and change the prompts. You have, there's so much stuff to do there that kind of slowly gets it up. Only a very small amount of it, you mentioned tailwinds, only a very small amount of it has been like core LLM performance. That's been like a couple of percentage points over the last two years. As the overall performance has gone from about thirty five percentage points at launch to about sixty five percent now. And so like, so much of it is just this testing and optimization process we do. And, you know, we have the data on that, we have the A B tests, and then we also have the sort of the cohort in view.

• 32:44Speaker 02

  And we can see that like recent customer cohorts get a similarly high resolution rate to customers that have been with us for a long time. There's always variance within those cohorts. Some customers have a very mature setup, but they've done a lot of optimization work with all these product features to help you optimize your help center content. But overall, you know, most of it is just improvements in the core engine of Finn that we have invested very, very heavily into. So the only kind of tailwinds would be the space as a whole, where, you know, there's more and more AI and it's available. And sort of the future is really, you know, for us, we've done a lot of work recently on investing in our custom AI models. We're sort of training our own models in-house for the first time. We've

• 33:26Speaker 02

  trained a custom retrieval model, a custom re-ranker model. We're very pleased that our custom re-ranker model, it beat out, you know, one of Cohere's top models, which we previously had used. And yeah, we're really happy with that. So we really think that's the future of our investment in Finn, is kind of taking all the data we have from all these different customers and using them to like, help Finn learn and make Finn better overall. And that's been working for us. It's been delivering resolution and we're really excited about continuing to invest in that.

• 33:59Nathan Labenz?

  Do you think that will continue going forward? Because that's, I would say, a quite different story from the most common one that I hear and frankly, that I tell, which is that, you know, for, to take my own company, Waymark, for example, we help small businesses increasingly bigger and bigger businesses because the quality of what we're able to deliver is improving, you know, quite fast as well. But it started with, you know, local brick and mortar, very small, you know, niche online retail. We helped them create video content. And you look back at what we were doing three years ago, when we first brought an AI video generation thing to market versus today, it's dramatically improved. I would say a lot, you know, a lot of credit goes to the team.

• 34:41Nathan Labenz?

  Certainly, like, we've tried a lot of things, figured a lot of things out. But I would say probably two thirds, if not more, goes to the fact that the models themselves have just got so dramatically better during that time. Everything has gotten easier, you know, just the quality, even when it comes to something like generating a voiceover, you know, it went from like robotic to still mostly robotic to now like very expressive, right? And we had to figure out how to prompt that and how to adopt it and how to get things to sync up time-wise. But there were fundamental advances that were just like, wow, you know, we're just, the giants on whose shoulders we're standing are getting taller, you know, at a, at a rapid rate. So. Yeah, I don't know. How do you see, you're telling a quite different story where that seems to be less of the lift for you?

• 35:26Nathan Labenz?

  How do you think about that? And do you think that will will continue into the future?

• 35:29Speaker 02

  Yeah, it's definitely a different story for us. And there's probably a few different reasons for that. And one reason for that is, you know, we built the first version of Finn on GPT-4. And we were like very early to that. We had like advanced access to GPT-4 for a couple of months. And it really took us like across the threshold that we wanted for like accuracy and quality. But it involved using this like big beast of a model. And we used to have to run it on dedicated hardware. And, you know, we would, we kind of, we had to really architect around it quite carefully. So I would say we have gotten some improvements due to improvements at the model layer in terms of our architecture. Our architecture is a little bit simpler and it can do like more powerful things.

• 36:14Speaker 02

  But yeah, in terms of the actual resolution rate, yeah, it's only a few percentage points that have been just due to like pure model improvements. And it's almost all to do with the like testing and iteration around it. But like, you know, that said, Finn is a different architecture as well. The architecture has changed as we've gone. So yeah, so it is, it's a complicated story, but we'll be, we'll be highly confident in that. Almost all of the improvement is in like, you know, what you might call the rag layer or the AI layer outside the core models. And core models have definitely gotten better and that's been great for us. It's been great for everybody. But, you know, that is, that has improved, like, it's reduced the cost a bit.

• 36:55Speaker 02

  And it's improved reliability a little bit, but it hasn't driven core resolution rate that much. But it has given us a platform to be able to go and like, you know, provide more flexibility to our customers. We had a feature called Guidance. Guidance is where customers can go and they can like, you know, make Finn more personalized to their brands. So let's just say talk into this tone of voice. And like, features like that would have been harder to achieve with the models and the architecture a year or two ago. So yeah, the, the, the reality is, is multifaceted and it's complicated. But I would say like, if you were to look at the You focus in on resolution rate, which is the core metric that we care about, that we build for, that our customers care about.

• 37:37Speaker 02

  I would say the story is a bit simpler and the story is is is less about the the underlying models. I think video would be very different than that. And G P T four is a really powerful model, you know, G P T four, which we had two years ago, is a very, very powerful model. It was a very big model. It was a beast of a model to run. And and so, yeah, we could be to talk through the whole history, G P T four turbo, which a lot of people have forgotten about was like much smaller and much more efficient. And but like very similar in terms of power to G P T four. And yeah, and then, you know, moving to solid, we got like a couple of percentage point improvement, which we care about a lot. But against the backdrop of the sort of like thirty five percent to sixty five percent that we've had over the two years.

• 38:22Speaker 02

  And the the actual models is is a relatively small part of that.

• 38:27Nathan Labenz?

  Yeah, that's fascinating. So is it, would it be accurate to say that intelligence is not the bottleneck? And would that also suggest that you are

• 38:40Nathan Labenz?

  able to use smaller, faster, cheaper models? Like, this might suggest that you are a Haiku user today or a two point five flash as opposed to needing to maximize intelligence. It sounds like you're maybe more emphasizing.

• 38:54Speaker 02

  And so if you think it goes to cost. Yeah, if you look at like Corfing, so again, it's complicated. But if you look at like Corfing, there is a sense in which customer service is a task that is less cognitively demanding than like, you know, the Max Olympiad or something like that. And so a given level of Max, or is it a given level of model intelligence will saturate that task earlier. And so, yes, I would say that like a large part of customer service probably was saturated by models of G P T four level intelligence. In which case, you know, yeah, you you know, you spend a certain amount of time optimizing that.

• 39:35Speaker 02

  And, you know, we have, we have went and we have trained our own small models to kind of to do parts of things. So for example, One part of Finn was summarisation. We summarise the end user query before we go and do rag out. And that was worth doing because sometimes end users, they put a lot of random stuff in their query. And you don't want to pass all of that to your like your search pipeline and your embeddings. So you do a canonicalisation or a summarisation piece first. That's always been very valuable for us. We used to use initially G P T three point five turbo to do that. And then over time, we used different models. We used, you know, G P T four point one. And we used Haiku for a while, didn't have great experience with Haiku.

• 40:17Speaker 02

  Recently, we've switched that. That model is now a combination of a proprietary encoder decoder model that we have trained ourselves. And then a fine tuned version of Quen tree that we have fine tuned to be really excellent at that summarisation task. That has made it cheaper and lower latency and more predictable. And more reliable and higher quality than we were able to get from third party L L M's. So yeah, so absolutely, we are taking some tasks that for which, you know, model intelligence is sufficient to saturate the task. And we're placing them with small models. They happen to be small models we've trained ourselves. Smaller third party models have never really given us the exact trade offs that we want.

• 41:00Speaker 02

  You know, they become less steerable and, you know, there's all these different complicated set of trade offs. But yeah, that's absolutely what we've been doing. We're very excited about that. And that's a big investment we made recently in Finn. Then there are other parts of Finn that are real frontier challenges. And so, you know, a bit part of that is like tasks, right, are procedures where Finn is like interacting with external systems. We have a very high reliability bar to hit there. And yeah, for that, we need frontier models. We use Anthropic's like excellent sonnet models for that sort of task at the moment. And yeah, it's great. Or like, you know, the hardest part of our answering prompts, we're still using third party models as well.

• 41:45Speaker 02

  So it's it's nuanced. Like our our product is pretty big at this stage. Finn was always ten fifteen prompts. And then we had a different architecture for email. Now we have Finn voice, which is a whole different story. So we really, we have this big cloud of A I services at this point. And so you end up in a nuanced discussion of each one. But yeah, speaking about core Finn, that narrative is correct, yeah.

• 42:10Nathan Labenz?

  Cool. That's excellent. And just the kind of thing people tune into this podcast for. I always hear that people, you know, they want these little nuggets of understanding. It's a pretty AI obsessed audience. So, and I think this is like a very timely and sort of thematically relevant conversation too, because there's this broad question around like, well, what's missing from AI that's gonna, you know, take it to the next step? Or if we imagine a drop in knowledge worker of the future, that it does seem like all the frontier companies are racing to create. What will that have that the current things don't have? And if we can take

• 42:53Nathan Labenz?

  just raw intelligence mostly off the table, you know, if you're at high 60s today and you've got 30 percentage points ish to go to, you know, fully automate your own customer service, and it's not intelligence itself that's gonna close those 30 point gaps. Like, what are the other gaps? How would you taxonomize, you know, that which is missing? And it may not all be on the AI side too. I mean, I guess with you guys, I would assume it would be more with your customers. I could imagine you might say, well, they gotta give us more information. They gotta actually use the features. But I assume you guys are using all the features you have, you know, pretty well to the fullest. So what is missing that you think will, will need to come online to get you, you know, climbing the rest of that 30%?

• 43:35Speaker 02

  Look, I would say that like, there are definitely intelligence bottlenecks still. So like, the core task of, given this article, answer the question from this article as well as a human customer support rep, we're kind of there, you know? The models are like, really, really good at that stuff. They're intelligent. They're human competitive at constrained tasks like that. However, the more complex tasks of like, given the whole bunch of external systems and a very vague, fuzzily defined policy about when to do a refund and when not to do a refund, and needing to do that reliably with common sense, not 99 times out of 100, but 100 times out of 100, that's still a frontier.

• 44:19Speaker 02

  Task, that's still a task where the whole ecosystem is leveling up. And so, you know, a bit like how with informational queries, we have to go and use RAG and we had to tune them quite well to get to the performance we needed. It's gonna be, the the right way to do that, the right way to attack that is gonna be to use the models as a building block. And then to go and tune the envelope in which the model works to be able to like, give the right performance. So we need to do that. That's thing number one. And then thing number two is this huge task of actually deploying these things. All the human factors, convincing the security team, making sure it's secure enough, penetrating the organization. And this cuts back to what Owen says earlier, people in the discourse around AI.

• 45:03Speaker 02

  You know, it's very easy to to fall in love with some metric on a back test and to not see all those messy human factors of deployment and adoption and penetration. And, you know, the more valuable something is, the faster it will penetrate. That's absolutely true. Amazing breakthrough technology penetrates fast, but it's still a process. And so that process needs to roll out. So that's gonna take, gonna take some time, yeah, you know.

• 45:28Speaker 01

  I I, yeah, I would also add that the more systems um, a product needs to touch, the slower it will penetrate the organization. And so like, in some senses, like the informational queries for a service were nicely isolated. And so it's easy to kind of pick that up, switch it on. But this future imagined knowledge worker that has to collaborate with many different teams and individuals and use different systems and pay attention to permissions and talk to external stakeholders too. That just sounds like it's a lot, way harder adoption.

• 46:07Speaker 01

  So the the first thing that comes to mind when you ask that question is kind of Fergo's answer at a higher level, which is that like, the the the recent trajectory of the actual foundational models tells me intuitively that.

• 46:25Speaker 01

  It's not going to be the base models themselves that just show up someday and they're good to go, but rather there will be companies that deploy them to specific use cases and build, as Fergal calls it, the envelope around that to do this work highly effectively. I mean, if someone built the sophistication of system that Fergal and the AI group at Finn have built for a range of different use cases, it would probably be ready today. I don't think that the technology itself is actually holding people back. It's the raw, hard work needed to point these things in the right direction and help them

• 47:08Speaker 01

  be effective and all of the work around the R and D to help companies adopt them. That is the difference between where we're at today and companies having, you know, real AI knowledge workers.

• 47:20Speaker 02

  I completely agree with that. I would also say that like, you know, I do think there's something maybe missing from some parts of the discourse, you know, when people talk about like the models are going to get so good, they're a, you know, a country of geniuses in a data center, that sort of thing, you know, like certainly there needs to be changes to the model capabilities to really achieve that. And so like, you know, the models are missing a whole bunch of key capabilities today. They're missing memory, right? They typically run in a stateless fashion and yeah, you can go and you can put memory outside and that kind of works, but it kind of doesn't. They're also, they're missing like what someone might call a system tree, that they're missing the ability to like grind on a problem and to like learn about that problem, right?

• 48:05Speaker 02

  So if you have like an intern, never mind, you know, some PhD level, you know, genius or whatever, if you just have like an intern out of school and you go and give them a task, they'll do pretty bad at the task the first time in a way that like probably Sonnet won't. Sonnet will give you consistent performance. But like the intern can learn and they can, you know, you can really learn in a task specific way over time. And like the models, they just can't do that. There are research prototypes where, you know, they're trained with reinforcement learning to go and like update their weights when they hit a certain point in a maths problem after doing a lot of chain of thought or something. That's all like, none of that's deployed yet. And so, you know, I think Crowe just did something

• 48:48Speaker 02

  interesting with like a live reinforcement learning system. I think people are going to be looking at that. But even, you know, that stuff is, you know, it's risky. And so, so we'll see. There are direct fundamental capabilities that are still missing from the overall intelligence layer. We'll need that. In the meantime, companies like us will have to like build around that. And even after we have those, as Owen says, you know, there's going to be a ton of work in building the envelope to make the system actually work for a business, you know. If you had some like crazy genius, you wouldn't want to give them, bring them to your company and let them just do everything, you know. You still want to train them, you know. Sometimes if they're, if they're a real genius, they need a lot of training, you know. There is a value. If you want to have someone come and do your customer support,

• 49:31Speaker 02

  do you want someone who has like 10 years of experience doing great customer service? Or do you want a crazy genius? Like, it's not obvious that you want a crazy genius to do your customer service, you know. So yeah, there's a lot to do here.

• 49:46Nathan Labenz?

  Yeah, I think that's really interesting. On the refund point that you mentioned specifically, you said, you know, not 99 times out of 100, but 100 out of 100. I wonder if that is, do you endorse that way of thinking for customers? Because one thing I've noticed when it comes to like self driving, for example, is Waymo is way safer than a human driven taxi. And it's like almost 90% maybe not quite, but like getting to sort of 90% ish reduction in accidents and in injuries. And yet we're just starting to deploy. So I wonder how often you see that essentially happening with your customers where, you know, to quote Biden, who used to say, you know,

• 50:30Nathan Labenz?

  don't compare me to the almighty, compare me to the alternative. How often do customers just fail to realize how accurate or inaccurate their humans are, how consistent or inconsistent they are amongst themselves? Again, at Waymo, to take a personal example, we had tried to evaluate aesthetically the image assets that our small business users would bring to the platform so we could make intelligent recommendations. We found that it was really hard to even establish agreement among people. Like, directionally, yes, there'd be correlation, but to say that everybody would rank them the same, like, far from it, right? You could, I always say, you could tell the top end of the curve and the bottom end of the curve, but in the middle, you wouldn't necessarily even know which way it was up or down.

• 51:13Nathan Labenz?

  So, all that to say, is there a way in which measurably, if people measured, the AIs are as good or better than what is happening today, but people, for one cognitive bias or motivated reasoning, reason or another, don't want to recognize that or are prepared to take the risk on the AI that they're already running with their humans?

• 51:37Speaker 02

  I mean, I can definitely give probably a couple of conflicting answers to that, because I think we all internally spend a lot of time thinking about this. Look, in one way, it's like, yeah, some customers have a nuanced understanding that their humans are imperfect, and some of them even will have, like, an error rate that they know their CS reps will do incorrect refunds for, and then that's the bar. So that's one answer. But another answer is that, like, if I was a PM at Waymo, I wouldn't be just trying to equal the human accident rate, I'd be trying to exceed that by two orders of magnitude, because that's what's required to really build a product that will penetrate very fast through the market. And you can spend all your time obsessed that people are holding you to too high of a

• 52:20Speaker 02

  bar, but for better or worse, people do hold new products to a higher bar. And I think, you know, we've always kind of engineered for it. We've always wanted to give not just good enough or not just human competitive error rates, we wanted to give the best error rates we possibly can, the lowest error rates we possibly can. And you know, the the the understanding changes over time. And then it's different for some customers. We have some customers for whom they're like, I'm a regulated industry. If, you know, a human makes a mistake, I can talk to the regulator about that. The regulator will understand it. But I'm worried that if the system makes a mistake, the regulator won't understand it. We have customers who are in that boat, and you know, regulators is also new to regulators.

• 53:03Speaker 02

  And then we've customers who are in the boat where it's like, no, I can make a judgment call. I know it's superhuman. I believe you. I've done the trial. Let's go for it. And so, you know, like the adoption of banning your technology gets complex. And Owen, I'm sure you have, you we talked about this a good bit as well.

• 53:19Speaker 01

  Yeah, totally. I think, you know, initially, as Virgil said, the expectation will just be remarkably high. There's just a great degree of kind of fear and skepticism. And so the lens, the microscope is really on every single interaction.

• 53:35Speaker 01

  It's almost expected that these new disruptive technologies just won't be as good, and so they need to oversell. But I think as soon as people start to realize the ways in which it's vastly superior to humans, maybe not all the times, but quite often, they'll give it a lot more permission for error. And Waymo has been in the market and with consumers a lot longer than the technologies that we are building. You know, people are not actively talking about the incredible customer service experiences that they've had. Often that doesn't even register that it's A I. And you'll see even with Waymo that people endearingly will forgive its mistakes. Oh, like stop behind this car that was pulled over and I got a little angry at it and then it kind of like turned, you know, pulled past it.

• 54:19Speaker 01

  And it's it's endearing. It's funny, right? So I I do think that just at large, we as a society will warm to the eccentricities of A I and be quite forgiven because we just know how effective it can be. And in customer service, for example.

• 54:37Speaker 01

  It turns out that people really hate having to reach out to human customer service. They know they're typically going to have to wait, you know, days, maybe hours, but probably days. They're going to reach someone who doesn't really want to do their job. They're probably going to get a crappy half answer that doesn't really answer the question. The whole thing is just really unpleasant. But when you have a snappy, happy, expert concierge agent, apparently very keen to hear from you and willing and ready to answer in seconds, they're going to ask a lot more questions. And so a lot of it is just us as a society building more meaningful relationships with these things.

• 55:18Nathan Labenz?

  Yeah, speed is really killer. In preparing for this, I took a look, and it hasn't been my department for a while, but I took a look into our intercom data. And, you know, for context, we have always really invested in customer service and, you know, we've got great people doing it and they actually do, you know, show up with a smile and interact with customers in a really authentic, thick way. And our customers have always really appreciated us for that and specifically, you know, called out individuals on our team an awful lot. And yet, you know, one thing that it is impossible for the humans to do at the level that the AIs can do is the speed of response. So we typically respond in two

• 55:59Nathan Labenz?

  minutes, which for a relatively small company with a small team is pretty good. And yet, it's just enough time for the person to tab away and do something else. And then when they get our response, you know, they tab back five minutes later. And the next thing you know, we're like 30 plus minutes to get to resolution, even with everybody being like attentive and doing a good job. And the AI's ability to just be there immediately is like, you know, that's a, that's a pretty dominant advantage that we're not going to catch up to anytime soon.

• 56:29Speaker 01

  It's true. I'll tell you a couple of funny stories about that, though. If it's too fast, people don't think it really is going to give it the right answer, that it's probably like a pre-baked crappy automated macro answer. So if it's too fast, people don't quite trust it. And if it's

• 56:51Speaker 01

  Too slow, but still way faster than a human. Let's just say 20 seconds. People will be pissed off because it's like, hey, this is AI. And the reality of where the technology is today is that if you actually gave it a couple of minutes as opposed to require that it responds in seconds, which I have pressured my team a lot to make sure that it does, it can actually do a better job if you give it a little bit more time. We've seen that with, you know, the latest models that are really just the same foundational technology applied, you know, at runtime to just have the permission to think a little longer.

• 57:33Speaker 01

  So the, the response time thing is actually super nuanced. And it also relates to people's understanding and expectations of AI. The, the one thing I will say though is that despite in some senses, my pessimism about how quickly the base models are going to improve, I do think that they will get just far faster and cheaper. And so as we build all these technologies together, we will be able to probably do the runtime stuff just quicker. We'll be able to provide that level of response, just in the amount of time that people expect.

• 58:14Nathan Labenz?

  How do you think about in the, if you kind of zoom out from where the product is today? And the, you know, particular architectural decisions that you've made and all of the sort of history. How do you know when to, this is like in some ways, the most human thing still relative to what AI is, is good at. How do you know when to be willing to consider changing the paradigm? You mentioned like voice, which is obviously a, you know, quite different paradigm than a chat. And I think another big trend that I'm sure you're thinking about a lot is what I call the choose your own adventure style of agent as opposed to the highly structured, a scaffolded, optimized, you know, controlled input output, more like a workflow agent.

• 58:55Nathan Labenz?

  Seems pretty clear that you started with like a very, you know, methodically task decomposition based approach to getting every little step in the chain working well. And it seems like Anthropic and others are now trying to push like, well, you don't need to do that anymore. Hopefully, we're going to make this thing so good that you just give it some tools and it'll choose the right tool, it'll figure it out. How would you, are you doing radical A/B testing where you have these two different paradigms in competition with each other? And how do you think about if and when it would be time to make such a big paradigm shift?

• 59:35Speaker 01

  I'll say one quick thing, which is that we're going to share at our Pioneer event in New York, which is our big customer event, which what date is it, Fergal? October?

• 59:45Speaker 02

  The 8th of October.

• 59:47Speaker 01

  October 8th. We're going to share basically what we've been working on and what comes next. And part of that is going to be a paradigm change, because I think that there is a big opportunity for stepping back a little bit and thinking more broadly about the problems that we're trying to solve. And I think that the kind of answer bot, Q&A bot thing is fine, but it's still quite limited. So I don't really want to say more about it than that, but these things will need to become more agentic and there's big opportunities for that. But Fergal can probably talk to some of the A/B testing that we do do, but I don't think it tends to be very radical. There's probably a spectrum from radical

• 1:00:29Speaker 01

  to micro optimizations. And I know that we do stuff in the middle at the very least. Fergal?

• 1:00:34Speaker 02

  Yeah, I'm really excited for that announcement at Pioneer. Look, to your question, specifically at a technical level, yeah, it's a very interesting time for architecture. I can tell you with high confidence that, so firstly, let me say that your synopsis of Finn is correct. Finn originally, very deliberately was this sort of like, or is, still is, this building block style architecture where you use the models in as building blocks and then you carefully isolate them and you carefully test them and optimize them. That's worked really well for us. And we've invested in training our own custom models for some of those building blocks recently that have been more performant and way better than

• 1:01:18Speaker 02

  some of the third-party models that we're using. So that's true. And some of those building blocks are definitely durable. So in a rank system, you need a retrieval engine, you need a re-ranker, you need a few other pieces like that that are definitely going to be durable. But I do think there is an interesting architecture switch or push on the cards in future and and Tropic and others have been pushing it. I can tell you that we're we're pretty confident that switching to that today would reduce the quality of Finn at kind of the question answering task versus where it currently is. And so like we're pretty happy that, you know, for the highest quality and that we are where we need to

• 1:02:02Speaker 02

  be at the moment. But we have built prototypes there. And yeah, if we get to the point where we have another generation architecture that is like that form you're talking about, yes, we would A B test that in the end. That is ultimately how we would get a sort of a full spectrum analysis of, you know, exactly its trade offs and its strengths and weaknesses compared to our current architecture. And that's how we've done architecture migrations in the past. Internally, we're on probably about generation four of core Finn at the moment. And each architecture change, we've done something like that. And some we've A B tested in production and scale. And some architecture shifts have been comparative in size to sort of what you're talking about there. So yeah, we'll

• 1:02:46Speaker 02

  see. But as Owen says, we will have things to share pioneer that are quite relevant to this from from a product perspective.

• 1:02:57Nathan Labenz?

  Cool. Well, we'll stay tuned for the next big update there. When it comes to the impact that Finn has had on your business, as far as I know, I think it was the first pay per outcome pricing scheme that I remember seeing with the, at least in my, you know, corner of the world, famous 99 cents per resolution pricing model. When we were first scheduled this, I was like, I wonder if they still have that. And sure enough, you do. How is that going? Like, if you could go back and do that all again, would you, you know, do that the same way? Would you price it at the same point?

• 1:03:41Nathan Labenz?

  And because obviously, like, a lot has changed under the hood, right? Like, we've got per-token prices are way down, but amount of tokens put in for, you know, context is, at least in many use cases, way up. Plus, you've got thinking, you know, tokens, which means outputs are way up. So I don't really know how to guess the cost basis has evolved, other than just to say there's clearly, like, strong forces pushing in either direction. But having anchored at that price point, you know, and probably being pretty reluctant to move it, you know, you've kind of got to manage against it, I would assume. So what can you tell me about what's going on under the hood? And then, you know, what would you say you've learned? And maybe, you know, any advice you would give to people that are thinking about a per-outcome pricing model?

• 1:04:26Speaker 02

  So, yeah, exactly as you said, and, you know, core LLM costs have definitely been falling. Cost per-token has definitely been falling. I guess the converse thing for us is that, like, Finn is doing a lot more work now than it used to two years ago when we launched it, right? So it's like resolving a much higher percentage of your inbound volume, but also it's resolving harder questions. And so, yeah, we have to spend more token budget on that for putting more and more things into our RAG system. And then there's all these others kind of supporting AI systems around Finn. So we've things like our Insights product, we have things like, you

• 1:05:07Speaker 02

  know, ways we kind of double-check and check, like, the content that goes into Finn. Our chunking strategy is way more complex and, you know, we've just, you know, we've added other pieces to kind of, you know, really make Finn better over time. And so, yeah, so I do think, yeah, absolutely, that the core, you know, cost per-token has come down a lot, but Finn is doing a lot more work than it used to, as we're trying to, like, push up the resolution rate more and more. And like, we're constantly struggling, like, our big North Star, a thing we really care about is like, you know, what percentage of your inbound volume are we actually resolving? Under Owen's direction, we're investing a lot in, like, making the product better and better

• 1:05:48Speaker 02

  and better. And that's the thing we're pushing towards, rather than optimizing purely for cost. So, yeah, that's kind of how I would say it from the applied level.

• 1:05:58Speaker 01

  I'll tell you a little bit about our story and thinking, but first I want to

• 1:06:05Speaker 01

  set us apart from some of the narrative in the market where people are talking about negative gross margins. When we launched Fan, I was told it was going to cost $1.21 per resolution. And we decided that, for reasons I'll speak to in a moment, that we must be more aggressive. And we really just, we just really liked the attractiveness of 99 Cent. And so we actually took a hit on each resolution at the start. But pretty quickly then turned that dynamic positive and then achieved software level gross margins.

• 1:06:45Speaker 01

  And so any narrative that these AI products, and particularly the agent products, need to be, you know, negative or even crappy gross margins is clearly not always true because we've shown that to be the case, even while there has been pressure because we're doing more work. Philosophically, we wanted simple pricing and we wanted pricing that would map to value. We learned a lot of hard lessons from having complex pricing in the past because we used to do a lot of things for different types of customers. And so we had many aspects to our pricing models that tracked different metrics. And it just became a nightmare for people to track and follow.

• 1:07:30Speaker 01

  So we really have been simple to a fault and probably far simpler than we need to be. There will come a point where certain classes of resolutions that just save their reps an hour or multiple hours of work,

• 1:07:48Speaker 01

  where we will charge more for those. They'll still be super cheap. I have no idea what it will be. Is it $1.99? Is it $2.99? Is it $3.99? Is it $4.99? I don't know. But we'll always make sure that it is easy to translate from a value perspective so that it's always a no brainer for customers. Our own work showed that we spend $26 per resolution all in. So that's salaries and everything like offices and benefits and all this stuff.

• 1:08:21Speaker 01

  And we've come across companies that say that they are as low as $5. We've come across some companies that say that they're lower. We don't think that they're doing the analysis correctly. And so we're highly confident this 99 cent is just a great deal. And what we really, really like about it, not only that it's simple or a great deal is that it also aligns our incentives. And so if we charged some sort of synthetic token or for every conversation, it would actually not be in our interest to make Finn more effective at solving customer problems. If we charged for every query we got, we're good. We're golden. We don't need to get better.

• 1:09:05Speaker 01

  Whereas when we charge per resolution, every time Fergal's team increases that by one percentage point every month, we make our customers way happier and we make our CFO way happier.

• 1:09:17Speaker 01

  So it's this beautiful alignment. The biggest cost to this outcome based pricing is that it's novel. And so there's a bit of education.

• 1:09:28Speaker 01

  We were certainly the first in the market. My favorite part of that story is that two other companies also announced that they were first in the market was Zendesk. And I think Sierra did the same, although I need to double click on Sierra, but they made a big deal about outcome based pricing. But we were certainly the crazy first people to do that. And we did experience an education cost, but when we went back and surveyed customers and tried to find out if they would prefer per conversation costs, which some of our competitors do, they said no. And so we think that outcome based pricing makes sense for all those reasons. And I can't see us about to change it.

• 1:10:09Speaker 01

  And certainly historically, when you read some of the kind of academic approaches to pricing, there's a great old book I read years ago called Pricing on Purpose. That analysis showed that value based pricing always yielded a higher profit than cost based pricing, even though cost based pricing has a baked in profit. The problem is that it doesn't properly price discriminate.

• 1:10:37Speaker 01

  People who get more value from your product than others will pay the exact same. But when it's value based and people say, okay, yes, I pay more for it doing more work. They're happy for it to do so because it perfectly maps the things that are important to them.

• 1:11:00Nathan Labenz?

  Yeah, that makes sense to me. We're trying to move toward value based pricing as much as and outcome based pricing as much as possible as well.

• 1:11:07Speaker 01

  The one thing I would say is that it's not just an academic problem. So you can come up with all these beautiful theories about how this is simpler and maps to value, but if the education and the friction that comes from that is too much, it might totally fail in the market. And so you need a degree of kind of artistic license with your pricing where it's not just science, there's real art and you need to be willing to kind of leave money on the table in a bunch of cases. For example, we have this really deep insights product, the best in the market, helps you understand exactly what's happening in your business using this modern AI, these modern AI technologies.

• 1:11:51Speaker 01

  But to this date, we've decided to not charge for it because we know it makes people better at using our product and it helps them get a higher resolution rate out of it. And so that's an example where we're, the science has been ignored at Intercom for the sake of simplicity and the art in pricing, which is making something easy to comprehend.

• 1:12:12Nathan Labenz?

  Yeah, well, it sounds like you feel like you nailed it and it certainly has a beautiful simplicity to it. So I can see why it's working. I, we don't have too much time left. I did in my prep for this, I did two things which kind of. converged on the same outcome for Waymark as a business. One was I went to the dev platform docs. And if I could make a customer request, it would be to create an L L Ms dot T X T. If you haven't in the last like six weeks since I did this. But what I did at the time was sent. I guess it was chat G P T operator. I don't know if it introduced agent at exactly that moment.

• 1:12:55Nathan Labenz?

  Sent it against the docs and said, visit all the pages on this docs website. Put them all into a Google doc for me. I think I got to, you know, however many hundred pages as it kind of went back and forth copying and pasting. Then I took that, put it into Gemini and said, create a consolidated, you know, just what I need to know. Cut all the crap, you know, that's repeated from this. And it did that and gave me basically an L M's T X T. And then I went to Claude and said, code up this data export for me so I can do an analysis outside the product. And Claude, by the way, one shot at that. And so I got my data exported into a C S V and that was pretty cool. And then I went into the product and I was like, oh, I mean, I kind of actually already created this

• 1:13:37Nathan Labenz?

  with these background insights, product type experiences. So I guess the main question, two questions there would be, are you seeing people doing a lot more casual hacking against the A P I's? And also then your own proprietary product work. How have you, you told a little bit about it in terms of just making people better. But I was kind of surprised to see that there was so much happening in the background that I wasn't even really aware of until I went looking for it. So I'm kind of interested in the, in the product philosophy there to the degree that there's more that you haven't already said.

• 1:14:16Speaker 02

  I, I, I don't know if our A P I usage has increased. I, I would say that like, you know, You're describing a pretty sophisticated story there, like you're not an outlier, but like you're a sophisticated user of A I in our customer base when you tell that story. And and so it might just be a little early yet. Absolutely internally, when in Intercom, we're really pushing ourselves to be early users of all these products. I remember being really blown away by Claude Code recently when I kind of saw it first. And, you know, we definitely tried to up level our own team to be good at those things. We've really built a big insights product and there's an awful lot of A I in the insights product to in order to deliver it.

• 1:15:00Speaker 02

  And so I I guess I guess I'll I'll say two things on that. The first thing I'll say is, you know, we built that with sort of, you know, custom or semi custom A I. Rather than doing something like just expose everything to, you know, a deep research style environment. We did that because we needed to do that to hit the quality bar, the performance, the efficiency. We have some customers that have very high conversation volumes. And, you know, all the models, they have context lens and the context lens get high. But the fidelity decreases when you try and use those very high context lens. And so I, in our opinion, eh, the way we built that was still the right way to build it, which was very much like.

• 1:15:41Speaker 02

  We use L L M's to process the conversations into kind of like topics and subtopics. And then we use sort of a a BERT style, you know, medium sized language model to deliver it at scale. And we think that's still the right way to build an insights product today. If you are a medium or a large customer, you can't just throw everything into a big context window. Like we've experimented with that, but the quality of output degrades. And you can end up with a product that looks nice superficially, but isn't really reliable. So we think that today, to get something really, a really industrial strength insights product, you still got to go the way we went. Um, now you're kind of like, oh, there was a big insights product there I hadn't really heard about.

• 1:16:23Speaker 02

  And yeah, candidly, um, it is difficult to market the depth of technology. We have this very broad product, it's very deep. And, you know, we, like everybody else in this space, are struggling to communicate against all, you know, you mentioned there you're keeping track of A I. There's just so much noise in this space and, you know, we're still struggling to find the right way to tell customers, look. We've built a deep product hearing. Clearly, you can see that and you can recognize the depth there and a lot of folks count. And so we're still working, we're still up levelling our product marketing game all the time in this new world of AI to get great at like telling people what it is. And I'm just really relieved that we've managed to improve our marketing

• 1:17:07Speaker 02

  of like complex queries and procedures recently. We had deep technology there for like a long time that was just, you know, people weren't ready for, or we weren't doing a great job at explaining it. But I think, you know, I think insights is a deep product as well. Yeah. Oh, and I'm sure you have thoughts on that.

• 1:17:26Speaker 01

  No, only to say that I agree with you that depth of usage is just a problem amongst all software vendors, you know, it's customers discovering the full things they've poured blood, sweat and tears into. It's just a problem. And it's not, you know, if anyone who imagines like some better onboarding pop-ups as a solution are completely wrong. I mean, yeah, just the amount of just attention to go around is just at a bare minimum. There's just so much contention for that attention. It's just a giant problem. So this is where

• 1:18:10Speaker 01

  some of these kind of high-touch, you know, more enterprise oriented businesses are being affected now because the vendor is stuck and stitched into the organization. The organization has committed very large amounts of money to use a product and they get the benefit of their attention and an opportunity to train them on the whole thing. But if you're kind of more in the kind of like upper end of mid-market, bottom end of enterprise where we are at, having everyone discover your features is hard. That said, You also want some degree of progressive discovery. You can't overwhelm people.

• 1:18:52Speaker 01

  And most people do find this functionality, as you finally did. And doing so brings a level of connection and kind of commitment to that product that is invaluable. And it'll be very hard for you now to go and move to one of our competitors because they don't have this Diamond Science product. So it's not a solved problem, and trust me, it's something that causes me a lot of heartache. Because you've got great people like Fergal killing themselves to build stuff, and then it's like, well, don't look at the data, but only 26% of people actually use that feature.

• 1:19:29Nathan Labenz?

  Yeah, that was, I mean, OpenAI just said that I think it was only 7% of people were using reasoning models before GPT-5, so you're in good company there. Well, let's maybe zoom out from all the AI product work, and I really appreciate you getting so deep into the weeds with me on that. And just think about the rest of the company. You mentioned like you've kind of slowed hiring on human agents and Fitch is doing a lot more of the work. Is there a similar impact when it comes to hiring junior developers? And how are you handling the adoption across functions? You know, is there a sort of everyone must use AI kind of mandate?

• 1:20:12Nathan Labenz?

  Is there, you know, a budget that people are kind of go forth and spend it on, whatever tools you think are interesting and cool? Or have you picked, you know, something that, you know, is sort of the official stack? Just really interested in how you're thinking about all that. And also, I guess, if I could tack on a one B, does this all mean that you will be potentially even more ambitious in terms of expanding into sort of adjacent niches? Like, one thing I do wonder about is whether companies with a certain platform will see, you know, maybe in the past I had to be focused, but now I can kind of layer on, you know, or append these adjacencies and actually feel like I have a good chance of making it work because I get all this productivity benefit from AI.

• 1:20:57Nathan Labenz?

  So there's a lot there. You can take it in any direction you like.

• 1:20:59Speaker 01

  But just to ask a number of things. Okay, I have a number of things to say to cover that. I think Fergal can probably speak to maybe developer efficiency, and there's certain things we're doing there. First thing I'll say is that I'm actually between two minds on the topic of AI usage in organizations. On one part, I think that companies that don't adopt AI broadly will become the dinosaurs of this industry. And I believe that young companies that are just AI native from the get-go have great advantages and a lot of efficiency and can move more dynamically and maybe even be more creative. That said, there's just so much AI out there, overfunded, overhyped. There's like 10 products in each category.

• 1:21:44Speaker 01

  And I don't trust, you know, a company of 1,200 people with a big budget to properly discern what's net valuable or not if they're told, "Use more AI." You can be guaranteed they'll buy a bunch of crap. And I've already seen us adopt some big AI platforms with big fancy names and I'm like, "Do we need this?" And probably we did, and maybe I'm like the old guy and I'm like too grumpy and jaded and cynical, but I think we're gonna, I'm quite certain that in a couple of years, we'll

• 1:22:24Speaker 01

  have a more nuanced view of what AI was outstanding for and what was kind of a bit of a pipe dream. So that's the broad piece. I'll speak to the idea of going into more areas and spaces. So this is part of what we're gonna start to talk about soon, but like, it's just patently obvious to us that people are not gonna have multiple agents talking to their customers. It makes no sense. You're gonna need a coordinated approach to solving customer problems.

• 1:23:00Speaker 01

  If you have a sales agent with one set of goals and a service agent with another set of goals and an onboarding agent with another set of goals, they're going to be competing against each other. And if they come from different vendors, they'll have different styles, different approaches, they may even have different interfaces. How can you track goals across all those agents and effectiveness and really coordinate and orchestrate them? It's not going to happen. It's not going to happen. So it's quite obvious that for Finn, which is the leading service agent by customer count, revenue, performance, we win all of our head to heads with our direct competitors. And in our benchmarks, it's quite obvious that we need to also be the leading customer agent. And so we'll

• 1:23:42Speaker 01

  work through the entire customer lifecycle and make sure that not only do people not have to split between different agents, but we can finally realize our dream of this beautiful, high-touch concierge experience for every single customer where they have the attention and personal treatment that we've all dreamt of giving our customers. It's not actually possible. So that's the way in which very holistically and in a very qualitative way, these agents are going to be so much better than humans. You're talking, we spoke earlier today about how they're faster, but that's actually quite quantitative. But you're going to see pretty soon when we really properly push the fact that Finn

• 1:24:25Speaker 01

  is a customer agent and many people are doing things with it today and we have it deployed in some other use cases, which we'll talk about at our event, you're going to see people then really start to get excited about the experiences that they're having, you know, incredible levels of attention and personalization. So that's the thing that we're really focused on. When it comes to engineering efficiency, what I'll ask Virgil to speak to, if he has any take on this, is the fact that our CTO, Derek Curran, did decide some number of quarters ago that we were going to actually two X our Our engineering output and our efficiency. And so I thought that that was quite cool.

• 1:25:08Speaker 01

  I actually didn't have a good read on how doable or not that is. Some people on the outside of the company said, oh, that's easy. But I actually think that's probably a pretty lofty goal. Have you any take on that, Fergal, on how far we are from getting there?

• 1:25:20Speaker 02

  Yeah, I mean, it's pretty funny. Earlier, you talked about just mandating overall improvements and like people are going to go and buy crazy tools. And look, you know, the two X thing, I definitely had mixed feelings on because, you know, I think we certainly put it in the perf system in some form. And yeah, that can always be a dangerous thing to do, right? You know, people will start optimizing for it in the wrong way. But on the other hand, it's like, there's a change in the world and like people are resistant to change. People will not properly adopt a new technology because they're busy and change is hard. And so, you know, you do need to kind of say, hey, this is a priority for us as a company.

• 1:26:01Speaker 02

  We are carving out time. We're going to make you focus on this. And so I really appreciated the push of it. Within my group, the AI group, you know, in one way, it's like, well, we're all AI technologists. So we're going to be great at adopting this new thing. But on the other hand, it's like, we're all AI technologists. We better not take it for granted. We better not assume we're great at adopting this new thing. And I remember using Claude Code, I came across it online, like really soon after it was first publicly released. And I played with it and I really, it took me out for that week. I spent all that time that week, like hacking on it, like late into the night, building prototypes and just being like, wow, there is a change here.

• 1:26:42Speaker 02

  And then I went and I, you know, demoed it to the AI group at my all hands on Friday. And like, some people were pretty skeptical. Some people were like, we already use Cursor. And I'm like, yeah, Cursor is great, but like, this is different. And, you know, so I think people do need a push. And you have to push yourself constantly because it can be fatiguing. And, you know, to experiment, a lot of those experiments will be wrong. And like, look, you know, how close are we to two X? I don't know. I don't even know if it's achievable and it's one of these, it's one of these soft things. I know for a fact there are things that we do now that we just wouldn't have done before due to the reduction in friction. I can like hack together a prototype.

• 1:27:27Speaker 02

  A designer can hack together a prototype. And so there's definitely qualitative changes there. You could fight endlessly about the quantitative impact, you know, whereas the bottleneck and those law like, you know, if you improve part of the system, how much does the system overall improve? But like, there's no doubt that there are incredibly powerful tools here. And then there's a mistake to ignore them. You've got to engage with them positively and optimistically. You've got to try it out. And then, yeah, you can't be dogmatic about it. You've got to be pragmatic and stirring in the end and be ruthless and like, okay, is this a toy or is it really valuable? And if it's valuable, double down on it.

• 1:28:06Speaker 02

  And I think that's it. That's the sort of the explore exploit process. We want all our people running. We want them to like not be stagnant. We want them to be curious and optimistic and like trying out the new technology. And then be rigorous about thinking, has it, did this really make me faster? Or did it make me slower? And when it works, double down and double down on similar things in future. And I think Dara's push, I think the company is moving to a posture like that. And there's skepticism, but there's optimism too. And I, I think that's probably the ideal thing, optimism and skepticism together.

• 1:28:46Nathan Labenz?

  I think that's a great note to end on. This has been a fantastic conversation, guys. I really appreciate the level of depth and detail and simultaneously the level of strategic vision that you've been willing to share. So definitely we'll keep our eyes open for your upcoming announcements at the Pioneer event. You want to tell us when and where that is again?

• 1:29:04Speaker 02

  It is October 9th in New York City and it's going to be a very exciting event.

• 1:29:09Speaker 01

  And it'll be live streamed also for anyone that wants to watch it online.

• 1:29:12Speaker 02

  New York City on the internet.

• 1:29:16Nathan Labenz?

  October 9th. Thank you again, Onikabe and Fergal Reid from Intercom. Thank you both for being part of the cognitive revolution.

• 1:29:26Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of A16Z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot ing.

• 1:30:10Nathan Labenz?

  And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts, so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI Meeting Notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI Meeting Notes free for 30 days.