Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

What changes when an AI mistake doesn’t just hurt a benchmark score, but affects someone’s housing or healthcare? EliseAI Head of Applied Research Mario Martone argues that deploying AI in essential industries requires a very different standard of reliability. Mario joins MTS to explain how EliseAI handles roughly 5 million phone calls a month, why LLMs aren’t the right solution for every problem, and how the team combines language models, traditional machine learning, neural networks, and internally built evaluations to decide what actually works in production. Drawing on 15 years studying theoretical physics, Mario also explains how the scientific method shapes EliseAI’s research culture, from testing new models against its own benchmarks to fine-tuning open-source models and automating increasingly complex workflows across housing and healthcare. And he makes the case that applying AI to the real world can produce problems every bit as intellectually difficult as the ones he encountered studying string theory.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I did not expect that, but even after coming from string theory, I would say the complexity of the problem of working now is probably higher now than it was before.

• 0:09Erik Torenberg?

  An AI system that works 90% of the time might look impressive in a research paper, but when you're handling millions of conversations about people's housing and healthcare, that remaining 10% becomes an enormous problem. In this episode, Elise AI Head of Applied Research Mario Martone joins MTS to explain what it takes to move AI from the lab into essential industries. Mario spent 15 years in theoretical physics, studying problems like black holes and string theory. Now he applies that same research mindset to production AI, where his team has to decide when to use an LLM, when traditional machine learning is actually better, and how to evaluate systems operating across millions of real-world interactions. They get into Elise AI's voice stack, its work fine-tuning open-source models,

• 0:54Erik Torenberg?

  and how the company has automated workflows that can take someone from looking for an apartment to touring it and eventually moving in. And for Mario, the intellectual challenge hasn't gotten easier. He argues that some of the problems his team is tackling in production AI may be even more complex than the ones he encountered in theoretical physics.

• 1:15Speaker 02

  Hello, everyone, and welcome back to MTS. Today, we are joined today by Dr. Doctors, because we have two PhDs.

• 1:22Speaker 00

  Two PhDs.

• 1:23Speaker 02

  A double doctor, Mario Martone. He is the Head of Applied Research at Elise AI, a company building conversational AI for essential industries like housing and healthcare. And he has a background in theoretical physics, physicist. He now leads a team applying the latest machine learning advancements to real-world problems. So Mario, welcome to MTS.

• 1:44Speaker 00

  Thank you. That's great.

• 1:45Speaker 02

  Yeah, welcome to the show. I did want to ask a little bit about your background, because you started off being a professor and theoretical physicist, and now you're actually applying all of that learning into applied AI. And I was just sharing, we were having a little conversation right before we went live about why it's so important to use these skills and apply them, especially to essential industries, like housing and healthcare are probably arguably some of the most essential industries. And so I wanted to just start off learning a little bit about you, why that change, why Elise AI, and what did you take with you?

• 2:21Speaker 00

  Right. Well, I think I agree with you that it's a big gap. So just to be precise, I mostly worked for 15 years on trying to understand the universe, black holes, string theory. Things are very, very kind of different from what I do daily on Elise AI. But I think there is something that actually is quite important, right? Like, we are living in this moment where AI is moving so fast, and it's really unclear, right, what the future will hold. And so there are extremely open questions. And so the similarity that I can see, while obviously, you know, working in string theory, things take forever, it's much slower than the pace, and here it's like bum, bum, bum, every day there's like a change. But the easy question of like, the people who are able to like really think deeply about the problems

• 3:06Speaker 00

  can be very creative, can constantly kind of come up with new ideas, have a huge edge, because AI is like a new field every day. So that's the thing that really, I think, brought me and here and allowed me to like really have a big impact.

• 3:19Speaker 02

  And tell me a little bit about the space. One of the big things, I mean, I'm a little bit biased, because we just had Mina Song on last week at the New York Stock Exchange. We're talking all about essential industries and why it's so important to actually use this intelligence and get it diffused inside these industries. Just having stronger models on their own inside labs doesn't necessarily mean the impacts of that intelligence actually gets diffused into the real world, into actual industries.

• 3:46Speaker 00

  In fact, you know, for me, it was like, you know, one alternative I could have done, because I was kind of interested in going to industry, is to, for example, try to join one of these like frontier lab, right? But the research that you do at frontier lab, it's kind of obviously different from theoretical physics, because it's much more alike. Like, you're working on this big problems. But one thing I really wanted is to have an impact in the real world. And I think Elise is like such a unique company in that respect, because, you know, you can have these LLMs that are getting growing capabilities, but we are facing every day with the real challenges of what it means for have like consequences of decisions. So we actually, the decisions that the LLMs make for Elise, for our products, they're not just like some random things that appears on the screen, but like drive impacts on people's life.

• 4:30Speaker 00

  And that could be very positive or very negative. So that's the thing that really excites me, and I think it makes Elise kind of such a unique space that we're in.

• 4:36Speaker 02

  Completely. I mean, it excites me too. I was taught, what's unique about your role, so you do applied research at Elise AI. I think right now, Elise AI's tech stack, it includes predicting renewal risks, and it also includes managing voice latency as well. And so because you're at this layer where you're the head of the applied research angle, I wanted you to just tell me a little bit more about this tech stack and what specifically is really hard about these engineering problems.

• 5:01Speaker 00

  Right. There is many, many things are very hard. I would say that you touched upon some of the best, you know, the kind of the best examples. Voice is, we were very early on on voice. We've been in it for three years. We definitely, and that's one of the things I love about Mina and Tony and their vision, we definitely started when it was way too early. So we were trying to make it work when the technology wasn't there. But voice is extremely difficult. Like, you're talking about doing, you know, tens, not, you know, hundreds of operations in the span of like a second and a half. And you're also touching upon another thing. Like, we really need to make sure that we understand people's behavior, right? In a very precise way. So that's another thing that is unique about about Elise,

• 5:46Speaker 00

  is that when you touched upon predicting renewals, we have a mixture of cutting-edge AI in the kind of more LLMs frontier-based, but we also use a lot of traditional machine learning, because we have a ton of data. And so for that, LLMs are not really the answer. And so we have really a full spectrum of tools that we're using, and we're using it on full force. So for example, the predictions of renewals, we're using really elaborate kind of neural network that we built in the house, but it has nothing to do with LLMs. So for me, as a head of applied research, it's kind of a privileged position, because I don't have to be like kind of tunnel vision on one thing. I can really see the advancement of the entire field of AI across the board.

• 6:29Speaker 02

  Wait, tell me a little bit more about this advancement you're seeing, especially because I think people try to throw LLMs at all these problems. And not all of these problems are fully probabilistic. You need some deterministic angles, you need a mix of all these different things. And so you get to have this bird's eye view of just like what's happening at a macro level, where the entire field of AI and research is going, and then you get to apply it actually at the company. So tell me a little bit more about that.

• 6:53Speaker 00

  Right. And then, you know, not only that, but like one of the major things is to make decisions, right? Or like, okay, we have a problem. Like, this is a problem renewing, you know, lot producing probability for a person that wants to renew their lease. Should we throw an LLM? Should we throw like a deep neural network? Should we throw like a very simple machine learning models? And I think that's kind of decisions that I, that me and other people have to make early on, right? And so here, it's like many people who have joined in very recently, they just do have one solution for everything, LLMs, right? Which it's, I understand the feeling, because like you, these LLMs can generate great answers, and it seems they're incredibly smart. But then when you kind of like dig a little bit deeper in the surface, like you see that some of the crap is like, hmm, not really good.

• 7:36Speaker 00

  And there are certain problems, like, for example, having, as you mentioned, determinism is not there. Like, trying to have the same behavior given their input, LLMs fail. And probabilities is also something that the LLMs have really hard time to elaborate. So I need to kind of like look at a problem, try to understand two things. Like, one, what's the one, the most impactful? And then, two, because we also have to ship at a very fast space, okay, what's, what's the solutions that can really get us almost as close as possible to the optimal outcome, but in the fastest way possible? And then we have to double down on that.

• 8:12Speaker 02

  Can you give me an example about this actual method that you're using within your team? So you're able to identify what is wrong with kind of the way that maybe traditional products are trying to build these things, especially when there's high risk here? You can't get it wrong? Like, it's, even if it's 99% accurate, that's not good enough? It has to be 99.999? And they joke that every nine that you add is another hill to climb? That's harder than the last one.

• 8:37Speaker 00

  It's kind of funny, because like, you know, I still read some papers from academia, and there you see really the difference. Like, when they're talking about reliability, they talk, okay, if the LLM can take it up to 90%. And then we say, 90%? That means for like 60 million of phone calls per month, per year, it means like an enormous amount of failures that is totally unbearable. So we could, you could really see the difference between people who are like thinking about this problem theoretically, and people who actually pay the consequences of those mistakes, us, right? We get inundated of like complaints, and also like the impact on people's life. And so what we did, which I think is quite exceptional, and to me at least, is what really engages me and really keeps me, keeps me going, is to build a mixture of expertise, right? So we have, at least, we have top academic talent,

• 9:22Speaker 00

  you know, including like myself, professors of top universities, as well as like top developers. So like, you have a team that has all eyes, and so they can understand deeply, you know, from coding to like kind of more academic questions. And we're trying to use all these tools to build evals, to understand, evals is kind of the answer to your questions. Like, can we, we have these millions of conversations every day. We make decisions, can we understand at scale the impact of those decisions? And then, once you start doing that, and you're doing systematically, kind of like the results speak back to you. So you run, really understand, you really see that difference. If you just like try it at small scale, like five or six conversations, all look very quite good. But at scale, when you get to the millions,

• 10:07Speaker 00

  then you see that different methods diverge significantly.

• 10:09Speaker 02

  Yeah, no, it's pretty interesting. It's almost like you're applying the scientific method to product design and how you actually build the product.

• 10:16Speaker 00

  I hundred percent agree with you. And I think, and I'm a strong believer that the scientific method is kind of one of the best ideas that the human currents have ever generated.

• 10:24Speaker 02

  I want to talk to you a little bit too about Elisa's AI voice model as well, because you specifically have, you're able to manage voice latency. I know you guys were working on voice and communication way before even some of the frontier labs were working on that. But one of the big issues with it is this latency issue, because it's not just a software problem, it's also hardware. You have all these considerations, especially you guys, I don't remember the exact number off the top of my head, but in terms of the customers that you have, who are interacting, it's like hundreds of million, what was that?

• 10:55Speaker 00

  We have about five million phone calls a month.

• 10:57Speaker 02

  Five million phone calls a month.

• 10:59Speaker 00

  Sixty million a year, which is pretty high.

• 11:00Speaker 02

  Exactly. And for those, you can't have this weird latency between how voice is actually interacting with these consumers. And so, how do you think about this problem? I'd love to get like into your head, into your teams. It's like, how do you think about this problem at Elisa AI? How do you break it down? How do you figure out what's the first thing you guys should work on to tackle it?

• 11:18Speaker 00

  Right, so again, unfortunately, this is like, so there are two parameters here, right? There is one, you know, knowing exactly the current stack. Like, you know exactly what the current stack? I built it mostly largely myself, so I have a really unique vision. And another huge advantage, I have to say, we built it entirely on our own. So we have full understanding of every single part of it. So you have the full stack, like you understand where the bottlenecks are today. What are the steps that takes the longest, right? And then, you are the outward-facing kind of perspective, where like you have a lot of different providers, different changes, like fine-tuning, open sources, or like newer speech to text, text to speech models. So you have a gist of possible solutions, and you try to match them to with the current stack.

• 12:03Speaker 00

  And basically, the idea is, okay, if we obviously have something that is really bad in our current stack, and it is a new solutions that you can apply, then you go in full force. Otherwise, you're trying to constantly understanding and prioritizing, okay, given the solutions that the market has built, this piece of our stack can be improved. And I'm going to say that another thing that we did to speed up that process is that we built a way to experiment. So we have a very closed loop, where something new happens, you know, somebody on Slack messaging me, is, dude, like I saw this post on X. We don't really have, I don't really have to believe it. I don't even look at the benchmark. The thing we do is like, okay, let's set it up, let's run and experiment internally.

• 12:46Speaker 00

  So we can actually see a performance of every model on our own internal benchmarks versus the other ones that you cannot trust.

• 12:54Speaker 02

  Wait, that's so cool. So it comes in on Slack, and then automatically, is this all automated? Or do you come in and then?

• 12:58Speaker 00

  We do have some human to loop, but we're working on automating it actually fully.

• 13:02Speaker 02

  Wow, that's so cool. And then, so tell me a little bit about this research background that you have. You're, I mean, a professor at some of the top, in London, right? One of the top universities. And so, tell me a little bit what you took from that entire framework for how you do research, especially theoretical physics research? And then, how does it actually apply, especially with your team? How do you actually use those same methods internally?

• 13:23Speaker 00

  Yeah, because we are able to, so again, the two things is, one, obviously, there is a gain of having some sort of technical strength, mathematics, and just understanding math, and it helps. And programming, those things are actually useful. But also, you understand what it's like to come up with a new idea. You know, like theoretical physics at the cutting edge of of kind of humanity, in terms of like reasoning and thinking. So coming up with new ideas, it's not easy. Like, how do you get an insight that really changed things? Like, if it was easy, everybody would have it. And so, like, there is a kind of a process. And then, this is very useful to do it, and then we bring it back, right? So for example, you know, we brought it this to fine-tune, right?

• 14:07Speaker 00

  There was like a big, one of the things that we invested on heavily in the last eight months or so, is to try open source models. And so, the thing was, okay, now we have the possibility that, you know, we try really fine-tuning on our end. So how do we do it? Where should we do it? What are the new things that we should really try? And because we have the background of being able to generate ideas independently, and you know that you've tackled one of the hardest questions that are out there, you're not scared of like tackling yourself. And so then, we set a group of researchers, and we started heavily tackling that. And we have been able to actually get incredible results with training our own models and delivering it to production.

• 14:51Speaker 02

  That's very exciting. Well, thank you for sharing that. I'm curious as well, with, you posted on LinkedIn this other post the other day, and it was talking about brilliant research backgrounds with minimal real-world impact. And this is also what we were talking about right before.

• 15:04Speaker 00

  It was me, like, up until three years ago, yeah.

• 15:06Speaker 02

  Exactly. And so, I think what's really cool, especially as you mentioned, and I feel your energy right now, just getting so excited about applying all this scientific rigor, but to real-world and essential industries, like housing and healthcare. And so, what are some of the hardest problems you say your team at Elisa AI right now, what are some of the hardest problems you guys are working on?

• 15:24Speaker 00

  Right, so I would say, I would name it two, right? And so, one is, for example, LLMs are moving fast, right? So now, one of the very legitimate question is, six months from now, what problems will be solvable with LLMs that are not solvable now? And then, push it to a year ago, or a year from now, and so forth. So like constantly prioritizing what are the things that we could solve with LLMs and we could automate. Like, and again, we're not talking about, you know, math problem, we're talking about real-world iterations. So like, you have to be sure that, oh, here, there is something that is really hard, that is real huge frictions for like millions of apartments, and now we can actually iterate. So like, this is one of the things that we're constantly spending time on.

• 16:09Speaker 00

  And it's very, and depends, kind of, it's a high-stake decisions, because moving early and moving fast allows us to be in a much better position six months from now. And then, the other thing that I think is extremely difficult is that, again, Elisa is growing fast, but we're still six hundred, you know, like a few hundred people in the company. And we operate at a scale where we automate millions, which we, one million phone calls a day, hundreds of thousands of emails and SMSs. So we really have developed loops which are fully automated, completely automated. And now, this is wonderful, it's great, it's glorious. But on the other hand, you need to really build some evaluations to, so that you know, like, we have tons of conversations

• 16:54Speaker 00

  that go on with our inputs. And these impact, again, these are housing and healthcare. These are not like some random thing. These actually impact people's life. There's consequential changes that happen if we make a mistake. And so, building those evaluations that we are able to really gather with few people, with ten people, like, they're able to like, get an understanding every day how the conversations are going, that's a really, really hard task, which we have largely cracked, to be honest.

• 17:19Speaker 02

  Give me some specific examples there. So what's an example of something that you fully automated this entire loop from?

• 17:24Speaker 00

  Oh, basically, in the entirety of voice calls is out of it. For example, like, the two main loops that we've automated, one is leasing people that are interested in apartments. They basically can do the entirety of the, of starting from like looking at a website and deciding that they are interested in apartments, to moving in. It could be kind of fully, literally fully automated. You know, they can, we have, for example, we have, you know, you can book a tour, then, you know, the AI sends you the code, you can go to the house, then you open the house with the code, you interact with the AI, the AI is explaining you the house. So like, you're doing the whole thing on your own, right?

• 18:08Speaker 00

  The other thing is resonant, for resonant, like calling and they have issues. There, there's still a gap that we've not been able to automate, because we need robotics to fully automate that. Because, you know, some of the fixes require actually human labor, and so LLMs can't really do that. But we have automated a large chunk of that, because we can now, you can now, you know, submit a work order to telling, okay, this, fix this thing. The AI is capable to understand what other requires a fix immediately, then escalates it, finds a vendor, calls up a person. And then, the last piece is a person, actual physical person, coming into your house, fixing it. Maybe we'll get to like automated the whole thing in a year or two.

• 18:49Speaker 02

  Wow, that's pretty interesting. So I want to know a little bit about what's next for you guys. I one of these big this big conversation I was having with Mina is around this vertical versus horizontal scaling. I think what's unique about Elise AI is vertically down the stack, when especially when it came to housing, because that's where you guys started, you fully went through every single problem, especially except, I guess, the robotics physically going into your home, but but you'll get there. And now you're applying it more horizontally. So now also going really deep down in healthcare. What's next? Are you thinking of another industry? Or going deeper on this vertical angle?

• 19:24Speaker 00

  Uh, yeah. So we, uh, we definitely, right now, we have, you know, a handful. I mean, healthcare, it's like forty percent of the GDP in the US. So, you know, we have room to grow a little bit. And especially, we, we do find that our best way to build AI that matters, right? Is because, you know, one of the things that unique about Elise is that we have these incredible things. Like, we have theoretical physicists like myself, who have spent fifteen years thinking about, you know, looking at the stars and think about black holes. And combined with they've acquired, combined with like very deep domain understanding and business knowledge, right?

• 20:03Speaker 00

  So we really do have this unique mixture of of of of skills. And so we want to continue to move that way. So we want to be driven by what has next big bet, based on what the client our client said. So one of the things, for example, that we are now facing very pretty heavily is to really cut down the costs of running affordable housing. This seems like a massive problems, because we are finding that affordable housing is a hugely complex, yet extremely impactful and relevant for especially the the component of the population that has the least means, right? The ones that is most vulnerable. And so we really want, you know, to be able to make them more affordable. And it's extremely complicated, right? It's basically not automated at all.

• 20:47Speaker 00

  The amount of tech depth that is in the currently in the industry is nearly nonexistent. And so we are really aggressively trying to, and it's a massive component of the entirety of your state. So that would be would be massively impactful. If we could bring down the cost of affordable housing significantly, so that will impact consumers immediately and make, you know, their lives much better.

• 21:09Speaker 02

  I know. That's pretty interesting. Some of the hardest problems attract the top talent. And one last thing I wanted to ask you before we go, I know your team is hiring, and I wanted to just better understand what you're looking for. You said earlier that you have a mixture of people with many different backgrounds, who have a mixture of skills. You come from the academia world. I just wanted to better understand, like, what are you looking for when it comes to how you grow your team?

• 21:31Speaker 00

  So we, we, the reasons why we are mostly hiring, especially on the technical side in San Francisco and New York, is because we want the best. I mean, that's, we're pretty high standards. And the two main things we have, we want best engineers. There's a ton of engineering that needs to be done, and we are so short on that. And in particular, in San Francisco, our second hub is growing, you know, fantastically well, but we need more. We just can't grow hard enough. And then secondly, we love, in particular, me, obviously, researchers. And the background there is people who have a very strong research background, but they really have this fire in

• 22:11Speaker 00

  them to wanting to use their ideas, their depth of thought, in actually impact. So we want, we want, these are kind of very difficult, very unicorns to find. People who have exceptional academic background, but yet they are really interested in in driving that into real world. And so researchers with a clear focus on impact people's life and make it better.

• 22:33Speaker 02

  That's exciting. All right. Well, thank you so much for joining us today, Mario. This is such a fascinating conversation. I think what's really interesting, getting to hear also, this is my second conversation right now at the least, and getting to hear from your perspective, also Mina's perspective, but just the different background and approach that you have, coming from that theoretical physics background, and then applying rich academia to some of the toughest problems we have now in AI.

• 22:59Speaker 00

  And I'm gonna say, I want to say to the people out there, like, I did not expect that, but even after coming from string theory, I would say that the complex, the complexity of the problem of working now is probably higher now than than it was before.

• 23:11Speaker 02

  Even coming from string theory?

• 23:13Speaker 00

  It's pretty, it's pretty, it's pretty hype, though.

• 23:15Speaker 02

  You're hearing here first. Well, Mario, thank you so much for joining us today on MTS.

• 23:21Erik Torenberg?

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z, and subscribe to our Substack at A16Z.substack.com. Thanks again for listening, and I'll see you in the next episode. This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product. This podcast has been produced by a third party and may include paid promotional advertisements, other company references, and individuals unaffiliated with A16Z. Such advertisements, companies, and individuals are not endorsed by AH Capital

• 24:05Erik Torenberg?

  Management LLC, A16Z, or any of its affiliates. Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy.