Why Your Data Isn't Changing Your Business Decisions

AI is helping marketers move faster, but not always smarter. Katrina Wong, Chief Marketing Officer at New Relic, explains how her team turns raw data into decisions leadership can trust. She covers using AI to analyze recorded sales calls for unbiased win-loss insights, deploying real-time in-product signals to personalize the self-service funnel, and building a human-in-the-loop fact-checking process to catch AI drift before it erodes trust in the data. See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

• 0:00Speaker 00

  The Martech Podcast is a proud member of the I Hear Everything Podcast Network. Looking to launch or scale your podcast? I Hear Everything delivers podcast production, growth and monetization solutions that transform your words into profit. Ready to give your brand a voice? Then visit ihereverything.com.

• 0:25Speaker 01

  From advertising to software as a service to data, across all of our programs and clients, we've seen a 55 to 65% open rate.

• 0:36Speaker 02

  Getting brands authentically integrated into content performs better than TV advertising.

• 0:42Speaker 03

  Typical lifespan of an article is about 24 to 36 hours.

• 0:46Speaker 01

  If we're reaching out to the right person with the right message and a clear call to action, then it's just a matter of timing.

• 0:53Speaker 00

  Welcome to the Martech Podcast, a member of the I Hear Everything Podcast Network. In this podcast, you'll hear the stories of world-class marketers that use technology to drive business results and achieve career success. Here's the host of the Martech Podcast, Benjamin Shapiro.

• 1:16Speaker 04

  78%. 78% of technology leaders report more production incidents after deploying AI-generated code, even though 93.5% say that the code appears to be higher quality during review. AI is helping businesses move faster than ever, but speed without confidence creates new challenges. How do you know the decisions you aren't making are actually the right ones? This isn't just a technical problem. Marketing teams are facing the same reality. We have more data, richer customer signals and more technical capabilities than ever before. Turning all the information and possibility into decisions you can trust is the biggest challenge in modern marketing. So how can you turn your data into decisions that you can trust?

• 2:00Speaker 04

  I'm Benjamin Shapiro and today joining me is Katrina Wong, who's the Chief Marketing Officer at New Relic. She's been through seven exits over her 25 years leading marketing organizations at companies like Twilio, Salesforce and SAP. And today Katrina shares how AI is changing the way marketers make decisions. Katrina, welcome to the Martech Podcast.

• 2:21Speaker 05

  Thank you, Benjamin. Thanks for having me on.

• 2:24Speaker 04

  Very excited to have you here. First off, you've been through seven exits, been wildly successful in your career and somehow you decided to land into a place where your customer set is the type of customer that is notoriously against marketers marketing to them. Why would you make this decision?

• 2:46Speaker 05

  Benjamin, I like challenges and after many years in the workforce, I think that unless I'm challenged, I'm not learning and growing. So it's been fun. And fun fact, this is actually my third company where we market to developers and they are the practitioners.

• 3:07Speaker 04

  You are an absolute glut for punishment. New Relic is a wonderful company. It does a great job marketing. I think you might be mildly insane, but I'm thrilled that you are here and that we get to talk about turning data into decisions. So you've been through a few things, like I said, seven exits and a whole bunch of companies that we've all heard of as the lead marketer. You've seen multiple technology waves. What is it about the AI transition and technology that makes you more energized than the previous technological changes?

• 3:42Speaker 05

  Oh, wow. It's the pace that we're able to move. And at the end of the day, for marketers, we've always wanted to innovate and move faster and really put our customers first. I call it the value exchange. What value are we truly providing to our customers in exchange for their time? And prior to AI, it just took a lot of work to do the mechanics of marketing. Data was harder to find. Decision-making was oftentimes delayed. And so we weren't moving as fast and maybe we didn't target correctly or we didn't get our message right. All of that has been so exciting for all of us in marketing these days with AI.

• 4:25Speaker 04

  I feel like there's three camps when it comes to this transitions. There's the people who are like, this is gluttonous. I can do everything. I have all of the information. The technical hurdles are gone. I built agents, they do the work for me. AI is amazing. I'm in this camp. There's the people who are like, okay, this is complex. I struggle to learn a new thing, but it's starting to make my existing processes better. And then there are the people who are like, no, this is terrible. I am so overwhelmed with possibilities. Where do you see a New Relic stands? And where do you see most marketers in that camp?

• 5:06Speaker 05

  I would say for us in marketing here at New Relic, because our product also has AI infused throughout it, we're very AI forward. We're not, we can't claim AI native, but every day we're feeling like we're more and more AI first. So there's creating and leveraging AI in how we want to solve problems for our customers. So as marketers, we do walk the talk. And what you're alluding to is really change management. So in other words, when there is a change, people learn and adopt differently at different paces. So we've done a lot within the marketing team.

• 5:51Speaker 05

  Year one, we made it very, very safe. So I joined two years ago where the AI that was in practice was really ChatGPT and maybe some wrapper applications around ChatGPT. So we made it really, really safe for marketers to learn together. And we started simply with just a bottoms-up approach. And I said to the team, we are all going to learn this together. It is not scary and let's do a bunch of lunch and learns, right? We're going to teach each other. Let's figure out the use cases that really help us. And then soon that turned into my monthly town halls where we had an AI showcase. And I'm so pleased to say, two and a half years later, we're still doing it. And guess what?

• 6:33Speaker 05

  It's the most subscribed part of my marketing town halls. We are having a lot of fun.

• 6:39Speaker 04

  So you've gone through this change management and look, New Relic is probably not the best example of the typical marketing team because of the technical complexity of your product to who you're marketing to. I'm assuming that you have a super advanced marketing team that is very technical and that they're probably very excited about making the change to AI. So I want to ask you a little bit about the way that your team has, has made the change, not about sort of the personality stuff. Specifically, talk to me about how the data feeding your marketing organization has fundamentally changed in the post-GPT era.

• 7:16Speaker 05

  Yes. So that has always been a little bit painful, getting the right data. And there's different types of data. There's being able to automate a tray show list of things and that upload. It sounds so boring and mechanical, but you can lose half your leads when that upload fails and what do you do, right? So that's very mechanical. Then there's leveraging ChatGPT-like capabilities where getting the data that perhaps we create, for example, the product marketing team for sales, instead of sales pinging on Slack

• 7:57Speaker 05

  product marketers as they are preparing for their sales calls because they want to figure out the right value pillar to use. We've created a way, and frankly for that, we Gemini gems, a way for the reps themselves to ask AI what they need and everyone's happier, right? So it's also, you know, us creating data as marketers and disseminating that to our stakeholders. So that's been really needle-moving and that's been really cool to see.

• 8:27Speaker 04

  I feel like there's the three different stages that you just highlighted. One is how do we get access to the data and fixing the mechanical portion?

• 8:37Speaker 05

  Yep.

• 8:37Speaker 04

  Two is how do we ask questions to the data when we need something? And then the third wave is how do I get the data delivered to me? So in advance, I know, or how do I use my agents to ask it questions about what data I actually need to have a conversation?

• 8:54Speaker 05

  That's right.

• 8:55Speaker 04

  We're at opposite ends of the marketing spectrum. I'm a guy in his garage, you're running New Relic's marketing team, drastically different, yet only a couple of miles apart.

• 9:07Speaker 04

  And it's funny, I was faced with this challenge today where somebody asked me in a sales call, how do you think about your audience profile, like more B to B, more B to C? And I've been using the same assumption-based messaging that of who our audience was. And I was like, I don't need to do that anymore. I can go and ask my agent, hey, look through all the data sources we have and come up with an understanding of where we skew B to B and B to C. I always said we're 50/50, maybe we skew slightly more towards B to B. Turns out it's like 70% B to B. Who knew? I was totally wrong on my data sources. If I had to ask my agent, go look at the data sources and tell me my information.

• 9:49Speaker 05

  We are getting insights all day long. Like a lot of our sales calls are recorded. So the insights, I mean, the old way before AI, we were interviewing reps, maybe customers for win-loss scenarios and how we could have done better so that I could tailor our messaging and positioning. And now when we analyze these calls, we're like, okay, here's what the humans are saying for what they perceive to be the issue. And here's what AI is saying. And it's been nice to use both the AI analysis, right, of qualitative information as well as triangulating that with a human. So that's been really cool to see. It does give us an extra unbiased layer of information.

• 10:34Speaker 04

  Yeah. It's interesting. You're you're turning your qualitative data into quantitative by moving the call recordings into something AI is analyzing.

• 10:45Speaker 05

  Yes.

• 10:45Speaker 04

  And then you're you're essentially still doing qualitative, like your win-loss analysis and seeing if they're matching. Are are they the same? Like, are what people inferring from a win-loss call the same as what AI is saying that happened on a call?

• 11:01Speaker 05

  I suspect that it might be a sampling error, right? Meaning I can only do a certain number of interviews or my team can within a quarter versus us being able to look at thousands of calls that we've made. But it is slightly different.

• 11:16Speaker 04

  Tell me a little bit about the differences.

• 11:18Speaker 05

  So perceived pricing and packaging, like, hey, we think it's a pricing and packaging issue. But actually, it's not. It's usage of a certain feature that we're hearing more. So there is the normal human bias, I think, you know, when it comes to selling or in any of our roles. Um, so that's been really interesting to see.

• 11:42Speaker 04

  It's the Glenn Galary, Glenn Gary, Glenn Ross problem where, you know, sales blames marketing for the bad leads and marketing blames sales for not being able to close. Uh, tell me a little bit about how you're turning data into better signals. You you have more of it, you're able to analyze it. Is it helping you make better decisions?

• 11:59Speaker 05

  It is. Um, in, uh, one was perhaps one of our first, uh, use cases was in our self-service funnel and with product-led growth. And so in order to create this flywheel, you are always looking to remove friction and answer questions and so on and so forth. So before AI, at best, maybe there's a non-AI-powered chat, if you have that. Or maybe it's a support ticket, you know, so high bar to, you know, kind of press a button and try to get an answer.

• 12:36Speaker 05

  So, um, I was an early adopter of one mine and what we ended up doing, um, I was a co-design partner of some sort. And they had, um, this, uh, this notion of an avatar. And that's when I went to Amanda and said, you know, that might be too much so early on. I don't doubt that maybe one day that will work. But for us right now, can you just make the interface like an orb, not very intrusive, similar to how ChatGPT was doing it at the time? And that's done wonders because now we're getting real-time signals as someone is walking through our product, onboarding themselves. Because with observability, you can do so much. It's a little bit of choose your own venture.

• 13:20Speaker 05

  So that's been really nice to take that information and know what it is that's sticking with our customers. Plus, in real time, be able to communicate. Um, so that's, that's been a whole wealth of learnings right there.

• 13:35Speaker 04

  I understand the the sort of real-time nature of being able to process your data, make decisions and and understand what's happening. I think that there's a trust component to it as well. You know, I I think of a lot of with marketing teams, like, we don't really need an analytics function or an or an engineering function in some capacities as much as we used to. I can go create my dashboards and I can create my reports and I can get access to my data and I don't need a NoSQL and I don't need an engineer to go build all this stuff. But what I don't necessarily know is if I can trust it. I can get a report and I can smell test it. And even if it's more in real time, the concern that I have and and most of the time, I think marketers are saying, great, I can just go do this.

• 14:20Speaker 04

  But the question is, is it right? They're like, can you fact-check it? Is it something that's sustainable? How do you think about building trust now that we can basically analyze our own data in real time without somebody else, you know, fact-checking whether it's correct?

• 14:36Speaker 05

  So we recently did this and we are fact-checking and we still go back to the source of truth. But with our normal forecasting reports, we were operating using Tableau that plugged into Snowflake that took data from numerous sources, mostly Salesforce. And there were still humans kind of analyzing the data and calculating pacing, so on and so forth. Now, AI does all of that for us, but we are still fact-checking because that's just where we are as a community. I still believe in human in the loop for sure. So you're right. Yeah, you're right. We don't know if it's a hundred percent accurate. And maybe one day there'll just be more and more.

• 15:18Speaker 05

  And I think some companies are doing this on some fronts, a confidence, you know, level or a confidence score. But unless it's a hundred, when we're talking about revenue, it has to be right, right? So.

• 15:32Speaker 04

  Yeah, I do feel like, you know, the efficiencies from using AI and honestly, the sort of pleasant nature of just having the work done for you and being able to set it and forget it has raised our error bars. And and we will actually say, look, our reporting doesn't have to be a hundred percent accurate because I get it consistently. And I can diagnose what, you know, essentially use intuition and a smell check to figure out if it's right or wrong. And I think that, you know, the concern for me is this idea of degradation over time. Is that if there's a little error up front that you don't recognize as it gets bigger and bigger and bigger, it's very hard to see. And we have this inherent trust with these new systems. And, you know, I think that there is some level of concern about whether they can be trust.

• 16:16Speaker 04

  How do you think about, you know, data accuracy as you're building more AI into your market?

• 16:23Speaker 05

  Yes, um, I also want to comment on what you're referring to as the, what psychologists are now saying, the automation bias. So that first stat that you shared where everyone's like, more and more code is generated by agents, but yet we think it's great. So there's this inherent trust where the manual job of just doing line-by-line co-review. So so the tail DR from that report is something like only sixty-two percent of all of the code now is being reviewed in the the survey that we conducted and and what we found, um, also in our systems. And, uh, however, the, um, the production failures have gone up.

• 17:07Speaker 05

  So it shows how strong, right, the automation bias is. So what you're referring to is, you know, not only is, um, the output, whether it's code or in the case that we were talking about, data being pulled and generated by AI. It's much easier and less painful for us to trust it. So what do you do? It comes back to, you know, we just have to, just like any other habit that you build. I I'm a huge fan of atomic habits, right? It's, I I love the speed of it, but I'm always still going to check and ensure that we have these checks in place, right? Even if it's like a checklist, because we have to, and that's kind of where we're at.

• 17:52Speaker 04

  For me, it's a sanity check on my reportings and actually going back to the single source of truth and and fact-checking. For you, there's a team of marketers who are building their own reports and it's probably impossible for you to personally go in and fact-check everybody's analysis.

• 18:06Speaker 05

  I hold my team accountable for that.

• 18:08Speaker 04

  Well, how do you, how do you do that? How do you make sure that when they're creating, you know, analysis or, you know, marketing campaigns based on analysis, that they're actually going back and checking their own book?

• 18:20Speaker 05

  I think a couple of things. If it is a campaign, right, I will to spot check. And very recently, actually, even today, I was like, you know, I've been looking in our image library.

• 18:33Speaker 05

  I'm like, it feels just a little bit off, like some of our graphics have, I don't know, I don't feel like it's quite on-brand. And that's when it hit me, I was like, because I took a look and because the majority of it was being AI-generated, right? It was starting to veer off a little bit. So, you know, maybe it's not scalable, you know, forever, but we have to. So my new saying for marketing is be a good neighbor. If you see something, say something, you know, like be a good neighbor because we have to ensure it is our fiduciary duty to not create AI slop.

• 19:12Speaker 05

  All right. Use AI, but with great power, right, comes great responsibility.

• 19:17Speaker 04

  I feel like your role as the CMO has changed in some ways where it used to be get it the most capable people to produce as much as they possibly can. You obviously still want the most capable people, but now your role, instead of getting them and sort of putting them in place and letting them run, you have to essentially be quality police.

• 19:35Speaker 05

  We talk about that internally a lot. What's the governance? And so there's some thinking that needs to be done there for most companies as well as ourselves, right? Whether it's agents, agents running a mock or overlapping agents and overlapping capabilities. Or, you know, recently I invited a past team member onto my virtual user conference. And he was talking about how different agents were updating the same field in Salesforce. So the human problem of, okay, maybe marketing wants the data to be this way, sales overrode it, vice versa. And now agents are doing that and he didn't know. So it is something that, you know, collectively we have to solve. Yeah.

• 20:17Speaker 04

  So we talked a little bit about how your access to data is changing the marketers capabilities on your team. Talk to me a little bit about the customer point of view. How does the customer experience change now that you've been making all these investments and using AI in your market?

• 20:32Speaker 05

  I think the end game one day is truly one-to-one marketing and perhaps one-to-one selling. Because I, I, I think as marketers, we've always aspired, getting back to earlier in the conversation, to do really great marketing and truly provide value to exactly what the customer wants. And we haven't always been able to do that. The technology was in there, so maybe campaigns where we thought they were specific, but still a bit more broad. So that is the vision. Now with our customers, we've been able to streamline, for example, in self-service. You know, that experience, you know, using one line a lot better.

• 21:17Speaker 05

  So they experience in a variety of ways. We've been experimenting with using AI and graphic design. And so for one of our data reports that we still do in-person events. And for our top customers, we, we asked them to join a customer advisory board. And someone was like, oh, I love all of your graphics. So we've been able to just iterate and produce more, which then also helps, right? Our customers consume in a more delightful way.

• 21:48Speaker 04

  Yeah, it's interesting. What I'm taking away from your response is before we were segmenting, what we called personalization was putting someone into a bucket with other people that, you know, looked, smelled and tasted just like them.

• 22:01Speaker 05

  Oh, yeah, that's right.

• 22:02Speaker 04

  And we are actually able to do personalization now. And that really is the big unlocked in terms of how you're using the data to improve the customer experience. It's a personal customer experience. Talk to me about some of the ways where you're implementing personalization to make something as technical as new relevant, something as technical as new relic, relevant to the individual person, not their persona.

• 22:28Speaker 05

  Yeah, that is a good point. You know, for us, because we are a complex sale and there's usually, you know, account team buying, there's just, there's the gathering of information that is pretty hard, right? Like imagine you're doing a sales call and there's like six people and humans are taking the notes. Are you going to jot down everything that everyone says? Now, we'll call recording and the ability to analyze, that's been game changing. Cause then that allows me to actually do ABM, not just for the account, but down to what different individuals need. So for example, recently we were talking to a European company and it was very clear

• 23:12Speaker 05

  that the different stakeholders in the room, and for some of these larger companies, you have a lot. Someone wanted more strategy and they wanted to understand what we know about retailers and what they were doing in this era, what AI innovations they are doing. And then meanwhile, several other team members just wanted to know what this one product capability would do and if it would do this and that. And I could just imagine the old world where you're in, you know, a Zoom or back in the days in a conference room, you're unable to capture all of that information all at once, because you're not recording these calls. In these days, it's pretty common to have, you know, calls recorded and have AI kind of tell you what's important.

• 23:55Speaker 05

  But for me, it's, it's, it's getting the primary data right. That will allow you to do better marketing and more targeted marketing and using AI to produce the, you know, the pieces too faster.

• 24:08Speaker 04

  As I think about the enterprise sale and how you're going through marketing, you know, it makes sense of like, well, we get somebody in a call and they tell us their pain points and now we have them documented. But also there's a little bit of the buying team mapping.

• 24:22Speaker 05

  Yeah.

• 24:22Speaker 04

  Where someone wants a strategy, someone wants a technical, you know, analysis, somebody wants alerts for, you know, their mobile app. And, and there's all these different teams, they might not be the financial buyer. You're able to actually map that out. And since we have AI to be able to scale marketing, now you can create multiple pieces of collateral that go to each person.

• 24:44Speaker 05

  That's right.

• 24:45Speaker 04

  Are you finding yourself or your, your marketing team being challenged by the amount of collateral and conversations and sort of the volume that AI is pushing them to? Are you having conversations with five different people in a, in a buying org? Or is it just one message, one person and sort of how it was before?

• 25:06Speaker 05

  AI has helped with the volume of work, right? But I care that it's more accurate because the input is more accurate. So, you know, it's now more effective, right? So everyone feels like they have something come out of the sales call that's tailored to what they need, right? Cause you take out the human bias of what's important and maybe the title. So for the marketers, it's been a little bit more gratifying because it feels more effective. And with the use of AI, they're also, they're also able to produce things quicker, prettier, better, more on point. So, yeah, so it's been nice. It's been nice.

• 25:45Speaker 04

  You've got your data helping you make better and faster decisions. There's a little bit of trust and, and sort of governance that now you are responsible for being the, you know, the sheriff in town. But it's helping you identify pain points and do more personalization. I still go back to your marketing towards engineers and they hate marketing and also hate marketers.

• 26:09Speaker 05

  Fun fact, I am married to an engineer. So fun fact, I don't think he loves what I do.

• 26:17Speaker 05

  I'm publicly saying this, but yeah, he, he hates marketing, not what I do specifically, but.

• 26:23Speaker 04

  A tribute loves you and your work and, and probably hates the, the, the role that you're in. But it, but it is a challenging role and I'm curious to hear how data is helping you earn the attention of the engineering community.

• 26:36Speaker 05

  We all strive to be authentic, but in particular, when you are sharing a message, I, I don't even want to say marketing, sharing a message or communicating to a technical audience.

• 26:51Speaker 05

  It's no jargon, no fluff, and it just has to be authentic. And it also has to solve a problem.

• 27:01Speaker 05

  A problem that they care about and, and specifically even in a way that they respect, right? The elegance of the solution, right? So you're just going to have, did the bar is high? You're just going to have to get very, very concrete and specific and pretty fast. It doesn't mean you can't have, you know, really beautiful taglines that become the hook. But underneath it, you go really deep, 101, 201, 301, pretty quickly. So for us, for our last keynote for a user conference, because we are also a data company, meaning we have so much data. Observability is about observing what's happening and we capture that as data.

• 27:43Speaker 05

  We were able to hone in on data insights that substantiate the problem, right? So to your point about, oh my gosh, more and more code is being written by agents. But yet we think it's all great and we're putting it into production, but the incidents are rising, right? So we call that agent debt, right? And by, so still we're doing something really clever. We're naming the problem. It's catchy, but we're substantiating it with data. And that's a very foolproof way for marketers, I mean, sorry, for developers.

• 28:26Speaker 04

  It seems like what you're getting at is that the original research is the way that you're implementing data.

• 28:31Speaker 05

  That's right.

• 28:32Speaker 04

  Right? And you're taking and turning the aggregate analysis of what's happening across the industry into information that's valuable to them. And I guess the question is, okay, great. You've got this wonderful data source of, you know, whose websites are crashing or what code isn't working? Or are people shifting from desktop to mobile because of AI? And you can highlight all of that information, but how does it actually impact marketing? How do you convert that type of story into not just awareness of the problem, but into, you know, credibility, more business for new relic?

• 29:05Speaker 05

  We do a lot of industry data reports, right? It's one of our highest generating, you know, tactics. You know, we run some campaigns around that, but people do see hungry for, am I missing out on something that I should know about? We actually do briefings of these data reports as part of our customer advisory board. So we do put a wrapper around the data and produce an asset that, you know, our audience likes genuinely and engages with.

• 29:39Speaker 04

  Yeah, I, I fundamentally, you know, even when you're doing something like using your data to support your customers through original research, we are all getting more technologically advanced. We are all using more data sources, using it to create a reporting, using it to build better customer experience, all of the things. Fundamentally, what I'm hearing from you is that there is a community aspect to this and that when you are a good active member of the community, you're producing original research that supports the engineering community. That helps the brand, that would fosters trust. And that's really what makes the connection beyond just, you know, this personalization and conversion metric type mindset.

• 30:22Speaker 04

  It's really a community building strategy as much as does anything.

• 30:26Speaker 05

  That's right. That's right. It's not transactional. We don't want to just market to you. It's the value exchange I was getting to earlier. What value are we offering in exchange for your time? So, you know, whenever you can help your audience learn in any way, shape or form, that's, that's a win.

• 30:45Speaker 04

  All right. That wraps up this episode of the MarTech Podcast. Thanks to Katrina Wong, the Chief Marketing Officer at New Relic. If you'd liked it, contact Katrina. You can find a link to her LinkedIn profile in our show notes or on martechpod.com. Or you can visit her company's website, which is new relic.com. If you haven't subscribed yet and you want a daily stream of marketing and technology knowledge in your podcast feed, hit the subscribe button in your podcast app or on YouTube and we'll be back in your podcast feed next week. All right. That's it for today, but until next time, my advice is to just focus on keeping your customers happy.