Todd Olson joins me to talk about making analytics worth paying for and relevant in the age of AI. The CEO of Pendo, an analytics SAAS company, Todd shares how the company evolved to support a wider audience by simplifying dashboards, removing user roadblocks, and leveraging AI to both generate and explain insights. We also talked about the roles of product management at Pendo. Todd views AI product management as a natural evolution for adaptable teams and explains how he thinks about hiring product roles in 2025. Todd also shares how he thinks about successful user adoption of his product around “time to value” and “stickiness” over vanity metrics like time spent.
Highlights/ Skip to:
- How Todd has addressed analytics apathy over the past decade at Pendo (1:17)
- Getting back to basics and not barraging people with more data and power (4:02)
- Pendo’s strategy for keeping the product experience simple without abandoning power users (6:44)
- Whether Todd is considering using an LLM (prompt-based) answer-driven experience with Pendo's UI (8:51)
- What Pendo looks for when hiring product managers right now, and why (14:58)
- How Pendo evaluates AI product managers, specifically (19:14)
- How Todd Olson views AI product management compared to traditional software product management (21:56)
- Todd’s concerns about the probabilistic nature of AI-generated answers in the product UX (27:51)
- What KPIs Todd uses to know whether Pendo is doing enough to reach its goals (32:49)
- Why being able to tell what answers are best will become more important as choice increases (40:05)
Quotes from Today’s Episode
- “Let’s go back to classic Geoffrey Moore Crossing the Chasm, you’re selling to early adopters. And what you’re doing is you’re relying on the early adopters’ skill set and figuring out how to take this data and connect it to business problems. So, in the early days, we didn’t do anything because the market we were selling to was very, very savvy; they’re hungry people, they just like new things. They’re getting data, they’re feeling really, really smart, everything’s working great. As you get bigger and bigger and bigger, you start to try to sell to a bigger TAM, a bigger audience, you start trying to talk to the these early majorities, which are, they’re not early adopters, they’re more technology laggards in some degree, and they don’t understand how to use data to inform their job. They’ve never used data to inform their job. There, we’ve had to do a lot more work.” Todd (2:04 - 2:58)
- “I think AI is amazing, and I don’t want to say AI is overhyped because AI in general is—yeah, it’s the revolution that we all have to pay attention to. Do I think that the skills necessary to be an AI product manager are so distinct that you need to hire differently? No, I don’t. That’s not what I’m seeing. If you have a really curious product manager who’s going all in, I think you’re going to be okay. Some of the most AI-forward work happening at Pendo is not just product management. Our design team is going crazy. And I think one of the things that we’re seeing is a blend between design and product, that they’re always adjacent and connected; there’s more sort of overlappiness now.” Todd (22:41 - 23:28)
- “I think about things like stickiness, which may not be an aggregate time, but how often are people coming back and checking in? And if you had this companion or this agent that you just could not live without, and it caused you to come into the product almost every day just to check in, but it’s a fast check-in, like, a five-minute check-in, a ten-minute check-in, that’s pretty darn sticky. That’s a good metric. So, I like stickiness as a metric because it’s measuring [things like], “Are you thinking about this product a lot?” And if you’re thinking about it a lot, and like, you can’t kind of live without it, you’re going to go to it a lot, even if it’s only a few minutes a day. Social media is like that. Thankfully I’m not addicted to TikTok or Instagram or anything like that, but I probably check it nearly every day. That’s a pretty good metric. It gets part of my process of any products that you’re checking every day is pretty darn good. So yeah, but I think we need to reframe the conversation not just total time. Like, how are we measuring outcomes and value, and I think that’s what’s ultimately going to win here.” Todd (39:57)
Links
Transcript
Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today I’m really happy to have the CEO of Pendo on the line, Todd Olson. Welcome Todd.
Todd: Thanks for having me. It’s great to be here.
Brian: Yeah, I’m glad we finally got this together here. I’ve been really excited to talk to you, and during our little intro, it was also fun to hear that you have some experience not just building an analytics SaaS, a commercial analytics product to help other SaaS companies improve their software, but you also have some experience with the internal data science and analytics efforts as well, so this should be really good for both of our audiences. I tend to think of the show having those two distinct audiences, so this should be really fun [laugh].
Todd: Awesome. Yeah, I’m looking forward to it.
Brian: So, just thinking really broadly, I know if you’re a decade in roughly at Pendo, something like that, you’ve built a successful commercial analytics software product here, and I’m sure along the way, you’ve probably had some times where maybe you overloaded users with data or metrics and had challenges making insights actionable—that’s an assumption; you can tell me if it’s wrong—but if it is true, when you look back over, like, a decade of this work here, were there specific things that you did to kind of address this analytics apathy, or just trying to make sure that this data actually gets used and it doesn’t just become another dashboard desert [laugh]?
Todd: Yeah. Yeah, I mean a lot of things. And look for a company like ours, about a ten-year-old technology company. When you start out, you’re really focused on the—you know, let’s go back to classic Geoffrey Moore Crossing the Chasm, you’re selling to early adopters. And what you’re doing is you’re relying on the early adopters’ skill set and figuring out how to take this data and connect it to business problems.
So, in early days, we didn’t do anything because the market we’re selling to were very, very savvy, they’re hungry people, they just like new things. They’re getting data, they’re feeling really, really smart, everything’s working great. As you get bigger and bigger and bigger, you start to try to sell to a bigger TAM, a bigger audience, you start trying to talk to the these early majorities, which are, they’re not early adopters, they’re more technology laggards in some degree, and they don’t understand how to use data to inform their job. They’ve never used data to inform their job. There, we’ve had to do a lot more work.
Like, just even, like, some basic fundamentals, which honestly to the audience, you’re like, really, that sounds so trite, but like, just good templates. Like, “What are you trying to accomplish?” “Oh, I want to”—you know we do like adoption—“Okay, here’s a feature adoption template that asks you a few questions and then creates a dashboard that directly helps you answer those questions, not only for yourself, but even potentially for other stakeholders.” And here’s the thing, on those dashboards, we actually include text-based descriptions of what these charts are and why they’re important and how you can interpret them, right? So, I think that’s a big piece of it is we’ve invested so much more in sort of explainability of each chart and graph, so you just know what action you should take when you’re looking at a number going up, or a number going down, or, you know, a cohort graph where you’re seeing, you know, drop off in, you know, month three.
Like, what do you do, right? So, I think helping connect those dots more directly is something we’ve had to do more recently. When I say, ‘more recently,’ the last few years. That’s been a big shift in our mindset is don’t just barrage people with more data and more power, and give people the basics. These people need the basics.
Brian: But with that comes, I mean, I’m sure, like, there’s people that probably want to, like, slice and dice, and I want to break this down by segment, and I want to break it down by region because we have different regional product offerings and blah, blah, blah. Like, how are your product teams, like, balancing all of this need, especially if it’s a loud whale customer that’s, you know, got a ton of seat licenses, and they really want to, like—I’m sure this has probably come up. I don’t know, but I’m guessing.
Todd: Look, at the end of the day, there can be what I call no dead ends. A dead-end is where I continue to have more and more questions, and I just can’t answer the question in the product. And bad dead-ends are when the user, who’s a smart user, knows we have the data, but we’re just unable to surface them in the right fashion. So, we have invested a lot, so regards to the questions or the drill through, regardless of what you wanted to answer, yes, the power user is always going to get their answers.
So, that was, kind of like, that was product strategy-wise, job number one. Don’t let a question go unanswered. That’s job number one. So, we have parts of our product, we have a thing called Data Explorer. You can go as deep and gnarly as you want, you can slice and dice.
But guess what? Only about 15% of our users use Data Explorer. Now, those 15% are maybe at those companies, the people that are then getting things exactly the way they want it, and then they’re publishing them to dashboards that their end-users are then consuming and using to make decisions. I think the challenge we saw is that not enough customers had those smart people at their shops, so they didn’t even know where to get started. So, what we’ve been trying to solve is, like, let’s get, like, a fast start and give you the data that we know you should look at.
And here’s the cool thing. After working with customers for five, six years—I can even say eight years, in some instances—we saw what people were doing. The smart people were creating dashboards that look like this, so why don’t we automate all this? And like, sort of give people what, the best companies, the smartest data scientists, what they think you should be looking at for some of these core use cases, and like, getting people out of the gate faster.
Brian: I guess the part of the reason I ask is because there’s always a design trade off with, like, when you start opening up, like, full access, data explorer, anything you could ever want is here, the complexity level for those novice users really goes up because it’s putting the onus of design back on the user to figure out, well, what’s actually useful here if you don’t know the information. So, I’m just kind of curious, was there any broad thinking or strategy about how to, like, not make the product experience worse for the 85% of people who are never going to go that deep, but also kind of opening up the power users capability to serve themselves.
Todd: Yeah, so what we have done, like, is it’s not just a basic template. We’re asking and interrogating a handful of, like, business-oriented questions, and then we’re taking those questions and we’re creating the Data Explorer reports for you, and then publishing them back to your dashboard. So, they are… they’re abstracting away a lot of the power. Now, what we’re doing today—like, literally, like, right now—is we’re building agentic versions of that whole workflow and use case. So, for example, now you’ll be able to go in, and you’ll be able to type in what you want to see, and instead of just giving you the answer, we’re going to give you the Data Explorer report as an example, and we’re going to give you how we built it.
So, part of our vision here is we don’t just want to, like—look, we can all debate what the future state is going to look like and how much of our UIs are going to be a hundred percent prompt based, or majority prompt based, versus not—but I think in the interim, we want to teach users how we built it. The added benefit of that is, then they’re getting the explainability of, “Oh, this report’s doing this.” Because sometimes the AI—candidly—builds the wrong report, and if I just abstract away how we built it and just show you a bunch of answers and it’s wrong, then we run the risk of leading people in the wrong direction, or in of data. And, you know, like, the danger of showing people wrong data, you know, the classic, “Lies, damn lies, and statistics.” Like, the last thing we want to do is, like, show something that is just—we misinterpret the question, or the AI did.
So, to solve that problem, we’re, like, giving you the explainability. We’ll show you how we built it. And that kind of bridges the gap. And then, over time, I think people are going to go right to Data Explorer, “Oh, AI taught me how to do this. I’ll just go do this myself. It’s so much faster.” So, I think we’re going to see this blended models for usage over time.
Brian: Yeah, I was going to ask you about this later in the show, but since you kind of went there, are you seeing the future of, like, we think about quantitative analytics and all of this, most of the time, users are often goal directed. They’re trying to get a specific answer to something—like, do I need to do something about X, for example—do you feel like moving to a prompt-based system where you’re basically putting a natural language processing layer on top of analytics, so it becomes an answer engine. It’s going to tell you what your growth was, and maybe some supporting evidence, or something like that. Are you deliberately trying to move the product in that direction or do you still think there’s a place for dashboards where it’s like, we’re going to surface the information just by default; we’re not going to wait for a prompt. Like, are you thinking at all about that experience right now and strategically, which way you’re going or is it just kind of still.
Todd: 1000% I’m thinking about it? Yeah, yeah. I think the answer lies somewhere in the middle of what you just sort of posed. If you posed, like, these two opposite ends of a spectrum. Yes, a hundred percent we’re investing in prompt-based UIs, and some of it is just to, like, help get people the answers they want and get things set up.
We are not abandoning or getting rid of dashboards anytime soon because what I want is I want to ask the system a handful of questions. I want to, like, get the business information the right way, then I just want to log in and see what the hell is going on. I don’t want to ask a question every single time I log in. That’s inefficient. I think the world’s going to have blended approaches.
And look, half the time when I log in, even our internal BI system—this isn’t even Pendo, our product—our internal BI system happens to be based on Looker. Like, I can self-serve and answer a bunch of questions with the business without having to ask people. Every once in a while, I have to go to our data science team and say, “Hey, you know, I didn’t see a drill through on this chart.” Or, “I really want a chart that looks at this parameter. Can you give me the power to do this?”
And then they’ll go in, they’ll either update the dashboard, they’ll either give me the chart directly or if it’s something I want repeatedly, they’ll just go update the dashboard for me, then I’ll log in and I’ll see that information. But that’s how it usually works. Like, we want people using data to make decisions. We don’t want people looking at data on a daily basis. The challenge with analytics systems—I’m sure you’ve seen this; this is what you do for a living—is that it’s very hard to measure what—and connect looking at data to outcomes. Like, very few times do I call a customer and they’re saying, “Wow, this one chart in Pendo saved me $10 million.”
Brian: Right [laugh].
Todd: You know? But—
Brian: If only it was that easy.
Todd: I wish it was that easy. But I have had customers that have said, “Wow, we made a handful of decisions, and this is the positive impact.” You know, maybe it’s a million dollar impact and—but broadly looking at, like, data in general, you know? But if you’re flying blind and have no information about your business, like, what is the impact of that? I think we would all agree you’re going to make worse decisions if you have no data about what’s going on.
Unfortunately, people don’t think of it quite like that. They sort of take it for granted that you have business information around what’s going on. That, to me, is one of the biggest challenges with analytics products in general, is that everyone can probably agree that they’re smarter by looking at it; you really can’t place a value and it’s hard to say what specific decision I made as a byproduct of looking at data, right?
Brian: Sure.
Todd: And because of that, we kind of struggle to, like, connect value to analytics and what’s it worth? What’s it worth to be smarter? So, that’s me, sort of the hard piece. The other thing is, I love is customers, we talk about data and analytics versus insights. When we think about insights, what’s an insight? An insight often is something that you did not know, or that’s kind of one of our definitions of it. It’s insight that, like, it’s interesting and valuable and you didn’t know before.
Well, hell, that’s a high bar, you know? Like, something I literally had no idea before, it’s going to like, like, like—but you see, good analytics systems are like, “Hey, we observe this trend that you may not have been looking at. You should go look at this deeper.” Okay, well, that’s valuable, but again, how much—how do you attribute value—how much value can you attribute that? And maybe nothing. Or maybe something. But the system says you should look at it, so there is value in doing all that. So, I don’t know this is helpful, but, I mean, I’m talking in circles, but like, this is at least how I think about the challenge. So.
Brian: Yeah, I mean, this value question is always a challenge, particularly for internal data product people. And my answer is, well, part of this, it’s not as much of a math question; it’s a little bit more on the art side than the math because the answer is very subjective. But I think you can talk to a team—the people that are using these solutions, we can do research with these people and understand, are we making your job and your life better with this information, worse, or is it neutral? And if you’re making—if they feel like they’re getting value out of it subjectively, then you could attach a financial value to that at some point. But it’s a subjective answer. It’s not a single math question answer where it’s $1.687 per whatever. Like, that’s not what it is. And that’s where, I think, [laugh] sometimes data people get hung up is they think there’s a right answer to this question, and it’s a little bit more subjective. In my—I mean, at least that’s my take on it. So [laugh].
Todd: It is. It is, but the end of the day, we all have budgets. We all have—
Brian: Sure.
Todd: —like, like, like—and so what we actually did recently is we actually just productized what we calling a Value Impact Report, and it’s a report on how you’re used to Pendo and improve your business and what we think the overall value was. And in specific to certain use cases, I think it’s pretty powerful. And look to me, is it, like, academically correct? It can’t be, but it’s a pretty good SWAG on how much value we delivered. And to me, it’s the beginning of a conversation.
And like, okay, let’s debate what in this report you disagree with, and then we’ll go adjust, like, the factors and the variables, and blah, blah, blah, blah, blah. But like, let’s at least try to have, like, an adult conversation on what we think the value actually is. Not just say it’s nothing. We all know that’s probably untrue, but how do we get some sort of alignment on how much value is there?
Brian: Yeah, yeah. Jumping over to, like, product management. Management for a minute, when you hire product leaders at Pendo, I’m curious, do you look for people that have a strong data background, maybe they’re a little less on the product management side, or would you rather have, like, a seasoned software product manager who maybe hasn’t done a lot with data science and analytics, they’ve never built an AI product before? Am I even asking the question the wrong way? I’m curious if you have a method in your head.
Todd: No, I think you’re asking it the right way. And we have a broad product. I mean, one area is very, very deep on analytics. Some areas are not, but we absolutely bias towards people that have analytical backgrounds when we’re more hiring product here at Pendo. And I don’t think every company would do that, but for the types of products we build, yeah, I think you kind of have to.
Like, you know, you know the difference between—I don’t know, this is, again, basic example—a mean and a median, like, you’re going to build bad product for analytics people, and that’s a very basic thing, but honestly, a shocking number of people have no idea the difference between those two different things. So, I think you just need to know the basics. I mean, some of our product managers go very deep with engineers on, like, how to design the analytics to make sure that they’re answering the questions that the customers and users want. And they’re getting on the phone with actual data scientists and, like, listening to what they want and need. So, if you can’t speak their language, you’re going to build worse products.
So yeah, not every product manager at Pendo has that background, but a bunch do, you know? At least a half a dozen do, and we do hire for that skill set. Now, we also hire for people that have been a product manager before. I think there is value in that. We tend to hire more technical product managers in general, so I think I tend to bias there. So yeah, yeah, I think you’re thinking of it the right way.
Brian: So, if I was play that back to you, it sounds like, at least for the people working on the hardcore data products, it sounds like you buy us a little bit more towards having a data background. Is that because you feel like we can, kind of, we can ramp up your product skills, maybe they have, like, one year of product management experience and ten years, like, working in hardcore data science. It sounds like that’s a more attractive answer than the opposite, the ten years of PM and, like, one year of, like, dabbling with something with, you know, basic analytics. Is that sort of right? And I know I’m at it’s making that really black and white, but—
Todd: Yeah the reason is I need someone who understands the art of the possible. What’s possible? What kind of questions could I answer? Like, how would I do it? Like, I think being able to go toe to toe with engineers and, like, challenge engineers—because engineers will say, “Well, we can’t do that.” But like, “Why not? What if you did this, this, this, and this?” You know?
You kind of, like, prototype it out yourself. I’d rather have someone with those skills and teach them the product management skills than otherwise. But look on the—then the flip side of it, you have to make decisions in your product. And you asked this question earlier. It’s like, how deep do you go? How deep do you go? That’s a product philosophy question is how deep do you go?
Like, in our products. We want you to go very, very deep. I said, you know, no dead ends is kind of our product philosophy, but we also don’t want to compete with, like, Snowflake or Tableau, you know? I know it’s different products, but, like, maybe a Tableau on top of it. It’s not a general purpose BI tool. It’s specific to the data set we have, generally time series data, so we try to construct an environment where, for that type of data, you can answer anything you’d want, but there are things we say no to.
It’s like, no, no, no, no, we’re not going to bring in some general data feed, create a join between what we have, and then, like, start answering a whole different set of questions. No, do that somewhere else. We have a data sync product. You can export Pendo data, put it into a data lake and throw a Tableau, or Looker around top and do the work that way. And in our internal data science team for internal work, that’s how they use our data, often, is they take data out of it, they use their own systems, and they munge it with other business information and take it.
I think that’s a perfectly fine metaphor for the expert data science team as well. And then it’s like, okay, how do I get our feeds? Do I get the right data exported in the right format to the right systems to make sure that those folks can take it and do what they want with it? So.
Brian: Yeah. I know you guys are starting to move into some of the these agentic-oriented products, and even measuring, you know, a company’s agent efforts, right? If they’re building agentic AI products. Did you feel the need to, like, I need to have, like, an AI product manager running this, like, someone who’s definitely worked on AI products, or were you like, no, we’re going to work someone that we have internally, and they’ll learn that along the way? Did it even come up as a question? I’m just kind of curious if—
Todd: Yeah. I mean, look, the overarching question I think you’re asking, too, is, like, do we have the right people on the bus to solve and address these problems? It’s a question I ask myself all the time, and here’s the reality of the situation of your question: like, there aren’t a ton of AI product managers out there, and the really good ones that are proven are being snapped and gobbled up by big tech right now, and I think that’s a challenge. Now, last July, we made our, I’d say, first real AI acquisition of Zelta AI. And the CEO of Zelta AI is our head of product for that agent analytics product you’re talking about, and, you know, built an AI-first business.
And, yeah, so I can solve this problem through acquisition, you know, which is an expensive way to hire an AI product manager, admittedly, but I think was one of the bold bets we made to get different talent in-house, and we’re going to blend that with the existing product management team we have. And honestly, I’ve been really pleased about how curious our existing product managers are around AI, and they’re working on building their skills and their craft to become AI product managers. So, I think acquiring a company bought us some time because we just quickly infused the skill set in our culture while the rest of the team sort of up-skills—or re-skills, so to speak—for AI technology.
So, I think it’s a blend of both. Is kind of how I’ve thought about this problem, but, man, if our strategy was only go out and hire AI product managers, we would be failing because it’d be really, really hard to do. Like, the Wall Street Journal had an article on Monday how, like, Mark Zuckerberg is personally reaching out to people and offering them $100 million packages. You know, like, we’re less smarter than Meta [laugh].
Brian: [laugh].
Todd: I’m not reaching out to anyone and giving him $100 million package to do that. So, like, that’s just not an option for me, and that’s okay.
Brian: So, people listening want $10 million packages, that’s totally fine, though?
Todd: [laugh]. Yeah, exactly.
Brian: Todd@pendo.io what is it? No, just kidding [laugh].
Todd: I was like, I can go back and I can get research, and I can get $100 million—
Brian: Right, right.
Todd: — pretty interesting. So, but I’m sure it’s probably harder than it sounds [laugh].
Brian: Yeah. I mean really—and I was less—I wasn’t necessarily trying to pry about your specific team. Really, what I was getting at is, like, do you even see—running a, you know, a date, a large data product and product organization, do you see AI product management as a distinct thing than just product management? Like, it’s different enough that it is a skill. There’s a substantial skill set that’s needed there you maybe you’ve seen through acquisition, they are working differently. It’s a different product that requires a different mindset. Or is it really that’s a lot of hype. Yeah, there is some stuff, but it’s really product management, and there’s a little bit of new domain knowledge, but it’s not, like—there’s too much hype attached to that extra label, the adjective in front of it.
Todd: Yeah. Look, I think AI is amazing, and I don’t want to say AI is overhyped because AI in general is—yeah, it’s the revolution that we all have to pay attention to. Do I think that the skills necessary to be an AI product manager are so distinct that you need to hire differently? No, I don’t. That’s not what I’m seeing.
If you have a really curious product manager who’s, like, going all in, I think you’re going to be okay. I think you’re going to be okay. Some of the most AI-forward work happening at Pendo—not just product management, but like, our design team is, like, going crazy. And I think one of the things that we’re seeing is a blend between design and product, that they’re always adjacent and connected; there’s more sort of overlappiness now. I mean, a product manager can go into a prompt, like, Lovable or Bolt and generate a whole working prototype.
Normally, they would call their designer and say, “Hey, I want to build a prototype.” They’re not doing that anymore. They don’t even need to. And we have designers who are coming up with their own ideas, that are again, prototyping themselves, and, like, you know, making it more interactive prototypes, and sort of understanding how it works. And look, some level of AI product management is like, how do you craft the prompts right? This concept of, like, collaborating with the engineers on this prompt engineering and the context and making sure it’s right.
Yes, you have to change your skill set and yes, you’re going to work a little bit differently, but like, look, fundamentally, it’s still really good skills to be able to understand how to work very closely with customers. Like, at the end of the day, I think the customer-centricity and, like, collaboration, I think it’s going to be even more important in this AI world, getting that context right, the prompt right. I mean, I think this idea of, like—I mean, what I do think is going to be really, really different is how we develop. The software development process is going to be very, very different. Like, we’re already seeing, like, how you build this stuff is just different.
But it can be taught, but you have to have the right mindset, the right culture, to almost unlearn how you were working before, and relearn a new set of skills in a new way. So, I think—and that takes leadership to force that to happen. You know, one of the things that we did—we were doing a localization project, pretty, pretty standard project, you know? People need to use our product in the language of their choice. It’s really important for, like, the Japanese market, et cetera.
And we gave the team the problem, we said, “Look, we really need to do this. It’s really important for the business. How long is it going to take?” Traditional estimates. And we’re like, “Well, how much would it take with AI?” It’s like, “No idea. Never done this before.” And guess what? That’s a really fair answer when no one’s done something before. So, we had to give them the space to spend, like, some period of time, like, a couple weeks, maybe a month, to really, really try AI and see what would the impact be. And sure enough, that estimate came way down, based on the usage of AI.
But you had to give people space you can’t just as a leader, like, snap your fingers and say, “Everyone do this,” and, like, magically expect better results. You have to give people space to learn something new and try it out and understand the gotchas. And I think—and the power of it. That’s a big piece of it. The other big piece of it is the technology is changing very, very rapidly.
Like, what was possible six months ago… like, I was talking to our CTO the other day, like, six months ago, he was wildly frustrated with using, you know, Cursor. For his job. Not every job, but his job. Now he’s, like, “I’m far less frustrated.” Like, as of today. You know, six months, there’s a big difference with some of these foundational models and some of the innovation we’re seeing. So like, you just got to stay with it and not give up on it. I mean, I think that’s ultimately what you need out of it.
But, yeah, it’s exciting times. And then I think, as a leader, like, you all need to play with it. Like, one of the things we’ve been—because we have an analytics team ourself—what could the do with our data, just natively, on its own, with a very basic set of prompts? And it’s pretty amazing. So like, our, like, basically head of analytics, who’s an expert, inarguably, you know, has gone in, like, trained an agent to, like, be him. Like, this is how I think about analytics. This is what I look at.
And then he’s giving it raw feeds of data, and see—like, wow. I mean, it’s pretty darn impressive what it can do. And guess what it’s free of bias [laugh]. I mean, we all know when we as business leaders, we have this idea in our head, and we just were convinced the strategy is going to work. We go to our data science teams, like, yeah, “I need some data to prove that, like, this idea is going to work.” [laugh].
It’s not, like, give me 100 ideas and, like, show me where the data leads me. It’s often, like, we’ve got a point of view, right? I mean, and that’s probably okay, but it’s kind of nice, too, when you have these large language models which don’t have a point of view, per se. They’re just, like, interpreting things. And they could interpret a lot more volume of activity than a human could, and that’s really where the power comes.
Brian: Yeah, just one kind of, like, follow up on this since we’re kind of jumping into the AI, or specifically, we’re talking about GenAI, really, it sounds like.
Todd: Sorry that we went there. So I—
Brian: No, it’s fine. I’m kind of curious, you know, as a product leader on the scale of one to ten, ten being I can’t sleep at night, and one being, essentially, I sleep just fine, where are you on the—when we open up the large language model capability in the product, we also open ourselves up to unknown responses, unknown experiences. We can’t design all of our experiences anymore and know exactly what’s going to happen, let alone the answers. There’s now a spectrum because it’s probabilistic. Does it concern you at all?
Like, where would you put yourself on that kind of one to—zero to ten I guess I should say? Like, are you not that concerned because you’re always providing this evi—you talked about kind of how, if you ask a query, we’re also going to provide all, like, here are the—you know, here’s how we came up with the answer, and what went into it. Is wrong data… is any of that, the you know, hallucination or confabulation is that much of a concern for you as a product manager, that it’s kind of out in the wild and we might give the wrong answer.
Todd: We don’t want to give the wrong answer. I think trust is one of the key things. I wouldn’t say—like, one, I don’t think I am sleeping well at night, so like, that’s the answer to the question—
Brian: Because of this?
Todd: —but not because of this specifically.
Brian: Oh, okay, so pretty mild level of worry on this [laugh]?
Todd: So, I would say… three or four there.
Brian: Oh, okay.
Todd: And why? I think it’s something not to bet against. Meaning, I’ve seen this play before, and this is a fixable problem and will fix through advancements in technology. I think it’s a short term worry, of course. I am more worried about doing nothing, about being complacent, around not moving fast enough.
That’s what I lose sleep over. Like, am I personally doing enough? Like, are we pushing enough? Are we investing enough? Like, are we moving in this direction fast enough for our customers and for our company, yeah. I mean, it gives me sleepless nights, you know? Not a ten, like, I can’t sleep at all. Like, I probably wouldn’t be functioning on this podcast if I wasn’t sleeping at all. But yeah, I mean—
Brian: The what’s keeping—like, what about this specifically, does worry—I mean, you talk about, are we doing enough, but I guess, like, strategically at a high level, what are you trying to deploy this capability against? What’s the enemy that you’re trying to slay on behalf of your customers, like, as opposed to just leading with a tactic, right? Like, “Oh, here’s a tool. It’s a new hammer at Home Depot. Go get ten of those, give every product manager one of them and let them hit shit with it.” You know?
Like, I’m sure that’s not what you’re doing, so, but what would they be hitting? Like, what problem are you trying to attack with that you think you’re—it sounds like you’re pretty bear—or, I’m sorry, you’re bullish on the technology here. What’s the kind of problem that you think can be solved here?
Todd: I think this technology is just so powerful. And you just worry about what’s the future of work, and how people are going to do it, and what’s our place in that future of work, so it’s more of, like, an existential worry than it is anything else. Like, what are we solving with this technology? I mean, I think we’re solving, like, what we opened the conversation talking about is, like, look, some percentage of my user base is experts, and I don’t really worry about them. They know how to use our products, they know how to use data, they know how to use—like, run our Data Explorer reports. Like, they are experts. They are great. They are really, really, really capable.
That is 10 to 15% of my users. You know, 85 to 90% of my users are, it’s new to them. They’re trying to figure out how to do it, and this technology is an amazing bridge to help guide people along this journey and sort of like be almost a junior product analyst agent, so to speak, like, a companion, sort of help them along this journey, you know? I think the future of work is going to be, we’re going to have a set of companions which would have helped offload a lot of the harder tasks, maybe the tasks we’re not quite as good at, feed us information, so then, you know, then it helps us focus on the more creative bits, the more critical thinking bits, the okay, given all this, what’s the decision? Right?
You know, I think a lot, you know, like, one of the things Jeff Bezos says a lot is that, as leaders—and I think a lot of people are leaders in organizations—you know, their job is to make a handful of, like, really important, high impact decisions. That is their job. I don’t think agents are making decisions anytime soon. People can probably disagree with me on that. That’s fine, but that’s my take.
But I think all of these agents will help you make better, smarter, more informed decisions if you’re using them correctly. And the people that end up using this will just make better decisions because they will be more informed, they’ll have more research, like, you’re just able to do more with all this. And like, and I want to make sure our data is part of that conversation. And if it’s not, that’s not good for us [laugh]. So, like that, maybe, I don’t know if that helps explain why I’m thinking about it a lot.
Brian: You talked about not sure if we’re investing, you’re always kind of wondering, are we investing enough here? Are we kind of staying, you know, ahead of the curve, or whatever it may be. Do you have a sense of, like, how you would know? How would you measure in the future whether you’re doing enough? Is there a signal you’re kind of looking for in terms of the product or some—a behavior change, a signal, something that would be observable, that would tell you those bets I placed worked. I feel like we’re now doing enough, like, we’re on the right track. Do you have a sense of, like, what that is, such that you would sleep better because you’d be able to tell yourself, you know, I set these goals, we hit them. How would you know, I guess, as a leader, like, how do you know?
Todd: I think it’s ultimately customer traction and adoption.
Brian: Okay.
Todd: Like, all of it is so new. If we, like, fast forward a couple years—you know, even a year—if, like, customers are using this day in, day out, and it’s changed the way they’re leveraging Pendo, that’s winning. That’s winning. That’ll make me feel good. I’ll be happy with the bets. Look, I think, yeah, and it ultimately is a more efficient way to use our product. Maybe we need fewer humans here at Pendo working directly with customers to help them unlock outcomes, right?
Because what happens now? What happens now at all? A customer is struggling to do this, we’re going to send an expert human in, and we’re going to, like, help train change management, get people on board, do all that work. We do that all the time now, and we’ll continue doing that for probably a long time because, like, plenty of customers will not be ready for AI. But if AI starts offloading some of that and it starts to get easier to see that level of change management without relying on our humans to do it, that, to me, is pretty powerful. That’s the promise of it.
Brian: So, less training requests, less how do I do X in the product questions things like that. Those might be some indicators?
Todd: Yeah. Yeah, yeah, yeah. The basics. There’s always going to be advanced situations we’re going to want to send our humans in and, like, collaborate on, but boy, if we can take some of the take care of some of the basics like that’s really, really, really key. Look, I think a big part of all this is we kind of, like danced around this topic, but how do we get more people to be data informed, data driven? How do we do that? How do we make sure data is part of every conversation?
Still, all too often, I encounter companies they’re like, “I don’t know if I really need data. My instincts pretty good. I talk to customers a bunch.” Like, really? You just trust, like, whatever conversation you had yet last week? Not like, again, recency bias, having a—like, don’t you looking at data? Aren’t you, like, managing it? How do you know that whatever decision you made last week was a good one?
Plenty of people don’t care. So like, I think we’re still on a mission to, like, help people change the way they work, you know? Leverage data more. I think AI can play a part in it because it, like, reduces the effort and knowledge to sort of start getting some basic data, some basic insights, you know? I hope. I hope it does. So.
Brian: Yeah. I guess it was funny, you were talking about, you know, both the, you know, reduction in customer service inquiries, also the product adoption, like, is it getting traction, and I guess in some ways, if you think about, one thing AI can do is probably reduce the amount of labor required to get the insights and decision support you’re looking for, which drops usage of Pendo or any other tool that’s leveraging this. So, I’m curious, like, and you talked about the quant—the kind of the quant, like, looking at data versus the qual, to me, the answer is, you need both because less use of the product can be a win because it actually takes me less time. I don’t need to spend as much time in Pendo because when I go in there, I get my information faster. Well, from a UX perspective, we might call that a huge win, but quantitatively, the dashboard is going to show, oh my gosh, our average spend time is, like, down 46% in the product.
And this is where I get concerned about, like, product analytics versus user experience outcomes because the map is not the territory. Like, the product never really measures user experience; it measures the product usage, not the humans experience using it. So, I’m curious if you think about that at all. Like, we’re going to actually give reasons less reasons to come in here and spend as much time, for example because we made the experience better of getting your insights out of it. I—[laugh] are you giving any thought to that? Or… [laugh].
Todd: No, you’re—but honestly, this is an age-old question. This is a question we’ve talked about forever. Like, if I—time is a delicate thing to be measuring, you know?
Brian: I know it’s counting adoption always feels weird to me. It doesn’t—
Todd: Was it—is it Campbell’s Law? Without those—you know, I was just reading a blog post about it, which is, like, be careful what you measure, you know, and then they were just saying in India, like, people were measuring cobras caught, and so people started breeding cobras, like, rampantly so they could catch more, so they could get more money from it. Anyways, like, it’s metrics dysfunction, like, story. Yeah, if I say, oh, time is the top measure, then, like, let’s find a way to, like, make it painfully terrible to use our products so that people spend all their time in it. Like, that’s a bad outcome.
So, yeah, so it comes down to outcomes. You said, UX outcomes. Pretty interesting that now in this agentic world, or AI world, people are saying that SaaS-based pricing, which is typically seat-based pricing, is dead. You know, I don’t know if—dead is a strong word, and I don’t know if it’s actually dead yet, but that people should be pricing on outcomes. And I do think I talk a lot about time to value, or time to outcome in our product, and that’s ultimately what I care about.
Like, if your goal is to have this impact on your business, and you could spend as little time in Pendo and still get that impact, that’s nirvana. That is literally what you want in a product. That’s good UX—
Brian: Value. Yeah.
Todd: —getting in and out. So, but I think we all know, too, if no one spends time in your product, that’s not a good thing, right? So, both can be true. So, I think about things like stickiness, which is—it may not be an aggregate time, but how often are people coming back and checking in? And if you had this, like, companion, or this agent that you just could not live without, and it caused you to come into the product almost every day just to check in, but it’s a fast check in, like, a five-minute check-in, a ten-minute check-in, that’s pretty darn sticky. That’s a good metric.
So, I like stickiness as a metric because it’s measuring, you know—it measures, like, are you thinking about this product a lot? And if you’re thinking about it a lot, and like, you can’t kind of live without it, you’re going to go to it a lot, even if it’s only a few minutes a day. Like, I mean, social media is like that. Like, I don’t—thankfully, I’m not addicted to, like, TikTok or Instagram or anything like that, but I probably check it nearly every day. That’s a pretty good metric.
You know, it gets part of my process of any products that you’re checking every day is pretty darn good. So yeah, but I think we need to reframe the conversation not just total time. Like, how are we measuring outcomes and value, and I think that’s what’s ultimately going to win here.
Brian: Yeah, yeah. Todd, this has been really fun to chat with you. I kind of wanted to give you the last word. Is there anything I didn’t ask you that I should have that you want to share?
Todd: I do think you were touching on something earlier that is an uber trend we’re going to have to all pay attention to in this world, generative AI. And it’s one of the challenges is because AI has reduced the overall cost to develop new things, we’re seeing more and more new things. And a lot of new things kind of sound and feel and kind of look like other things. So, we’re going to have more choice, probably. Certainly in the near term, we’re going to.
The question is, how do we know when the answers are better? How are we judging it? How are we grading it? When you have three things it all kind of sound like, what data is the best data? Like, how do you actually know? I think there’s going to be a new… way to evaluate things. We talked about the internal use case. We were looking at a few data science tools that were all AI based, and they all kind of sound the same, we all thought that they were same, and the vendor came back and said, “No, try these three data points.” Wow. Amazingly different results.
Like, quality is going to be a really interesting question going forward. And how do we measure it? How do we know it? How do you build for it? Like, how do I make sure that my agents are just better quality? And how do I prove it to someone, maybe as prospects, to people that aren’t even customers yet, and then certainly to customers over time?
So, I think quality is something I’m thinking a lot about right now. Like, how do I make sure our stuff is just, like, amazing, like, really good data? Like, how do you measure accuracy, you know? You mentioned accuracy. Part of it’s just, like, what’s the query look like? Kind of, look at the underlying SQL, blah, blah, blah, like that of course, will do, but, like—
Brian: Technical quality, yeah.
Todd: But if you’re trying to service an insight, like, a real insight, like, what’s the quality of that insight, you know, versus others? I think it’s going to be a fascinating world. And when we all have to, like, the great products are going to be the ones that are highest quality, of course, but like, how do you know it? So, I don’t know.
Brian: That’s a great way to close because I think how will we define quality upfront such that we will know if we hit it or not, it takes a lot of thinking, and I think especially when we talk about generative AI products, I always like to think about it, it’s now a continuum, and you can go negative, too. Like in the past, you didn’t usually go negative unless your product just totally bombed and you made things worse, but like, now you could go negative, [laugh] but you could deliver amazing, too. Like, the world is bigger here, so I think having a sense, especially as a product leader, about defining what success looks like? What are the metrics for that? How would we measure and know for both user, you know, the business side—yeah, sure, you can measure sales and growth or retention, or whatever those things are, but like, particularly from a user experience standpoint, how do you know if we made this situation better for this person’s life?
Todd: Exactly.
Brian: That’s going to be an interesting—and I think it’s really important to do that because we can possibly make things—we can screw stuff up with this technology, as well as we’ve seen many public examples of this, you know? But, that’s another episode [laugh]. But Todd, thank you so much for sharing Pendo’s story with us. You know, LinkedIn, you mentioned social media a little bit. Is that a good place if people want to follow your work?
Todd: LinkedIn, X… yeah. I’m just tolson, or todd@pendo.io so reach out. So, love to hear from you.
Brian: Got it. Pendo.io for the product, I assume?
Todd: Yep, a hundred percent.
Brian: Awesome. Todd, thank you so much for coming on Experiencing Data with me.
Todd: Thanks, Brian, for having me. It’s great to be here.