Transcript:

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, I want to talk about how adoption usage of analytics and intelligence products is not the right way to measure user experience and satisfaction of customers. If you’ve listened to the show before, you’ve heard me talk about customers. This is kind of a loaded term. We could be talking about fiscal buyers, we could be talking about beneficiaries of products that are being sold, we could be talking about end-users who may not be direct beneficiaries necessarily and may not be buyers. So, in B2B contexts, it’s important to know what we’re talking about here.

And so today, I really want to focus on the actual users, not necessarily the fiscal buyers of your B2B analytics product or automated intelligence application or tool. I really want to talk about the people that are hands-on-keyboard, so to speak. So, what do I mean when I say adoption is not equal to value? Well, let’s jump in here. And this episode is inspired by an article I sent out to the mailing list. I do write to the list every other week, so every Tuesday, I write a little—my latest thinking to the list, and then on the next Tuesday, I release a podcast and cycle back and forth, and that’s generally my rhythm.

And I got some replies about this, and at least one person, a couple, had asked about some concrete examples of how else we can measure user experience quality beyond adoption and product usage metrics. So, let’s jump in. And frankly, it’s the only answer I ever get to the question, “How would you measure success for your end-users?” When I ask that question on a discovery call with a prospective new client, the general answer I get is, “Adoption.” And that’s it. Not even how much, not what kind of adoption.

And I get it, you probably believe that’s a reasonable place to start, and that if they’re using it, then they must be getting value. Except counting adoption, which is probably happening via analytics on your analytics product—go figure; getting meta here—it’s not actually telling you anything useful. Because beyond the information that adoption is not zero, what are you going to do with the information? If it’s 16 units of adoption or 52 units of adoption, or every user once a week, every user once a month, half the users all the time, half the users never. Now, what?

What number would make you change anything in the product about what you’re doing? The design of the product, the features that are in it, addressing trust issues, what are you going to do with product usage analytics in order to fix user experience problems, at least? So, here’s the deal. Adoption matters and usage does matter, but if you want to create a data-driven product that users find indispensable, you can’t boil this down to how many logins there were, how many sessions there were. Usage of these types of products—or really, any products—can signal a tax on your customers and your users.

Frequent usage might mean this product is hard to use and it takes forever to get anything done. And similarly, lack of usage as reported by product usage analytics might mean, hey, it’s so great, the users really never have to log in or even think about it. I mean, think about something like your agent. Like, the agent’s off doing a bunch of stuff, like, can we just look at the amount of usage to know something? And what about the human in the loop that’s leveraging an agent? How much usage should there be?

In theory, so many companies are basically beginning to use these tools to reduce busy time labor, or to offload certain kinds of labor that they think machines can do, and so by definition, you’re trying to reduce usage. So, the story is more complicated here. The better way. What is it? Well, I’m going to tell you.

You need to understand the level of frustration, satisfaction, or delight that they have now. You need some kind of baseline reading of their experience. And this is often a mix of both qualitative and quantitative measures because product analytics never going to tell you why, and they don’t really tell you anything, they don’t give you any context for anything, and they’re definitely not going to tell you whether somebody was satisfied, how they felt because the computers can’t track that. So, when you know your customers needs well, and you know what their baseline experience is with your solution—or even your competitor’s solutions, if you’re, you know, considering going after a competitor you can do some of this analysis with a competitor solution—then it’s much easier to define what an improvement in their life would look like, such that you could observe it. If you don’t know your starting point, then you really can’t identify what an improvement is, and so you’re not going to know if you’re making progress. You’re basically just guessing.

Fiscal value and sales metrics are easy to see and observe, and they’re lagging indicators a lot of times. Sometimes they’re a leading indicator, but then we don’t realize what’s happening with the UX. We make the sale, but a different set of humans are the ones that are going to be hands-on-keyboard. So, UX improvements—and I’m borrowing heavily here when we talk about user experience improvements are really user experience outcomes. I want to give credit to Jared Spool for this whole framing which I think is an excellent way for non-user experience people and even UX professionals to think about user experience in a way that’s observable, that’s objective, that we can all talk about in simple language. So, I want to give a shout out to Jared, since I’m leveraging some of his ideas here—but user experience improvements changes in people’s lives that make them love your solution, that make it feel indispensable or delightful, these improvements are easy to observe if you have done the proper discovery work to define what the current state is and what a future improved state might look like. So, most of you listening to this show, you probably somewhat identify as a data person, so stop telling yourself that counting logins and sessions is really telling you anything useful beyond it’s not zero. There are better ways to measure, place good product bets, and observe if your product is working.

So, how do we do this? So, I wanted to, like, jump into, like, a realistic scenario here, and just give you some thinking about how you might approach this. So, one concept of a place to start is starting with a baseline and not a feature list. So, you can’t define better without really knowing where you’re at. So, if you don’t know where your users are at in terms of satisfaction, frustration, or delight, then you really don’t know how to make an improvement in their life. All you’re doing is adding features, but we don’t actually know whether the experience of using any of these capabilities is actually making things better or not.

So Jared, as I mentioned earlier, will often draw a chart for this, a simple x-y axis where the x axis is in the middle. So, if you imagine this curve, you draw a curve on it, and anything below the x axis, let’s say that’s zero, right, anything that’s right on the line there, we’ll just call that satisfaction. And over time, they use this product—so imagine you’re drawing out the chart data, and you’re just drawing this line from left to right, when the line goes down and users have frustration. We call that lower part of the graph frustration, and we call anything above the x axis delight. And so, your solutions, your products, are always experienced over time.

And that might be a very short term task, like coming in and checking some numbers like they do every single morning to see if anything interesting has happened, or it could be thinking about this at a larger, more strategic level, which is, like, you know, we’re doing, you know, financial planning and it takes two months to do this for the company at the end of the year, there’s multiple touch points with the software, there’s Excel, there’s all these other programs in the loop, they’re using AI to do some of the stuff, and your product is part of this, like, ongoing process of doing some really complicated, you know, business planning or something like this. So, we can be really down at the ground level, we could be at the 10,000 foot level, but the point is, experiences happen over time and people are going to likely be drifting from points of frustration, sometimes into points of satisfaction, and then sometimes to points of delight. And it’s good to know where they’re at right now. So, this is that baseline is understanding in their workflow, and I would probably recommend starting with something at, like, a task level workflow where there’s a clear start and end to something and eventually you want to level up to more strategic. This is where user experience strategy can come in and really help you out is thinking more strategically over longer periods of time about how your product really fits into a larger context of use, beyond just tasks and jobs to be done and this kind of thing, but I think if you’re not a user experience professional, this, the kind of workflow level is a good place to get started with this.

And frankly, I think a lot of product teams just skip the baseline entirely. They jump right to solutions. They don’t necessarily—if the product is selling, that does tell us something, that there’s a level of satisfaction, at least to the fiscal buyer, but if the fiscal buyer is not the same as the user, this is where attrition can happen, or apathy, or dashboard deserts, all these things that we hear about that happen constantly with analytics solutions, and they happen with more data science solutions and using leveraging machine learning that they just a lot of this stuff ends up not getting used. And so, I think this is where, when we don’t have good information about what people are actually experiencing in the wild who are hands-on-keyboard, this is where things go wrong. So, without a baseline, you’re not going to know if you’re making progress except through lagging indicators, like, oh, someone else bought, another person bought something, or some random person stopped you in the hallway and/or sent you an email and said that they liked it.

So, let’s take all this and make it even more concrete. So, let’s get into a specific example. So, I want to imagine that the products, the product that we’re building, or you’re building, is a marketing analytics platform for, like, e-commerce campaign managers or something like this. So, the primary user here, their persona, is like a campaign manager who’s trying to make good decisions about their advertising, targeting, creative, and spend how much money they’re spending to try to drive new business into the e-commerce platform.

Now, their real job is actually not to do that stuff. It’s really, don’t blow the budget, don’t look like I’m spending a lot of money and not getting any results, be able to justify my decisions to the business and primarily my manager, and run smart experiments. This is what their personal incentives are this is the kind of stuff that’s not going to show up and product usage analytics. What are their incentives? What drives them? What makes them look like a hero? What are they worried about? These are all things that we cannot get by looking at quantitative data captured by analytics tools.

So, what is a person like this doing all day long in their job? Like, out in the wild, what is this campaign manager’s life like? And they might be exporting data into Excel, uploading CSVs to Claude or ChatGPT, and asking for advice on it. They’ve got their own workarounds that they’re using. They’re tweaking, you know, copy on the ads or trying new geotargeting, whatever it may be, there’s a whole world of stuff that they do, and the question is, how informed are you at a real, fairly intimate level, about what it’s like to do that work and what their baseline experience is using your product, or whatever the status quo is?

When you find out that there are workarounds and they’re bypassing your solution and not using it for some of its intended purposes, this can be some signal about where your product might be failing them. So, what might we actually measure here, beyond adoption? Again, product usage analytics and are not telling us about what people’s perceived experiences. So, some things we could find out are perceived time on task. So, how long is it taking them to do this work?

And one of the key things here is the perceived amount of time and less about the actual amount of time. So, what you might think is a ridiculous amount of time to do something, the question is, are they feeling like they’re making a reasonable investment of time to do that work or not? So, their perception about it is what matters. And this is why, you know, sometimes you’ll hear, like, oh, there shouldn’t be more than three clicks to get anything done. We just heard this on a show.

And I understand where that comes from. We’re trying to boil the world of usability down—and experience—down to this, like, three-click rule, but what really matters is, does the customer feel like they’re making progress? Do they feel like they’re getting value? Do they feel like the amount of time investment and effort is reasonable for what they’re getting back? So, it really gets into what the perception is.

Another one—so we talked about perceived time on task—decision confidence. This is really key in the space of analytics. Did they feel safe pulling the trigger on, say, a spending decision? Like, I’m going to double the budget, or I’m going to create another campaign over in this geo with slightly different language because I’m pretty convinced that your tool, which you know, ran all these simulations somehow, and did some really fancy number crunching and used AI told me some stuff. And so, how much confidence do customers feel when they actually are taking this advice, or perhaps taking the advice of an agent or some intelligence that’s in the software? How much confidence do they have in that information? Again, this is another feeling, a perception that they’re having.

And similarly, reporting confidence. When these users have to explain the results to a boss or a client and that kind of thing, did the tool or product, did it give them what they needed to justify the decisions they made in the real world? Say in this case, like, let’s say the campaign underperformed and a lot of money was wasted, and things didn’t go so well. Are they able to create a reasonable story there to explain this? And this really gets into the details about, like, well, what would they be sharing with a boss or client? Are they building a PowerPoint deck? Are they sitting down and showing their boss some dashboards and some numbers?

And then that really gets into, like, well, how informed is the boss about all the domain jargon in marketing analytics? Is there a view that’s suitable for an executive stakeholder who may not know the ins and outs of campaign management, but they do know that. You know, we spent $2 million here, and we really did not get the number of leads or purchases or site visits or whatever it was that they’re looking for. They do understand at a high level when performance targets haven’t been met. So, this would get into, well, what? What does that e-commerce manager need?

How could we make that experience at least satisfying, but ideally delightful in some way because delights is what’s going to make people stick around and find the tool indispensable, not to mention possibly talk about it or drag it along to their new job. When they leave the company, they go somewhere else that they might say, I can’t do my job without that amazing Betacorp product or whatever the heck it was, right?

So, what else? Well, level of worry. Are they anxious the system will go off the rails? And this is especially true as we start to use automation and agents and things like this. How are they feeling about outsourcing work, for example? If they’re starting to use machines to do this, are they worried about this? Are they confident? What’s the risk level they’re having? Again, you can’t look at, oh, you know, on average 0.8 user—or 1.2 users per week, created an agent and deployed it to do work, and here’s the token count. None of that is going to tell you what the level of worry the humans in the loop who are responsible for the decisions of that agent, nothing is going to tell you that unless you go out and actually interact with these end-users.

Another one, another thing you can measure. Inaction as a signal. If users aren’t taking the actions the product is designed to support, then that itself is a metric. You know, if it’s like traditional analytics, are they stuck in analysis paralysis? Maybe there’s too many variables there. Or as is the case, sometimes on products that I’ve helped design and with clients I’ve worked with, sometimes the product creators know the domain or the vertical so well, and they feel like we need to expose all this information that we’re collecting because the right user would know how to put this together into a story that’s insightful and that would help with some kind of downstream decision.

But in reality, there are some, if not many, users who don’t know, hey, why are you showing me all this information? Why are you telling me any of this? They are not connecting the dots between the stuff that’s being shown in the tool to value that they can feel and understand. So, this can result in simple inaction and apathy. Like, I guess I’ll wait for the phone to ring when someone’s mad because I don’t know what to do with any of this, no one’s really asking me to do anything with this, and it’s kind of like, yeah, so what? So, there’s the inaction.

So, are we aware of where they’re not taking inaction? We don’t know that. You can’t tell, really. Yes, you can measure that a page didn’t get used or some—you know, you can go in and look at this kind of stuff, but what we don’t know is about why did they not take that action. We need to understand that in order to figure out how might we change the product to enable the forward momentum and the value creation that we want for the user.

So, what’s the theme here? You know, most of the signals that will tell you that you’re delivering a positive UX are qualitative metrics that you can only understand through direct observation and inquiry, not through metrics that are collected by computers, also known as analytics. So, product usage analytics just tell you about usage, but they don’t tell you about experiences, feelings or frustrations. They don’t tell you where people are on Jared’s curve, right? Are they most of the time hanging out below that x axis in the land of frustration? Are they cruising at just kind of good enough, which is satisfaction, right in the middle of the chart? Or are they spending most of the time above the x axis in the land of delight?

And obviously, I’m sure you would love to have your products up in the delight category most of the time, and at a minimum, in the satisfaction level. And of course, there might be parts of the product that are satisfactory and other stuff where you really hit it out of the park. The question is, do you know where those places are, and are the places that are delightful actually the most important parts of the product, or are they parts of the product that rarely get used? Like, wow, you have this amazing onboarding and setup wizard and it makes you feel really good to get through it, and you don’t need any—you don’t have to call anyone, there’s no SDR, there’s no, like, sales engineer that—it’s great.

And then as soon as you get in, it’s like, you’re waiting for value, or just blank screen, don’t know where to start. So, you hit a home run on the setup thing, but as soon as the thing is set up, the time to value becomes really slow because the user has to take over, and it’s no longer this guided workflow. And this is actually where it really matters because the customer didn’t buy the product to set it up and install it. That is not a goal. That was a tax they had to get through. And we made that experience good, if not maybe delightful. Maybe it’s really fun. Maybe you made it just dumb, easy, but does it matter?

Well, again, you’re not going to know. You can go look at the time on task, you could time the wizard and all that, and if you’ve done some qualitative testing, you might know, you know, what a reasonable amount of time is based on your conversations. And fine, at some point, there might be some measurable analytics parts that can go along with this, but does it matter? These are product strategy questions that the analytics are not going to answer.

So, given all this stuff with AI, let’s talk a little bit more about the whole AI layer here. You know, trust—when we bring AI into the products, I mean—trust is necessary, but it’s not sufficient because value creation is still the goal. So, when we talk about AI and agentic AI, you know, I think users are shifting from slow analytical thinking in the traditional ways when we deal with data to try to make decisions, to more of this curation mindset. So, instead of slowly building a case for decision and closely analyzing evidence, it’s almost like I have an assistant doing that work for me. I am not putting in the same level of cognitive thought into what’s going on, and now I’m mostly reviewing and approving what the system proposes to me.

And so, I think this is changing—and a lot of this is, I just want you to know, these are my opinions, this is a very active space, as many of you know, so this is my current thinking on it, and I just want to state that. This is not based on deep scientific studies or rigor or anything like that. These are just my opinions and how I’m thinking about this stuff right now and how I would talk to clients about these things. But this changes the core UX question from, you know, can they find an insight or something useful in the data to do they feel safe acting on it or allowing a piece of automation to act on it, right?

So, key questions the product team needs to answer: how much evidence does the system need to surface for the users to feel confident letting it do its thing? What risks does the user not yet know about? And how does the product make those visible or invisible? How does the weight of the decision affect how much transparency is needed? You know, we probably can posit that high-stakes decisions require more evidence, right?

But at the same time, we don’t want too much evidence and poor design to make the high stakes blurry such that they can’t actually understand it because there’s so much information being displayed to users. So, there’s a trap, though, with all this, and that’s that a system can be trustworthy, fully explainable and well evidenced, but also still not deliver value in the eyes of the end-user. And this is what I call technically right and effectively wrong, which I’m sure many of you have heard this phrase on the show before. The AI recommends a solid action backed by solid data, but the user actually doesn’t care about it.

So, let’s switch context. We talked about a marketing analytics tool. Let’s go into this mythical world of sales software, like CRM tools, stuff like this. I often use this because I think a lot of data people have probably dealt with a CRM and know what that is. So, let’s imagine that your tool, your product, can help reawaken old leads, and it knows which ones should be dug out of the boxes deep in the database and surfaced to a salesperson so that they can try to reawaken some new business, right?

So, let’s say it’s going out and it’s finding, like, two year old dead leads that apparently have a lot of strong signal that’s relevant right now somehow, but they’re old, and the salespeople don’t know how to surface these at the right time. And so, your tool, like, leads with these AI-drafted emails, and there’s enough signal, there’s enough, you know, markers and the profile data of the customer, and maybe there’s some kind of activity tracking that their visits on the website, or whatever, there’s enough data to theoretically justify that this lead should be brought to the attention of the human sales person. But in the end of the day, the salespeople in this organization don’t believe those contacts are actually worth their time. So, we spent a bunch of tokens, a bunch of software engineering, bunch of time, opportunity cost, surfacing low-value automated actions that are evidence based, but we’re actually just creating more noise. We’re not actually creating any value.

We just have this tool that’s now surfacing stuff that users actually don’t care about. It’s not making their life better at this current moment. The salesperson doesn’t feel like their life is being improved by the product. Whether it’s technically right or not, it doesn’t matter. The measure of success hasn’t changed.

The measure of success is still, did we make the end-user’s life meaningfully better when we released our product, or this new feature, or this new capability, or this AI-powered whatever it is? Did we make it meaningfully better or not? Where did we start from? Where did we say that we wanted to bring the user to in this journey? Did we reach our goal? When we imagined what a better experience would be, did we achieve that or not?

So, to complicate all this and kind of close the loop on the AI thing, an extra challenge with all this, especially with agents and all of this, is that we can actually make things worse. I think it’s possible to much more easily make things worse because poor decisions made by automated systems and all this, they open up the potential for real messes. Whether it’s creating more work to clean up, or, you know, spending money incorrectly, or creating harm downstream, or whatever it may be, the point is, we’re kind of at a new period here, where in the past, I think in general, with deterministic software, and we do a lot of testing, and we think through all the workflows, and most of those scenarios are all pre seen, the game was, really, does anyone care, or—and the answer might be yes, they’re using it and they love it, yes, they’re kind of meh, or nope, they’re not using it, and those were kind of the only options here. But I think with AI because of the probabilistic nature, there’s also a chance we can make things worse here.

So, that’s just another thing I want to leave you with, and why we need to go talk to users, and why you can’t look at usage analytics for this stuff and really get anything that’s useful as a product leader or a founder. It’s not telling you anything particularly useful about how to change your product strategy or your user experience strategy. And these things are roughly the same thing in my worldview. I tend to look at all this stuff as kind of intertwined. Product strategy obviously goes beyond just user experience, but for all intents and purposes, for this episode, we can think of them as sort of one thing here.

And the point is, are you getting good information to change stuff, or are you only looking at fiscal buyer knowledge? And I understand landing the sale. Sometimes you need to close that deal just to stay in the game tomorrow, so you can work on some of this. I understand all of that stuff. But if you’re trying to build a company that’s going to last, something that’s going to get sold, or something that’s going to IPO one day, you can’t be looking at one sale at a time and just praying that you’ll make it better tomorrow, you’ll make it better tomorrow, you’ll make it better tomorrow.

This kind of pattern, rarely it’s like, okay, now we’re finally here, and now we’re actually going to go and make this experience really great, and now we’re going to go focus on actual value to the end-users. I don’t think it happens that way. I think that kind of sales-driven perspective is just about catch the buyer when you can, get their dollar as fast as you can, and then get the hell out of there and hope that they renew. And I think there’s a better way, if you want to create a product and a company that’s going to be around for a long time, that’s really going to create positive impact. It could be financial impact, but also making people’s lives better, creating other kinds of value, I just think there’s better ways to play the game.

So, let’s just review real quick here. You know, in B2B software, you’re never really just designing for one person. You’ve got this fiscal buyer. You have beneficiaries of the solution, like, imagine, like, in our sales context, this could be the VP of Sales who’s maybe not doing a lot of sales work, but he is responsible for the sales organization. Then we might have end-users, like the actual salespeople in this case, and we have to think about their experiences, and we need a baseline in order to understand what a possible improvement would look like, so we’re not just randomly and occasionally shipping something that does make their life better, but we don’t even really know it unless they write us an email or we run into them in a conference or whatever it may be, we have some random happenstance, like, meetup with them or whatever. It’s all very reactive.

And there’s proactive ways to really understand this stuff, by having the right amount of customer exposure time at the right points, using the right methods to do this kind of work, but frankly, going out and observing, asking good questions, listening, but really watching usage of the product or the status quo products or tools, whatever they’re using now. This stuff can reveal a lot of this stuff quite quickly.

So, I didn’t really get into, like, how you collect some of this stuff. So just, I want to kind of try to make this tactical for you. A lot of this is qualitative research, right? We’ve talked about it’s not looking at an analytics dashboard on your analytics dashboard, [laugh]. It’s not that meta thing, right?

So, conducting observational research and interviews across the full journey, look at the setup process. Look at what the honeymoon period looks like after they’ve gotten on board, or with the POC process. What’s the time-to-value like? What was the last thing they did when they logged—last time they used the tool? When you meet someone, meet a new customer for the first time that’s been using the solution, ask them to do a replay. What are they working on right now? If they’re setting up an agent, how is that experience for them? Are they frustrated? Are they seeing any value right now? Are they kind of just, like, replacing low-level tasks, and it’s like, yeah, that’s kind of nice. I mean, it drafts an email for me, okay? That saves me a little time. Are they actually delighted with any of this work? This is all stuff we can do with observational research and doing customer interviews over time.

You can develop scoring rubrics to actually track these improvements over multiple customers. So, while the computer isn’t necessarily tracking it, you could just develop some simple scoring things through your conversations with them. Or—I’m not a big fan of digital surveys for this stuff. I think surveys can be supporting evidence at scale, but I do, especially for small, mid-sized companies, I think qualitative is the way to go, but you can develop some little scoring rubrics and kind of run those when you have a chance to get in front of a customer and kind of see how you’re doing with this stuff.

Product analytics do play some supporting role in there. So, for example, a spike in usage of you know, let’s say you have that automated campaign tool now for the first time, and there’s the machine created campaigns there, well, if there’s a huge spike in usage there, that’s a signal to go investigate. It’s not an answer to what the experience is, but abnormal usage of anything in your product, whether high or low, this can be a really insightful signal. So, where are they not using stuff that you think is amazing, or where is there a lot of usage in the product about stuff that you were kind of surprised about. You’re like, that’s not even where the good stuff is. The real feature is X-Y-Z. It’s not A, B and C, yet they’re hanging out over in the A-B-C land.

It’s not an answer. It’s not telling you anything about good or bad. It’s just telling you there’s some signal that’s worth investigating. So, the quant signals tell you where to look. The qualitative stuff is telling you people’s lived experience. It tells you about what could possibly be better. We can start to imagine what that would be. Customers will love to tell you what could actually be better. They’ll frequently give you their opinions.

Take that stuff with a grain of salt. I would really focus on their current experience right now, how they’re talking about it, how they’re feeling about it, kind of put it in storage off to the side, any of their solutioning ideas. Sometimes those solutions are good, sometimes they’re not, so just take them as information, not necessarily instructions on what to go make, but it’s still valid to collect that information. And when qualitative improvements do start to appear through your adjustments in your product and design, you can look back for correlated business metric improvements. So, you can then make the case that this product work, this investments in making the product easier to use and making the experience better is actually working.

We can start to correlate, hey, you know, the time to value during the POC used to be really long because we required them to set up, you know, 15 different connectors inside the business and import all their CRM Salesforce data and on and on and on. And what we really found out through our research is that they want to validate this stuff within, like, a day or two, and they’re willing to use sample data to do that. In fact, they’d rather not connect their CRM yet because they don’t even know if they want to trust the data exchange there. They want to use some kind of test data. And all this stuff is stuff that you’re not going to figure out by looking at charts and your Google Analytics tool or whatever that you’re using. It’s not going to tell you any of that kind of stuff.

But over time, after you learn this information about, say, this POC process and how the customer actually wants to evaluate value, let’s say you go out and you make those changes in your product, and you get rid of the Salesforce connector, and you reduce the number of third-party systems that have to be connected before the tool can do any of its amazing magic, and instead, you really limit that world down to a small thing and say, you know, first get this right. If you love this, then we’re going to open the floodgates and give you access to everything. But what you’re doing is, you’re delivering experience that helps them know, is this product right for me or not right for me? And you’re doing that in a reduced amount of time, hopefully you’re making that process delightful and easy to use, and when you do that, you might see those lagging metrics in terms of time in the POC, sales conversion rate, the salesperson’s opinion about the excitement of the buyer or the end-users. Is the salesperson detecting any type of emotional change in how they’re talking about it?

All these things can become the measures and the signals that your product and design investments are actually working. And I tend to believe that, you know, when you get the user experience and customer experience right, when you get from satisfaction level up to the delight level, you end up with a product that’s a lot easier to sell as well because you can demonstrate the value at the end-user’s perspective, probably at the beneficiary’s perspective, too, especially if they’re touching the product and you’ve done this work, and the fiscal buyer, if they know anything about what their team’s doing, and if they’re truly the one writing the check and they’re making the business decision to invest, they’re probably going to be able to relate to what the end-user’s experience is, and understand why their staff is so happy or would be really happy, using a solution like this. Oh, wow. Your tool automatically formats, you know, the sales data for GAAP accounting methods, and it has a connector to Microsoft 365, they don’t have to adjust the columns, and it puts the general ledger codes in automatically. It’s like, I’m not an accountant, but I have seen people do that, and wow, you guys give a lot of thought to that little capability inside the tool, and it’s like a little extra thing, and it only happens once a year, but they notice these kinds of things. And it might be a big deal to someone that’s in the organization.

But the cumulative nature of making all these little touch points better, this is when we create products that are really delightful is that it’s not usually just one killer feature or one killer workflow; it’s the sum of approaching your product development process with this customer experience in mind, about making people’s lives better, and this becomes a form of value of a user experience outcome, a transformation. It used to be like this, and now with your product, my life is like this, and it’s so much better, and I would hate to go back and do things the old way. It’s a lot easier to sell a product if that’s the kind of reactions that your end-users are having.

So, in closing, you know, I know a lot of teams aren’t going to do this work. They ship it, they see if it gets used, and they call it done. So, you have to decide, is that good enough for your long-term business plans? Is that a reliable way to place bets and see what’s going to work? Are you always in reactive mode, or are you potentially in proactive mode?

Because delivery work right now is getting so much easier with all these coding agents. It’s so much easier to ship stuff, but ultimately, good product management design isn’t about shipping the stuff. It’s really about shipping outcomes and transformations, benefits. Are we helping this customer transform from this place of misery to this place of satisfaction or delight? Are we helping them move along in the ways that they care about, solving for problems that they really care about right now?

You want to get out of that reactive posture and get into more of that proactive nature, so you’re not just chasing the latest little, tiny signal that you got from one user and throwing more product at it. Without this kind of qualitative research and customer exposure time, the way I tend to see this in my head, it’s like looking at your customers experience through, like, a tiny straw, and you’re peeking into the straw, and you see at the end, and you could see what’s directly in front of you, but you’re totally blind to the larger context of what their world is like, their job, their tasks, the their incentives, what motivates them, what they’re frustrated about. You have no idea because your viewport is this little half-centimeter straw. And so, you’re never really going to be aware of their frustrations and the latent needs that they have and possible ways that you could improve things there—improve aspects of their life that would make them genuinely love your product. You have to just run into that inspiration as you look through the straw and you have that little viewport. You just kind of have to wait until you run into something and you’re back to that kind of reactive posturing here.

So, if you don’t know what better looks like, you’re probably not going to get there. You might even get there and not even know that you’re getting there because the better has just moved somebody from frustration to delight to—I’m sorry, frustration to satisfaction, but they’re not going to write you an email about that. They’re probably not even going to mention it. It’s just like that was good enough, but that’s not making me tell anybody about it. That’s just, like, table stakes for them.

For you, you might have thought it was a really huge deal. I think until you really understand current state, you can’t really pre-define what a better state is, and really to try to go beyond just what better is, but really push into that delight category where users find your product indispensable. Ultimately, that’s where most of you probably want to be. You don’t want to see attrition, you don’t want them leaving. You don’t want to find out they’re going over to a competitor and all that kind of thing.

So, you got to know current state before you can define what better looks like. Design is a process of helping imagine what different possibilities for better could look like, and it’s never been a more fun time to actually exercise our creativity and to create these new possible solutions because it doesn’t take as long as it used to. But do you actually know if it’s better or not? That’s the real question. There are ways to measure it, it’s just not with product usage analytics. Adoption is not equal to value. The goal is not to build an analytics or an AI product. The goal is to build a valuable product. So, go out there and do that.

And if you feel like all this stuff I just talked about is just a lot to take in, it feels like a lot of work, you’re not really sure, you have to, kind of like, where do I get started with this? It sounds really good, but I don’t have time for that, or this sounds really frustrating. I don’t want to just, like, start this and not know if it’s working, I’m happy to help. You can hop on a free discovery call with me. There’s no charge for this. I’m happy to spend some time with you and do a quick assessment of your situation and suggest possible ways that I might be able to help you, if I think that I can.

My goal is to help you turn technical complexity into commercial clarity to help create data products that sell themselves. So, if you’re trying to move those sales metrics forwards or, you know, get that redesign happening so that adoption goes up and you don’t have, you know that contract that’s coming up for renewal at risk because you know all those seat licenses aren’t being used, if you’re worried about that stuff, feel free to reach out. Of course, there’s tons of resources on my website as well, designingforanalytics.com/resources. Podcast episodes, my frameworks, you can read about the invisible intelligence gap, all that good stuff there. For now, I will leave you with that link. designingforanalytics.com/contact, if you’d like to get in touch. Until next time, thank you for listening to Experiencing Data, and watch out for those technically right, effectively wrong products. Don’t get stuck there. Cheers.