Today I am going to try to answer a fundamental question: how should you actually measure user experience, especially with data products—and tie this to business value? It's easy to get lost in analytics and think we're seeing the whole picture, but I argue that this is far from the truth. Product leaders need to understand the subjective experience of our users—and unfortunately, analytics does not tell us this.

The map is not the territory.

In this episode, I discuss why qualitative data and subjective experience is the data that will most help you make product decisions that will lead you to increased business value. If users aren't getting value from your product(s), and their lives aren’t improving, business value will be extremely difficult to create. So today, I share my thoughts on how to move beyond thinking that analytics is the only way to track UX, and how this helps product leaders uncover opportunities to produce better organizational value.

Ultimately, it’s about creating indispensable solutions and building trust, which is key for any product team looking to make a real impact. Hat tip to UX guru Jared Spool who inspired several of the concepts I share with you today.

Highlights/ Skip to

• Don't target adoption for adoption's sake, because product usage can be a tax or benefit (3:00)
• Why your analytical mind may bias you—and what changes you might have to do this type of product and user research work (7:31)
• How "making the user's life better" translates to organizational value (10:17)
• Using Jared Spool's roller coaster chart to measure your product’s user experience and find your opportunities and successes (13:05)
• How do you measure that you have done a good job with your UX? (17:28)
• Conclusions and final thoughts (21:06)

Quotes from Today’s Episode

• Usage doesn't automatically equal value. Analytics on your analytics is not telling you useful things about user experience or satisfaction. Why? "The map is not the territory." Analytics measure computer metrics, not feelings, and let's face it, users aren't always rational. To truly gauge user value, we need qualitative research - to talk to users - and to hear what their subjective experience is. Want *meaningful* adoption? Talk to and observe your users. That's how you know you are actually making things better. When it’s better for them, the business value will follow. (3:12)
• Make better things—where better is a measurement based on the subjective experience of the user—not analytics. Usable doesn’t mean they will necessarily want it. Sessions and page views don’t tell you how people *feel* about it. (7:39)
• Think about the dreadful tools you and so many have been forced to use: the things that waste your time and don’t let you focus on what’s really important. Ever talked to a data scientist who is sick of doing data prep instead of building models, and wondering, “why am I here? This isn’t what I went to school for.” Ignoring these personal frustrations and feelings and focusing only on your customers’ feature requests, JIRA tickets, stakeholder orders, requirements docs, and backlog items is why many teams end up building technically right, effectively wrong solutions. These end user frustrations are where we find our opportunities to delight—and create products and UXs that matter. To improve their lives, we need to dig into their workflows, identify frustrations, and understand the context around our data product solutions. Product leaders need to fall in love with the problems and the frustrations—these are the magic keys to the value kingdom. However, to do this well, you probably need to be doing less delivery and more discovery. (10:27)
• Imagine a line chart with a Y-axis that is "frustration" at the bottom to "delight" at the top. The X-axis is their user experience, taking place over time. As somebody uses your data product to do their job/task, you can plot their emotional journey. “Get the data, format the data, include the data in a tool, derive some conclusion, challenge the data, share it, make a decision” etc. As a product manager, you probably know what a use-case looks like. Your first job is to plot their existing experience trying/doing that use case with your data product. Where are they frustrated? Where are they delighted? Celebrate your peaks/delighters, and fall in love with the valleys where satisfaction work needs to be done. Connect the dots between these valleys and business value. Address the valleys—especially the ones that impede business value—and you’ll be on your way to “showing the value of your data product.” Analytics on your data product won’t tell you this information; the map is not the territory. (13:22)
• Analytics about your data product are lying to you. They give you the facts about the product, but not about the user. An example? “Time spent” doing a task. How long is too long? 5 minutes? 50? Analytics will tell you precisely how long it took. The problem is, it won’t tell you how long it FELT it took. And guess what? Your customers and users only care about how long it felt it took—vs. their expectation. Sure, at some point, analytics might eventually help—at scale—understand how your data product is doing—but first you have to understand how people FEEL about it. Only then will you know whether 5 minutes, or 50 minutes is telling you anything meaningful about what—if anything—needs to change. (16:17)

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, I want to talk to you about a basic framework for measuring user experience. A lot of people following the show in my mailing list working on data products in particular for enterprise B2B use cases, they typically find my work because they are somewhat problem aware.

They’re aware products are not being used a lot, that they’re working on, or they’re hearing frustrations come from users or stakeholders that their solutions are too difficult to use, or they don’t trust them, the various human reactions to the problems, and so they start to feel like, hey, this isn’t a technical problem. We have a different kind of problem here that has something to do with the human-computer interaction, so to speak. But how do we measure whether we’re doing a good job with design? Because design sounds like it’s this very subjective thing. And for the purposes of this, I’m going to talk about design mostly as a verb, an action that we take in order to yield a particular outcome, which is an experience.

So, people will use the design after we design the solution, and the use of this design becomes the user experience. So, you could say, in some ways, we can sort of shape the user experience, but we can’t really make it. The experience is a subjective thing that our customers and users have when they use our solutions. But how do we know whether or not it’s good? And why do we care it’s good?

Well, first of all, that’s probably a really good place to start. The reason we should care if the design is good is primarily if usage of the product is tied to the business value, or the organizational value that you’re trying to produce. And for most of you, people not using your solutions means business value is definitely not being created. As such, addressing the usage of the product should have, and most likely will have, positive effects on the business value that product people are typically tasked with generating on behalf of their own organization or their clients or their business. Most of the time, this means your product.

So, how do we measure it? Well, that’s what we’re going to get into today. The first thing I want to talk about is that low adoption of data products is a well-known problem, but we don’t want just adoption for adoption’s sake. Because usage of the product does not necessarily mean value has been created. And the reason of this is, think about something like, for most of us who aren’t accountants, doing our taxes.

So, I’m in the middle of getting all my tax data prepared for my accountant, and more usage of my tax, and expense tracking software, and QuickBooks and all this, that is not making my life better. This is a pain I have to go through, and I want it to be as painless as simple. So, if you were to track my usage of my QuickBooks software, that does not tell you whether or not I’m having a great experience. And in fact, I want the usage to be as absolutely low as possible. And so, the value to me is in effectively automating as much as I can, and never having to log into these tools, except when I absolutely have to, which mainly is at the end of the year, when I need to pull data and feed it to my accountant.

So, you all need to stop tracking things like what Google Analytics does, and thinking that that’s telling you something about your user’s experience because it’s actually not. And the best metaphor for this that I know is that the map is not the territory. And I’m sure most of you are familiar with this idea, but there’s a very big difference between walking the streets of New York City or Paris, and looking at a map of Paris or New York City, and feeling like you understand what it’s like to be there. There’s a very big difference. So, the map is not the territory, in the same way the analytics are not really telling us anything about the quality of the experience of our data products.

So, where does that leave us? Well, just to go a little further on this, analytics people, in general—and most of you are data people and analytics people—analytics track computer metrics on things, things that we’ve decided are theoretically worth tracking, but they’re not directly tracking how people feel. And even if they were, we need to be careful about thinking that all of our customers and users are always thinking very rationally and analytically because they’re not. Subjective experience is something that we can test, primarily through qualitative one-on-one interactions with our users. And that’s effectively how we’re going to get signal about whether or not we’re making improvements or not because we do need to get people’s attitudes and feelings out about their experience using the solutions that we’re creating.

And if we don’t, then we’re basically relying on proxies, other types of signals, and hoping that those signals are actually telling us useful information. But most of the time, analytics are not doing that. So, analytics on analytics products is not the way to tell you whether your analytics [laugh] product is working correctly, okay? The map… is not… the territory. And even—there are—for most of you out there, and I’m speaking mostly to a non-design, non-user experience professional audience, there are things like usability testing out there, and other surveys, and things like this, self-reported surveys, et cetera.

It’s not to say there’s no value in running a usability study, which really, if done properly, only tells you one facet, which is, this thing usable or not? It’s not really telling you a lot about whether it’s desirable. It’s not really telling you about the experience somebody had. It’s really just explaining whether or not someone was able to complete a set of tasks that were given to them. So, this is a valid thing to go do to see if your design is on the right track, but it’s really not telling you the full picture of whether or not your experience is good, and therefore whether or not adoption is going to go up or down. Because it’s probably not going to go up unless you’re addressing the user experience problems that you have.

And the only way to know what those are is through talking and observing—talking to and observing—your users. So, if analytics are bad—and I’m saying analytics are bad, you understand what I’m saying at this point in this particular podcast—but if analytics are not giving us a real story about someone’s subjective experience, then how do we get this out of them in conversation and through research? Well, one of the good things about this is that it does take time. You are going to go out and have to actually speak with real people, and you’re going to have the product with them, and you’re going to have to build a habit of doing this on some routine basis because, generally speaking, you can’t just go out and do this one time and then everything’s solved. That’s not really going to tell you’re definitely not going to be able to track improvement because if you only do this once, and you find out where problems are, how will you know if you improve them just because you release something? You won’t.

You may think that you had the problem correct, and you may think that the solution that you’re providing will correct all those issues, but we don’t necessarily know what new things are we going to uncover, now that we’ve addressed this issue. So, this is almost like a never end—like, product, the product is never really done. It’s just on a continuum of constant improvement and change. So, one of the big messages I want to get across to data professionals listening to this show is that we need to really let go of our analytical mind here. Your ability to think about measurements and how we should measure experience, that’s still super valuable. It really is valuable.

However, we need to understand that the feelings people have about has my life been improved by this feature that you gave me or this new version that you gave me, that’s ultimately the thing that we have to track. Did we make this person’s life better or not? And it sounds very touchy feely, and I totally understand that. And my buddy Nick Zervoudis, who I know, many of you probably know who he is, and he’s been on this podcast before, and he’s a member of the Data Product Leadership Community—I love Nick—we had a little back and forth about this on LinkedIn—you can probably find it if you just look at my page and look at my past activity—and we had a really interesting conversation about this. He was showing a schematic that had to do with giving—about the different facets of data products, and how he thought usability needed to have its own carve-out as an area of focus.

And so, I had come to him and said, I think usability is almost too narrow of a lens; we need to be thinking about user experience outcomes, which, again, are this idea of, did we make someone’s life better or not when we made this improvement to the product? So, making somebody’s life better. Well, what does that mean, especially if you’re working on, you know, enterprise products and B2B products? Well, I’m sure you’ve had to use software and applications and tools that you absolutely dread, that you wish you never had to deal with before, and you wish that those things could be better, or you wish that some solution would remove a bunch of unnecessary tooling, and work that you had to do, or maybe you’re spending a lot of busy time prepping data, and you’re like, “Man, I don’t even get to do analysis work anymore. All I’m doing is data engineering work and data cleaning work, and I really want to get to do analysis work because that’s what I went to school for, and I’m really good at that.”

Or you’re a data scientist, and you’re really good at building models, but all you’re doing is this, kind of like, data engineering work and prepping stuff and getting access to the data from the IT team, and blah, blah, blah. Well, right there, if you were building a product for this mythical data scientist I’m talking to you about, you can see right there what their frustrations are. Their frustrations are that they don’t get to spend a lot of time building models. So, if you were an internal team, say you work on a data platform product inside a large organization, or you sell your own data platform product, a commercial product, how would you improve the life of your data science members so that they can spend more time building models and producing the value that the business really hoped to get out of those people that have extensive training and are hard to find. How could you improve their life?

And the question is, we have to look at, well, what’s it like currently to use the product to do data science work? What does their workflow look like? Where are they finding frustrations? Where do they get angry? Where are they annoyed?

And this can be both in usage of your product and as well as the shoulders, the things that come before and after your solution. And it’s really good to always be thinking temporally over time. Because someone doesn’t just sit down and all of a sudden decide to just use your data product one day. There’s a trigger for them to log into your system. Well, what’s the trigger? What came before?

And when they’re quote, “done” using your solution, well, where did they go? And are they bouncing back and forth between two applications? Like, what are they doing in their work? So, by understanding the range of tasks and activities that they’re doing, including your product, which is kind of in the middle of this, by evaluating current state, we can start to learn where their frustrations are. And I heard a really cool thing on one of Jared Spool’s… videos, or so—I can’t remember where I saw it, and he talks about this concept of a roller coaster chart, and it’s a way that he likes to plot user experience on a scale of what he calls frustration to delight, and I think this is a great way to think about it.

So, if you just imagine, like, you have a line chart over time, and the x axis at the high end is delight, and at the very bottom is extreme frustration, and in the middle is somewhere kind of meh or indifferent. And so, as someone goes through your product and starts using it, you could draw a line from frustration to delight, depending on what the activities are that they’re doing. So, let’s say, again, you’re a data platform product, and you’re supposed to help data scientists build more models, and run more experience more quickly. And so, your product gives them access to the data sets, and it lets them explore the data sets online, and understand all the metadata about them, and it lets them refresh them, and request access and on, and on, and on. Well, there’s a bunch of steps there that this fictitious data scientist customer that uses your product needs to do.

And so, by observing them do the activities that your product is supposed to enable, we could start to plot, well, where were they frustrated? In here, when we watched them do this thing, they talked about how easy this was, and how the old way that they used to do it was so cumbersome, and now it’s really great. So, you draw the line more up on the delight side. And basically, whenever there’s a period in the user’s journey using your solution that tends to be on the lower part of this y axis, this roller coaster chart, down in the frustration zone, that tells you where you have problems. Now, mapping those frustrations to business value may get easier.

And it depends on the context, of course, but a very simple metric is just how much time are we wasting? This is a pretty rudimentary one, but if your product is supposed to quote, “save people time,” and save companies labor cost by making things more efficient, then that’s a very easy way to start tying a user experience outcome to a business metric, and to some type of business value or business promise. The same thing could go with an analytics dashboard, right, decision support. How long does it take before someone in procurement that needs to go and buy—like, how much printer pa—I need to sign a contract to buy, I don’t know, like, licenses for all of our Microsoft products and all of our Adobe products and all this other stuff, and we need a projection of what our staffing is going to be, but we also don’t know who doesn’t actually need licenses. So, I need some data in order to make this purchasing decision, which I do one time a year.

Well, you need to know what it’s like for this procurement person to go and do this purchase planning work. How do they want to go about doing that, and does your analytics application, or dashboard, or tool, does it allow them to go and easily do that? And by tracking where they’re spending lots of time—actually, not even where they’re spending lots of time; where they feel frustrated, that’s what’s important is the feeling of frustrations—by addressing that, you’re making this procurement worker’s life happier, and by doing that, we now have a business outcome. Again, the labor and time and cost of the procurement person, plus the changes in the purchase that they might ultimately make for these licenses. Maybe they’ll find out, wow, there’s 1,500 Microsoft Office licenses we don’t need because those are for workers who drive trucks all day, who are not actually using Microsoft Office. We need to stop buying Microsoft Office for every employee in the company because these people spend all day driving a delivery truck around.

Well, how much is 1,500 licenses of Microsoft Office at the enterprise rate worth? Or 15,000, right? So, we have labor changes, we have a happier procurement person working less on this particular thing, and we have the savings that are associated with them making a better decision because they have the right data to do that. This is how we’re tying user experience changes over to business value. But the feelings of the procurement person matter a lot.

It doesn’t really matter how long it takes. It matters how long they felt it take, and this is why counting things—again, getting back to my complaint about analytics as a way to measure user experience as being wrong—counting how many procurement people logged in and how many sessions they had isn’t telling us almost anything. About the only thing it’s telling us is, did somebody use it or not? And if the usage is absolutely zero, well, now we have some signal. And if 1% of people are using it, that’s probably not great, but we might find out the 1% of people using it actually are having a pretty good experience, and that might mean you have a different kind of problem.

You don’t have a design problem. Maybe you have a communications problem. Maybe people can’t find the tool, they can’t access the tool, they don’t have permission for the tool. The tool itself is actually well designed, but only seeing 1% adoption, 1% login, and your analytics tool about your analytics tool, it’s not telling you what the experience is. So, that’s not the place to measure this.

It’s very rudimentary, and I know it’s harder because we want to just look at a dashboard. We want to track some data, especially when we’re thinking about multiple users, it feels like it’s going to take a lot of time. But what’s most important is, are you getting any useful signal such that all of your data product development efforts are actually going to yield an outcome, an improvement in this person’s life, which is the thing that is tied to whether or not the business metrics are going to change in a positive way or not. I have yet to see an example of where—and it’s not to say user experience is the only thing that matters of the product, and if you get that right, everything else takes care of itself. There are viability issues, and there’s the marketing and communication of it, and there’s some organizational change things that sometimes even design can’t fix.

So, there are other facets of product management. However, there’s a lot… I think there’s a whole lot that can be done in the space of B2B data products, analytics tools, and AI products as well, where fixing the user experience problems and the low adoption problems can get you some easy business wins because you’re really focusing on whether or not the solution is linked to what people care about. Are they getting value out of this tool or not? If the answer is no, then the business value is not going to follow.

It will take time. You are going to have some type of repeated habit and process of engaging with end-users, and stakeholders, and understanding what these improvements look like. You need current state, and you need a design vision of future state. You then need to slowly implement the future state, and check your progress. As you begin to implement design changes, which are a hypothesis about what would be an improvement, we validate the hypothesis by doing continuous engagement with our end customers. In the ideal situation, the design remedies that we apply to our data product result in improvements in people’s lives. And if they do, the business value has a lot better chance of improving itself.

So, that’s about it. That’s the main thing I wanted to talk about in this episode. We can’t use analytics to track adoption because it doesn’t really tell us anything about someone’s experience. And experience is what is most tied to business value. Adoption quantity of adoption, or specifically quantity of use, does not tell us whether something is good or bad. It just tells us a number. It tells us a product metric, but a product metric is not a measurement of someone’s subjective experience.

The teams and companies that decide to focus on what someone’s experience is are the ones that are going to build solutions that people find indispensable, and they can’t live without, and they’re the ones that people are going to pay more for. And this, in turn, to you, dear product team leader, this is what builds trust that you know what you’re doing, so that when you want to do something that’s a little different, you want to challenge the status quo with your stakeholders, if you have them, you’ll be able to have more backup to do that. You’ll have created the trust to do that because you’ll have results that you can show for it. You can say, “This is how we work. This is how we create business value. By tracking how people are doing using the product with a strong framework for measurement, we can show business results. Let us do this again. We’re going to try it this way. We have a new idea.”

And someone asked me yesterday, what do I think about improving SaaS analytics tools? Should we improve a SaaS analytics application user experience, or should we let agents do this? And to me, the framing of that is wrong. What is an improvement in the target customer’s life? What does that look like, and does or does not an agent help do that?

If offloading the work that the human would normally do doesn’t create more taxes and new problems that are worse than the current challenges the customer has, then an agent might be the right implementation method. But you can’t just know whether a feature or product is better by looking at the product metrics. We have to go back to what was the outcome? What was the change that happened? What is the change that we want? That will answer the question whether or not AI is the right implementation method or not.

Anyhow, short episode today. That’s all I wanted to get across to you. I hope this helps you user experience outcomes by having this model of how you’re going to create metrics and measure your data product. This is a different way to think about it, but I think it’s the right way to think about it, and I think you’re going to find out you’re able to create more business value if you get this piece right.

This is hard to do by yourself, if you don’t have resources because it takes time. You have to spend time with customers, you have to synthesize the research. Someone has to then just propose and design what the improvement to the product looks like, and then you have to build the improvement, and then you have to repeat the cycle. And this is what good user experience research and design teams do all day. So, if you’re on your own and you don’t have design resources, that’s okay. I don’t know a better way to do this work, so either you have to hire people or you have to find time to do this.

Because if you’re not, then you’re just basically putting out guesses and hoping that they work. And at some point, that tends to be an expensive and slower way to work, and so you have to decide whether or not you can just build, hope, build, hope, build, and hope is a reliable way to innovate and to create more value. Maybe it is for you, I don’t know, but the tools to do this have actually gotten easier. There’s easier tools for doing prototyping. There’s great tools now for doing a synthesis and analysis of qualitative research. So, AI is actually helping us build better AI products.

So, some of the labor here has actually gotten easier, even if you don’t have, you know, the resources of really mature software product teams in the user experience space. So, that’s all I got. If you need help with this, you’re always welcome to reach out to me, designingforanalytics.com. There’s a ‘Contact Me’ button. You’re welcome to set up a free discovery call if you’d like to talk through some of these challenges with your own product.

And there’s, of course, more resources on my website, including this podcast and my weekly insights mailing list, which you can get on at designingforanalytics.com/podcast. Until then, take care, and remember, shut off your analytics. Stop looking at analytics on your analytics and data products. [whispering] It’s not helping.