In this episode, I unpack essential UX principles for data product managers working in analytics and AI. Since many data teams lack dedicated design and research resources, product managers often need to step into this role. This episode gives you practical fundamentals to guide your work.

Highlights/ Skip to:

• The kind of customer and stakeholder exposure you need to embrace (4:03)
• The importance of stating visible and measurable outcomes for users, not just the biz/org (9:02)
• Internalizing design as a method for refining problems, not just solutions (15:03) 
• The role of management in automated systems (27:03)
• Prototyping mindset   (30:59)
• Conceiving behavior change as a data product “requirement” (37:24)
• Ethical considerations in AI and data products, ”what gives you the right”? (44:46)
• Design as a team sport  rooted in serving you by serving others (47:47)

Quotes from Today’s Episode

“Analytics and AI solutions often fail, not because the math is wrong, but because the workflows don't match how people actually do their jobs. No matter how advanced the tech is, success depends on alignment with real-world processes. Technical accuracy alone won't drive your user adoption”

"Outcomes must be defined, measurable, and visible, because user experience outcomes are often the stage-gate to business outcomes. UX isn't a 'nice-to-have'; it may directly determine whether business goals can be met. Tracking and revising your desired outcomes as you build ensures progress stays tied to real impact."

"Well-defined problem spaces unlock value faster. Requirements are often friendly lies that mask incomplete or misunderstood needs. By using design to uncover the real problems, teams avoid wasted effort and build solutions that address what actually matters to people."

“A trusted, good-enough model will always beat a perfect but untrustworthy one, because trust drives adoption. Users generally care more about explainability and reliability than technical perfection. Exceptions exist, but a model that's 90% accurate but understandable will outperform a flawless black box model much of the time."

"User and stakeholder research doesn't have to be perfect, it just has to be routine. Lightweight is okay; the habit is what matters. Regular, small check-ins build stronger products than waiting for large-scale studies. Normalizing research as part of the design process beats seeking the 'perfect' research moment."

Links

• My flagship instructor-led training seminar for data teams trying to adopt product and UX-design methods in their data product work:  Designing Human-Centered Data Products 
• New in 2025: join my Cross-Company Group Coaching program for data and AI product management professionals

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today I want to share—I’ve been thinking about foundations and user experience design as it relates—you know, this is, what, September 2025—to advanced analytics, AI work. Many of you listening, obviously, are building data products—and I include AI solutions in that definition—and, you know, I’ve got a course that I’ve written about designing human-centered data products, and I’m launching this new cross-company group coaching service as well, so I’ve just been thinking a lot about education in general recently.

And I realized, I think it’s been a while, I can’t even remember if I’ve ever done an episode that was really about foundational user experience principles for data and AI product managers. So, that’s literally what the title of this episode is today: “Foundational UX Principles for Data and AI Product Managers.” These principles, many of these would apply to formally trained designers and user experience professionals as well. I really tried to take a focus here on what I think the most important things are for non-designers in particular, especially because I think with the advent of AI and AI helping us with product development work and delivery work in particular, we’re going to have more generalist-type solution creators, product teams that have wide skill sets, that are doing design research, maybe even some lo-fi coding or vibe coding, or even some production-level coding, hopefully not for many of you. You probably shouldn’t be doing that too much, unless you’re at a startup. But even then, the point here is the skill sets are kind of blending even more.

And I’ve always been a designer who tends to think that the circle between product management and product design or user experience design, these are heavily overlapping circles. And some people would disagree with this, and they would say how design is a completely separate thing and product people should stay in their lane and vice versa, but I just don’t tend to see it that way. So, let’s jump in here. This is for those of you that do want to ramp up your user experience design skill set because ultimately you are the one doing that work. Perhaps you have some fairly young talent on your team that’s right out of school or something like this, or maybe you have no design talent or resources to work with, whether it’s contractors, employees, or whatever it may be.

You might be doing all this work yourself because maybe you’re the only one that even sees that human adoption, delight, usability, simplicity, utility, that all of these qualities are actually what’s so often holding up our analytics and AI solutions from creating the value and impact that everyone’s hoping that they’re going to have. But sometimes management doesn’t see that, and design sounds like some kind of decoration skill that you bring in to make things look pretty. And I really hope for most of you listening that you’re well beyond that perspective. Maybe you’re new to the show, though, and you haven’t been listening to Experiencing Data for the last nine years, or however long we’ve been going here, but this one’s for you. So, let’s jump in.

I’ve got about seven key ideas that I want to get across here in September of 2025 as I think about this. So, foundational UX principles for data and AI product managers. The first one is that user and stakeholder research is a must. It must be routine, but it’s okay to do it lightweight. The professional researchers out there will probably cringe at this, and that’s okay.

I would rather see you get to the gym, start doing some workout, build the habit. Just get in there and start doing some work and don’t worry about doing the perfect workout because just getting in there and getting moving is creating some value for you and you’re building the habit and you’re also beginning to learn what you don’t know about how important that work is. So again, let me restate number one: user and stakeholder research: lightweight is okay. But it needs to be routine.

Why this matters? Analytics and AI solutions often fail not because the math is wrong, but because the workflows don’t match how people actually do their jobs. So, we’re trying to design these solutions, we’re focused on governance and data quality and pipelines and all this technical stuff, but we’re often rarely spending the time to see how this solution is going to fit into the natural workflow or the jobs to be done of the people who are supposed to use it. And without this exposure, without our ears turned on and our eyeballs observing, it’s really hard to get this right on a regular basis. And you might guess your way there correctly, but guessing is not really a reliable strategy here.

So, you know, some tactical things here: try to commit to even you know, one 30-minute qualitative research session—what I mean by that is, you’re not doing a survey, you’re having an open-ended conversation or observation session, watching one of your users or a prospect or an employee from a team that you’re about to serve with some kind of work, if you’re doing internal work, watching them do their job when they’re running the books, if they’re in finance and they’re doing the books, ask if you can observe them for a half an hour or an hour once a week, one person, and just see the tools they’re using and how they make decisions and ask them questions. Ask them what slows them down? Ask them what would save them time. Ask them to tell you a story about the last time they got really frustrated with one of their data products or data-related solutions that they’re using. The importance of having technical users participate in this work, I can’t underline that enough here. What I mean is, if you’re in product management, you’re probably in control of or have a lot to say about the engineering and data science and data engineering work that needs to get done to build the product.

These people need to be part of this work. You need to get their eyeballs on these end-users so that they can start to develop some empathy there and they know that there is real human beings at the end of the solution that are going to actually use this model or this dashboard or whatever the application that you’re building is. That empathy will help them start to think differently. So, you’re going to want to rotate these people in. So, maybe every week, you pull a different person in, and you do a two-on-one session, you ask the question you have them come and observe.

I like recordings, but most people aren’t going to spend the time to watch it, and it’s just kind of like watching a recording of a concert; it’s not the same thing as seeing the Grateful Dead live. The tapes are great—if you like that kind of music, at least; I’m actually not a Deadhead—but it doesn’t replace actually going to see the live experience, even if the recording is really high quality, so trying to get them involved on a regular basis. Capture your insights in a really lightweight artifact. You can record them. We have all this AI technology that can summarize key findings for us and make them searchable, and things like that.

I do recommend taking some notes. One common tactic is sticky notes: one idea for sticky note. You give everyone that’s coming with you a stack of those and a pen, and it helps them keep the ideas really short and the observations there. And in short, treat this UX research as an operating cost, an ongoing cost of doing the work that you do, and think of it as insurance for your team. It is insurance to ensure that the data team, the data science and analytics team, or the data product team, is not just out there sinking more and more cost into technology and not actually delivering any value. This is kind of your insurance against low use or no use solutions. Okay?

Number two, outcomes and projects and products must be defined, measurable, and visible. Revise them as you build and know that the user experience outcomes are often a stage-gate to business outcomes. So, why does this matter? Well, let’s say an AI-powered product, you know, optimizes KPIs, but you know, it frustrates your end-users, right? The one that delights users but doesn’t move business metrics won’t get funded, but the one that doesn’t do anything to help users want to use it or even enable them to use it, this isn’t going to create any value for the business anyways, so all you’re doing is setting a clock, running the clock down, and spending time and money building a solution that nobody wants.

So, both the business outcomes and the user out the user experience outcomes need to be defined up front, or I should say they’re defined up front, and they’re revised as you build because as you’re going to learn a second, we’re actually designing to clarify the problem, especially early on, so design isn’t just about solutioning, it’s also about problem definition. So, let me get a little bit clearer here because this—I’m still thinking about the best way to explain some of this to you, but the churn prediction model, for example, is only successful if your sales reps actually change how they prioritize doing outreach, right? So, in this example, we don’t get any of the business value of the model until sales reps are willing to change how they do their work, which is influenced by the model, and when they change how they do their work, only then, might we actually reduce customer churn. So, embedded in all that is this behavior change of the sales reps. So, we need to understand, well, what does it mean to deliver an outcome, a change that we can observe for the lives of the sales reps? That is a stage-gate to getting the business value, which is the churn reduction.

So, these outcomes should be written in plain English, or whatever language you speak at work, and they need to be kept visible during your product work. Outcomes need to have a from and a to component, so there is a delta. There is typically going to be—not always—but there’s typically going to be a change, an observable change that we should be able to measure. And if you can observe it, you can measure it. I don’t want to get too far into this, but I have a whole episode on how to measure anything. It’s one of my favorite episodes. But if you can observe it, you can measure it. It just gets into how accurately you can measure it. So, you can climb through my old—I don’t have the episode number right now about that, but I highly recommend you taking a look at that if you’re getting stuck on how to measure some of these things.

Also on this topic of outcomes, is this idea of user adoption. User adoption is somewhat of a rookie UX objective, counting things like sessions and logins and how many users did something with the application or tool that you’re building, but adoption is not actually an outcome. An outcome is actually downstream of user adoption. And the reason it’s downstream is that adoption, as tracked through product analytics or your solutions own analytics such as logins and time spent using the tool, adoption might actually signal that your solution is a tax and not a benefit, and this is why we can’t just look at product usage metrics to tell us whether or not we’re designing a useful solution that’s actually creating a benefit or outcome for our users. So, we have to actually have direct exposure to the humans in the loop that are using these solutions.

A log of these wins over time is really powerful evidence that your team is having impact, so the act of defining these outcomes, making them measurable and making them visible, and then actually going back and looking at the results and spending some time to evaluate did we actually create the impact here, keep a log of these things. This is really good for the team, it’s good for your career, and it shows that you are being outcome-minded simply in the fact that you’re actually tracking some of this stuff. So, that’s kind of my spiel on outcomes. Again, I’m going to restate that one: outcomes must be defined, measurable, and visible. Revise them as you build and know that the user experience outcome is often a stage-gate to the business outcome that you seek.

Because no use or low use may be the gate to the business getting the value that it seeks to get and most of the time, business stakeholders, most customers, are not thinking about user experience as a requirement. They’re not at all thinking about that; they’re just thinking about the downstream outcome that they want, not the work that goes into it. Your job is to intercept that and understand that the humans-in-the-loop have a lot to do with the organization getting what it wants.

Okay, number three, design to refine the problem, not just the solution. Well-defined problem spaces unlock value faster. So, why does this matter? Well, a lot of times, analytics and AI projects will start with a request for data or the ability to interact with data that is really a solution that’s masked as a problem. So, reframing this, for example, by asking, well, what decisions do you need to make faster, and where do you lack confidence, and how do you know when you have enough confidence to make a better decision? At what scale do you need to make these kinds of decisions? How would you know whether or not it was easier to make this decision with a new solution versus the way you do it now? These are all getting into feelings and changes that we can observe, and we talked about that earlier.

You want to translate vague requests such as, “Build an AI chatbot,” into problems such as, “We want to reduce the tier one support tickets by 40% and we want to validate that our CSRs’, our customer service representatives’, job frustration has reduced within four weeks of releasing this data product or this feature or this sprint.” And we would do that perhaps with qualitative interviews with the CSRs to see how are they generally feeling now that you have this AI bot that’s intercepting some of these low-level support tickets. In turn, this might help the business then realize the benefit of retained staff, and that might be part of what they want is to reduce this job frustration because they don’t want to have to hire new staff and retrain them. Another example here: leadership might say, hey, let’s use AI to predict which customers will buy more. Well, you might think, okay, let’s go figure out what made people buy in the past, build a model around that, and then, you know, we’ll be off to the races with a data science project.

If you actually go through a design-led discovery session here, what I sometimes call a design jam, we might discover that by doing this work with sales reps in the room, the people that would actually perhaps use this predictive algorithm or predictive score, we might find out that they don’t really need a generic score, like an index, a number. What they might really want is specific reasons why a flagged customer might expand their purchase. So, things like, you know, their usage has been higher than their license limits for, like, three months and they’re paying extra fees. Or, you know, we’ve seen that accounts like this one have adopted the gold-tier package instead of the silver tier package, and so you may want to call this customer because accounts like them have adopted the new integration or the new feature, whatever it may be. And it’s this reasoning, it’s the explanation behind it that makes it powerful enough to get the sales reps to actually change their behavior.

And when we talk about designing to refine the problem and not just the solution, part of this is we do this because we accept that requirements are often friendly lies. So, refining the problem as we design might get you and your team to—might get you to the point where you have an AI model that focuses on explainability and action-taking over data science accuracy, mathematical accuracy. You might end up with a solution that is built with existing tools, inside of existing tools that users already like and live in. So, in the sales example, maybe instead of a new dashboard or a new application or a stand—you know, whatever it may be, you integrate this right in the CRM that they’re living and breathing in all day long, and we don’t create some friction there by introducing yet another dashboard or another solution that they need to go and get a login to and blah, blah, blah; we’re embedding that right into the place that work is already happening. So, we’re asking them to make the least amount of change possible.

And we might, as we design to refine the problem, what we might end up designing is a product that’s smaller. We might find that there’s actually a smaller set of technical requirements because in the original requirements we had, the customer was speccing out this solution and all the data they needed. I need all these columns, and I needed to be able to do X, Y and Z with it, et cetera, et cetera. And that’s because they were kind of imagining a solution in their head. But customers aren’t designers.

And while you may be like, “Well, I’m not a designer either,” if you’re listening to this podcast and you’re trying to actually take some action with these insights, then you’re on the way there, and you’re beginning to do the work of design. And that’s what matters, is we’re starting to think a little bit omnisciently about this, we’re designing with intention, instead of byproduct design, which is, well, whatever the engineers and the data people made, whatever kind of comes out at the end of the project, that’s the design. I call that byproduct design, and it’s designed without any type of intention. So, the fact that you’re having any intention just by listening to this, you’re on the right track here. And yeah, I’m going to come back from my little soap box there. Trust me, I’m off it. I just stepped off, and I did sprain my ankle, but it’s getting better already.

Okay. And the last thing I want to say on this number three here is, test your problem framing with both your business sponsors and your users. Misalignment, really, it breaks trust with stakeholders. Nobody wants projects and products that are going off the rails and they’re, you know, one year, two years behind, all this kind of stuff. So, keep in check with the problem space and make sure you’re constantly talking about that with the product team and up to the business and/or stakeholders.

Also remember that as we design to refine the problem and not just the solution, the problem space is probably going to change and you may find out that while we set out to solve problem X, what we’re really realizing is that this data product we’re trying to build, it really needs to solve problem Y. And that might have not have been stated up front, and it may be that Y is greater than X, it’s a more important problem and it might be just stated in a framing that no one had ever really thought about before. And that’s normal, so accept the fact that as we do our design work, we are refining the problem and not just focusing on making the solution go from rough to fine, and okay to great. You’re really living in this problem space early on, before you really get into solution refinement. Yeah, so the misalignment kills projects later. Problem space does change as we build.

And then finally, just remember, you know, new solutions will create new problems. And I think this is really true with AI, especially with the language model type implementations here, we may find out that, like, you know, if you start automating things or using agents to do, you know, quote, busy work that humans used to do, we may find that this starts to introduce new types of problems, new work that never used to have to be done because a human being used to do all of it, but now they’re only going to do 10% of it. But it’s not really just 10%; it’s 10% of the old job, and now there’s an extra 15% which is managing the agent that’s doing this automated work. That’s a very agentic AI specific example, but I’m using that because if you imagine throwing an agentic data product into the workflows of your end-users, you could imagine how this thing isn’t really fully automated with no oversight, it lives in code only, and if it goes off the rails, no one can do anything about it except to deploy new code. Chances are that’s not how that agent works. It’s going to live within a deterministic application, and there’s going to need to be control points.

And I have a whole episode about my MIRR—M-I-R-R—UX, framework for agentic applications. It’s a few episodes back. Because of all of this, you are introducing all of that oversight now over machines doing work for people. So, a new problem space has now been created that needs to be designed for. So, that’s normal and it should be expected, and the fact that you’re actually discovering these things, I think is a good thing because we’re just getting more awareness of the world around us, and particularly the world around the customers and the people that we’re trying to serve. Okay?

Number four, build trust as a first-class feature, especially with AI solutions. Why does this matter? Well, AI adoption really hinges on trust and not just accuracy. So, users need transparency, control, reassurance, et cetera. I think as of this time in 2025, I think your default approach as the user experience person on a data product team, your default framing here should probably be that there’s going to be some excitement, but probably skeptic—excitement might be more from the business imagining all these amazing returns on value they’re going to get.

From end-users, you probably should assume some skepticism there, possibly even fear. And part of the work of doing good design work is to address these fears and the skepticism so that people aren’t reluctant to use these solutions, particularly if you’re quote, “Taking work away from somebody,” like we just talked about these automation just a second ago. Taking work away from someone doesn’t necessarily mean you’re doing them a favor if they feel like their entire job and career is attached to that work. So, you need to think of trust as a first-class feature, something that needs deliberate attention. It’s not something that we write with code, but it is a quality attribute of the system and the data product that we’re creating.

So again, how do we do this? Again, my MIRR UX framework—I just mentioned that a second ago—that has some ideas, particularly for agentic AI applications, so I’ll just state that’s not a broad framework, but it is relevant in the AI space. The reminder that explainability often trumps accuracy when it comes to increasing trust, giving users a way out of solutions, particularly when they’re involving AI, so making them feel like they’re in control, that they can override automations, that they can flag problems there, they have a way to audit certain information there. They want to feel like they’re in the command seat, and they should be. And my broad perspective here is really you should be resisting automation unless it’s work that everyone is really excited to automate.

And I mean ‘everyone’ means that includes the end-users whose work that you’re actually taking away and that we’ve had intentional product management and design effort given to managing this automation as a product. So again, this gets back into this idea that even if you deploy agents or an automated system, there’s still probably going to be a management interface or application that is governing these solutions. There has to be some transparency into what is going on, what these agents are doing here. And so, if there’s deliberate attention given to designing that system around the people who are then managing the solutions that are now partially or fully automated, that takes a lot of work, and I really think that you should be trying to resist automation or using it sparingly, and double-checking that you’re actually getting to the benefit and the outcomes that those end-users seek, and the business is getting the downstream value that it wants. Because you can do damage here, just in the form of sunk cost because people don’t use it. You could also create resistance if you’re automating the wrong things, and people feel like they’re being like—particularly internal customers—feel like they’re being threatened.

I’m going to talk a little bit more about the damage stuff here that can happen in just a second, but really, this trust factor is something we want to be measuring over time through interviews and observation. I really think qualitative work here, qualitative research work is really important here. I would rely less on things like surveys and self-reported data where you’re not there to actually hear their feelings and attitudes and the way they’re talking about things. So, I wouldn’t use surveys for this. I really think qualitative is probably the better way to go, at least until you know what the right signals are, such that you could, at scale, measure trust, but you really need to start qualitatively first. And just remember that a trusted, good enough model—particularly when you’re talking about predictive analytics and AI—good enough probably beats an untrustworthy, perfect model.

So, more and more technical precision, technical work to make models more precise, if you’re only playing the technical quality game, you’re not necessarily building trust by simply increasing the quality of the technology that’s underpinning all these solutions that we’re building. Trust is a feeling, and it involves direct intervention and interaction with human beings. Sorry if you’re one of the people that don’t like to do that work, you’re probably listening to the wrong show, or at least if you are listening, I’m hoping you’re starting to realize, if you’re not ready to make changes to how you work and realize that people don’t really care about that the tech, they just care about the benefits from the tech that you’re building, I mean, that’s really a cornerstone of this whole show. So, I hope you’ll get something out of that and a little light bulb will go on for you, and you’ll start to realize that they don’t really care about your technical chops. Only your peers probably care about that, and frankly, they probably don’t really care either. Everyone just cares about what’s in it for them, so if you’re playing that game, you’re probably going to get ahead.

Anyhow, number five. For AI, think about a portfolio of rough products and have a prototyping mindset, with users in the loop from creation all the way to actual use. I’m going to say that again. For AI in particular, think about having a portfolio of rough products and keeping a prototyping mindset with users in the loop from creation to use. So, why does this matter? Well, enterprise work in particular, just B2B in general, analytics and AI workflow, these are complex things, and if you wait until you’re already doing all the data science and engineering work, you’ve already sunk major cost.

Research, user and stakeholder research, design work and implementation, these are all activities that we need to do with users and not for them. And I mean, we’re doing this work with them while we’re creating it, and we’re involving them in the evaluation or sometimes called acceptance testing, UAT. I don’t really like that framing, but the point is, the users and customers are involved from start to finish. There’s no big, giant reveal at the very end. If you want to do that kind of work, you’re taking a lot of risk.

Companies like Apple that do very secretive types of design work and all this, the difference between you and them is that most of you listening to the show, your company doesn’t value design work at all. Now, you are a thousand times different than a company like Apple that puts a premium on design. That is most likely not the organization you are working in, and so you need to design a lot more on fact and a lot less on skills and opinions and feelings of experts because frankly, you don’t have those. Very few companies have those. But let’s get back to this idea of a portfolio of rough AI products and a prototyping mindset, right?

Sketch out your workflows on paper or whiteboards before getting into algorithms, and have users involved in these sessions. And I like workflows in particular because they get us thinking about experiences, and experiences happen over time. User interfaces are kind of like an instance in time, and so while UI design does matter, the UX professional is really thinking about experiences that occur over time. And workflows are tasks that one thing follows another thing, follows another thing, follows another thing. Getting into the headspace of your users’ tasks and jobs and work that they do, this is really an easy way to unlock value and to make sure that the solution you’re building is going to create the value that you intended it.

Host short design jams. Get your engineers, your technical people and the users in that same room. You don’t need fancy software to do this kind of work, but we’re also, by involving our technical implementation people there, particularly the senior ones, your architects and the principal solutions engineers, whoever the really senior people are, it’s really valuable to get them to have a non-technical working session, to really think about user experience and to think about customers, tasks, and jobs, and ask them to keep their implementation hat outside the door, keep that in the little you know, like you have to, you know, in the cold—what do they call those special rooms on the spy shows, where you have to check your phone in, you know, outside the room, you’re not allowed to take your cell phone in—I want you to, instead of taking your cell—well, yeah. Put the cell phone in the little cubby hole as well because that doesn’t belong in a design jam, but you can also take off your implementation hat and ask everyone else to take theirs off and check those at the door.

And emphasize validation of these workflows. “Hey, dear user, does this actually fit how you work today, or when we build this new, you know, AI solution, is this going to change the way you all like to do your work right now? Like, what kinds of problems might we create if we give you this solution that we know that you asked for? But we’re curious, what might be different here? How is this fitting into what your work and your job is today?”

We want to iterate quickly. We want to fail in shorter periods, and I think if we have a prototyping mindset and a portfolio of products, we accept that, first of all, not all the products are going to create value and win, and that’s okay. If only a portion of them, even a fraction of them, create some value, that can be a really good thing. And it could be very significant value, even if the cost above the other things was pretty great. The prototyping mindset also helps us iterate quickly, get things out the door before they’re perfect, and focus on learning and not just value creation because part of what we’re doing here when we’re designing these solutions is learning. We’re trying to figure out what’s working and not.

And when we learn why something is not working, particularly if we’re focused on a set of customers or end-users that are consistent and don’t change over time—like, let’s say that you work inside a large enterprise, you’re regularly serving the marketing team or a sales team or finance team, over time, you’re building a knowledge about those audiences that you serve. And when we iterate quickly here, we’re increasing the speed of these feedback loops. And the more of this knowledge that we can have about others, the easier it is for us to actually start to anticipate needs and to move from simply providing satisfactory experience to actually delighting people. And when we start to delight people, this is where larger value unlocks can actually happen in terms of business goals and business value metrics, financial metrics, things like this. So, that’s the last one on number five.

Number six—I got two more here—behavior change is hard. I want you to consider this—like trust—to be a product requirement, and I want you to actively design for it. Why does this matter? Well, if you’ve listened to this show, you know, solutions can easily—data products can easily become technically right and effectively wrong if we don’t think about behavior change. And particularly if we’re doing enterprise data product work and you’re serving internal customers, what is the status quo? What is the culture around here? How do things get done? How do people do their job? Why are things the way they are?

We need to understand all of that in order to design for behavior change. So, it’s a requirement. Make it a requirement and make sure the technical team understands it’s also a requirement, not just the SLAs, not just the model requirements, not just the streaming data has to come in this fast, and error rates need to be like this, da-da-da-da. No one wants the—scary thing about the behavior change is, your technical team is not going to know how to directly address that. And that’s okay, but as a product leader, you really need to hold up that might be the stage-gate to getting to the business value, so it needs to be considered as a requirement.

You can ask if this works, if this data product works, that we’re building, what will we observe that is different inside the user or the organization, and by when? Like, what would we be able to see? And this gets back to those outcomes that we were talking about. We should be able to observe this behavior change if it’s happening. And if we can observe it, we can measure it.

Quantitative measures stand out. I know we talk about behavior change and metrics and measurability, we tend to think about numbers and all this kind of stuff, but qualitative results and stories are super memorable and they can be really powerful at persuading people like our executive stakeholders to understand what kind of impact is the data team had. And, you know, delighting just a few key users early on, that might be enough to justify increased investment in your product vision, or letting your team run with the direction it wants to go, even if it’s not what was quote, “Ordered” by the end customer or quote, “Ordered” by the business. If they can see that benefits are starting to occur, observable, desirable benefits and outcomes are emerging, you’re buying some trust there. And the stories that these end-users tell you—or even your stakeholders—these stories about their feelings and how much easier their job is to do, and how I used to hate doing this, and now it’s really easy, or it’s like I feel so empowered to make better decisions, and on and on, don’t underestimate the power of these stories, particularly as it relates to communicating up to your management and to your senior stakeholders.

So, there might be 50 salespeople, but if you can get the three directors to become your cheering squad, and they’re excited and they’re doing their own watercooler chats at the upper management meetings and stuff like this, do not underestimate the value of that, even if the actual sales metrics are hard to pin down. Maybe it’s hard to say, well, churn went down and, you know, by, you know, 10% which equals, you know, $50 million a year, and it’s directly attributable to this analytics metric and all of that, sometimes that’s going to be really hard, but the fact that you have three excited sales directors that might give the business leadership all the confidence it needs to know that you’re creating desirable benefits with whatever it is that AI and analytics stuff that we don’t know how all that stuff works and where all the math people sit, go ahead, keep doing that stuff. We don’t know exactly what it is, but we can see that you’re creating value. And that then, in turn, lets you have more autonomy, and it lets you begin to anticipate the right kinds of projects and data product work to do, and on and on. It snowballs.

Number seven. Actually, I lied. There’s two more. There’s actually eight in total, so here’s two more here. When you’re tackling UX design yourself, I want your minimum goal of your work to be don’t create harm and do increase usability and utility. However, as I hinted earlier, the big value unlocks come when we move from satisfaction to delight. I’m going to say this one again. When you’re tackling the UX design work yourself as a data and AI product manager, your minimum goal is no harm and increasing usability and utility. However, the big value unlocks, the big financial impacts, the stuff the business is really going to like, that’s more likely to come when you’re moving from basic satisfaction—usability, utility, that kind of thing—up to delight.

And yes, this is hard work to do. It’s even hard for trained designers, but that’s where the big unlocks tend to be. User experience is a measurable thing. We can measure how frustrated or satisfied or delighted our end-users are. This can be measured, and it does need to be measured so that you can track how you’re doing. Why does any of this matter? Well, complexity feels impressive, especially when we’re doing AI work, but it can bury the value. And simplicity, this is what accelerates adoption in analytics and AI is when people feel like the work that these data products are actually serving them.

So, it’s not about you and all the amazing technical work that went in under the covers. The end-users are not going to celebrate that work and they’re not going to care about it, frankly. They just care about what’s in it for them. And if it’s not simple, then there’s not a lot in it for them. Simple is hard, but simple unlocks adoption, and adoption is the start of the behavior change and the value unlock that we’re trying to get.

As you mature your design prowess, you’re going to know when a data product really needs to focus on delight and not just satisfaction, and there’s going to be other times where simply getting to basic usability and utility, that might be enough, and it’s time to move on because not every project is—particularly for internal data products—you’re not going to have the time and budget to do all of that, and the value, the ROI, on moving from satisfaction to delight, it might not be there. It might be good enough to just get rid of the blockers and make something really simple and inviting to use. I did want to call out here this idea of no harm, and this is particularly relevant with the AI thing. So, you know, we can harm with breaching privacy, whether it’s we promised not to share certain kinds of information, and that information got out anyways, or users might have had an expectation of a certain type of privacy, and that privacy was breached, even though there wasn’t a commitment necessarily from the product or the organization to keep that information private. Both of those things are real, whether we like it or not, and you can say, well, that’s what our policy—for the last one, you could say, well, that’s… we didn’t commit to that.

All I’m saying is the harm can still happen, so you have to decide whether or not that type of privacy harm is something that the business can afford to do. Harm can come from poor automation decisions and lack of oversight. Harm can come from a lack of empathy as to how our data products might introduce negative externalities inside of our org or even with our customers. So, you know, think the classic example here is like the, you know, getting a loan approval for a mortgage, for example, and if, you know, the data was based on biased, you know, racist decision-making in the past, well it’s going to generate, if we don’t correct for that ethically, the model is going to be biased in the results that it gives. And so, there are lots of different ways to create harm, particularly with AI, so you know, data product teams really need to also be asking, should we be building this? Should we be doing this work? What or who gives us the right to do this kind of work?

Yes, there will be times where you don’t need to ask those big questions, and they some of these things may feel really giant, but if you think about things like healthcare, should we be doing this? What gives us the right here? There are some really difficult incentives in certain types of industries. Again, US healthcare is a great example. For-profit companies that are also saying, we’re for patient outcomes as well, but when our billing system is based on the number of procedures that we do, we’re charging for effort and not for results, we’re charging for procedures and tests and activities performed and not necessarily health outcomes, then there’s going to be times where these questions do need to be asked because the interests are multiple, and the interests are sometimes at odds with each other, and I think data is a big part of this going forward. And so, I’m getting really heavy, I’m going to come back down to ten-foot, hundred-foot altitude, I don’t know, and get out of the ten-thousand foot space.

Number eight design is a team sport rooted in serving. Serving end-users, stakeholders, and ultimately paying customers. And I’m stating that because I know a lot of you are working on internal data products where you may be a fair number of hops away from actual paying customers. Why does any of this matter? Well, I can tell you that your executives, your most senior business people, the ones that have P&Ls, the ones that are responsible for results and not just building out tech and saying, “Hey, we shipped it on time. Whether it worked, not our problem,” someone’s responsible for that when you go high enough up. Maybe that’s you.

I hope and I think most executives will value teams that are ultimately focused on downstream customers’ experiences and the benefits that we’re providing to those paying customers. So, when you can start designing for and thinking about how your data product work is affecting those possibly downstream paying customers because let’s say, “Well, I’m really serving the sales team right now, but ultimately the sales team is going to talk to the paying customer. How does my work make that paying customer’s life better?” That’s the ninja level stuff. I want you to be thinking about that and just at least being—for now, be aware that your work is ultimately going to affect those people. How, I don’t know, but you should be mindful about how it will.

The second thing on this, your career will shine when you realize that much of this game of design and product design, product management—I just think of all this as product-y work—it’s rooted in serving others and enabling outcome for others. And you shine, your career shines, when you’re helping others get what they need. So, it’s really not about you. The path to your shininess comes through others, making improvements in the lives of others. So, when we do all of this work right—here’s the kind of tough thing—when you do all this design stuff right, and we get a lot of this kind of stuff right, a lot of it, we can’t see it.

The solutions, the data products we actually build, the final technology that might ship, it might feel logical, obvious, really simple, like, kind of not a big deal. That might even be particularly true if you’ve been designing all along as a team sport with customers, not for customers. And I mean end-users here, when I say the word customers. When they’ve been involved all along, there isn’t a giant reveal at the end, and it’s kind of like, oh yeah, this thing just went out the door, and we’ve been working on it, they know we’ve been working on it, and they finally have access to it. And then we move on to the next thing, or the next sprint, or the next data product, and it just almost feels like it’s not that significant.

And that’s part of the thing about design is, a lot of times the good stuff is invisible. It will feel logical, obvious, and simple. But it’s really hard to do simple well, and so it doesn’t mean you didn’t do a good job if you don’t have a gleaming, pretty UI at the end of the day. That’s not the kind of design that I’m talking to all of you about today. I want you to get to that almost place of invisibility, where the solutions you created, created obvious value, measurable benefits, outcomes that we could see and observe, the solutions feel obvious, usable, useful, and hopefully delightful.

Anyhow, that was a whole lot of stuff. I hope this was useful to you. I do do training on this for teams. I have a seminar called, “Designing Human-Centered Data Products,” so if you and/or your team is looking for live training with me, you can head over to designingforanalytics.com/seminar.

And by the time this rolls out, I may have publicly launched my cross-company group coaching, but I’m okay just giving you the link to this, now. I have soft-launched this new service as of again, it’s September 2025 to a very small audience, just the people that were involved in my own UX research. So, if you feel like you got some of these skills already down, and you’re kind of dealing with all the organizational friction and cutting through the red tape and getting access to end-users, you want to do this work, but there’s different kinds of blockers in your org, my cross-company group coaching is really a small group way of getting help from me on your own challenges and helping you figure out how to navigate those things, unlike the seminar, which is really about knowledge transfer, a structured teaching over a defined amount of time and that kind of thing. So, if you’re interested in cross-company group coaching, the link is designingforanalytics.com/groupcoaching. And that, again, is brand new, and it will be launching soon. The first ten people in that will get lifetime early-bird pricing as well. So, if you want to lock in lifetime early-bird pricing with that, I hope you’ll join me in the first cohort of that new service. That’s enough. Until next time, thanks for listening to Experiencing Data.