Transcript:

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, what to change when product and engineering are moving six times faster, but your sales growth is unchanged. If you’re like a lot of software companies right now, you’re probably shipping features faster than ever and it probably feels really good to burn down that backlog. All that stuff that you never thought you’d probably get to, well, you’re getting to it, so why isn’t the new business growth moving just as fast?

Well, that’s what I want to go into today. I’m going to try and explain to you how the idea of quality has to be defined before you can determine if you’re putting out better product or just more product. And secondly, you have to go back and observe the change in the lives of your buyers and your users that these releases made or didn’t make, as is often the case, such that you’re connecting the outputs that you’re shipping to the desired outcomes of these people. And I’ll also make the point that this act of measuring the quality along the way is often identical to or a proxy for what your buyers and users are actually judging you by. And finally, well, I’ll try to give you an example… I’m not going to try; I am going to give you an example of what this might look like for a fictional B2B analytical pricing product, a pricing tool, so that you can try to apply these concepts to your analytical product.

If your team skips these steps, you’re more likely to end up with a product that unintentionally creates what I call the invisible intelligence gap for your customers, whereby the truly sophisticated analytics, machine-learning models, or AI capabilities that your team has created aren’t perceived as useful, usable, or valuable. As the Brits like to say, mind the gap.

So, let’s kick this off. First off, speed does not equate to value. Now, we’ve all heard speed wins, right? I don’t think so. I don’t think of it that way. Everyone right now is deploying code fast these days. Features are flying out the door, or they can fly out the door. Engineering isn’t a bottleneck. So, you’ve probably heard that judgment and product sense matters more than ever, so deciding what to build that will actually create desirable impacts.

So, I did want to say, and I got to tell you, I’ve written more code in the last six to 12 months… or rather, Claude has written it for me most of the time… across several applications I use for both personal and professional reasons, and it’s really amazing how fast you can bring these ideas to life with working software. However, I also think some of these tools are like crack for an engineer right now. If you’re technical, you probably felt like you were always behind, and now you finally can get ahead. Like, there’s not a bunch of people waiting on you to get the work done and to show you progress and all of this. In fact, I have one client right now telling me that their bottleneck is actually product management, not engineering, so take that product people [laugh] listening to this show. You had your 20 or 30 years of complaining about engineering velocity, and I guess payback sucks, right? Now, you’re the problem [laugh].

So, you know, if you’re sitting though above all this in the C-suite, you’re probably wondering why isn’t my business growth, my sales traction moving that much faster, given all this stuff that we’re able to deploy so much faster. Today’s episode, I am going to answer that. I just wanted to tell you, you know, today’s episode was inspired by a chief revenue officer that I was interviewing as part of a product strategy initiative that I was doing with this client that I mentioned earlier, and as part of these engagements that I do, I look outside the product and engineering teams and work, and I also look at positioning, marketing, and sales, the way that’s done inside the organization to see if product is aligned with the promises that marketing and sales are making, and vice versa. And during our chat with this CRO, he mentioned how delighted that he was and the whole business was in the new engineering velocity that they’ve seen. And he said something along the lines of, like, “It must be, like, six times faster than what we used to be. We’ve never gone this fast before.”

But the kicker is that their leads and their sales aren’t moving any faster. And in fact, he actually went on to talk about how, you know, their customer service is actually struggling to keep up sometimes because they’re not even up-to-date on everything that’s going out in the product because so many features are being released. So, I want to a tangent into that, but I want to talk about this idea of fast versus good. So, changing the goal from faster engineering to better products that produce value; let’s talk about this.

You know, for years, CXOs and software companies wanted faster, and boy, did Cursor and Claude Code and all the other tools help them get their wish, right? And if you’ve been in the product field as long as I have, you can probably remember… and you’ve done enterprise work, you can probably remember these projects that would go six months, twelve‘ months, two years, three years longer than their original goals and deadlines. Nobody could estimate well and the desire was always to accelerate the build, accelerate development so that the product could get out faster. And now that everyone has speed on their side with all these AI tools doing coding for us, and the fact that prototyping and code is almost as fast as thinking up what to prototype in the first place, I think a lot of leaders are realizing that speed alone doesn’t win. People don’t buy the fastest B2B analytics product. They buy products that solve their neediest problems, they buy products that self-quantify their value so that they can defend the investment to their stakeholders, and they buy products that make their lives better and improve their status.

So, deploying features faster doesn’t guarantee that any of these changes in the world will actually occur. The issue is in whether, when, and how quality is defined and whether anyone is responsible for producing outcomes, the outcomes being changes in the world that you and your team can observe and measure in some objective way. So, you know, like the show is called Experiencing Data; my company is called Designing for Analytics, as most of you know, so let’s talk about this in the analytics and data products kind of context here. The agentic BI and analytics trap, the right answer, faster, but is it worth paying for? So, let’s get into this whole agentic thing with BI right now.

For years, most data leaders inside enterprise have told me something that I think is flat wrong, and that is that the way to measure the quality of analytics work is by measuring the adoption or usage of the dashboards and applications and analytical outputs that are used by users. So, the idea here is they think usage equals value. Except it doesn’t work that way. Usage can be a tax, not a benefit. And I went into detail on this in “Episode 192: Analytics Product Usage does not Equal Value”.

Analytics, including analytics about your analytics product… which is my favorite topic, of course; I mean, it doesn’t get more meta than that… this kind of information doesn’t tell you anything about buyer perceptions nor end-users’ subjective feelings, attitudes, and perceptions of your solutions. All it does is count things computers can count. It counts product usage in time units and other computer quantifiable metrics. The analytics are not inside the user’s brain or the buyer’s brain. So, we’re not really getting… we’re not measuring experience or any other subjective feeling about the product.

When I think about all the modern AI solutions, including things, like, you know, data prep tools that convert data into AI-ready data or tools that use large language models to facilitate talk to your data type experiences, we tend to look at the labor reduction only, I think, in the market and the wow factor of being able to ask a question in plain English… or hopefully the language of your choice… and get a math-right answer. However, what many enterprise organizations, the customers of many of these kinds of tools have struggled with is getting any action to be taken, changes in their business, getting the change in the business to occur as a result of these analytical insights. So, while I’m not saying there isn’t money to be made here in say, you know, simplifying the context layer for AI and the enterprise, for example, it’s really important to define what quality means in the eyes of the people that matter, in your buying and using loop.

So, we have to ask, quality for whom? Who is the end-user here? The business user on the end of an LLM query, the person who sits in the tool, ensuring that those calling LLMs or users know when and how to access this middleware layer so that they generate correct analytical answers? What about the head of data looking for reduced data engineering or analytics engineering overhead? They’re trying to get their people working on other stuff. What about somebody who’s responsible for analytics adoption or quote, “Value from data,” which we’ve been talking about for 25 years? Do you have to satisfy the buyer, the user, and the beneficiary? Because sometimes we have at least three, if not more personas than that. Beneficiaries sometimes don’t touch the software. How will these different people each measure the value and the impact of your product or lack thereof?

Well, there isn’t a single answer here. The point is that your homework is understanding who has a vote and a voice in the buying of and the usage of and the benefiting of your product. So, if you’re promising the agentic analytics experience, which for the record, almost every analytics and BI product seems to be doing as of this recording in September 2026, which means it’s not really unique, so then it makes sense to understand, is the agentic capability being seen by and valued by the users and buyers that matter, or are you just doing it because everyone else is and because being able to talk to your data seems like a logical way to get better answers, faster, with less work? I might argue that if all these products are all making it easier to get more correct answers faster with AI… and I mean these large language model-based user interfaces… then you basically have a commodity product on your hand, a product that has no unique differentiation and one that users might just begin to treat, like, oxygen.

Of course it works that way. Of course it has talk-to-your-data capability. So, how is that unique and worth paying for in and of itself? The only thing you can charge for in that scenario is crystal-clear outcomes, observable positive changes as defined by your customers. The agentic capability itself is just a means to an end and offers little pricing power or value-add in and of itself.

So, let’s talk about rethinking how quality can be defined here. So, acceptance criteria isn’t value. Rethink how quality is defined in the eyes of buyers and users. So, let’s look at the acceptance criteria, AI evals, and what I think the right way is to measure quality. In the old Agile lingo, we talked about… and I’m not saying agile is wrong or bad still… but we talked about acceptance criteria as a means of determining whether a feature is ready to go into production or not.

These days, with AI powered products, we hear a lot about how product leaders in particular need to spend their work defining evals: evaluations. So, you have to define ways to measure the responses for when we have AI capabilities in our intelligence products, we have to find ways to measure the responses. Were they math correct? Were they programmatically verifiable? Do they pass that kind of vibe or gut check? All that stuff is interesting and necessary, but it’s also insufficient when it comes to evaluating the impact on buyers and users.

And here’s why. I believe evals make a lot of sense if you’re deploying AI-powered products, but you need to understand you’re evaluating the product here, not the impact of the product, and that’s the kicker. The thing your team should be measuring lives downstream from the product’s performance against your internal eval criteria. So, focusing on evals alone is like checking that all your app’s UI design and visuals are sitting on the grid with the proper padding and margins and font sizes as dictated by your style guide or your design system. You’re measuring something that largely doesn’t matter. Indirectly. But these indirect measurements don’t matter much if you’re not measuring the most important aspects that are directly relevant to the humans in the loop. Visual design does matter, but it doesn’t matter in isolation, and it doesn’t matter if the most important aspects to the humans in the loop aren’t solid.

I realize this is a little conceptual, so let’s use a data-driven product example to make things concrete. So, my hypothetical example here is turning an AI powered pricing tool into a high impact, high value product, okay? So, let’s imagine that your product is a dynamic B2B pricing tool that can mine the web with AI for price information from competitors, and even look at related inputs such as the cost of materials in the product, and it can also look at your historical pricing to date, such as your sales history, and feedback about your pricing from customer feedback, reviews, phone calls with customer service, stuff like that. And so, from there, this magical tool produces the highest acceptable price range it thinks your customers’ products could be listed for. So, your customers in this case are companies selling products online with a website, maybe offline too, but you’re providing this pricing intelligence for them.

Now, since a number out of context is often not a trusted number, your product, of course, cites all of its evidence, like, all good AI products do now. It gives feedback on the range of prices in the market, and it explains how the product would be positioned against its competitors at that price. Is it now the cheap one, is it, kind of, in the average range, or is it going to be seen as premium? And it also helps you understand the likely impact on sales volume since, you know, setting your prices higher might mean you get fewer sales, but your revenue could stay the same or even go up. So, in theory, all that stuff probably sounds pretty nice, right?

So, how is the B2B buyer at the other end of this thing, your prospect, how are they going to evaluate if this is a good investment with clear ROI? And what about the end-user… now I’m talking about the users of this application… who are sitting in front of a pricing tool user interface… which maybe is just Claude or ChatGPT, for example, and maybe there’s no dedicated UI, it doesn’t really matter… how do they measure if this tool is beneficial to them? The analytics-minded person can see all the reasons why all this quantifiable evidence in the pricing tool should be obviously valuable to a customer, but that’s theoretical and it’s guessing. And right there is why feature after feature, even at six times velocity, as I mentioned earlier, this doesn’t always produce any change in your company’s business metrics: because you’re operating on guesses and theoretical quality that isn’t rooted in empathy for your buyers and users. It’s also possible that your execution was off here. There’s a lot of different reasons here, but the question is, is all this speed leading to the desirable changes in the KPIs you’re tracking for your business?

So, let’s talk about defining quality outcomes with an empathy-driven approach. While your AI-powered pricing analytics product uses the most sophisticated crawlers and knowledge graphs and modeling under the covers, this technical sophistication alone isn’t necessarily commercially valuable. You have to translate your technical complexity into commercial clarity. So, what does success look like to the buyer? Well, in this pricing tool example we’ve been talking about, quality here from the fiscal buyer’s lens may be the ability to, for example, conduct an A/B test with your product so they can get some hard evidence in order to get procurement to sign off on their purchase of your product. And we all love procurement, don’t we? I’m sure there’s probably some, like, procurement analytics people listening to this. Hey, you’re doing your job, right? But I digress.

Let’s say your tool lets your customers do dynamic website pricing on their e-commerce site. So, they set up a test whereby 50% of their e-commerce website’s traffic, their customers are going to see prices the way they currently set prices for the next quarter, and the other half are going to see the pricing set by your AI product. And in 30 days, this business buyer, your prospect, your lead here, wants to evaluate the results, and they’re looking for a minimum of 8% greater revenue with no more than a 10% drop in sales volume, right? So, revenue can go up because prices are higher, but they also don’t want too much sales volume drop, in this case 10%. Why 10%, you say? I’m so glad you asked.

Because more than 10% to your human buyer… remember, you’re always selling to the person. You’re not selling to a business. A business isn’t something that has a wallet. A person in the business has a wallet… this person will make other stakeholders worried that pricing has been set too high and something is wrong because these other stakeholders are incentivized on keeping sales volume at a certain level, not on revenue. 10%, it doesn’t have to be an analytically computed number. In this case, in this buyer’s subjective mindset, anything more than a 10% drop would not make him look good. This will cause concern. This will block the sale.

It’s 10% because they said it’s 10%. There is no right number here. It’s just subjective. So, the big takeaway here, this type of test that the buyer, that this lead or this prospective buyer is evaluating your product on, perhaps quietly without you knowing, you may not even know that this is what these numbers or what the criteria is, this is the exact type of success metric that you should have been writing before you started building. I’m going to say that again. The way they’re evaluating your POC in order to understand its value, this should be informing what your product team is doing in terms of setting success metrics prior to development and building.

So, let’s talk about the end-user now of this product, not the fiscal buyer, but the end-user. What does success look like to the end-user, sitting in front of the tool’s interface? On the user side of things, the incentives are often different, and while the end-user is an individual contributor and does care about, you know, their company’s financial health to a degree, their job is really to only take the most minimum amount of risk that won’t get them in trouble, it won’t get them fired. They’re not being compensated on taking risk and on increased revenue or profitability, as they might just see pricing as mostly a guessing game today by the business, and so management has decided that these pricing-related roles shouldn’t really be bonus eligible. It’s, kind of like, closer to gambling and bluffing. There’s really no way to know what the market is going to pay, and so you’re really just guessing, so it’s not really a strategic role, and so there is no extra compensation if you do a great job pricing right.

So, what is a positive impact by your product look like for this end-user whose job it is to basically price as high as possible, but don’t ever mess up. Well, we would look at how they do this pricing today without your tool and how they feel about the prices that they set today, especially when they’re increasing them. Are they frustrated? Are they worried about it? Is it hard or easy to come up with the numbers to type in? Is there anything fun about it? Like, what is the baseline experience for someone that does pricing work?

So, that’s what I want to get into next: establishing the baseline user experience of our pricing employee so that you can measure the future improvement. You have a place by which to measure improvement. The point is we have to have a baseline in order to measure the improvement, otherwise, you can’t really quantify anything. And we want to make an improvement in their lives, an improvement that can be observed and measured.

The model I like for doing this is called UX Outcome Model, which I first heard from Jared Spool, and I’ve been using other language for this myself for a long time, but I like his model quite a bit, and I want to give him credit. And he’s been on the show in the past too. If we go and observe these users and listen to them, watch them, and observe how they use the software and listen to them, we might learn that this is really high stress work that is unfortunately seen as mostly tactical by the business because they see it as basically guessing and it’s not particularly strategic. The pricing employee, though, worries about going too high all the time with prices and then having to justify their pricing choices when things don’t go well. This means a lot of time is spent not just figuring out the right price numbers to type into the price tool, but also what evidence they will need to back it up if things don’t go well.

And that evidence is a drag for them to put together. Nobody wants to really read it, and the evidence is also largely subjective or open to interpretation, meaning it doesn’t provide a lot of support to this user if they have to go and justify what they did to their stakeholders. As such, our pricing staff mostly sees their job as to either keep the status quo or at best, small incremental raises to the lowest risk products that they’re responsible to price.

So, to just roll this up, prices are therefore set primarily based on what won’t get them fired, not on what might allow the most revenue to be captured within a sales volume tolerance that business leadership cares about. So, let’s imagine now, and this is where design comes in particular, and our ideation work here, establishing the aspirational user experience for this pricing employee. This is the improvement that you seek to make in their lives by better design, a product that works better. When we apply the UX Outcome Model to this, we’re in a different territory. Now, you’re asking, what would an improvement in this pricing user’s life look like if we did a good job with our next product sprint?

Well, you have your baseline measurement, which we took earlier, and by talking with them, you might have learned that fear of job loss is the key driver for how they price, and secondarily, the price justification evidence collection is a real drag. We talked about this. So, if that could be made simpler and easier for the people who are asking for this justification, that would reduce busy work that they don’t particularly like doing. What we might also find out is that if our product could actually get better data and write more compelling arguments, like, data storytelling with AI that makes the human persons, the human employee doing this work makes their pricing choices look super defensible, then they might not just be satisfied, they might actually be delighted.

So, what we’re doing here by trying to design for these outcomes here is we’re moving them along a scale, a scale which on the low end, is frustration and in the middle is satisfaction, and on the high end is delight. And I’ve talked about this before, and again, this is Jared Spool’s model for thinking about this. Delight at the top end here… think of that on, like, the Y-axis, up and down… delight is where we get indispensable solutions that customers tell their bosses they can’t live without. It’s the tool that gets dragged over to their next job when they get promoted by leaving the company and going somewhere else. The key point here is that the business user, your fiscal buyer, is looking at the financial results of the pricing experiment that they ran to judge your tool’s value in the POC.

The end-user who does this pricing labor work has different goals. Your product needs to account for both of these humans’ needs, if that pricing user has a voice in the purchase of the software. Now, I will say there are definitely examples, lots of examples, I remember having to use Lotus Notes when I was at Lycos. Oh, my God. People, this is the worst software I ever used my entire life, but someone decided that’s what the company’s going to use. And clearly they did not spend any time with users actually testing [laugh] what it’s like to use Lotus notes. I’m dating myself here.

The point is whether or not they have a voice in the purchase or not, if this tool sits there unused or bypassed, it’s still not producing any value for the business, even if they don’t have a say in it. And I guarantee, at renewal time, if your product hasn’t spoken for itself because it didn’t get used, then guess what? This end-user who didn’t have a vote, did have an impact on your business because their lack of use of the product directly impacted you when you didn’t get the renewal.

Now, in reality, there could be many more personas in the buying and using loop, particularly for enterprise sales and large organizations and things like this, but you get the idea that the interests are different and your product may die if it underwhelms one of these key personas. So, if you’re interested in more information on all many of the different personas you might need to think about if you’re selling a B2B analytics product, you can check out the last episode I released, episode 202. Called “Surface Tension”, which looks at the five essential control surfaces and related personas that I think you should be aware of almost exclusively. If you’re in the B2B and the analytics space, selling an analytics product, you should definitely be aware of all five of those. There could be more, there could be less in your context, but you should definitely be aware of those.

Anyhow, moving on. Evaluating if the tool’s AI capabilities produced good financial results is actually pretty easy because the math is going to speak for itself. The UX outcomes are actually trickier, but let’s look at what those improvements might look like for our pricing employee. So, in the new world, with your amazing AI powered pricing product, they no longer see pricing as their job. Work is now fun because they feel like they’ve got a secret edge over the market competition, and with all this data, they’re going to look like rock stars to management when their price changes produce more revenue.

Why rockstars? Because not only can they show business ROI, they also don’t have to do so much labor to generate useful evidence and justification for their price choices. What used to take a week to gather now takes under two hours per product that they’re pricing because they didn’t have to manually combine analytics from multiple tools and data sets and reading search logs and, you know, customer service logs and reviews, et cetera. Now, this AI assistant can draft really compelling data stories that make these price changes defensible.

So, as a product lead, if you’re thinking about your organization and your product, what were the user experience metrics to measure here? Well, they might have looked something like this. Number one, reduce the evidence gathering workflow from five days to two hours while ensuring that the resulting ‘justification report,’ as we’ll call it, is reported verbally by our users as feeling like a compelling start, if not a complete finish to their work. So, cut the time down to create this justification report and make sure that they feel like it’s a pretty compelling start, if not finish, after they use the tool to generate it.

Number two, after one month of use, at least 50% of users report feeling… right here: feeling… an increase in self-reported confidence in the pricing they gave their management, or that management verbally said they noticed their pricing staff seeming more confident in their readout meetings. So, here we are measuring increased confidence and pricing, whether it’s reported by management seeing it in the staff or whether the staff themselves self-report feeling that way. Third one: after 30 days, by using analytics, ironically, the business will be able to see greater risk-taking with the pricing by staff without a greater than 10% reduction in sales volume. So, greater here could be defined by looking at, you know, the deviations and in price-setting behavior as compared to the past. The exact math doesn’t matter here to make the point.

The point here is there would be an analytical way to see, are people taking more risk and is the risk paying off? And if they’re taking more risk and we’re not seeing a negative reduction and a negative reduction; that’s a double negative, I guess, right [laugh]? We’re not seeing an unacceptable reduction in sales volume, which was considered 10% was our magic number here, then the business would feel good about that because they have staff that are doing higher value work now. So, your bar for quality is your bar for value, and I’m hoping by now that you can see how the idea of a better product has to be tied to the subjective value as seen through the eyes of the people that you serve. At the most basic level, for B2B analytics solutions, we’re talking about users and fiscal buyers.

The process of defining clear measurements, even if they’re partially qualitative in nature, and then ensuring you have a routine in place in your product development methodology to go back and evaluate your features that you deployed, this process, this habit, creates a learning cycle whereby you can objectively evaluate if you’re just going fast or if you’re going faster and better. If AI is letting you deploy capabilities way faster than you used to be able to, but the work deployed isn’t having a business or user impact, a desirable impact, then it’s a meaningless stat to the business that you’re pushing out features faster. Remember my client’s CRO who said he was observing a six-fold increase in engineering speed at the company with features flying out the door faster than they can keep up with.

Well, I’d much rather see that 6X pace go to a 4X pace and use the additional 2X time to do this methodical product and design work, whereby success metrics, value, and impact are measured up front. I’m sorry, success metrics and goals for value are defined up front, not measured up front. They’re defined up front, the targets, and then they’re validated on the back-end after the deployments happen or after the product design work is done or the prototyping is done. The point is there’s a goal, there’s some building, and there’s some learning happening. Whether they’re showing improvement or not, the main point is that you want to have this feedback loop built into your product development culture and your product development cycle so that you’re learning from both your wins, what’s working, and your losses.

So, that was a whole lot of stuff. I’m going to just wrap this up with a couple bullets, a couple things that you can take away today and hopefully apply today, okay? So, how can you apply these today? Well, first, define your success metrics in ways everyone can understand objectively. Number two, define those metrics for both users, what we call those user experience outcomes and business outcomes, which are usually ROI-related financial metrics.

Number three, establish a baseline so it’s easy to compare the before and after. Number four, find out how your customers are going to evaluate your POC or demo and turn those criteria into success metrics that define whether your product delivery work is good or not, or is it on track? Number five, understand that usage does not equate to value for users. End users are often not buyers in B2B contexts, and their incentives and needs may differ from your buyers. Number seven, if you’re moving five to ten times faster with AI-based coding inside your engineering team then, you know, repurpose, you know, 2x of that capacity towards product management and user experience design and evaluation work. Get those engineers, your technical staff involved with just defining success, measuring it, and observing customers in the wild.

And the last one here is, reframe your code deployments and feature releases from endpoints to starting points. I’m not talking about API endpoints. What I’m saying is when it goes out to customers, when you finally ship the thing, the output, the feature, the capability, when you ship it, you’re at the very beginning of testing whether what you just built has any value at all. It feels like you’re done with something because you finished building it, it’s code complete, it passed the evals, but you’re at the beginning of the journey from a customer perspective.

If you don’t know how to get started with all this, or if you feel like, you know, you’re trying to read the label from inside your own jar, well, I can help. You’re welcome to set up a free discovery call with me at designingforanalytics.com/go, and I’ll help you figure out how to go from faster engineering to faster and better analytics products that sell themselves. Again, that’s designingforanalytics.com/go.

Thanks for listening to Experiencing Data, and I always appreciate it if you share this episode. If you know someone who needs to hear these words as spoken by yours truly, please pass it along. I really appreciate it. Thank you for all the reviews and thank you for supporting me in getting past episode 200. Take care.