Transcript:

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, the Invisible Intelligence Gap. I have been thinking that one of the things that it’s been missing from the podcast and from my writing to the mailing list—and I hope most of the listeners here, if you’re not aware, I do have a free weekly insights mailing list at designingforanalytics.com/list—anyhow, I was thinking that one of the things that might not be clear to new audiences that are finding my work is kind of an overarching worldview about how product and design—product management and user experience design issues can cause problems with adoption and sales in commercial revenue generating analytics products. So, today’s episode is really an attempt to kind of give, almost like a 101 primer at a very high level about why should you care about any of this stuff, and what’s my perspective on all of this.

I have released multiple frameworks over the years to help people design better data products that sell themselves, that get used, that drive outcomes for both users and for organizations and businesses. But I kind of felt like there was a high-level perspective piece that kind of pulls all these ideas together that was missing. So, that’s what today’s episode is going to attempt to do. My goal is that, if you’re listening to this, that if you were to refer, you know, someone to my work, this would almost be like the 101 start here episode. And it’s kind of funny because I think this is, I don't know, episode 189. I don’t number them until after they come out, so I think it’s around episode 189 and it’s kind of funny. You’d think that someone would have gotten to this before 189 episodes. But hey, that’s how I roll.

Sometimes you write to think. That’s something that I had to learn. I always thought you premeditate all the ideas and then you write them down and publish them, and nope, that’s not how it works. You actually write to think, and I write before I do podcast episodes to clarify my thinking and to keep bringing you something that is from a real human being. So, not that I don’t use artificial intelligence myself all the time in my business and to challenge my ideas and to challenge clarity in my thinking, and this is no exception to that. So anyhow, enough about the meta stuff. Let’s jump into the Invisible Intelligence Gap.

So, the Invisible Intelligence Gap, why sophisticated B2B analytics products don’t sell despite their technical excellence. So, I’m going to use the word sell here. I’m also going to sometimes talk about adoption. These things can be commingled depending on your business model, whether you have end-users that can actually trial software on their own, or whether you have a higher touch POC or evaluation process, you’re running on-prem or private cloud software. You know, sometimes there’s a little bit of usage that happens in front of a sale, and sometimes it’s the other way around, where you just got to get your foot in the door and end-users are actually later in the game after your fiscal a sponsor, you’ve passed that first gate with them, but I will try to be clear in my language.

So, I’ve worked with a lot of teams building analytics and insights products and decision support systems, and the pattern I keep seeing isn’t that the math and the data, the analytics themselves are wrong, or that the machine learning and AI models are weak. A lot of the time, the technology is just fine. The challenge is that all of that not always artificial intelligence, per se, but just we’ll call it intelligence, if we think about our insights that we generate, they’re not surfacing as value to your customer. And when I say customer, your customer is really a collection of your fiscal sponsor, your fiscal buyer, your end-users, who are not always the buyers, if you’re in one of those kind of eval software situations, right? You have a product that requires some type of internal usage POC evaluation before there’s a purchase made, and sometimes those end-users have stakeholders of their own. I kind of collectively put all this under the word customer.

When your product’s value is opaque, either new customer acquisition suffers, or once you get in the door with the fiscal buyer, the end-users don’t find the product useful enough to change their current behavior and workflows. So, you end up with dashboards that look impressive, AI features that demo well, pilots might get strong reactions, and then usage stalls, sales cycles drag on, and teams quietly revert to spreadsheets and other tools that they might be using. Buyers, or rather, prospective buyers say that they like your vision, but deals don’t seem to move into that closed one stage. So, if your gut tells you the primary blocker is not your sales process, and it’s not pricing and packaging, it’s not procurement issues, which I know can be difficult, or data quality or risk and compliance, then you might be suffering from what I call this Invisible Intelligence Gap.

Your customers are always the judge of value, so when they don’t buy it can mean this isn’t valuable enough for us, full stop, but it can also mean this might be valuable, but it’s too hard or too risky or too disruptive to adopt it consistently. And so, this Invisible Intelligence Gap is really about that second kind of objection, that second no. The Invisible Intelligence Gap shows up when your data-driven product produces value, but that value is conditional on people changing how they work. In other words, your system may generate the right answers, data, or insights, but the customer only benefits if they interpret it correctly, they trust it, they integrate it into their workflow, and they act on it consistently. And with the probabilistic nature of GenAI, the additional burden of not making the customer lives worse also has to be stated.

So combined, these are often a much higher bar than many product teams realize. Buyers evaluate your solution as if value—which is always defined by them and subjective—that this value should be obvious, but users experience your product as things such as another user interface or experience to learn, another process, another decision to own, and another risk to take. And when the cognitive and political cost of changing behavior outweighs the perceived improvement, you’re competing against a rival, and that rival is the customer’s status quo, how stuff gets done currently, what tools we use now. Hello, Excel [laugh].

So, what are some of the common symptoms of the Invisible Intelligence Gap? If you’re building a B2B analytics or AI product, you may have seen these symptoms before during sales conversations or your evals. Examples look like long enterprise sales cycles despite strong demos, great pilot feedback that never seems to convert to a full rollout, dashboards that are verbally praised but rarely used or rarely seen or used for actual decisioning in the wild, requests for dashboards, despite just having built an LLM-powered chat interface so they could talk to their data, back channel Excel worksheets and requests for data exports that just won’t die, users saying they trust your product’s insights models and data, but in reality, these insights end up illuminating uncomfortable truths, processes, or company culture that isn’t ready to change, simply because those issues are now visible, and you provide massive work reduction or efficiency with AI, while also accelerating the customer’s confrontation with that next big and often tougher challenge that used to be downstream or more of a tomorrow problem. Now, while some of these issues here can be attributed to marketing problems, they’re unlikely training issues. If you’re getting leads but the deals aren’t closing, the root cause may be product management, design and user experience challenges.

The humans in the loop, your ICP buyers, and/or the end-users see too much risk despite the technical excellence of your software. So, what’s actually happening? Most of the time, analytics, insights, and AI systems require human behavior change to unlock their full value. A process has—let me give you some examples here. So like, a process has to change, one that has dependencies, change risks, unaligned incentives, or status-quo bias, or a pivot away from gut experience or seniority-driven decision-making given the cold, hard facts of what your product likely illuminates, or trusting a model backed up with solid evidence that challenges long held beliefs, or forcing accountability for when your AI-powered product is inevitably wrong and possibly wrong at scale, and finally, accepting that machines may be better at busy work and the remaining humans in the loop may have to have more accountability and responsibility for value creation than was previously expected.

The more your product automates, the more it can collapse one part of a workflow and push new, higher order decisions into the user’s lap. That can be powerful, but it also creates new friction. So, while your product team is thinking, hey, we’ve eliminated, you know, 50% of the work from the user, that’s great, and a fiscal buyer can likely quantify that value fairly easily, the end-users may be thinking, “Great, now I’m responsible for the hard part.” And so, when that gap isn’t intentionally designed for, sales can stall or usage can stall.

So, let’s talk about three amplifiers of the Invisible Intelligence Gap. If the value of your analytics insights or AI is not coming across to users, as evidenced by low usage or sales traction, there are often just three primary amplifiers contributing to your solution’s intelligence remaining hidden in the eyes of the user. And I should say here, when I talk about hidden intelligence, I’m not talking about this in a positive situation because there can be times when we, especially in design, where we intentionally obfuscate things that the system is doing, and particularly with AI, and we think about agentic AI, how much feedback do you provide to the human beings that are in the loop of these systems? What is too much information? What is not enough?

You know, for any of you that have used a terminal before to write code or to do whatever it is, and Linux or Unix that you need to do, you’ve seen command line stuff before. Well, when you’re like, you know, installing packages, for example, or you’re running a job, or whatever it may be, somebody decided, how much do you get to see about what’s going on? How much logging is there going to be that’s pushed to the screen? Do you really need to see, you know, 5000 lines per second going by? Do you want to see a dot, dot, dot, slowly crossing the screen?

The point here is, when I talk about invisible intelligence, I’m talking about your product is probably promising some kind of insights or intelligence edge that they can’t get somewhere else, and I’m saying the value of that intelligence is hidden. I’m not saying the background engineering processes and, you know, the reasoning that the model is doing, or any of this kind of stuff, I’m not talking about that being hidden. Some of that should be hidden. I’m talking about the value of your intelligence is not obvious to customers. So, now with that clarity, let’s jump back to the three amplifiers of this Invisible Intelligence Gap.

Number one, the value translation gap. Customers and users can’t clearly connect what your product does to what they personally need to achieve. So, the insight is there, but the meaning isn’t obvious. Especially important to be acutely aware of this in enterprise context, where fiscal buyers and end-users often are not the same people. And ironically, while you and/or your sales team are likely focused on the budget holder and winning the sale, the value to the budget holder is often primarily dependent on meaningful usage of the product by the end-user population.

This value translation problem can surface itself pre-sale, I.e., in your marketing and messaging and how you talk about yourself during the sales process and calls, during trials and evals, or even after the sale, particularly when the value of the product depends on that usage from those users who had no say or no presence in the purchasing process. So, that’s the value translation gap.

The second amplifier of the invisible intelligence gap is the workflow alignment gap. So, the product’s ideal usage pattern doesn’t match how work actually gets done inside the organization or inside the team that’s using it or the individual. So, using your product introduces friction or resistance to change or even inaction. In short, the product has to make users’ lives better, which is largely dependent on how well the product’s UI and UX was designed. In order to make it better, it also means that you have to have great clarity on what your baseline is from what you’re going to improve.

And this is what many founders can’t tell me when we first do engagements, is they haven’t actually—they might have talked to customers, but they’ve rarely actually observed customers using either their product or a competitor’s product or their existing tool chain to really understand what the baseline is such that an improvement and that user’s live could actually be objectively seen, and the team could agree, did we or did we not make things better? So, that’s the workflow alignment gap.

And the third amplifier of the Invisible Intelligence Gap is the trust and control gap, so users don’t fully understand how the product reaches conclusions, what its limits are, what data went into it or didn’t go into it or when and how to override any automated decisioning resulting from analytics data. So, they hedge and they double-check, or they disengage. This can arise from a combination of feature-focused engineering, poor data visualization, what I call by-product design, as opposed to product design, rushed development, where speed trumps quality, which itself is vaguely defined to begin with, an AI product and user lacking context of each other, and perhaps the root cause of many of these items, which is your team does not fully understand the problems, the attitudes, and the feelings of your target users and your ICP. So, that’s the trust and control gap.

So, what do we do about this? And what do we do about these amplifiers if you feel like one of those is representative of your situation? I’m so glad you asked. And fortunately, I have some solutions here, or at least I have some frameworks and some content that should help you. And as I was saying before, a lot of this actually is already available because I’ve worked on a lot of those pieces, but I didn’t really have this Invisible Intelligence Gap model to kind of bring it all together and how these things fit together into my worldview about how we make data products that sell themselves.

So, how to solve the invisible intelligence problem? Well, if your gap problem is the value translation, so making your analytics, your charts, and your UI, your data visualization, all that kind of stuff, worth paying for or worth renewing, as is sometimes the case, then one of my frameworks that might help is the CED framework, and that’s easy to find on my website. You can also just go to designingforanalytics.com/ced, and you can either read about it or you can listen to it. I have both audio and written versions of that.

So, how can that actually help you, though? Let’s get into that. Just in brief, since there’s already an episode. I won’t go deep here. But it is a user experience design framework for advanced analytics applications, and it’s intended to help teams see beyond data visualization and turn your product really into an indispensable decision support aid.

And so, it really gets into this idea that a lot of products tend to surface evidence all over the place, but they’re rarely providing conclusions that can actually drive decision-making. And so, much of why we provide analytics and insights is to influence decision-making. So, I have a worldview that our tools and our products need to be opinionated, they need to push for conclusions, and then only after that, provide evidence to back that up. And the third tier is really the raw data that goes into the evidence. So, it’s a three tier model. So, that’s the CED framework.

The next gap problem, if you are trying to solve for workflow alignment, and that’s the challenge you’re having—so this is about aligning your product with how users do their jobs and do their work and what their needs are—the frameworks that are probably best for this is what I call FOWA, or the F-O-W-A, Flow of Work Alignment. So, if you’ve been listening to this show for a long time, I actually just released these recently, so this is episode 187 and episode 188. And since I know you’re listening to the podcast already, I know you know how to find those. So, they are, of course, on designingforanalytics.com. If you go to the podcast page, you can see all the episodes there in numerical order, so check those out.

And how can the FOWA or Flow of Work Alignment help you? By minimizing the behavior change your product imposes on users, you can make the product and user experience feel more aligned with how users work today, so that improvements with your product in the loop, the value is obvious to them. So, check those out.

And the third one, if your gap problem is perceived user trust and control over your AI, if this is where there are concerns you want, you may want to try out the MIRRR user experience design framework for trustworthy agentic AI applications. Mir is spelled M I triple R. M-I-R-R-R. This is easily findable. It is episode 175. And how can this help you? Well, is your product’s design, is it mostly a prompt box? Because chances are that is probably a problem, unless you’re a foundation model company. And even then, most of the people listening to this are, I’m guessing, not working at foundation model companies. You are, if anything, building products on top of this capability, or using this capability to enhance traditional analytical data products or product or products, as it may be.

So, if you’re just relying on a prompt box here, you probably have challenges, and so the MIRRR framework will help you see what product design and product management considerations you need to be thinking of when designing for probabilistic interfaces. Meaning—and I think most of you probably know what I’m talking about here—but probabilistic meaning, we don’t always know how the system is going to respond. We do not deterministically code in all the edge cases, we know how every possible state of every screen would be because it’s all in the flow of the code. Instead, we don’t really know what’s going to always come out, so we can also do damage. And so, we need to actually put some intentional guardrails in place, and not just data privacy guardrails and all that, but I think of it as design and user experience guardrails. So, the MIRRR framework will go into that and help you solve that perceived trust and control issue that an AI product of yours may be experiencing with your ICP.

So, that’s really it. That’s all I have for you today. That’s kind of my summary of this Invisible Intelligence Gap challenge. I’m really hoping that this helps you avoid creating technically right, effectively wrong software. Nobody needs that, and it’s fairly easy to do.

So, if your B2B analytics or AI product works in theory, but it’s struggling in the real world, where sales feel harder than they should, and user adoption never really quite matches your vision, you may not have a feature problem. You may have an invisible intelligence challenge whereby your product value simply isn’t being seen or felt by the buyers and users and the stakeholders, aka your target customer. The first step may not be adding more capabilities from your backlog or improving your models, per se. It’s making sure the value is tangible, the product is useful and usable, and perhaps most of all, it’s desirable.

Hope that helps. If you found this episode useful, I’m really hoping you’ll consider sharing it. Again, this is meant to be kind of a high-level overview to my whole worldview about building indispensable data products that sell themselves. That’s really what my goal is for B2B commercial software product leaders in the analytics and AI space, so please consider sharing. And for now, that’s all, and we’ll be back soon. Cheers.