Are you prepared for the hidden UX taxes that AI and LLM features might be imposing on your B2B customers—without your knowledge? Are you certain that your AI product or features are truly delivering value, or are there unseen taxes that are working against your users and your product / business? In this episode, I’m delving into some of UX challenges that I think need to be addressed when implementing LLM and AI features into B2B products.

While AI seems to offer the change for significantly enhanced productivity, it also introduces a new layer of complexity for UX design. This complexity is not limited to the challenges of designing in a probabilistic medium (i.e. ML/AI), but also in being able to define what “quality” means. When the product team does not have a shared understanding of what a measurably better UX outcome means, improved sales and user adoption are less likely to follow.

I’ll also discuss aspects of designing for AI that may be invisible on the surface. How might AI-powered products change the work of B2B users? What are some of the traps I see some startup clients and founders I advise in MIT’s Sandbox venture fund fall into?

If you’re a product leader in B2B / enterprise software and want to make sure your AI capabilities don’t end up creating more damage than value for users,  this episode will help!

Highlights/ Skip to

• Improving your AI model accuracy improves outputs—but customers only care about outcomes (4:02)
• AI-driven productivity gains also put the customer’s “next problem” into their face sooner. Are you addressing the most urgent problem they now have—or used to have? (7:35)
• Products that win will combine AI with tastefully designed deterministic software—because doing everything for everyone well is impossible and most models alone aren’t products (12:55)
• Just because your AI app or LLM feature can do ”X” doesn't mean people will want it or change their behavior (16:26)
• AI Agents sound great—but there is a human UX too, and it must enable trust and intervention at the right times (22:14)
• Not overheard from customers:  “I would buy this/use this if it had AI” (26:52)
• Adaptive UIs sound like they’ll solve everything—but to reduce friction, they need to adapt to the person, not just the format of model outputs (30:20)
• Introducing AI introduces more states and scenarios that your product may need to support that may not be obvious right away (37:56)

Quotes from Today’s Episode

• Product leaders have to decide how much effort and resources you should put into model improvements versus improving a user’s experience. Obviously, model quality is important in certain contexts and regulated industries, but when GenAI errors and confabulations are lower risk to the user (i.e. they create minor friction or inconveniences), the broader user experience that you facilitate might be what is actually determining the true value of your AI features or product. Model accuracy alone is not going to necessarily lead to happier users or increased adoption. ML models can be quantifiably tested for accuracy with structured tests, but because they’re easier to test for quality vs. something like UX doesn’t mean users value these improvements more. The product will stand a better chance of creating business value when it is clearly demonstrating it is improving your users’ lives. (5:25)
• When designing AI agents, there is still a human UX - a beneficiary - in the loop. They have an experience, whether you designed it with intention or not. How much transparency needs to be given to users when an agent does work for them? Should users be able to intervene when the AI is doing this type of work?  Handling errors is something we do in all software, but what about retraining and learning so that the future user experiences is better? Is the system learning anything while it’s going through this—and can I tell if it’s learning what I want/need it to learn? What about humans in the loop who might interact with or be affected by the work the agent is doing even if they aren’t the agent’s owner or “user”? Who’s outcomes matter here? At what cost? (22:51)
• Customers primarily care about things like raising or changing their status, making more money, making their job easier, saving time, etc. In fact,I believe a product marketed with GenAI may eventually signal a negative / burden on customers thanks to the inflated and unmet expectations around AI that is poorly implemented in the product UX. Don’t think it’s going to be bought just because it using  AI in a novel way. Customers aren’t sitting around wishing for “disruption” from your product; quite the opposite. AI or not, you need to make the customer the hero. Your AI will shine when it delivers an outsized UX outcome for your users (27:49)
• What kind of UX are you delivering right out of the box when a customer tries out your AI product or feature? Did you design it for tire kicking, playing around, and user stress testing? Or just an idealistic happy path? GenAI features inside b2b products should surface capabilities and constraints particularly around where users can create value for themselves quickly.  Natural hints and well-designed prompt nudges in LLMs for example are important to users and to your product team: because you’re setting a more realistic expectation of what’s possible with customers and helping them get to an outcome sooner. You’re also teaching them how to use your solution to get the most value—without asking them to go read a manual. (38:21)

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, the topic: the hidden UX taxes your AI product or LLM features are imposing on your data products customers. All right.

Coming to you solo again today, wanted to talk about what these hidden taxes are. These are points of friction, usability issues, risks to your business, things along these lines that may not be totally obvious when you’re implementing an AI product or a deterministic piece of software that has AI capabilities in it, particularly large language model type features or small language model, whatever type of model it is, and particularly in the B2B context. I’m also thinking about analytics products as well. So, insight solutions, things of this nature as I was going through, writing up my notes for this episode. So, let’s jump in.

The key message, I think, today that I want to bring to you is that without user experience design attention—and I mean deliberate design attention as opposed to what I call byproduct design—AI is likely creating benefits, but it’s also creating taxes. And again, some of these may be invisible to you as a product leader unless you’re intentional about addressing them. So, the reason this matters is because these friction points can lead to loss sales, or attrition, or low adoption rates if they’re not addressed as part of your product development process. It’s probably worth stating that again: customers only care about benefits and outcomes. And by customers here, I mean both our business stakeholders or our business buyers who aren’t always the users if you’re in the commercial space, as well as the actual end-users. They all are really focused on getting a benefit or an outcome from the solution.

And measuring improvements in the lives of the user is really the only way to know if you’re doing a good job with user experience, and that requires a solid baseline understanding of what the status quo is, whether that’s prior to using your product and their alternatives, whatever their current solution methodology is, or it could be your starting point, your first time you’re getting your product in front of them and establishing and measuring what that baseline is. Inflated expectations that aren’t met early in the UX can create a negative brand association from the start. That brand can be your internal data product team or it could be the brand of your company if you are a software company building some kind of commercial tool. There’s brands for both, and we don’t want to be damaging that because of the product.

So, I’m sure some of you can probably relate to this if you’ve seen what happens when these AI features are slapped into products. You know, maybe you’re writing a social media post or you’re writing something, and then you let the service rewrite it for you or you’ve been prompted to let it write the stuff for you, only to find that it’s made it worse. And then you start to wonder, is it really worth my time to redo all of this, and tweak it, or should I just start over with something of my own. When these solutions are kind of thrown in with the everything box, there can be taxes that customers are paying for.

So, some big ideas to get across today in this episode. First one is improving your AI model accuracy improves the outputs, but customers only care about outcomes. So, a good example of what I mean by this is that think about the difference between using a co-pilot for coding—and I’m using trying to use a technical example that this audience can probably relate to—cutting and pasting stuff between Claude or ChatGPT to get coding insights or debugging errors, things like this, versus an integrated experience such as what you see in this application called Cursor; there’s other co-pilot applications out there as well. But the point is they made the experience of using the model and what it can do to make software engineering more efficiency, and they really bundled that up and wrapped it up in a piece of software that helps improve the experience so you’re not bouncing back and forth between multiple windows and applications, losing context, things like this.

So, you can make the model better at writing better code, but you could make something perfect, you could make one of these large language models perfect, say, at—hypothetically speaking, of course, but let’s say it’s perfect at writing JavaScript code—but you wouldn’t actually be addressing the user experience pain of doing all this back and forth bouncing here. So, you have to decide how much effort and resources should you put into model improvements versus improving the user experience, and figuring out which of those is actually going to create a higher benefit or a better benefit that someone is worth paying for to get access to that solution. Obviously, model quality is really important in certain regulated industries, and medical, and financial things like this, and you have to make those appropriate decisions as well when doing any type of work in that space, but when errors and confabulations are lower risk to the user and simply are creating friction, the user experience that you facilitate might help really determine the value that your AI features or product is actually going to deliver. So, we have to let go of this idea that model accuracy alone is going to necessarily lead to happier users.

And I’m sure the analytics folks here know that even when you make the data more accurate, and you’re building dashboards or reports or things like this, and you get this quantitative structured data, which many of you are typically working with, and you know it’s more accurate and you’ve captured all the edge cases and all the ways somebody could poke holes in it, and yet sometimes the stakeholders still don’t use it to make any decisions, or they deny the reality of it, or whatever. That’s back to technically right, effectively wrong, and we don’t want to go there now that we’re in the AI space. The other thing here with the model accuracy thing is that—and this is a dear message to the analytical thinkers listening to this episode—is that because the models themselves can be tested for accuracy and quantified with things like structured tests and predictive power and all this kind of stuff, it makes it feel like there is an objective way to measure quality, but in reality, measuring user experience quality is a little bit more tricky and it’s a little bit more subjective, and we can’t simply run a test and know whether or not we’ve improved somebody’s life.

So, the next one is that AI-driven productivity gains also put the customer’s next problem into their face sooner. So, are you helping them solve it? I’m going to state this one again. AI-driven productivity gains also put the customer’s next problem into their face sooner. Are you helping them solve it?

So, with certain B2B AI products and especially, like, analytics application, some of you may realize that there’s always, like, another problem right around the corner. So, let’s say that you’ve built a propensity model that will predict what product this customer record of the database—for every customer you have in the database, what are they most likely to buy next? And let’s say you’ve got 100,000 customers in your CRM, right? Now, what? So, in the past, you couldn’t do this predictive power, you didn’t have this predictive power, and now you have it here. How is the sales team going to react to this information that they never saw before? Did you figure out how to fit this into their workflow?

Because in the past, maybe there was a way for them to go and make an educated guess by going through different sales transactions and linking up the data, and let’s say you’re selling really expensive products and knowing this information would be really valuable, and there is a high-touch sales process involved, how are they doing that work now? And if you accelerate the process of figuring out what that—automatically predicting what the next product is, how is the sales team going to change their behavior of about this?

Another example of this is of pushing the next problem around the corner right into their face to deal with. Let’s imagine you’re building an insurance claim application that’s using AI, and it does some cool summarization of the claims and prioritization, and it’s allowing claims adjusters to kind of work in bulk, at scale in a way that they used to have to do serially. A model-first AI-driven approach, which I’m still not in favor of, this might mean that you end up looking at this technically, and then you might report back to management that, hey, our new model can now parse a thousand insurance claims per adjuster, and it can summarize them, what’s in the claims, and then it can help the adjuster know which ones to address first. And so, we’re talking about the power in terms of the volume of work that the AI seems like it can do. Now, you take this further, and if this is a commercial software application, then you start seeing things like this in the marketing copy about how we’re AI powered and we give you a one thousand times increase in your claims adjusters’ productivity, right?

Well, a designer user experience approach to this problem would look at this more like asking the following questions: if we reduce the effort of parsing through insurance claims, how would we make a claims adjuster’s life better? What do they find challenging or annoying or difficult about this work? And if we introduce something that’s going to help them at scale, and you know, they’re supposed to manage a thousand claims at a time versus one at a time, or maybe they’re handling, you know, five to ten at a time or whatever, how is their work going to change as a result of this? And what’s the new problem that our solution just created as a result of the AI doing a bunch of the probably more manual predictive tasks, some of the analysis work? And does your product have to solve for that or facilitate this new work that the human is now needing to do? Is that within the boundaries of what your product’s promise is? Should it be?

And then over time, this ability to do the claims analysis going to become the commodity feature, the basic thing that all of the other insurance claims applications that do this kind of stuff already have, such that your value that’s worth paying for starts to become not in that analysis work, but this next problem, whatever that is in the claim adjuster’s life and in their workflow. When I say ‘life,’ I mean their work life and their job. So, without doing the right user research, it’s really hard to know what to build and where to put the boundaries within your product, so understanding what these boundaries are is really important, and that’s a product management question that you need to figure out. Where do we stop here so that we’re not trying to boil the ocean here, especially if you’re a smaller company and you can’t take on replacing the entire work of this target customer. You have to decide what that is.

So again, just to restate this topic was about when you get the productivity gains of AI, you’re also pushing the next problem into their face sooner, and that next problem might be a new problem that never existed before because the scale issue was never real until now. And you’re not going to know this until you go out and talk to customers and understand how they would want to address or how would they address these problems in that situation.

The next one, products that win will combine AI with tastefully designed deterministic software because doing everything for everyone well is impossible, and most models as standalone things aren’t products. So again, products that win will combine AI with tastefully designed old-fashioned deterministic software because doing everything for everyone really well is impossible, and most models aren’t products in terms of standalone products. So yes, there are exceptions. ChatGPT and OpenAI’s Claude, and of course, you know all the big names here. Most of you listening to that are not working at these places and you’re not in the foundation model space.

And even those are trying to be everything to everyone, and in reality, when we start leveraging these more as services that provide a feature that we can contain inside of our product, our deterministic traditional software application, I think this is where stand-out products worth paying for are going to prosper. You need to deliver exceptional value, and this is especially true now with what AI is doing with a lot of different industries and types of work. Standing still is really difficult, and if you want to get ahead here, you need to figure out, well, what are customers going to be willing to pay now to get access to our solution here, if it’s starting to be commoditized because of different AI capabilities that now exist out there. So, I think you need to have tasteful, opinionated choices that are made by a sharp product and design team that’s done really good research. This is likely to always trump a foundation model experience that really hasn’t been designed for a specific type of task and work that someone needs to do.

And this is putting aside all the other issues around data privacy and all those kinds of things. So again, if you think about it, that Cursor, the example I gave with Cursor, this AI-powered coding co-pilot versus bouncing back and forth between your current IDE and ChatGPT or a Claude or whatever it may be, well, Cursor still is a downloadable file. You run the executable on your computer, and it opens a browser window, or it has an application GUI to it, and menus and all the traditional stuff that software applications have, and then it’s tastefully inserted the AI capabilities where it makes sense to improve the developer’s life and save them time. That’s a really good example of this. So, trying to just AI everything for the sake of AI, I don’t think that’s a winning strategy here.

And part of the reason I really believe in that is because whenever we’re using AI here, I think defining what quality means to the customer and to the user experience, that gets even harder to do when we no longer have complete control over every experience that might happen in the application. So, quality needs to be even more understood by the team: what is our current situation and what does a quality improvement look like? I’m not talking about data quality or QA testing, engineering quality or any of that; I’m talking about quality as perceived by the buyers and the end-users of the solution.

The next big idea, just because your AI app or LLM feature can do X doesn’t mean people will want it or will change their behavior. Just because your AI app or LLM feature can do X doesn’t mean people will want it or change their behavior. So, I’m bringing this up because I find technical founders who are often the ones that are leading B2B software product efforts, or they’re working on internal data products, or yeah, technical founders tend to have this more than other types of founders. But I think some of you all know that I do advising for MIT’s Sandbox Venture Fund—so this is, like, an incubator for MIT student founders—and I’ll often see that there are teams that are looking at this from the standpoint of, hey, AI can now do X. I want to try to build a product around some capability that AI now has.

And I’m going to pull an example out from an actual call that I did, an advisory call with a really cool idea I think that has a lot of potential and I want to recognize that there, and I think it’s also just a positive—the products not trying to directly necessarily just monetize for the sake of money. But this was, I believe, a data scientist type founder who was interested in computer vision work, and what we could do to help farmers look at satellite data of their farms to understand, where on my farm, what plots of my land are underperforming, and what type of intervention might be needed, particularly even just solving the problem of where do I need intervention, which plots are underperforming the rest? And so, satellite imagery now apparently can do some of this work, or at least that’s the IP that he was working on. So, he wanted to know, how should I visualize my dashboards and show these heat maps or there was a, you know, a visualization of the satellite imagery with some overlays there to try to show which plots, you know, were negatively performing and this type of thing. And approaching this as a data visualization dashboard design question, while I understand that eventually you may end up having some UI, some data visualization, some dashboard, there probably will be some type of interface, eventually, starting the answer to those questions is going to come from understanding the farmers in this case, or the agribusiness owner, and really understanding what it’s like to be them, and this should really inform how the dashboard should be designed beyond just basic UI and UX heuristics and data visualization heuristics because that’s only—heuristics and best practices is only going to get you so far.

So, I really tried to push this founder to understand, what does the farm owner do today and how would having this insight change their business, or say they’re staffing the farmhands and the people that are actually out in the fields, either the type of intervention or how he assigns work to these people. Having this insight may actually change some of the processes that this farmer has, and really understanding that first, and not getting so lost and how the dashboards may need to design, that will almost take care of itself if you can really understand how the farmer’s life is going to change in their work. I also really encouraged him to go and find out, what do these agribusiness owners struggle with right now, and what change are they ready and willing to make, regardless of whether or not AI can now do X, which in this case is that there’s enough computer vision that can apparently identify underperforming crops relative to the neighboring crops there. So, is that necessarily the—is the farmer’s biggest problem not knowing where to send his resources, and say maybe wasting time? I don’t know.

And while I can imagine these kinds of things and we can do designing on guessing, we can also do more designing on fact, by spending the requisite time with these business users to really understand this problem space of crop intervention. So, I also think prototyping solutions here instead of I know it’s easy to build solutions now, but even in situations like this, I think low-fidelity prototyping can help you have really useful conversations, research conversations, and learning and problem-surfacing conversations. And you also could potentially do mixed prototypes here where you might have a low-fidelity Figma, or even a hand-drawn sketch, or dashboard, or something like that, and then maybe you actually use a real foundation model like a ChatGPT or something like this, and you might give the customer some different tasks. Like, given this dashboard or given this plot that I’m showing you here, what kinds of questions would you ask the solution if you could interrogate the system? And letting them use a real LLM, and monitoring the types of questions they ask, and their attitudes, and their frustration levels, this could reveal a quite a bit of interesting information. Again, not necessarily just judging the outputs of the LLM, the actual functioning LLM, but rather looking at it as a chance to surface the problems that the farmer is starting to ask. That’s where we’re getting hints about what we actually need to build.

Next one here is AI agents. Right now, we’re hearing a lot about agents and these sound great, but user experience has to enable trust and intervention at the right place with these, especially if there is meant to be a human in the loop. And eventually there’s almost always going to be some human beneficiary of the system. The agents to me is a really overhyped name for basically allowing computers to do parts of jobs that traditionally were done by humans, and just sometimes at scale. But a task completed does not mean we’ve necessarily created an outcome or improvement in the user’s life.

So, successful agents use AI to accelerate the creation of a positive UX outcome and/or a business value that’s worth paying for. That’s when we know whether or the AI agents are really doing their job. So, in terms of UX though, what do we need to know here? Well, agents create new friction and problems that have to also be addressed for the value of this AI capability to be transformed into a benefit. So, just some examples here: should we provide transparency into these background processes when these agents are doing their work, or not? How much transparency needs to be given or not? Should users be able to intervene when the AI is doing this type of work? And what happens if it’s a fairly complex task and a human being jumps into the middle of it, in terms of, do we finish that workflow? Do we cancel it? Like, just thinking about all those situations where the human might want to intervene with the system.

Facilitating refinement loops. So, if you’re allowing the user to kind of set up one of these agents through a GUI in your application, we might really need to think about all the refinement loops that happen in terms of, ‘Oh, that’s not what I meant’ or, ‘Oh, I need to give it more context here so it does X, Y, and Z.’ There could be a lot of refinement loops that you need to design into the process of making the agent if that’s what you’re facilitating. Obviously, handling errors is a classic thing we do in all software, but what about retraining and learning so that the future user experiences is better? Is the system learning anything while it’s going through this, and is that learning being retained? Is the human beneficiary of the agent, is it obvious or not? Has it been surfaced? What information is being learned by the agent such that we know, yes, keep doing what you’re doing, and you’re on the right track, versus, oh, this is going in the wrong direction and I’m finding out about it early enough that I don’t have to wait until either damage is done, or I’m getting back bad information, or my tasks aren’t done correctly because I didn’t have visibility into that?

Obviously, there’s data access and permissions issues and, of course, ethical guide rails here, so we have to decide to what end are we going to allow these agents to go achieve the goals that we’ve given to them. And we also have to think too, if we’re really designing here with an intention of making these solutions work for humans, well, what about all of the downstream beneficiaries, or affected parties, or other human touch points within the agent’s work? So, a simple example of this might be, let’s say you’ve probably heard about these bots that can make a phone call and schedule a dinner reservation for you and this kind of thing. Well, what’s the downside risk if the language the bot is using is inappropriate when it calls the restaurant, or it’s becoming really annoying, or it’s called ten times in the single day, leaving messages, sending text messages, relentlessly doing anything it can to get that dinner reservation, or to find out who is the head of purchasing at this Fortune 500 company and it’s calling the administrative assistant on the end of the corporate phone number over and over and over again, and we’re not aware of that because it doesn’t know that it’s not supposed to nag these people this often to achieve the goal. So, we need to think about what are we doing to our brand, but what are we also doing to the other people that might be interacting with these agents, or their own businesses, or lives may be somehow touched or affected by them.

The next one here, I’ve never overheard this from customers: “I would buy this or use it if it had AI.” Again, not overheard from customers: “I would buy this or use it if it had AI.” Marketers right now really love leading with AI, but again, I have yet to meet anybody who is deliberately buying solutions because it has AI in it. It’s very natural to want to be experimenting with this in our products right now, and I think that absolutely makes sense to do—there’s a lot of potential that we can unlock here—but each AI capability that you add may seem to offer significant feature improvement, but we have to remember all these taxes that we’ve been talking about in this episode that have to be addressed if the buyers or the stakeholders of these solutions are going to get those benefits. Customers still care—especially buyers—care about raising their status, making more money, saving time, making their job easier, and being empowered or being backed up. Like, there’s feelings involved with why people buy stuff. They don’t just buy stuff because it rationally makes sense. Otherwise, everyone are you listening that has a product that says, “I’ll solve your data lake integration issues and I’ll make your pipelines ten times faster.” If it was just about the technical values that your solution can provide, why wouldn’t everyone just always be buying it?

Well, it’s because the timing is wrong or these other values are not being addressed in the solution. I actually think AI might eventually even indicate a tax or a burden on customers because of the inflated expectations that I think we’re creating with this. And it may even signal that some customers are going to be less in charge than they used to be, and depending on the context, some users may not want to relinquish their power, or even some of their work, to an AI-based solution. So, the other thing I wanted to just say about this, in terms of figuring out what the right product is to create and not necessarily just thinking that because it has AI that someone is going to buy it, Marty Kagan wrote about this recently, and I think he had a great framing which is that, product teams, we’re still going to need our engineers, and our PMs, and our designers, and I think as necessary when there’s a heavy data component, the data scientist person, the product development part, the building and the solutioning part, that’s going to continue to accelerate in speed, and really what powerful product teams are going to be doing, the ones that really stand out is spending more time in the discovery space, and figuring out what work actually needs to be addressed and built into a solution. What should we actually build? What are the actual problems that really are worth someone’s money to pay for right now versus just something we could do? Because now we can do so much more than we used to be able to and we can do it faster, so figuring out what to make and what the problems are, that’s going to be where more of the time will be spent.

Next one—I’ve got just a couple more here—adaptive UIs sound like they’re going to solve everything. So, this is this idea of artificial intelligence kind of generating a customized UI when you interact with the solution. This sounds like it’s going to solve everything, but to reduce friction, they’re going to need to really adapt to the person and not just the outputs of the model, any type of response that the system is returning back to the user. So, if you think back to, like, how many times you heard that, “Oh, you can customize it, don’t worry,” in all the software that we use today, but then you never actually went out and did any of that customization, or when you put it in front of users, they didn’t do it, or when they tried to customize the solution, you might have realized that really what customizing often does when it’s not implemented properly is it pushes the burden of design back onto the customer. And if you ask them if they want it, of course they’re going to say yes because it sounds like, great, I’ll be able to get any work, any value that I need, I’ll be able to get it as long as it has customization.

And I think adaptive UIs also sound like this on the surface, but I think the difference is that we need to, if you’re going to build a solution like this and it makes sense to have that capability to improve the value of the product, it’s going to need to adapt to the person and the type of work they’re doing, not to the model’s outputs. So, what do I mean by this? So, let’s use an example here. Using natural language processing to query—or an LLM—to query facts from a structured database, right—so this is, like, your BI tool that has a natural language front-end on it—well, the first level of this probably is, can you even return the right chart visualization or data visualization based on the type of data that was requested? I realize even getting that right is fairly difficult, but the next level here to make this better may be understanding, well, what precise rows and columns should actually be returned here based on the role or the types of work that this person does?

Like, over time, ideally the product has stored in persistent memory something about this user such that we know, hey, this is a product analyst, not a sales person, and so when we—or this is a marketing person—so we might want to return these other columns, or we may want to change the filtering and sorting controls because this person is probably not an analyst, they’re probably a marketing person, or things of this nature here. To take this to another level, even outside of, like, say the analytics realm, imagine you’re building a custom application for say, a hedge fund or a financial services company that does, you know, institutional trading or something like this. And let’s say that they’ve put, you know, a large language model, or some kind of language model—interface on top of all their structured and unstructured data that they have about companies that they invest in or that they’re considering investing in, right? So, you might be able to—a trader might want to ask a question about a certain stock using its symbol or its name, for example. And so, while the LLM might just be able to spit back a simple answer—“Well, you know, the current beta is, you know, 1.2,” or, “The, you know, last bid and ask was this,” like, simple quoting information, sure, it can return this kind of stuff, but what it might be better off doing and provide really good value is that certain UI templates that have been designed with intention might be called at the right time, whether the AI is knowing how to do this from training data or whether or not you use traditional software engineering, you say, “Hey, this request included a stock symbol. When you return the response, return it in this UI format.”

And imagine something like one of these, like, little card interfaces, like when you do a Google search, you might get an answer card back that actually maybe it has the last trading price, the total outstanding volume, the name of the company, whatever the standard metrics are that a hedge fund wants its traders to have in their mind whenever they’re doing certain types of queries. And you could have a whole library of these different UI cards. And the point is, it’s a standardized UI pattern that the company or this user base is used to, and having that along with whatever the LLM’s response is a much better value than just answering, one by one, different questions that we might expect or we would get by default if we put a language model search interface into the product there. So, what are those different UI cards? Well, obviously, customer research and knowing the audience, or even looking at their traditional applications to date, we would need to think through all those use cases and then make sure we’re designing the appropriate interfaces given the certain types of prompts and questions that are asked, and then obviously evaluate whether or not we’re making the improvement in the user’s life by providing those intentional UI artifacts at render time.

So, I just wanted to say too, in terms of analytics applications, stuff like this, I’m fairly certain that while so many BI tools and classic analytics tools, and especially in the business space, there’s so much of them that can be well designed using very basic data visualization techniques and methods, bar charts, line charts, small multiples, spark lines, things like this, I’m fairly certain that more complex data visualizations that are actually going to really be useful and create real value are still going to require manual designers for the foreseeable future because really understanding what the visual part of analytics and what data needs to be comparable in the viewport at the same time when answering questions, or even what data viz may need to be retained while we’re, say, drilling down if we’re using an AI-based interface to drill into some data, well, do we need to retain, like, the parent data or maybe if you’re using, for example, my CED method, the Conclusions Evidence Data Framework, you might want to keep the conclusion of this analytics solution, whatever the insight is, the conclusion that you’ve been trying to derive with the stashboard or application, you might want to keep that in context and visually present, even if another part of the screen is perhaps getting into the drill-down part of that. My point is here, it’s not going to be a simple answer here, and I don’t think we’re going to create amazing value, and all of a sudden analytics is going to use so much more, if it only had a natural language query prompt in front of it and it would just return our questions with a nice row of table data and a chart rendering on top of it. I don’t think that’s going to solve all the adoption problems with analytics solutions. So, that’s my opinion right now.

The final one here to talk about is that introducing AI introduces more states and scenarios that your product may need to support that may not be obvious right away. So, what are some of these? Well, I think maybe one of the most critical ones, particularly for software companies for those of you that are actually selling your wares, is the first time experience. What kind of experience are you delivering right out of the box, the first time they do a trial or the first time they’ve installed the solution? I think that as soon as you open up the doors and you don’t have as much control over this experience because there is some type of generative AI component that’s built into the experience, I think you should expect users to kick the tires, to play around with unexpected prompts and queries, and they’re going to probably be stress testing the system in their own way.

So, you need to design that experience well because you’re only going to get so long to make that first impression and you want it to be good so that they stick around. So, we can counter this by surfacing capabilities and constraints. So, what is the tool and particularly the AI features inside it? What’s possible within the solutions, and what are its constraints? Can you give hints about what types of work or tasks the human can do that will show results really soon?

What should I try out in this solution that will show me some value really quickly? We want to try to get to a win fast so that they build some confidence, especially if they’re switching from the status quo, whatever their old solution was, and they’re trying your solution out, we want to get them some value as quickly as possible. Understanding and surfacing the limitations of the system, and where necessary preventing them from hurting themselves or others, or allowing the AI capabilities to waste their time because they’re going down a rat hole for example, and we don’t want them to spend a whole bunch of time only to realize they’re going to hit a dead end at some point.

Other aspects here with this are things like abusability testing and red-teaming. So, this is getting into how might users either accidentally, or maybe even intentionally in some cases, try to use this service in a way that it was not designed for? And do you have the proper guardrails in place to counter this? So, this is a really big conversation, and I know this is a conversation in both the UX space, the product space, and in the data science space here, so I’m just going to touch on it right now in the interest of time. But you can have an internal team be your red team here and try to use it for nefarious purposes, and then you can start to plug those holes as you learn where all the little holes in your AI capabilities are that you haven’t properly accounted for.

Obviously, you’re going to want to watch your users use these solutions in a lab setting or in some kind of observational setting, as well as looking at logs. If you’re using prompting, for example, you’re going to want to look at what are they prompting about? Can you interpret what the overall task is? You’re going to want to get that insight there and then react to it. The other things here with this one are, have the designers considered both the context window memory and the persistent memory, such that over a longer period, the application of the product is understanding the type of user experience that the customers are actually having?

So, this gets into what is this tool remembering about me when I’m using it, and is it adjusting to me along the way, or is that compartmentalized? Like for example, as you know with ChatGPT or any of these tools that store different threaded conversations, there’s the context window memory, which is just local to that particular thread, but then there’s also this idea of persistent memory where the AI capability knows information across all of these interactions. So, there’s not only the question of should you be doing that, but it’s also do you need to surface any of that information so that the end customer is aware of whether or not this tool is adapting to me in a positive way or in a negative way, such that they can then correct for it.

And then you have that whole experience. Can they even correct for this? Can they delete the information that’s been stored? Can they retrain the system in some way? And retraining is a whole other thing. When is the system actually learning anything from the customer’s dialog that it may be having, if it is a large language model implementation? It’s very opaque right now how these systems work, and what they’re remembering and what they’re not remembering, and so depending on your product and the context of your users, these might be things that, if addressed by either intentionally surfacing them or maybe even intentionally hiding them, you might be able to provide an outsized experience that creates an outsized value, something that sets your solution apart from the competitors.

And then the final thing on this is thinking about auditing, right? So, especially in regulated environments, there’s always that audit user experience. So, if I need to go back and explain something at a later time, can I do that in the system? You know, this is a wide—almost a very narrow feature that may or may not apply to every situation, and obviously regulated industries, this is much more important to have, and this also determines partly what type of technology implementation is used in terms of black box models versus more interpretable ones and things like this, but someone is going to audit the system, and so there is yet another experience there. And if the customer, the actual end-user is the one doing that audit, you have to decide, well, how easy or frictionless should that experience be? Is it okay for that to be a really painstaking experience or is that something they should be able to do quite well themselves?

Anyhow, that was a lot. I hope this was helpful, surfacing some of these hidden user experience taxes that your AI product or features might be imposing on your customers and what you need to do to start addressing these in your design. So, if you found this useful and you’d like to get more insights like this or one-page summaries of these podcast episodes, I do publish those on my insights mailing list. You can join that over at designingforanalytics.com/podcast. There’s also a link there to record an audio question, which I may answer on a future episode of Experiencing Data. So, thank you again for chiming in, and I’ll see you on the email list, or maybe at a conference near you, or maybe in person one day. Take care.