Ready for more ideas about UX for AI and LLM applications in enterprise environments? In part 2 of my topic on UX considerations for LLMs, I explore how an LLM might be used for a fictitious use case at an insurance company—specifically, to help internal tools teams to get rapid access to primary qualitative user research. (Yes, it’s a little “meta”, and I’m also trying to nudge you with this hypothetical example—no secret!) 😉 My goal with these episodes is to share questions you might want to ask yourself such that any use of an LLM is actually contributing to a positive UX outcome  Join me as I cover the implications for design, the importance of foundational data quality, the balance between creative inspiration and factual accuracy, and the never-ending discussion of how we might handle hallucinations and errors posing as “facts”—all with a UX angle. At the end, I also share a personal story where I used an LLM to help me do some shopping for my favorite product: TRIP INSURANCE! (NOT!)

Highlights/ Skip to:

• (1:05) I introduce a hypothetical  internal LLM tool and what the goal of the tool is for the team who would use it
• (5:31) Improving access to primary research findings for better UX
• (10:19) What “quality data” means in a UX context
• (12:18) When LLM accuracy maybe doesn’t matter as much
• (14:03) How AI and LLMs are opening the door for fresh visioning work
• (15:38) Brian’s overall take on LLMs inside enterprise software as of right now
• (18:56) Final thoughts on UX design for LLMs, particularly in the enterprise
• (20:25) My inspiration for these 2 episodes—and how I had to use ChatGPT to help me complete a purchase on a website that could have integrated this capability right into their website

Quotes from Today’s Episode

• “If we accept that the goal of most product and user experience research is to accelerate the production of quality services, products, and experiences, the question is whether or not using an LLM for these types of questions is moving the needle in that direction at all. And secondly, are the potential downsides like hallucinations and occasional fabricated findings, is that all worth it? So, this is a design for AI problem.” - Brian T. O’Neill (8:09)
• “What’s in our data? Can the right people change it when the LLM is wrong? The data product managers and AI leaders reading this or listening know that the not-so-secret path to the best AI is in the foundational data that the models are trained on. But what does the word *quality* mean from a product standpoint and a risk reduction one, as seen from an end-users’ perspective? Somebody who’s trying to get work done? This is a different type of quality measurement.” - Brian T. O’Neill (10:40)

• “When we think about fact retrieval use cases in particular, how easily can product teams—internal or otherwise—and end-users understand the confidence of responses? When responses are wrong, how easily, if at all, can users and product teams update the model’s responses? Errors in large language models may be a significant design consideration when we design probabilistic solutions, and we no longer control what exactly our products and software are going to show to users. If bad UX can include leading people down the wrong path unknowingly, then AI is kind of like the team on the other side of the tug of war that we’re playing.” - Brian T. O’Neill (11:22)
• “As somebody who writes a lot for my consulting business, and composes music in another, one of the hardest parts for creators can be the zero-to-one problem of getting started—the blank page—and this is a place where I think LLMs have great potential. But it also means we need to do the proper research to understand our audience, and when or where they’re doing truly generative or creative work—such that we can take a generative UX to the next level that goes beyond delivering banal and obviously derivative content.” - Brian T. O’Neill (13:31)
• “One thing I actually like about the hype, investment, and excitement around GenAI and LLMs in the enterprise is that there is an opportunity for organizations here to do some fresh visioning work. And this is a place that designers and user experience professionals can help data teams as we bring design into the AI space.” - Brian T. O’Neill (14:04)

• “If there was ever a time to do some new visioning work, I think now is one of those times. However, we need highly skilled design leaders to help facilitate this in order for this to be effective. Part of that skill is knowing who to include in exercises like this, and my perspective, one of those people, for sure, should be somebody who understands the data science side as well, not just the engineering perspective. And as I posited in my seminar that I teach, the AI and analytical data product teams probably need a fourth member. It’s a quartet and not a trio. And that quartet includes a data expert, as well as that engineering lead.” - Brian T. O’Neill (14:38)

Links

• Perplexity.ai: https://perplexity.ai
• Get the Book @ Amazon: Ideaflow*
  (*Paid Link - As an Amazon Associate, I earn from qualifying purchases)
• My article that inspired this episode

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. I’m back with part two of my earlier episode, which is number 147, on designing for LLMs. I haven’t talked too much about generative AI on the show, and I wanted to share some ideas that I have about how we might design for LLMs, particularly within enterprise applications. So, go back, pause right now, listen to episode 147 if you haven’t already. That’ll give some additional context, and then you can hop right back over here for part two.

So, where are we starting in part two? A practical example here. I’m creating a hypothetical use case here around designing better tools for insurance adjusters, so let’s jump into this. Let’s assume we have an internal digital UX product data science team, something like this. It doesn’t really matter who’s on it, as long as the right work is being done. But they’re at a large insurance company.

And let’s assume that this team is building both customer and internal customer-facing technology solutions. So, this team has gone out, and they’ve collected a bunch of raw transcript data from talking to customers. So, think about one-on-one qualitative research, actually talking to real insurance adjusters inside the business about how they process customers’ claims. And the goal of this effort is to get customer claims processed faster, and to process the correct claims first. So, those are the business objectives, and we’re also trying to improve the life of the adjuster and the work that they do.

So, the research team is, say, in a separate department, and they’re aware that many of the 20 or so—and I’m just making this up—20 or so internal product and technology teams, do not regularly seek advice from the user experience research team, or their research questions are generally too hard to answer, given the velocity that the build team runs under. And yeah, they’re of course they’re using agile, right, but we all know that Agile does not necessarily guarantee that we’re producing outcomes. So, in this case, we have a bunch of user experience research, these interviews that have been done with the adjusters. That work is done, but a lot of times, it’s rarely used. So, it’s effectively sunk cost, right?

So, let’s also assume that the CEO of this company, or the executive team, has recently stated that closing claims more quickly, by 20% on average, is a primary goal for the next quarter, as the time that’s spent on any one claim costs the company more money. And the company wants to rebrand itself around being really easy to file claims with. And so, the team doing this research—I’ll call it the research team or the UX research team—believes it likely has insights that would be helpful for the design, and engineering, and data teams working on the tools that the insurance adjusters use. After all, this labor constitutes—again, making numbers up—80% of the time consumed by an open claim.

However, many of the product and engineering teams don’t know what it’s like to be a claims adjuster. They haven’t talked to one recently, and maybe they even find the whole thing rather boring. Maybe they want to focus on the analytics of the data science, or all the math parts that go into insurance, and they really don’t spend a lot of time with actual adjusters. Those people are over in a different town in some building downtown, and we don’t know much about them, right? Well, I can think of a lot of zero-to-one use cases here for which an internal LLM or AI application in a large organization like this fictitious insurance company might be helpful to teams building these decision support tools and products, especially if we compare this to the possible alternatives that these technology teams might otherwise engage in, such as building solutions with no research at all because it’s too slow, and we got to ship it by X date. And there’s nothing here that answers the questions we have about the features that we were told to build in this sprint. Things like that.

Or building product with partial research exposure. So, “Hey, I went to two of those interviews your team conducted.” Well, they missed the other 12 that happened, and they never really saw the research team's final recommendations on what they found out after they had talked to 14 or 15 of these insurance adjusters, right? So, partial research exposure. Or the classic, “We’ll change it and round two or version two,” they said because the research insights were useful, but they came too late, and somebody higher up in the food chain committed to some date already, and we have to ship something this quarter, et cetera, et cetera. So, you’ve probably heard all this before.

So, back to our LLM. How might an internal LLM-driven application improve or accelerate access to primary research findings and get more non-user experience people, those who are controlling the engineering resources, and the data resources, and the implementation teams, to guess less, and to actually factor in research findings into their work early? Given the pressure these internal teams might be under to meet the CEOs goals, they may want to know as soon as possible questions like this, “As an engineering leader, or as a data science lead on this project, I have no idea what this whole UX research panel”—a panel, think of that is, like, a study—“What this panel was about? I don’t know what the questions were, I don’t know who was interviewed. I don’t know what the results of it was. Is anything here relevant to the widget, the use case, the dashboard, the tool that I’m working on for, let’s say, the auto collisions adjustment team?”

Another question. “Are there any obvious patterns and the findings of this research panel? What was the overall tone and attitude of the panel? And maybe extra nice if the solution could look at audio-video, as well as text, right? What questions were asked of these adjusters the most? Where’s all the primary data about topic X in this panel?”

So again, let’s say I’m on the auto collisions team, maybe I just want to look at any time topics around car collisions came up, right? “Is anything here different or new based on what my team already knows about the auto collisions team and those adjusters?” And maybe, “What questions should I even be asking an LLM?” So, I don’t even know much about this data, and I don’t even know what I might want to be asking it, but I’ve heard that maybe there’s something useful in all this data that can help me make better tools—or data products, or whatever it may be—for our colleagues in the adjustment team.

I just want to say, this is a little bit of a meta example because I’m talking a little bit about UX research here, and I’m also talking about designing for LLMs. And in this case, the LLM—you can kind of see where this is going—the LLM’s training data here is a body of qualitative interview transcripts that was done. So, I hope I’m not losing you a little bit in the, kind of, meta example. But I thought this would be, kind of, an interesting and fun use case to look at.

So, jumping back from my aside right there. If we accept that the goal of most product and user experience research is to accelerate the production of quality services, products, and experiences, the question is whether or not using an LLM for these types of questions is moving the needle in that direction at all. And secondly, are the potential downsides like hallucinations and occasional fabricated findings, is that all worth it? So, this is a design for AI problem. If we consider the domain and the context of use, what if this LLM solution was enabled to do things such as warn users when their prompts are likely to return inaccurate information because we know what types of questions this GPT is not good at asking, and we want to educate our users about that.

Or attach citations to any summarizations such that users have at least some capability to determine the accuracy of an LLM summary without directly consuming all of the research data. Quick aside. I think Perplexity.ai is doing some really interesting things with search because they’re combining search results or search answers with citations of the underlying sources. So, that’s a little aside.

Jumping back to another one, notify the team who owns the LLM solution about trending prompts around specific panels or research topics so that they know what questions product teams are curious about, and can also get ahead of potential problematic interpretations. Or allowing researchers to have their own analysis challenged by the LLM. So, for example, “Hey, here’s my analysis of these 15 transcripts. This is what I got from you. Are any of these conclusions or interpretations not supported well by the raw transcript data?” Or put up appropriate guide rails, given the audience it was designed for. So, for example, is the proper education around this tool in place, either as part of the underlying user experience, or some other educational program, or whatever it may be.

So, what’s in our data, right? Can the right people change it when the LLM is wrong? The data product management and AI leaders reading this article, or listening to this episode—and I will be posting the full article—I will be adding part two to part one of the article, and that’s cited later on in the episode notes for the show—the DPM and AI leaders reading this or listening know that the not-so-secret path to the best AI is in the foundational data that the models are trained on. But what does quality mean from a product standpoint and a risk reduction one, as seen from an end-users’ perspective? So, somebody who’s trying to get work done.

Of course, we all have the obvious data governance and privacy aspects to consider with any solution like this, leveraging a large language model, however, another type of data that may be necessary here is the domain knowledge of subject-matter experts, particularly if a custom GPT will be trained, as it probably would be necessary for many enterprise use cases. So, when we think about fact retrieval use cases in particular, how easily can product teams—internal or otherwise—and end-users understand the confidence of responses? When responses are wrong, how easily, if at all, can users and product teams update the model’s responses? Errors in large language models may be a significant design consideration when we design probabilistic solutions, and we no longer control what exactly our products and software are going to show to users. If bad UX can include leading people down the wrong path unknowingly, then AI is kind of like the team on the other side of the tug of war that we’re playing.

While we try to design a useful, usable and trustworthy solution, the LLM is always tugging us in the opposite direction as it spits out lies, funny retorts, or, quote, “Believable,” but actually misleading information. Now, I said, may be a significant design challenge here because there are times when accuracy maybe doesn’t matter as much. An example of this would be creative cold-start, what I call zero-to-one tasks. So, I’m facing this blank canvas, and I need to get from this zero state to a one state, or situations where I’m looking for inspiration and not necessarily an answer, right?

So, let’s look at a deeper example here. If I need to present my team’s quarterly results to the boss, and I want some creative titles for my PowerPoint deck, does it matter how off the titles are? Well, not really. And similarly, if we look at a marketer working on designing ad copy for advertisements, what may be helpful for them is generating a sheer volume of options from which the marketer can then take over with their brain to develop the final options, right? And according to Jeremy Utley, co-author of the book Ideaflow, and past guest on episode 106 of the show, one of the core ways to innovate is to begin with a large volume of potential ideas or solutions to a problem, such that derivative ideas have a chance to surface themselves.

It is often within the derivative ideas that sparks will fly. But as somebody who writes a lot and composes music—I’m talking about myself now—one of the hardest parts is this zero-to-one problem of getting started, and this is a place where I think LLMs have great potential. But it also means we need to do the proper research to understand our audience, and when or where they’re doing truly generative or creative work.

I want to talk about Star Trek for a second. Let’s get creative here. One thing I like about the hype, investment, and excitement around GenAI and LLMs in the enterprise is that there is an opportunity for organizations here to do some fresh visioning work. And this is a place that designers and user experience professionals can help as we bring design into the AI space. How might we improve the use case or user outcome by leveraging GenAI or LLMs? Or the fun version: how might this experience work on Star Trek?

If there was ever a time to do some new visioning work, I think now is one of those times. However, we need highly skilled design leaders to help facilitate this in order for this to be effective. Part of that skill is knowing who to include in exercises like this, and my perspective, one of those people, for sure, should be somebody who understands the data science side as well, not just the engineering perspective. And as I posited in my seminar that I teach, the AI and analytical data product teams probably need a fourth member. It’s a quartet and not a trio. And that quartet includes a data expert, as well as that engineering lead.

Whether that is one person or two people, I will leave that up to you, but the user experience quality of AI solutions, which has a lot to do with adoption at the end of the day, that’s largely influenced by the data. So, I want to see more interactions between the data science and design professionals on enterprise software and data product teams. All that said, as of right now, here’s my, like, overall take on LLMs and enterprise software.

As of the publishing of the article that I wrote on this, and this podcast, while I’ve definitely seen some positive capabilities of GenAI firsthand, I’m still skeptical on how much improvement in users’ lives we’re going to make with this technology at this current time, given some of the challenges that they present around transparency and hallucinations, and in particular, the mostly CLI-style user interface that many implementations are limited to, right, the CLI being the Command Line Interface. This is not to say that there are not going to be some discrete use cases that could definitely be aided by these solutions. And the one I just gave, maybe there is a way for teams who are currently using no research to build data products or any type of tools for their insurance adjuster colleagues, maybe if those teams had an LLM to interact with in order to extract research findings and knowledge out of raw or processed research data—so, say, the findings documents and things that the research team may have put together—maybe that net puts us in a better place because in the past, the team was doing no research at all. They were building requirements that somebody passed down, they were feature teams—add this feature, add this widget, give me this report, give me this table, give me this chart—and they don’t really understand how it’s being used by an insurance adjuster. Are we net making it better if we’ve given them access to a tool that may sometimes generate the wrong insights, false information, but on the other hand, it is giving us some correct insight, and it’s making the access to the insight that much faster? Is that better or not? That is a complex problem, and it’s one that only you and your team can answer.

So, as for the companies and vendors out there, I wonder how much the companies who own these foundation models are really going to always have the upper hand here, and potentially the ability to compete with point solutions that are leveraging these LLM capabilities. And the second point I want to make is that given how many enterprise use cases depend on information accuracy, and especially question or problem spaces that include quantitative data or analytics, everything I know from technical experts I trust is that even with things like RAG—RAG or Retrieval-Augmented Generation—in place, LLM user experiences will struggle to deliver accurate, trustworthy information from data stores that are holding structured data. I think most of the non-design data professionals listening to this show already know this. They know what the risks are, and how these LLMs are effectively next-word guessing machines, and they’re not great at doing fact-based retrieval, using SQL or structured data methods for retrieving data from databases and things like that. I think that crowd knows that risk, but this is something that I think designers need to be very well aware of, particularly when we think about things like enterprise software.

So, I hope this was useful information, just to get your gears turning in your head and the creative juices flowing, some questions and concepts to think about, as you consider deploying some type of large language model into one of your internal tools or one of your commercial software products. I would love to see a lot less GenAI buttons next to large text area boxes where I just type something in, and then it lets me rewrite the solution. This feels like very much feature-first thinking, and we want to be able to slap an AI label on to our marketing, so what’s the quickest way to do that? Well, find any place in the product where a user is typing in text, and provide them a button to have that text somehow modified, improved, summarized, or changed by a large language model. This is not what I think great design for LLMs looks like, particularly in the enterprise space.

So, I look forward to seeing more examples of this in the wild that are actually producing real value. If you’d like to share any of those with me, please feel free to shoot me an email, or you can record an audio message or question for me at designingforanalytics.com/podcast. You can also get on my insights mailing list there.

And I just also wanted to leave you with this, kind of, last personal reflection. What was the inspiration for this episode, and for my recent article on this? I actually hacked together a solution here. I think I was just using ChatGPT, I can’t remember which one, but I was shopping for trip insurance for my summer vacation to Poland, and I realized, this is a real pain. It’s a pain in the ass shopping for trip insurance. There’s so many plans out there, there’s so many different terms out there, and trying to figure out, like, what’s the right plan, it just sucks. There’s nothing fun about buying trip insurance.

And part of the reason we need that—at least for Americans—you know, most of our health care coverage is not usable overseas, so not being an EU citizen or a Polish citizen, I don’t have access to health care, unlike my wife, and my kid, so I need to at least have that kind of coverage there, and stuff. So, I went over to I think it’s insuremytrip.com, and they have a fairly useful, faceted search interface for comparing the different terms and coverages of different travel or trip insurance plans there. But it wasn’t quite what I needed there, and what I realized is that I could pop the actual legal agreements, the actual policies, which are in PDFs that you have to, like—you know, when it says you get $100,000 in medical coverage, or whatever, you know, there’s a link there to open up the actual policy language and look at that section to see what the fine print looks like.

So, I found myself kind of using their interface, which does not seem to use any type of LLM technology or anything to distill all this legal, and insurance language down into simple—a more useful interface, I find myself basically using their tool to narrow down the number of plans to a handful, and then I loaded those documents into—I think it was ChatGPT, again; I can’t remember, it’s been a couple months—and then I was able to ask it to do some comparisons, and to ask it some questions about my situation there, and also tried to get some sense of third-party reviews. I think I might have used Perplexity for that once I had narrowed down the different vendors here to try to get a sense of, like, you know, the quality in terms of filing claims and stuff like that. And I found that to be a really useful use case, and I was thinking, that seems to be a really practical way that they could be leveraging an LLM to summarize these terms and allow a more human experience in terms of the shopping here.

Now, I can of course understand the issues here when the answers that an LLM might give you back about these policies are wrong. And again, this gets back into what does it mean to be good enough, and is good enough, in this shopping experience, does that trump the potential risks that may be associated with selecting the wrong plan because some people don’t actually ever check the underlying policy coverage based on, you know, the LLM says, “Oh, the coverage is X for Y situation,” but that information was simply incorrect for some reason. So again, this is a complex space. You need data people here, designers, product people, you probably need your legal team involved with this kind of thing, and I don’t know, if insuremytrip.com, or whoever it was that I was using, I don’t know if they’re looking into this kind of thing, but I thought that was an interesting, useful application more on the summarizing of documents that nobody really wants to spend their time reading, particularly as they think about something fun like vacation.

It did help me narrow things down more quickly. And I didn’t take everything with a grain of salt as truth, but it allowed me to get some direction that oh, this plan might be a little bit better because of X. Then I can go in and manually validate that to my own satisfaction based on actually reading some of the underlying coverage there. So, for me, it was less about the LLM giving me the final answer, but rather narrowing down the world to a more reasonable amount of work, and then I could go kind of do the final mile myself. And this is this idea of the designing an assistant here as opposed to replacing the work here, right? We’re trying to use the AI to assist the user’s task, which, in this case, was find and compare the right plans for my situation, which might be different than yours. So anyhow, that was the inspiration for this recent article and these two episodes on designing for LLMs in the enterprise space.

So, until next time, good luck with your generative AI work that you’re doing. Watch out for the lies, keep that outcome-oriented mindset alive, both on the user experience front and on the business front. That should guide you in positive ways. But just remember because we don’t control exactly how these things work, there’s a new level of risk and concern that we need to be very mindful of, and we need to have ways of validating that to understand what is acceptable and what is not. So, that’s it for today. I hope your ears and your brain will be back for more Experiencing Data in the future. Thanks.