Let's look beyond the “rewrite with AI” button and chatbots!

Generative AI in 2024 is all the rage and for the most part, to date, I haven’t seen or heard about many great examples of these making a substantial impact to user experience.

While many product teams continue to race to deploy some sort of GenAI and especially LLMs into their product, the general sense I get is that this is more about FOMO than anything else. So, beyond the “rewrite this with AI” button you keep seeing popping up (hello LinkedIn), is there anything here more than hype? Can we think beyond yet another terrible customer support chat bot?

Today I want to share some ideas that I would be considering in the design of software that is going to leverage LLMs to improve value and outcomes for the humans in the loop.

Foundations

No matter what the technology, a good UX design foundation starts with not doing harm and hopefully going beyond usable to being delightful. Adding LLM capabilities is no different. We still need to have outcome-oriented thinking on both our product and design teams when deploying LLM capabilities. This is a cornerstone of good product work: understanding what an improvement to the business and to a user’s life looks like.

 Now, the human-centered design perspective also means we are (ideally!) thinking beyond just the business we work for or serve , and the users — because there may be other humans in the loop affected by our design choices. It’s easy to dismiss this until it bites you in the ass with your legal team, a newspaper article you didn’t hope to read, or some other surprise — and it’s all the more reason why having designers and researchers involved is important. They’re providing a natural check to all the technical work—work that may now mean even more engineering and data work relative to traditional software efforts. The extra burden on your technical team by definition means that UX is likely to get short-changed because the impacts are harder to see, especially if you don’t have a testing plan to validate what “quality” means in your LLM use cases.

So, the key point here is that you still need to define what a successful outcome to users and the business looks like before you implement an LLM and consider it successful.

Challenges with LLM UI/UXs

The most obvious challenge to me right now with LLMs is that while we’ve given users tremendous flexibility in the form of a Google-search like interface, we’ve also, in many use cases, limited the UX of these to a text conversation with a machine. The CLI is back.

The machine tends to return the same GUI every time: another text prompt that returns yet more text.

Many of these systems do not store context either, nor do they know how to help the user refine their question (e.g. “you may want to try asking X.”)

Even if they do, not all information is best conveyed in written format, and so the UIs right now are immature with LLMs. As such, and as with just about any other technology, product managers need to be thinking about context of use.

What is the outcome/benefit here and how is progress being measured before and after we insert an LLM into the workflow?

The thing is, LLMs may also come with these taxes: the made-up data (silly, serious, and ridiculous), frustrating filtering/narrowing experiences, and a UI that tends to always return the same type of response format (more text). You may be thinking “but we can also create images and videos.” Yes, that’s true—but in a lot of enterprise use cases, particularly when thinking about data and information, what we may need are better visualizations of accurate information.

This then forces us to visit the interaction medium: text/visual, voice, or other format?

In many enterprise situations when designing for knowledge workers sitting at desks, we’re probably still looking mostly at visual UIs for the foreseeable future (vs. voice etc.) Why? If we take a problem such as, “I have all this information about X, and I need to figure out what to do/instruct my team to do/give my boss a recommendation on next steps,” a voice interface is generally not a good way to convey this information.

When you take the CLI-style interface, the hallucinations/guessing, the challenge LLMs have with doing calculations or retrieving structured (i.e. tabular) data that is often part of knowledge work that we want queried with the proper tools (e.g. SQL) and we have what’s known as a “complex problem.”

It’s complex now, and probably will be for the considerable future.

Hence, we must as product leaders constantly revisit the question, “are we making things better for the lives of our customers and users such that the value to the org/business is likely to go up?” What about the other humans in the loop such as affected third parties? What about the workers doing this work now by hand—what is their role and purpose going forward?

LLMs: How Much Does Accuracy Matter in the UX?

Anyone who has played with an LLM such as Chat-GPT have experienced hallucinations and incorrect information. In fact, you can take almost any response from an LLM and reply to it, “sorry that’s not correct,” and the LLM will accept your criticism as correct and produce a new response.

So, this gets to the question of what does it mean to measure “accurate enough” to be useful, safe, valuable or whatever parameters are most important in your context?

I’m going to go into a quasi-hypothetical scenario here, hopefully not confused by using a somewhat “meta” example. The topic is using (or not!) an LLM to summarize customer interview  transcripts from research panels.

Should we use these to take large quantities of text from lots of users and find actionable trends/insights to inform product development?

Hypothetical Use Case: Using LLMs to Summarize Raw Customer Research

For reasons besides privacy concerns, I saw a post recently about just how wrong it is to use LLMs to process information such as raw customer interviews (i.e. transcripts) and summarize findings.

A lot of the ramifications were concerns that the wrong types of shortcuts would be taken and lead to misunderstanding of the primary research data that had been captured—effectively, nullifying much of the value of doing primary research.

I also implied from this post (and prior ones from other UX researchers I know) that there is a sense that an LLM/AI agent could not do as good of a job as a human researcher, and could potentially introduce false information and misleading insights from a panel. In short, the sentiment I’ve been exposed to is primarily that “A human researcher has to be involved” and even if you control for things like data privacy, an LLM simply cannot provide useful, credible information.

My take as of right now is that this position seems extreme, but it doesn’t really matter for the purposes of this article. I’m not here to take a position as to whether you should or should not use LLMs for this purpose. However, in our hypothetical, let’s assume an internal team contemplating building their own custom GPT for this purpose. What questions might they need to ask to determine whether to proceed?

I might suggest asking:

• How is “accurate” human analysis defined today such that we know we can trust that the researcher is providing “good” info?
• How much does speed (time to insight/reaction) matter?
• Is it better or worse if the recipients of research insights (i.e. product managers, engineers, business stakeholders) are now more interested in conducting primary user research because time-to-insight (or product experimentation time) is reduced because of easier access to insights?
• How easy is it to undo a product decision that was mis-informed (e.g. a product strategy / feature / change was implemented based on a bad or incorrect interpretation of research insights from an LLM?)
• If we enabled 10 product teams to move more quickly by having faster access to user research - but largely through an LLM - and in the next quarter, 5 of them could show positive value (UX+biz) as a result, is that a 50% win or a 50% loss?
• If the other 5 shipped “bad” product that had no measurable impact on the org’s key KPIs, in part because at some later date, we found out the customer research insights this team use used were fabricated by the LLM, does that change our 50% win (or loss) score as an org?
• What if just one win was a huge outsized success, or one of the failures was deemed “horrible for the brand”?
• What game are we playing? Is our product or products a portfolio of “bets,” to use Annie Duke’s parlance?
• What is the role of a researcher at this org? What is their highest value and are they spending the majority of their time doing that type of work?

There isn’t an absolute right/wrong answer here; this is a leadership question and the context, industry and domains matter.

The team will have to decide what “safe/good enough to use an LLM” means. Teams of humans. Logically, health and medical use cases will be different than using an LLM to generate tag lines for creating eyeglass advertising copy.

Fabrication and hallucinations aren’t going away any time soon, so…

As I understand it, and I’m neither a data scientist nor an LLM researcher, we’re not close to guaranteeing that LLMs won’t hallucinate and make mistakes. Whether it is a “good job” is a metric that has a range of values; it’s not binary. The model will predict certain things we cannot control because it is probabilistic – so teams leveraging LLMs have a new factor to consider that doesn’t exist with traditional software: misinformed use.

This is one of the unique differences between designing for AI and LLMs vs. traditional software: we now have to consider informed use, no use, and misinformed use. That’s part of the price, and challenge, of designing for AI and especially for LLMs.

So, let’s pretend we’ve broadly accepted that the pros here outweigh the cons, and an internal team is going to build its own custom GPT with an overall goal to:

• Increase usage of primary research across the product teams
• Accelerate the shipment of product that is informed by research such that outsized wins have a greater chance of emerging
• Reduce how much “bad product” is shipped as a result of relying upon guessing (no research) or because “we were told to build it”

Let’s look at a practical example of this set in the insurance industry.

Practical Example: Designing Better Tools for Insurance Adjusters

Let’s assume we have an internal digital/UX/product/data team at a large insurance company. They build both customer and internal-facing technology solutions.

The research team has a collection of raw transcript data from a bunch of 1x1 qualitative research sessions interviewing insurance adjusters about how they process claims. The goal is to get customer claims processed faster, and to process the correct claims first.

The research team is in a separate department and is aware that many of the 20+ internal product/engineering teams do not seek advice from the UX research team, or their research questions are generally too hard to answer given the velocity that the build teams run under. (Agile, of course!)

So, the research is done, but rarely used. It’s sunk cost.

The CEO has recently stated that closing claims more quickly (by 20% on average) is a primary goal for the next quarter, as the time spent on any one claim costs the company more money and the company wants to rebrand itself around being “easy” to file claims with.

The research team believes it likely has insights that would be helpful for the design/engineering teams working on the tools the insurance adjusters use. After all, their labor constitutes 80% of the time consumed by an open claim. However, many of the product/engineering teams don’t know what it’s like to be a claims adjuster. They haven’t talked to one recently, and find the whole thing rather boring.

I can think of a lot of “0 to 1” use cases here for which an internal LLM / AI application in a large org. might be helpful to teams building these products—especially if we compare this to possible alternatives that the eng/product teams might otherwise engage in such as:

• Building product with no research (“it’s too slow, gotta ship by X date” “there’s nothing here that answers questions we have about the features in this sprint”)
• Building product with partial research exposure (i.e. “I went to 2 of those interviews your team conducted”) (the missed the other 12 though and never saw the researchers final recommendations email)
• “We’ll change it in v2” they said, because the research insights were useful, but they came too late given somebody higher up the food chain committed to some sort of date already

How might an LLM improve (accelerate) access to primary research findings and get more non-UX people (“who control the engineering resources”) to guess less and actually factor in research findings into their work early?

Given the pressure these internal teams might be under to meet the CEO’s goals, they may want to know as soon as possible:

• I have no idea what this research panel was about - the questions, people interviewed or the results - is anything relevant to the widget/use case I am working on for the “auto collision” adjustment team?
• Are there any obvious patterns in the findings of this panel?
• What was the overall tone/attitude in the panel (extra nice if the solution could look at audio/video as well as text) ?
• What questions were asked of the adjusters the most?
• Where is all the primary data about topic X in this panel?
• Is anything different/new?
• What questions should I be asking the LLM? (I don’t know what this thing can do for me?)

If we accept that the goal of most product/UX research is to accelerate the production of quality services, products, and experiences, the question is whether or not using an LLM for these types of questions is moving the needle in that direction at all. And secondly, are the potential downsides like hallucinations and occasional fabricated findings worth it?

It’s a design for AI problem.

If we consider the domain and context of use, what if this LLM solution was enabled to:

• Warn users when their prompts are likely to return inaccurate information—because we know what types of questions this GPT is not good at asking and want to educate our users.
• Attach citations to any summarizations such that users have at least some capability to determine the accuracy of an LLM summary without directly consuming ALL of the research data (aside: Perplexity.ai is doing interesting things with search + webpage citations)
• Notify the team who owns the LLM solution about trending “prompts” around specific panels or research topics (so they know what questions product teams are curious about and can also get ahead of potential problematic interpretations)
• Allowed researchers to have their own analyses challenged by the LLM (i.e. “here is my analysis of these 15 transcripts; are any of these conclusions or interpretations not supported well by the raw transcript data?”)
• Put up appropriate guide rails given the audience it was designed for (i.e. is the proper education around the tool in place, either as part of the UX or some other program?)

What’s in our data? Can the right people change it when the LLM is wrong?

The data product management and AI leaders reading this article know that the not-so-secret path to the best AI is in the foundational data that the models are trained on.

But, what does quality mean from a product standpoint and a risk-reduction one (as seen from an end user’s perspective who is trying to get work done?)

Of course, we have all the obvious data governance and privacy aspects to consider with any solution like this. However, another type of data that may be necessary here is the domain knowledge of subject matter experts, particularly if a custom GPT will be trained (as is probably necessary for many enterprise use cases).

When we think about “fact retrieval” use cases in particular, how easily can product teams (internal or otherwise) and end users understand the confidence of responses? When responses are wrong, how easily if at all can users and product teams update the model’s responses?

Errors in LLMs may be a significant design consideration when we design probabilistic solutions and no longer control what exactly our products and software are going to show to users. If bad UX can include leading people down the wrong path (unknowingly), then AI is kinda like the team on the other side of the tug of war we are playing. While we try to design a useful, usable, trustworthy solution, the LLM is always tugging us in the opposite direction as it spits out lies, funny retorts, or “believable” misleading information. Now, I say “may be a significant design challenge” because there are times when accuracy doesn’t matter as much. Examples include creative, cold-start, “0 to 1” tasks.

For example:  If I need to present my team’s quarterly results to the boss, and I want some creative titles for my PPT deck, does it matter how “off” the titles are? Not really. Similarly, if we look at a marketer working on designing ad copy, what may be helpful for them is generating a sheer volume of options from which the marketer can then “take over” with their brain to develop some final options. According to Jeremy Utley, co-author of the book Ideaflow and past guest on episode 106, one of the core ways to innovate is to begin with a large volume of potential ideas or solutions to a problem such that derivative ideas have a chance to surface themselves. It is often within the derivative ideas that sparks fly, but as somebody who writes a lot and composes music, one of the hardest parts is sometimes the 0-1 problem of getting started. This is a place where LLMs have great potential—but it also means we need to do the proper user research to understand our audience and when/where they’re doing “generative” creative work.

Star Trek Anyone? Let’s get creative folks

One thing I like about the hype, investment, and excitement around GenAI and LLMs in the enterprise is that it is an opportunity for organizations to do some visioning work—and this is a place that designers and UX professionals can help as we design for AI. “How might we improve [use case / user outcome] by leveraging GenAI”? Or the fun version: “how might this experience work on Star Trek?”

If there was ever a time to do some new visioning work, I think now is one of those. However, we’ll need highly skilled design leaders to help facilitate this in order for it to be effective. Part of that skill is in knowing who to include in exercises like this, and in my perspective, one of those people for sure should be somebody who understands data science well (not just the engineering perspective). As I posit in my seminar, the AI and analytical data product teams probably need a “4th” member—a quartet, not a trio—that includes a data expert as well as an engineering lead. Whether that is one person or two I’ll leave up to you, but the UX quality of AI solutions is largely influenced by the data, and so I want to see more interactions between the data science and design professionals on enterprise software and data product teams.

All that said…as of right now, here’s my overall take on LLMs in enterprise software:

My opinion on LLMs as a capability in enterprise software

As of the publishing of this article, while I’ve definitely seem the positive capabilities of GenAI firsthand, I’m still a little skeptical as to how much improvement in users’ lives we’re going to make with LLMs at this current time, given some of the challenges they present around transparency, hallucinations, and the CLI-style UI that many implementations are limited to.

This is not to say that there are not going to be some discrete use cases that could definitely be aided by these solutions. As for companies and vendors, I wonder how much the companies owning the foundation models are really going to always have the upper hand and potentially the ability to compete with point solutions that are leveraging LLM capabilities.

The second point I want to make is that given how many enterprise use cases depend on information accuracy, and especially question spaces that include quantitative data or analytics, everything I know from technical experts I trust is that even with RAG (retrieval augmented generation) in place, LLM UXs will struggle deliver accurate, trustworthy information from datastores holding structured data.

I think most of the non-design data professionals listening to this show already know this. They know what the risks are, and how these LLMs are effectively next-word guessing machines, and they’re not great at doing fact-based retrieval, using SQL or structured data methods for retrieving data from databases and things like that. I think that crowd knows that risk, but this is something that I think designers need to be very well aware of, particularly when we think about things like enterprise software.

So, I hope this was useful information, just to get your gears turning in your head and the creative juices flowing, some questions and concepts to think about, as you consider deploying some type of large language model into one of your internal tools or one of your commercial software products. I would love to see a lot less GenAI buttons next to large text area boxes where I just type something in, and then it lets me rewrite the solution. This feels like very much feature-first thinking, and we want to be able to slap an AI label on to our marketing, so what’s the quickest way to do that? Well, find any place in the product where a user is typing in text, and provide them a button to have that text somehow modified, improved, summarized, or changed by a large language model. This is not what I think great design for LLMs looks like, particularly in the enterprise space.

So, I look forward to seeing more examples of this in the wild that are actually producing real value. If you’d like to share any of those with me, please feel free to shoot me an email, or you can record an audio message or question for me at designingforanalytics.com/podcast.

Want more insights like this article? Subscribe below: