This is part 1 of a 2-part episode. Here is part 2.

In this episode of Experiencing Data, I introduce part 1 of my new MIRRR UX framework for designing trustworthy agentic AI applications—you know, the kind that might actually get used and have the opportunity to create the desired business value everyone seeks! One of the biggest challenges with both traditional analytics, ML, and now, LLM-driven AI agents, is getting end users and stakeholders to trust and utilize these data products—especially if we’re asking humans in the loop to make changes to their behavior or ways of working.

In this episode, I challenge the idea that software UIs will vanish with the rise of AI-based automation. In fact, the MIRRR framework is based on the idea that AI agents should be “in the human loop,” and a control surface (user interface) may in many situations be essential to ensure any automated workers engender trust with their human overlords.

By properly considering the control and oversight that end users and stakeholders need, you can enable the business value and UX outcomes that your paying customers, stakeholders, and application users seek from agentic AI.

Using use cases from insurance claims processing, in this episode, I introduce the first two of five control points in the MIRRR framework—Monitor and Interrupt. These control points represent core actions that define how AI agents often should operate and interact within human systems:

• Monitor – enabling appropriate transparency into AI agent behavior and performance
• Interrupt – designing both manual and automated pausing mechanisms to ensure human oversight remains possible when needed

…and in a couple weeks, stay tuned for part 2 where I’ll wrap up this first version of my MIRRR framework.

Highlights / Skip To:

• Introducing the MIRRR UX Framework for designing trustworthy agentic AI Applications. (00:34)
• The importance of trust in AI systems and how it is linked to user adoption (01:27)
• Cultural shifts, AI hype, and growing AI skepticism (03:06)
• Human centered design practices for agentic AI (04:13)
• I discuss how understanding your users’ needs does not change with agentic AI, and that trust in agentic applications has direct ties to user adoption and value creation (06:48)
• Measuring success of agentic applications with UX outcomes (11:32)
• Introducing the first two of five MIRRR framework control points (15:26):
  • M is for Monitor; understanding the agent’s “performance,” and the right
    level of transparency end users need, from individual tasks to aggregate views (16:29)
  • I is for Interrupt; when and why users may need to stop the agent—and
    what happens next (20:29)
• Conclusion and next steps (28:02)

Quotes from Today’s Episode

• " We have to think about designing and meeting people, meeting our customers where they're at, especially in terms of experience [with agentic AI applications]. So as such, it basically means you'll need to give more transparency than you might otherwise. And hopefully over time your users get more familiar with these systems and trusting their outputs and, and the benefits that they're supposed to be producing
• "You may very well have a deterministic software application that wraps around your agentic AI application that provides a monitoring user experience. So you'll have to make classic UI and UX design choices about what the right level of transparency is around the automation system that's actually inside and the meat and potatoes of your application."
• "An example of automated breakpoints might be a situation where sensitive data such as PII or financial personally identifiable information or financial data about a customer appears in the agent's task that otherwise shouldn't, perhaps due to a human error earlier upstream in a workflow or a job to be done. Or an unexpected result from an AI interaction upstream. The challenge here is obviously understanding where to programmatically check for these violations."
• "Conceptually, I'm in favor of designing for AI in the human loop instead of vice versa. The agents are always in service to humans, and absolute total autonomy is usually not the goal or should not be the goal as such."
• "It's important for teams to understand the risks of an AI system that goes off the rails in part or in full, well-designed or otherwise. The issue here is the scale at which damage can be done. If the system is doing a hundred times the incorrect work of a human that would be doing that work, problems can compound at a much faster rate."

Links

• Listen to part 2 of 2
• MIRRR is one way to close a specific part of the Invisible Intelligence Gap: the moment when users do not fully understand how an AI product reaches conclusions, what its limits are, or when they should override it. If your AI features demo well but hesitation shows up during evaluation or real-world use, read The Invisible Intelligence Gap for the full model. Then use MIRRR to address the trust and control side of the problem.

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, I’m rolling solo again, and I’m going to be sharing with you the MIRRR—M-I-R-R-R—UX framework for designing trustworthy agentic AI applications. And this is probably part one of a multi-part podcast episode, and it’s really targeted to those data and AI product leaders out there who are thinking about using agents to offload work from humans, and for those of you who probably are realizing that even if we’re talking about automation, there are still human touch points, there are still humans in the loop, there are still experiences that are occurring because you if we’re having trust problems and adoption problems, that means that there are humans in the loop.

So, you can’t really get rid of them fully. We’re not there yet where the machines are just doing work for the other machines and everybody’s at the beach—I don’t think that’s actually the goal, anyways—but how do we think about designing for trust in this environment? So, this is a rapidly evolving space, and I’ll tell you right now, I sometimes have imposter syndrome even sharing my ideas with you ahead of application. But, you know, GenAI, while new, it doesn’t mean that every single thing that we’ve learned in the last 30-plus years of designing good experiences for people on the web and digitally has to be thrown out. So, I’m sharing my ideas, subject to change and hopefully subject to your feedback and stories.

The first thing about agentic AI and design that I want to dispel is that we may imagine that user interfaces—or UIs—will disappear as there will be all this automated AI-driven work happening invisibly in the cloud, or maybe on-prem. And I’m pretty sure that, especially right now with where we are, particularly with knowledge work, that almost every attempt to jump to invisible, full automation is going to be doomed on multiple levels. Lack of adequate planning and scenario handling, especially when scenarios can now be infinite in some cases, or at least the idea of edge cases as a normal state where, like, we have to think about all the edge cases as just being normal because there are many when we’re working in that environment, well, you’re doomed, in some ways, to run into scenarios that we haven’t thought about. So, what happens then, both with the value we’re trying to create, as well as the impact on users and trust.

I should probably walk back a moment too and say that I’m trying to read a lot about agents and the hype, and there’s a lot out there with companies talking about singularities and AGI out there. And meanwhile, there are many companies that are not making a dime on any of this technology at the moment. So, there’s just a lot of hype, and I think there’s a fair amount of noise out there as well, and inflated expectations. That said, when we look at the cultural changes going on with how humans are using LLMs, the dramatic change in Google as the default answer space on the internet, I think it’s foolish to think that this is all just completely a giant fad. Researching both skeptics on this and applying creative thinking about what might be possible with the software and data products that we create now, to me, that feels like a healthy path to be on. So, I applaud people that are experimenting in this space right now and looking to see what’s possible here.

So, with that, let’s jump into some ideas for all you data and AI product leaders out there. I’m sure there will be some product designers and UX professionals listening to this, too, but frankly, I’m starting to feel like, you know, particularly if we keep seeing the accuracy improvements we are and can learn culturally how to balance human and machine work effectively, the roles of product design and product management will continue to merge, particularly around delivery and interface design work. What we will need are creative thinkers who can think in business terms and understand the human factors and the experience design considerations in a B2B context—and that’s always the context we’re talking about on this show—and people who can work with the technical and data team, the leads on those teams, to solve business problems in a unique way. And, I don’t know, maybe that’s a unicorn, but I tend to assume that, you know, given this show was originally designed to help data science and analytics leaders learn how to apply product-y and human-centered design practices to their work to drive adoption, trust, and usage, that you know, I have a lot of listeners like you out there who are already plenty capable on the technical side. So, if you’re not, look elsewhere for that advice, like John Cook or—I mean, is that guy a machine or what? I don’t know [laugh]. But you can get the technical side of this somewhere else. I want to talk to you about the product value side, the human factor side, the experience side, all of that.

So, later in this episode, I will go directly into the UX framework called MIRRR that I hope will help you think about how to check your agentic AI applications for quality, not in terms of code quality or just model accuracy, but in terms of a high degree of quality to the people using them and the people paying for them. But we got to talk about some other stuff first. Oh, and one other thing, in this episode, I’ll use agentic AI and AI and agents interchangeably to mean applications that plan and execute tasks with partial or full autonomy, leveraging LLMs or other AI reasoning components to achieve a user-defined goal or making context dependent decisions along the way and producing non-deterministic outputs. I think most of you know what I’m referring to there. It means we are not coding in all the business logic and we do not always know the routes and what the destinations will look like inside the jobs that these solutions are doing.

So, big number one here: trust and benefits creation will determine if your AI agent gets used within its human ecosystem. So, this probably rhymes with a lot of what you hear me talk about on Experiencing Data, but when we get back to first principles, which is key as we think about how to adjust the capabilities of AI and software, we have to come back to the problem space. Regardless of how we solve the problem or need, with AI or not, we have a couple factors to think about here. First, are we creating a benefit to a problem that both the humans in the loop care about solving and not just the financial beneficiaries? For those of you that are not in marketing, which is probably most of you, not many of you, this is an important concept to understand.

You may see the world through the technical lens of what’s possible right now with AI and agents, but that’s not how buyers and users see the world. Buyers and more specifically users don’t care how work and problems get solved, typically. What matters first is, do they actually want to solve those problems right now, relative to everything else that might be considered an unmet need or problem facing them? Why, you might see that, you know, for example, you could automate parts of contracts in the procurement flow at your company. Well, a procurement specialist might not want you doing that across the job to be done, or the tasks or the workflows that those people do day-to-day.

They may, however, find certain parts of their work extremely annoying and time consuming, and perhaps they perceive those things to be low value, and this is what the smart product leader is listening for in their primary market or user research: a problem or the problem that the user is eager to solve, not the one that could be solved now that we have AI capabilities because this is leading with a solution instead of leading with a problem, and you’re going to be rowing the boat uphill if you’re trying to lead with a solution. AI or not, if end-users do not trust this your system or this agentic AI application, or they don’t want it because they don’t resonate with the need that it’s trying to fulfill, regardless of whether or not you think they have a need, then your agentic AI application isn’t creating any value; it’s just another piece of shelfware with the latest technology. Product leaders need to be aware of the viability and desirability of pieces, particularly in the B2B space. Do you have alignment across the human ecosystem? Is the user of the application, the financial stakeholder or sponsor, and any third-party downstream beneficiaries all in alignment on what the agent is going to do in terms of the benefits it will create?

Using a simple insurance claim example, whereby parts of making and approving claims could be automated by agentic AI, this would mean thinking about, one, the insured person, two, the claims adjusters, some of the employees who might be the most hands-on, “End”—quote—“User” of the agentic application and any interfaces that help to monitor and manage the agent that’s doing some of the claims adjusting work, and three, the business stakeholders who oversee the claims department and have made the investment in agentic AI to get usually a financial outcome of some sort. There could be many other relevant parties and stakeholders as well. For example, you know, imagine scenarios where a claim is made, you know, overseas, in a different country, and additional laws and protocols exist that the agent may need to interact with, possibly invoking additional human checkpoints along the way that wouldn’t be necessary if that claim was made in the country of origin.

The point is, the agent’s success is defined by multiple parties, which could include at least three key personas, right? The business outcomes. Did the insurer, the company, the organization, get the value it sought, such as, you know, in this case, probably reduced claim processing time or more accurate claims processing, and less human work required to process these claims. There would be metrics here that we would track, but not necessarily just analytics, which our metrics computers track. That’s only one kind.

Number two, the UX outcomes. Specifically we are evaluating, did we improve the lives of the users who oversee the agent by removing work from their plate and offloading it to this agent. And so, you might call this a quote, “internal customer.” Did the claims adjuster and staff involved in processing claims now feel like work that they disliked got removed and they were still empowered to oversee and control the model or this agent, as opposed to feeling like they are a cog in the wheel of the AI agent, and they’re actually the ones that are kind of being manipulated by the machine? Note that the user’s objectives can sometimes be at odds with the business outcomes.

So, good product leaders should be looking at this friction, especially if usage of the solution is integral to the downstream business value being generated. Some things you can prototype or listen for here in terms of UX outcomes and metrics, could be evaluating through your user research, which I hope you’re doing, A, where is there existing work that this user base hates doing and they’re actively seeking help, or B, can you score the annoyances? Figure out how much the different taxes cost or create pain or problems or challenges for these users. Can you score these annoyances? And C, is this process fairly mature and apolitical? Can you evaluate this alongside the annoyance level and the user’s willingness to allow agentic AI to actually help them out?

And going back to these, kind of, three personas, we’re talking about, or these three different, yeah, personas, people that we’re trying to satisfy here, the beneficiary’s outcomes. This is the third group of people who are supposed to be benefiting here. In this case, this might be the customer—the paying customer, not the internal customer—the insured party. This is the person that bought a policy from your company, your insurance company, which we’re pretending, the person who now seeks to get coverage for a claim because something happened and they want to get paid for that because that’s what the insurance policy promised to do. Did utilizing the agent directly or indirectly produce a benefit for this person, the actual beneficiary, the paying customer?

For example, faster time to resolution, or more transparency about where the company is in the claims process, which effectively might be a way of giving customers a better experience. And not just a better experience, maybe that transparency means they don’t have to call into a human or even go to the website to start trying to run a chat and asking questions about what’s going on with my claim. AI doesn’t change the fact that within the machine ecosystem, we have human actors, sometimes with competing goals and desired outcomes, so first you need to figure out who all the important stakeholders and actors are, and then define the problem spaces. And then finally, what does success mean in those different contexts? And once this is in place, it’ll be easier to tell if your agentic AI is producing value or not because you will know how to measure this beyond simply, did we automate or not automate some tasks, and are the automations considered successful or not based on some model accuracy or some computerized method of testing the results? Because this doesn’t actually tell us whether or not the business or people got any value; this is like measuring the system against a standardized test.

Number two, the framework. I’m calling it MIRRR for now—M-I—triple-R—a UX framework for agentic AI, humans in the machine loop. So, this is a framework or a model for a control framework that can be thought of as a UX framework for agentic AI applications. And these letters, the M-I—triple-R, stand for Monitor, Interrupt, Redirect, Rerun, and Rollback. And regardless of the vendors you are using in your ecosystem, we can think of these agnostic verbs or terms as considerations that we probably need to design for in order to deliver on the outcomes that we established earlier.

So, in practice, here’s what these look like. I’m going to conceive of these verbs as activities that might exist inside of a wrapper application that helps visualize and expose the control surface for your agentic application. So, let’s jump in. First one, M is for Monitor. What’s the system doing and how is the system performing? How much transparency into the agent’s work is needed, and at what altitude? If the machine is processing multiple instances of jobs at once, do we need aggregate views of what is happening, possibly with analysis? Do we need to explore each job separately? And at what point are we giving too much detail?

You know, I’m sure many of you have probably seen a Mac OS startup screen and a terminal window with a CLI, a Command Line Interface. These are extreme differences in terms of the verbosity of transparency into what the system is doing. In the Apple UI, you get a login screen. It’s maybe a smiley face, a login prompt, and at most, maybe a progress bar once in a while, if a software update is going on. You’re not seeing any detailed logging in the system about what’s happening in the background.

In the CLI we see completely the opposite. We see tons of detail all the time, for the most part. Now, I can’t actually give you a recipe to follow, and just follow the recipe, and everything will just work out perfectly for your application, but I can advise a few things. First, early on in your customer’s exposure to this technology, you will likely need more verbosity and detail as to what the agents are doing. Why? Because trust with these systems is still being developed. They’re not things that we can just take for granted any longer.

It’s kind of like, if we remember, you know, is it safe to click on this link in this email, or is it safe to use PayPal? Can you actually send money and is it safe? I mean, look at the way people Venmo and Cash App and share money today. It’s second nature, and the concerns about safety, a lot of that has kind of gone into the past. Right now, I don’t think that’s where we are with these systems, and so we have to think about designing and meeting people, meeting our customers where they’re at, especially in terms of experience.

So, as such, it just basically means you’ll need to give more transparency than you might otherwise, and hopefully over time, your [users 00:18:44] get more familiar with these systems and trusting their outputs and the benefits that they’re supposed to be producing. Now, transparency can be overdone. I’m sure you have seen ChatGPT or the other foundation model chat bots who will sometimes provide way too much detail in certain contexts. And these applications have their entire user interface dedicated to single response output. You may very well have a deterministic software application that wraps around your agentic AI application that provides a monitoring user experience, so you’ll have to make, you know, classic UI and UX design choices about what the right level of transparency is around the automation system that’s actually inside and the meat and potatoes of your application.

And you might also have to consider whether your application wrapper is providing analytics and KPIs in the aggregate. So, think about manager or director views or executive views versus operational viewing for day-to-day use. So again, the claims adjuster, you might have hundreds of them working on cases, claims, and then you’ve got department heads that are looking in aggregate, how fast are we processing claims, and you know, how much money are we paying out? And then you’ve got someone that’s even higher up that’s looking at aggregates of aggregates and things like this. That’s a very basic way to think about it, but these all become relevant, particularly when we think about scaling here, which is one of the benefits of agents, right? It’s not like you develop an agent and it takes the work of one employee. Oftentimes, the whole point of this is to get a scale-level impact.

The next letter I. We talked about Monitor, we’re moving on to I, this is for Interrupt, the active and passive pausing of an AI agent instance. The great capability of probabilistic systems is their ability to react to a multitude of possible use cases and make next-step decisions. However, there will always be times when humans in the loop—or humans in the machine loop, rather—need to interrupt an agent’s work to take it over, to adjust parameters, or to explore, hey, why is it doing what it’s doing? Because in this honeymoon period, your end-users—again, these are, in our case, our end-users are internal employees, claims adjusters in this insurance use case—they’re likely kicking the tires and wanting to observe how the system works, particularly if it has direct impact on their job or their job security or the performance of their work or other aspects.

So, why would we want to pause a system, either manually or automatically in some situations? I am sure you have likely experienced one of the foundation models, particularly the newer reasoning models, running off in a direction that isn’t what you intended. And then you realized your instructions probably need some refinement, your prompts. And I could think of several reasons why a manual interrupt affordance needs to be made available, not just because of the probabilistic nature of use cases that the AI agent is handling, but because of human reasons, too. So, let’s look at manual interrupts, in other words, to pause execution of the agent.

Here’s two examples of when manual interrupting may need to be afforded in your application. The system account encounters a use case that the business has not contemplated before or is infrequent, and because of regulatory safety or significant business risk, a human moderator needs to step in. What is significant business risk? Well, again, this is where product people need to think about all these details. Another example, a business rule, policy, or even a law has changed, but the agent doesn’t know this or have access to this information yet, so for a period of time, certain jobs may require manual intervention at certain times without necessarily disabling the entire agent from doing its work.

Now, what about automated interrupts? This is sometimes referred to as policy break points. Here’s a couple of these. I think we can all imagine scenarios where a situation needs to be escalated for resolution or approval. Credit lending is a good example. Let’s say an individual contributor employee isn’t allowed to approve credit, or if we go back to our insurance example, a claim, maybe there’s a claim, if it’s over $100,000 director sign-off is required.

Your AI agent can also be programmed with this break point, too, as a place for a human to come back into the loop. If approving large credit lines is a complex process that is not easily repeatable yet, or simply due to the company culture, the comfort level within the company culture, the business isn’t ready to automate this yet because it feels uncomfortable or unnecessarily high risk, then this is also a great example of where a good product leader would not boil the ocean and try to make the agentic application cover every possible scenario. And we wouldn’t want it to use its own reasoning to decide what to do, especially if it’s paying out money. Instead, you’d take the win, providing value on a subset of the of these lending decisions or claims payouts and helping the staff, business, and customers achieve their desired outcomes. Gradually over time, you might increase the scope of what the agent can do as human trust is built, always coming back to the success metrics for users, beneficiaries, and those fiscal sponsors or stakeholders.

From a human standpoint, one of the qualitative success metrics from a UX perspective may be that staff handling these credit decisions or claims payouts feel like they’re mostly dealing with really important work, and the important decision-making on behalf of the company, and they’re no longer taxed with, you know, approving small loans or small claims and things like this. The employees may feel like their instinct and skill is being better utilized, and their impact is greater than it was earlier when they were handling every single loan request or claims payout that came across their queue. But here’s a hint for you data folks: software analytics isn’t going to tell you any of this information, and I don’t recommend surveys either. Your qualitative research with these employees, these end-users, is essential to validate feelings. And feelings right now are things that, for the most part, computers can’t track. They don’t directly measure people’s feelings about these things, and feelings have a lot to do with how people make decisions, so don’t ignore them [laugh].

Another example of automated break points might be a situation where sensitive data, such as PII or financial personally identifiable information or financial data about a customer appears in the agent’s task that otherwise shouldn’t, perhaps due to human error earlier upstream in a workflow or a job to be done or an unexpected result from an AI interaction upstream. The challenge here is obviously understanding where to programmatically check for these violations, although one thing that regulated industries have going for them is laws clear, or at least clearer, directives on what constitutes crossing a line that should not be crossed. One final note to think about with these interrupts is that there is a good chance that there may be situations where the system does not automatically get interrupted and perhaps it should have been. We all know that human error can occur, and given the scale that AI can work at, particularly when doing automated work, it’s important for teams to understand the risks of an AI system that goes off the rails, in part or in full, well designed or otherwise. The issue here is the scale at which damage can be done.

If the system is doing a hundred times the incorrect work of a human that would be doing that work, problems can compound at a much faster rate. As such, conceptually, I’m in favor of designing for AI in the human loop instead of vice versa. The agents are always in service of humans, and absolute total autonomy is usually not the goal, or should not be the goal. As such, the goal is, design in relevant human touch points and affordances, and to assume that there will always be times at which humans need to manage the AI agent.

That’s all for today. I know we didn’t get through the full framework. We will dive into the triple-Rs—[talks like a pirate] ARRS—in the next episode. I hope this was a good opening for you and provided some help on designing good experiences for agent Agentic AI applications focused on building trust. So, this is Brian, signing off for now. Stay tuned for episode two on this topic.